Bug #11752069 (former bug 43152)

Assertion `bitmap_is_set_all(&table->s->all_set)' failed in
handler::ha_reset

This assertion could be triggered if two connections simultaneously
executed two bitmap test functions on the same bitmap. For example,
the assertion could be triggered if one connection executed UPDATE
while a second connection executed SELECT on the same table.

Even if bitmap test functions have read-only semantics and have
const bitmaps as parameter, several of them modified the internal
state of the bitmap. With interleaved execution of two such functions
it was possible for one function to modify the state of the same
bitmap that the other function had just modified. This lead to an
inconsistent state and could trigger the assert.

Internally the bitmap uses 32 bit words for storage. Since bitmaps
can contain any number of bits, the last word in the bitmap may
not be fully used. A 32 bit mask is maintained where a bit is set
if the corresponding bit in the last bitmap word is unused.
The problem was that several test functions applied this mask to
the last word. Sometimes the mask was negated and used to zero out
the remainder of the last word and sometimes the mask was used as-is
to fill the remainder of the last word with 1's. This meant that if
a function first used the negated mask and another function then
used the mask as-is (or vice-versa), the first function would then
get the wrong result.

This patch fixes the problem by changing the implementation of
9 bitmap functions that modified the bitmap state even if the 
bitmap was declared const. These functions now preserve the
internal state of the bitmap. This makes it possible for
two connections to concurrently execute two of these functions
on the same bitmap without issues.

The patch also removes dead testing code from my_bitmap.c.
These tests have already been moved to unittest/mysys/bitmap-t.c.
Existing test coverage of my_bitmap has been extended.

No MTR test case added as this would require adding several sync
points to the bitmap functions. The patch has been tested with
a non-deterministic test case posted on the bug report.


include/my_bit.h:
  Removed my_count_bits_ushort() which is not needed anymore.
  Added my_count_bits_uint32().
unittest/mysys/bitmap-t.c:
  Extended test coverage of my_bitmap.

1 files changed, 5 insertions, 4 deletions

diff --git a/include/my_bitmap.h b/include/my_bitmap.h
index ab69b2d671d..42f985c8918 100644
--- a/include/my_bitmap.h
+++ b/include/my_bitmap.h
@@ -1,4 +1,4 @@
-/* Copyright (C) 2000 MySQL AB
+/* Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -149,9 +149,10 @@ bitmap_is_set(const MY_BITMAP *map,uint bit)
 
 static inline my_bool bitmap_cmp(const MY_BITMAP *map1, const MY_BITMAP *map2)
 {
-  *(map1)->last_word_ptr|= (map1)->last_word_mask;
-  *(map2)->last_word_ptr|= (map2)->last_word_mask;
-  return memcmp((map1)->bitmap, (map2)->bitmap, 4*no_words_in_map((map1)))==0;
+  if (memcmp(map1->bitmap, map2->bitmap, 4*(no_words_in_map(map1)-1)) != 0)
+    return FALSE;
+  return ((*map1->last_word_ptr | map1->last_word_mask) ==
+          (*map2->last_word_ptr | map2->last_word_mask));
 }
 
 #define bitmap_clear_all(MAP) \