BUG#11392 - fulltext search bug

Fulltext boolean mode phrase search may crash server on platforms
where size of pointer is not equal to size of unsigned integer
(in other words some 64-bit platforms).

The problem was integer overflow.

Affects 4.1 only.


myisam/ft_boolean_search.c:
  my_match_t::beg is unsigned int, that means type of expression
  (m[0].beg - 1) has unsigned type too. It may happen that instr()
  finds substring in the beggining of passed string, returning
  m[0].beg equal to 0. In this case value of expression (m[0].beg - 1)
  is equal to MAX_UINT.
  
  This is not a problem on platforms where sizeof(pointer) equals to
  sizeof(uint). That means ptr[(uint)-1] = ptr[(uint)MAX_UINT] = ptr - 1.
  
  On some 64-bit platforms where sizeof(pointer) is 8 and sizeof(uint)
  is 4, wrong address gets accessed. In other words ptr[(uint)-1] is
  equal to ptr + MAX_UINT.
mysql-test/r/fulltext.result:
  A test case for BUG#11392.
mysql-test/t/fulltext.test:
  A test case for BUG#11392.

1 files changed, 2 insertions, 1 deletions

diff --git a/myisam/ft_boolean_search.c b/myisam/ft_boolean_search.c
index f1ff8f6d886..fad25abcc6c 100644
--- a/myisam/ft_boolean_search.c
+++ b/myisam/ft_boolean_search.c
@@ -446,7 +446,8 @@ static int _ftb_strstr(const byte *s0, const byte *e0,
   {
     if (cs->coll->instr(cs, p0, e0 - p0, s1, e1 - s1, m, 2) != 2)
       return(0);
-    if ((!s_after || p0 + m[1].beg == s0 || !true_word_char(cs, p0[m[1].beg-1])) &&
+    if ((!s_after || p0 + m[1].beg == s0 ||
+         !true_word_char(cs, p0[(int) m[1].beg - 1])) &&
         (!e_before || p0 + m[1].end == e0 || !true_word_char(cs, p0[m[1].end])))
       return(1);
     p0+= m[1].beg;