Bug #45386: Wrong query result with MIN function in field list,

WHERE and GROUP BY clause

Loose index scan may use range conditions on the argument of 
the MIN/MAX aggregate functions to find the beginning/end of 
the interval that satisfies the range conditions in a single go.
These range conditions may have open or closed minimum/maximum 
values. When the comparison returns 0 (equal) the code should 
check the type of the min/max values of the current interval 
and accept or reject the row based on whether the limit is 
open or not.
There was a wrong composite condition on checking this and it was
not working in all cases.
Fixed by simplifying the conditions and reversing the logic.

mysql-test/r/group_min_max.result:
  Bug #45386: test case
mysql-test/t/group_min_max.test:
  Bug #45386: test case
sql/opt_range.cc:
  Bug #45386: fix the check whether to use the value if on the
  interval boundry

1 files changed, 16 insertions, 4 deletions

diff --git a/sql/opt_range.cc b/sql/opt_range.cc
index d52db41d64c..571a342fa17 100644
--- a/sql/opt_range.cc
+++ b/sql/opt_range.cc
@@ -9395,8 +9395,14 @@ int QUICK_GROUP_MIN_MAX_SELECT::next_min_in_range()
       /* Compare the found key with max_key. */
       int cmp_res= key_cmp(index_info->key_part, max_key,
                            real_prefix_len + min_max_arg_len);
-      if (!(((cur_range->flag & NEAR_MAX) && (cmp_res == -1)) ||
-            (cmp_res <= 0)))
+      /*
+        The key is outside of the range if: 
+        the interval is open and the key is equal to the maximum boundry
+        or
+        the key is greater than the maximum
+      */
+      if (((cur_range->flag & NEAR_MAX) && cmp_res == 0) ||
+          cmp_res > 0)
       {
         result = HA_ERR_KEY_NOT_FOUND;
         continue;
@@ -9511,8 +9517,14 @@ int QUICK_GROUP_MIN_MAX_SELECT::next_max_in_range()
       /* Compare the found key with min_key. */
       int cmp_res= key_cmp(index_info->key_part, min_key,
                            real_prefix_len + min_max_arg_len);
-      if (!(((cur_range->flag & NEAR_MIN) && (cmp_res == 1)) ||
-            (cmp_res >= 0)))
+      /*
+        The key is outside of the range if: 
+        the interval is open and the key is equal to the minimum boundry
+        or
+        the key is less than the minimum
+      */
+      if (((cur_range->flag & NEAR_MIN) && cmp_res == 0) ||
+          cmp_res < 0)
         continue;
     }
     /* If we got to this point, the current key qualifies as MAX. */