MDEV-24033: SIGSEGV in __memcmp_avx2_movbe from queue_insert | SIGSEGV in __memcmp_avx2_movbe from native_compare

The issue here was the system variable max_sort_length was being applied
to decimals and it was truncating the value for decimals to the number
of bytes set by max_sort_length.
This was leading to a buffer overflow as the values were written
to the buffer without truncation and then we moved the offset to
the number of bytes(set by max_sort_length), that are needed for comparison.

The fix is to not apply max_sort_length for fixed size types like INT,
DECIMALS and only apply max_sort_length for CHAR, VARCHARS, TEXT and
BLOBS.

1 files changed, 2 insertions, 0 deletions

diff --git a/sql/sql_class.h b/sql/sql_class.h
index d693cfa2727..2727b4c84d5 100644
--- a/sql/sql_class.h
+++ b/sql/sql_class.h
@@ -5449,6 +5449,8 @@ struct SORT_FIELD_ATTR
 {
   uint length;          /* Length of sort field */
   uint suffix_length;   /* Length suffix (0-4) */
+  enum Type { FIXED_SIZE, VARIABLE_SIZE } type;
+  bool is_variable_sized() { return type == VARIABLE_SIZE; }
 };