glsl: rewrite glsl_type::record_key_hash() to avoid buffer overflow

This should be more efficient than the previous snprintf() solution.
But more importantly, it avoids a buffer overflow bug that could result
in crashes or unpredictable results when processing very large interface
blocks.

For the app in question, key->length = 103 for some interfaces.  The check
if size >= sizeof(hash_key) was insufficient to prevent overflows of the
hash_key[128] array because it didn't account for the terminating zero.
In this case, this caused the call to hash_table_string_hash() to return
different results for identical inputs, and then shader linking failed.

This new solution also takes all structure fields into account instead
of just the first 15 when sizeof(pointer)==8.

Cc: mesa-stable@lists.freedesktop.org
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
(cherry picked from commit 31667e6237d30188d0b29e17f5b9892f10c0d83a)

1 files changed, 13 insertions, 10 deletions

diff --git a/src/glsl/glsl_types.cpp b/src/glsl/glsl_types.cpp
index b4223f4f06e..970bcd82501 100644
--- a/src/glsl/glsl_types.cpp
+++ b/src/glsl/glsl_types.cpp
@@ -690,24 +690,27 @@ glsl_type::record_key_compare(const void *a, const void *b)
 }
 
 
+/**
+ * Generate an integer hash value for a glsl_type structure type.
+ */
 unsigned
 glsl_type::record_key_hash(const void *a)
 {
    const glsl_type *const key = (glsl_type *) a;
-   char hash_key[128];
-   unsigned size = 0;
-
-   size = snprintf(hash_key, sizeof(hash_key), "%08x", key->length);
+   uintptr_t hash = key->length;
+   unsigned retval;
 
    for (unsigned i = 0; i < key->length; i++) {
-      if (size >= sizeof(hash_key))
-	 break;
-
-      size += snprintf(& hash_key[size], sizeof(hash_key) - size,
-		       "%p", (void *) key->fields.structure[i].type);
+      /* casting pointer to uintptr_t */
+      hash = (hash * 13 ) + (uintptr_t) key->fields.structure[i].type;
    }
 
-   return hash_table_string_hash(& hash_key);
+   if (sizeof(hash) == 8)
+      retval = (hash & 0xffffffff) ^ ((uint64_t) hash >> 32);
+   else
+      retval = hash;
+
+   return retval;
 }