Bug#38296 (low memory crash with many conditions in a query)

This fix is for 5.0 only : back porting the 6.0 patch manually

The parser code in sql/sql_yacc.yy needs to be more robust to out of
memory conditions, so that when parsing a query fails due to OOM,
the thread gracefully returns an error.

Before this fix, a new/alloc returning NULL could:
- cause a crash, if dereferencing the NULL pointer,
- produce a corrupted parsed tree, containing NULL nodes,
- alter the semantic of a query, by silently dropping token values or nodes

With this fix:
- C++ constructors are *not* executed with a NULL "this" pointer
when operator new fails.
This is achieved by declaring "operator new" with a "throw ()" clause,
so that a failed new gracefully returns NULL on OOM conditions.

- calls to new/alloc are tested for a NULL result,

- The thread diagnostic area is set to an error status when OOM occurs.
This ensures that a request failing in the server properly returns an
ER_OUT_OF_RESOURCES error to the client.

- OOM conditions cause the parser to stop immediately (MYSQL_YYABORT).
This prevents causing further crashes when using a partially built parsed
tree in further rules in the parser.

No test scripts are provided, since automating OOM failures is not
instrumented in the server.
Tested under the debugger, to verify that an error in alloc_root cause the
thread to returns gracefully all the way to the client application, with
an ER_OUT_OF_RESOURCES error.

1 files changed, 28 insertions, 3 deletions

diff --git a/sql/thr_malloc.cc b/sql/thr_malloc.cc
index 392db9224c3..4ebcf1c50af 100644
--- a/sql/thr_malloc.cc
+++ b/sql/thr_malloc.cc
@@ -21,10 +21,35 @@
 extern "C" {
   void sql_alloc_error_handler(void)
   {
-    THD *thd=current_thd;
-    if (thd)					// QQ;  To be removed
-      thd->fatal_error();			/* purecov: inspected */
     sql_print_error(ER(ER_OUT_OF_RESOURCES));
+
+    THD *thd=current_thd;
+    if (thd)
+    {
+      /*
+        This thread is Out Of Memory.
+        An OOM condition is a fatal error.
+        It should not be caught by error handlers in stored procedures.
+        Also, recording that SQL condition in the condition area could
+        cause more memory allocations, which in turn could raise more
+        OOM conditions, causing recursion in the error handling code itself.
+        As a result, my_error() should not be invoked, and the
+        thread diagnostics area is set to an error status directly.
+        The visible result for a client application will be:
+        - a query fails with an ER_OUT_OF_RESOURCES error,
+        returned in the error packet.
+        - SHOW ERROR/SHOW WARNINGS may be empty.
+      */
+
+      NET *net= &thd->net;
+      thd->fatal_error();
+      if (!net->last_error[0])                  // Return only first message
+      {
+        strmake(net->last_error, ER(ER_OUT_OF_RESOURCES),
+                sizeof(net->last_error)-1);
+        net->last_errno= ER_OUT_OF_RESOURCES;
+      }
+    }
   }
 }