MDEV-29579 Fix CONNECT ASAN hits (#2277)

There are currently two things causing ASAN hits on CONNECT engine when
the plugin is used as a dynamic module. These are libxml2 and libodbc.

libxml2 has some quirks when not the first and last thing called in the
main thread of an application, some of the global memory isn't cleaned
up correctly. The same is assumed of libodbc but this does not have
explicit API for this.

This is being fixed in two ways.

First we are removing the libxml2 cleanup call. This is because the
current one is messy and whatever it fixed has gone away. But also
because if this is called and libxml2 is used again this can cause
issues. For example if two different plugins to MariaDB both happen to
use libxml2.

The second fix is a hack that exploits `STB_GNU_UNIQUE` so that when
compiled with ASAN the plugin will remain in memory after dlclose().
This allows libodbc to cleanup and has the added advatage that we will
get clean stacks from ASAN for CONNECT when the leak is detected at the
end of execution.

Details of the `STB_GNU_UNIQUE` method can be found here:

https://web.archive.org/web/20100730094324/http://www.redhat.com/archives/posix-c++-wg/2009-August/msg00002.html

2 files changed, 24 insertions, 38 deletions

diff --git a/storage/connect/ha_connect.h b/storage/connect/ha_connect.h
index 3f1ecc18af7..846f289a40c 100644
--- a/storage/connect/ha_connect.h
+++ b/storage/connect/ha_connect.h
@@ -551,3 +551,27 @@ public:
   uint    int_table_flags;            // Inherited from MyISAM
   bool    enable_activate_all_index;  // Inherited from MyISAM
 };  // end of ha_connect class definition
+
+
+/* This is a hack for ASAN
+ * Libraries such as libxml2 and libodbc do not like being unloaded before
+ * exit and will show as a leak in ASAN with no stack trace (as the plugin
+ * has been unloaded from memory).
+ *
+ * The below is designed to trick the compiler into adding a "UNIQUE" symbol
+ * which can be seen using:
+ * readelf -s storage/connect/ha_connect.so | grep UNIQUE
+ *
+ * Having this symbol means that the plugin remains in memory after dlclose()
+ * has been called. Thereby letting the libraries clean up properly.
+ */
+#if defined(__SANITIZE_ADDRESS__)
+__attribute__((__used__))
+inline int dummy(void)
+{
+  static int d;
+  d++;
+  return d;
+}
+#endif
+
diff --git a/storage/connect/libdoc.cpp b/storage/connect/libdoc.cpp
index aef7b551d65..54715da4ddc 100644
--- a/storage/connect/libdoc.cpp
+++ b/storage/connect/libdoc.cpp
@@ -290,46 +290,8 @@ if (!rc)
 /******************************************************************/
 /*  XML library cleanup function.                                 */
 /******************************************************************/
-/*
-  This is a copy of xmlCleanupParser() from the libxml2 sources
-  with xmlResetLastError() commented.
-
-  xmlResetLastError() called from the original xmlCleanupParser() causes
-  valgrind to report memory leaks. This happens because
-  ha_initialize_handlerton() is called from the main thread in mysqld.cc,
-  while ha_finalize_handlerton() is called from a non-main thread.
-  libxml2 gets confused because of xmlInitParser() and xmlCleanupParser()
-  being called from the different threads.
-
-  Perhaps the code in mysqld.cc should eventually be modified
-  to shutdown plugins from the main thread.
-*/
-static void
-xmlCleanupParser_replacement(void)
-  {
-  xmlCleanupCharEncodingHandlers();
-#ifdef LIBXML_CATALOG_ENABLED
-  xmlCatalogCleanup();
-#endif
-  xmlDictCleanup();
-  xmlCleanupInputCallbacks();
-#ifdef LIBXML_OUTPUT_ENABLED   
-  xmlCleanupOutputCallbacks();
-#endif
-#ifdef LIBXML_SCHEMAS_ENABLED
-  xmlSchemaCleanupTypes(); 
-  xmlRelaxNGCleanupTypes();
-#endif
-  //xmlResetLastError();
-  xmlCleanupGlobals();
-  xmlCleanupThreads(); /* must be last if called not from the main thread */
-  xmlCleanupMemory();
-  }
-
-
 void XmlCleanupParserLib(void)
   {
-  xmlCleanupParser_replacement();
   } // end of XmlCleanupParserLib
 
 /******************************************************************/