MDEV-20377: Introduce cmake -DWITH_MSAN:BOOL=ON

MemorySanitizer is a compile-time instrumentation layer in clang and GCC.
Together with AddressSanitizer mostly makes the run-time instrumentation
of Valgrind redundant. It is a little more tricky to set up, because
running with uninstrumented libraries will lead into false positives.

You will need an instrumented libc++, and you should use
-stdlib=libc++ instead of the default libstdc++. To build the
instrumented library, you can refer to
https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo
or you can adapt these steps that worked for me, for clang-8 version 8.0.1:

cd /mariadb
sudo apt source libc++-8-dev
cd llvm-toolchain-8-8.0.1
mkdir libc++msan; cd libc++msan
cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory \
-DCMAKE_C_COMPILER=clang-8 -DCMAKE_CXX_COMPILER=clang++-8

Then, in your MariaDB build directory, you have to compile with
libc++ and bundled libraries, such as WITH_SSL=bundled, WITH_ZLIB=bundled.
For uninstrumented system libraries, you will get false positives for
uninitialized values. Like this:

cmake -DWITH_MSAN=ON -DWITH_SSL=bundled -DWITH_ZLIB=bundled \
-DCMAKE_CXX_FLAGS='-stdlib=libc++' ..

Note: you should also add -O2 to the compiler options, or you may
get crashes due to stack overflow.

Finally, to run tests, you must replace libc++ with the instrumented one:

LD_LIBRARY_PATH=/mariadb/llvm-toolchain-8-8.0.1/libc++msan/lib \
MSAN_OPTIONS=abort_on_error=1 \
./mtr --big-test --parallel=auto --force --retry=0

Failure to do so will report numerous false positives related to
operations on std::string and the like.

This is work in progress. Some issues will still have to be fixed
for WITH_MSAN to be usable. See MDEV-20377 for details.

1 files changed, 5 insertions, 1 deletions

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 75f9d22d747..0dcc2a75587 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,5 +1,5 @@
 # Copyright (c) 2006, 2017, Oracle and/or its affiliates.
-# Copyright (c) 2008, 2018, MariaDB Corporation
+# Copyright (c) 2008, 2019, MariaDB Corporation.
 # 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -219,6 +219,10 @@ IF (WITH_UBSAN)
   MY_CHECK_AND_SET_COMPILER_FLAG("-fsanitize=undefined -fno-sanitize=alignment -U_FORTIFY_SOURCE" DEBUG RELWITHDEBINFO)
 ENDIF()
 
+OPTION(WITH_MSAN "Enable memory sanitizer" OFF)
+IF (WITH_MSAN)
+  MY_CHECK_AND_SET_COMPILER_FLAG("-fsanitize=memory -fsanitize-memory-track-origins -U_FORTIFY_SOURCE" DEBUG RELWITHDEBINFO)
+ENDIF()
 
 # enable security hardening features, like most distributions do
 # in our benchmarks that costs about ~1% of performance, depending on the load