MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional

In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.

A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.

MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.

Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.

So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.

We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.

WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.

ut_crc32_byte_by_byte: Remove (unused function).

legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.

buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.

4 files changed, 29 insertions, 26 deletions

diff --git a/storage/innobase/include/buf0buf.h b/storage/innobase/include/buf0buf.h
index 8e13b3876e4..da344a3216c 100644
--- a/storage/innobase/include/buf0buf.h
+++ b/storage/innobase/include/buf0buf.h
@@ -716,14 +716,12 @@ buf_block_unfix(
 @param[in]	read_buf		database page
 @param[in]	checksum_field1		new checksum field
 @param[in]	checksum_field2		old checksum field
-@param[in]	use_legacy_big_endian   use legacy big endian algorithm
 @return true if the page is in crc32 checksum format. */
 bool
 buf_page_is_checksum_valid_crc32(
 	const byte*			read_buf,
 	ulint				checksum_field1,
-	ulint				checksum_field2,
-	bool				use_legacy_big_endian)
+	ulint				checksum_field2)
 	MY_ATTRIBUTE((nonnull(1), warn_unused_result));
 
 /** Checks if the page is in innodb checksum format.
diff --git a/storage/innobase/include/buf0checksum.h b/storage/innobase/include/buf0checksum.h
index 0bac2b911ee..06eb37906d2 100644
--- a/storage/innobase/include/buf0checksum.h
+++ b/storage/innobase/include/buf0checksum.h
@@ -1,7 +1,7 @@
 /*****************************************************************************
 
 Copyright (c) 1995, 2016, Oracle and/or its affiliates. All Rights Reserved.
-Copyright (c) 2017, MariaDB Corporation.
+Copyright (c) 2017, 2018, MariaDB Corporation.
 
 This program is free software; you can redistribute it and/or modify it under
 the terms of the GNU General Public License as published by the Free Software
@@ -29,19 +29,26 @@ Created Aug 11, 2011 Vasil Dimov
 
 #include "buf0types.h"
 
+#ifdef INNODB_BUG_ENDIAN_CRC32
 /** Calculate the CRC32 checksum of a page. The value is stored to the page
 when it is written to a file and also checked for a match when reading from
-the file. When reading we allow both normal CRC32 and CRC-legacy-big-endian
-variants. Note that we must be careful to calculate the same value on 32-bit
-and 64-bit architectures.
-@param[in]	page			buffer page (UNIV_PAGE_SIZE bytes)
-@param[in]	use_legacy_big_endian	if true then use big endian
-byteorder when converting byte strings to integers
-@return checksum */
-uint32_t
-buf_calc_page_crc32(
-	const byte*	page,
-	bool		use_legacy_big_endian = false);
+the file. Note that we must be careful to calculate the same value on all
+architectures.
+@param[in]	page		buffer page (srv_page_size bytes)
+@param[in]	bug_endian	whether to use big endian byteorder
+when converting byte strings to integers, for bug-compatibility with
+big-endian architecture running MySQL 5.6, MariaDB 10.0 or MariaDB 10.1
+@return	CRC-32C */
+uint32_t buf_calc_page_crc32(const byte* page, bool bug_endian = false);
+#else
+/** Calculate the CRC32 checksum of a page. The value is stored to the page
+when it is written to a file and also checked for a match when reading from
+the file. Note that we must be careful to calculate the same value on all
+architectures.
+@param[in]	page			buffer page (srv_page_size bytes)
+@return	CRC-32C */
+uint32_t buf_calc_page_crc32(const byte* page);
+#endif
 
 /** Calculate a checksum which is stored to the page when it is written
 to a file. Note that we must be careful to calculate the same value on
@@ -69,6 +76,5 @@ const char*
 buf_checksum_algorithm_name(srv_checksum_algorithm_t algo);
 
 extern ulong	srv_checksum_algorithm;
-extern bool	legacy_big_endian_checksum;
 
 #endif /* buf0checksum_h */
diff --git a/storage/innobase/include/page0zip.h b/storage/innobase/include/page0zip.h
index d72d5662f78..f458b29c4b4 100644
--- a/storage/innobase/include/page0zip.h
+++ b/storage/innobase/include/page0zip.h
@@ -492,16 +492,17 @@ page_zip_parse_compress(
 @param[in]	data			compressed page
 @param[in]	size			size of compressed page
 @param[in]	algo			algorithm to use
-@param[in]	use_legacy_big_endian	only used if algo is
-SRV_CHECKSUM_ALGORITHM_CRC32 or SRV_CHECKSUM_ALGORITHM_STRICT_CRC32 - if true
-then use big endian byteorder when converting byte strings to integers.
 @return page checksum */
 uint32_t
 page_zip_calc_checksum(
 	const void*			data,
 	ulint				size,
-	srv_checksum_algorithm_t	algo,
-	bool				use_legacy_big_endian = false);
+	srv_checksum_algorithm_t	algo
+#ifdef INNODB_BUG_ENDIAN_CRC32
+	/** for crc32, use the big-endian bug-compatible crc32 variant */
+	, bool				use_legacy_big_endian = false
+#endif
+);
 
 /**********************************************************************//**
 Verify a compressed page's checksum.
diff --git a/storage/innobase/include/ut0crc32.h b/storage/innobase/include/ut0crc32.h
index 36b389b5bd2..175a55e51c0 100644
--- a/storage/innobase/include/ut0crc32.h
+++ b/storage/innobase/include/ut0crc32.h
@@ -1,7 +1,7 @@
 /*****************************************************************************
 
 Copyright (c) 2011, 2015, Oracle and/or its affiliates. All Rights Reserved.
-Copyright (c) 2016, MariaDB Corporation.
+Copyright (c) 2016, 2018, MariaDB Corporation.
 
 This program is free software; you can redistribute it and/or modify it under
 the terms of the GNU General Public License as published by the Free Software
@@ -47,13 +47,11 @@ typedef uint32_t	(*ut_crc32_func_t)(const byte* ptr, ulint len);
 /** Pointer to CRC32 calculation function. */
 extern ut_crc32_func_t	ut_crc32;
 
+#ifdef INNODB_BUG_ENDIAN_CRC32
 /** Pointer to CRC32 calculation function, which uses big-endian byte order
 when converting byte strings to integers internally. */
 extern ut_crc32_func_t	ut_crc32_legacy_big_endian;
-
-/** Pointer to CRC32-byte-by-byte calculation function (byte order agnostic,
-but very slow). */
-extern ut_crc32_func_t	ut_crc32_byte_by_byte;
+#endif /* INNODB_BUG_ENDIAN_CRC32 */
 
 extern const char*	ut_crc32_implementation;