summaryrefslogtreecommitdiff
path: root/storage/innobase/fil/fil0pagecompress.cc
Commit message (Collapse)AuthorAgeFilesLines
* Merge 10.4 into 10.5Marko Mäkelä2020-05-181-1/+0
|\
| * Merge 10.3 into 10.4Marko Mäkelä2020-05-161-1/+0
| |\ | | | | | | | | | | | | We will expose some more std::atomic internals in Atomic_counter, so that dict_index_t::lock will support the default assignment operator.
| | * Merge 10.2 into 10.3Marko Mäkelä2020-05-151-1/+0
| | |\
| | | * MDEV-21336 Memory leaks related to innodb_debug_syncMarko Mäkelä2020-05-141-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This essentially reverts commit b393e2cb0c079b30563dcc87a62002c9c778643c. The leak might have been fixed, but because the DEBUG_SYNC instrumentation for InnoDB purge threads was reverted in 10.5 commit 5e62b6a5e06eb02cbde1e34e95e26f42d87fce02 as part of introducing a thread pool, it is easiest to revert the entire change.
* | | | MDEV-21133 follow-up: Use fil_page_get_type()Marko Mäkelä2020-05-071-2/+1
| | | | | | | | | | | | | | | | | | | | Let us use the common accessor function fil_page_get_type() instead of accessing the page header field FIL_PAGE_TYPE directly.
* | | | MDEV-21133 follow-up: More my_assume_aligned hintsMarko Mäkelä2020-05-071-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fsp0pagecompress.h: Remove. Invoke fil_page_get_type() and FSP_FLAGS_GET_PAGE_COMPRESSION_LEVEL directly. log_block_get_flush_bit(), log_block_set_flush_bit(): Access the byte directly. dict_sys_read_row_id(): Remove (unused function).
* | | | MDEV-21907: Fix most clang -Wconversion in InnoDBMarko Mäkelä2020-03-111-6/+9
|/ / / | | | | | | | | | | | | Declare innodb_purge_threads as 4-byte integer (UINT) instead of 4-or-8-byte (ULONG) and adjust the documentation string.
* | | Merge branch '10.3' into 10.4Oleksandr Byelkin2019-05-191-1/+1
|\ \ \ | |/ /
| * | Merge 10.2 into 10.3Marko Mäkelä2019-05-141-1/+1
| |\ \ | | |/
| | * Merge 10.1 into 10.2Marko Mäkelä2019-05-131-1/+1
| | |\
| | | * Merge branch '5.5' into 10.1Vicențiu Ciorbaru2019-05-111-1/+1
| | | |
* | | | MDEV-18644: Support full_crc32 for page_compressedMarko Mäkelä2019-03-181-169/+378
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a follow-up task to MDEV-12026, which introduced innodb_checksum_algorithm=full_crc32 and a simpler page format. MDEV-12026 did not enable full_crc32 for page_compressed tables, which we will be doing now. This is joint work with Thirunarayanan Balathandayuthapani. For innodb_checksum_algorithm=full_crc32 we change the page_compressed format as follows: FIL_PAGE_TYPE: The most significant bit will be set to indicate page_compressed format. The least significant bits will contain the compressed page size, rounded up to a multiple of 256 bytes. The checksum will be stored in the last 4 bytes of the page (whether it is the full page or a page_compressed page whose size is determined by FIL_PAGE_TYPE), covering all preceding bytes of the page. If encryption is used, then the page will be encrypted between compression and computing the checksum. For page_compressed, FIL_PAGE_LSN will not be repeated at the end of the page. FSP_SPACE_FLAGS (already implemented as part of MDEV-12026): We will store the innodb_compression_algorithm that may be used to compress pages. Previously, the choice of algorithm was written to each compressed data page separately, and one would be unable to know in advance which compression algorithm(s) are used. fil_space_t::full_crc32_page_compressed_len(): Determine if the page_compressed algorithm of the tablespace needs to know the exact length of the compressed data. If yes, we will reserve and write an extra byte for this right before the checksum. buf_page_is_compressed(): Determine if a page uses page_compressed (in any innodb_checksum_algorithm). fil_page_decompress(): Pass also fil_space_t::flags so that the format can be determined. buf_page_is_zeroes(): Check if a page is full of zero bytes. buf_page_full_crc32_is_corrupted(): Renamed from buf_encrypted_full_crc32_page_is_corrupted(). For full_crc32, we always simply validate the checksum to the page contents, while the physical page size is explicitly specified by an unencrypted part of the page header. buf_page_full_crc32_size(): Determine the size of a full_crc32 page. buf_dblwr_check_page_lsn(): Make this a debug-only function, because it involves potentially costly lookups of fil_space_t. create_table_info_t::check_table_options(), ha_innobase::check_if_supported_inplace_alter(): Do allow the creation of SPATIAL INDEX with full_crc32 also when page_compressed is used. commit_cache_norebuild(): Preserve the compression algorithm when updating the page_compression_level. dict_tf_to_fsp_flags(): Set the flags for page compression algorithm. FIXME: Maybe there should be a table option page_compression_algorithm and a session variable to back it?
* | | | MDEV-12026: Implement innodb_checksum_algorithm=full_crc32Thirunarayanan Balathandayuthapani2019-02-191-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MariaDB data-at-rest encryption (innodb_encrypt_tables) had repurposed the same unused data field that was repurposed in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN) field of SPATIAL INDEX. Because of this, MariaDB was unable to support encryption on SPATIAL INDEX pages. Furthermore, InnoDB page checksums skipped some bytes, and there are multiple variations and checksum algorithms. By default, InnoDB accepts all variations of all algorithms that ever existed. This unnecessarily weakens the page checksums. We hereby introduce two more innodb_checksum_algorithm variants (full_crc32, strict_full_crc32) that are special in a way: When either setting is active, newly created data files will carry a flag (fil_space_t::full_crc32()) that indicates that all pages of the file will use a full CRC-32C checksum over the entire page contents (excluding the bytes where the checksum is stored, at the very end of the page). Such files will always use that checksum, no matter what the parameter innodb_checksum_algorithm is assigned to. For old files, the old checksum algorithms will continue to be used. The value strict_full_crc32 will be equivalent to strict_crc32 and the value full_crc32 will be equivalent to crc32. ROW_FORMAT=COMPRESSED tables will only use the old format. These tables do not support new features, such as larger innodb_page_size or instant ADD/DROP COLUMN. They may be deprecated in the future. We do not want an unnecessary file format change for them. The new full_crc32() format also cleans up the MariaDB tablespace flags. We will reserve flags to store the page_compressed compression algorithm, and to store the compressed payload length, so that checksum can be computed over the compressed (and possibly encrypted) stream and can be validated without decrypting or decompressing the page. In the full_crc32 format, there no longer are separate before-encryption and after-encryption checksums for pages. The single checksum is computed on the page contents that is written to the file. We do not make the new algorithm the default for two reasons. First, MariaDB 10.4.2 was a beta release, and the default values of parameters should not change after beta. Second, we did not yet implement the full_crc32 format for page_compressed pages. This will be fixed in MDEV-18644. This is joint work with Marko Mäkelä.
* | | | MDEV-18493 Remove page_size_tMarko Mäkelä2019-02-071-2/+1
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MySQL 5.7 introduced the class page_size_t and increased the size of buffer pool page descriptors by introducing this object to them. Maybe the intention of this exercise was to prepare for a future where the buffer pool could accommodate multiple page sizes. But that future never arrived, not even in MySQL 8.0. It is much easier to manage a pool of a single page size, and typically all storage devices of an InnoDB instance benefit from using the same page size. Let us remove page_size_t from MariaDB Server. This will make it easier to remove support for ROW_FORMAT=COMPRESSED (or make it a compile-time option) in the future, just by removing various occurrences of zip_size.
* | | Merge branch '10.2' into 10.3Sergei Golubchik2019-01-031-4/+4
|\ \ \ | |/ /
| * | Merge 10.1 into 10.2Marko Mäkelä2018-12-211-4/+4
| |\ \ | | |/
| | * MDEV-18025: Detect corrupted innodb_page_compression=zlib pagesMarko Mäkelä2018-12-201-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In MDEV-13103, I made a mistake in the error handling of page_compressed=1 decryption when the default innodb_compression_algorithm=zlib is used. Due to this mistake, with certain versions of zlib, MariaDB would fail to detect a corrupted page. The problem was uncovered by the following tests: mariabackup.unencrypted_page_compressed mariabackup.encrypted_page_compressed
| | * MDEV-13103 Deal with page_compressed page corruptionMarko Mäkelä2018-06-141-510/+181
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fil_page_decompress(): Replaces fil_decompress_page(). Allow the caller detect errors. Remove duplicated code. Use the "safe" instead of "fast" variants of decompression routines. fil_page_compress(): Replaces fil_compress_page(). The length of the input buffer always was srv_page_size (innodb_page_size). Remove printouts, and remove the fil_space_t* parameter. buf_tmp_buffer_t::reserved: Make private; the accessors acquire() and release() will use atomic memory access. buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter. Do not acquire any mutex. Remove the allocation of the buffers. buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf(): Refactored away from buf_pool_reserve_tmp_slot(). buf_page_decrypt_after_read(): Make static, and simplify the logic. Use the encryption buffer also for decompressing. buf_page_io_complete(), buf_dblwr_process(): Check more failures. fil_space_encrypt(): Simplify the debug checks. fil_space_t::printed_compression_failure: Remove. fil_get_compression_alg_name(): Remove. fil_iterate(): Allocate a buffer for compression and decompression only once, instead of allocating and freeing it for every page that uses compression, during IMPORT TABLESPACE. fil_node_get_space_id(), fil_page_is_index_page(), fil_page_is_lzo_compressed(): Remove (unused code).
| | * Remove traces of the non-working MDEV-6354Marko Mäkelä2018-06-131-3/+2
| | | | | | | | | | | | | | | | | | | | | MariaDB never supported the MySQL 5.7 compression format. FIL_PAGE_TYPE_COMPRESSED: Remove. This was originally added as FIL_PAGE_COMPRESSED.
* | | Merge 10.2 into 10.3Marko Mäkelä2018-11-191-1/+0
|\ \ \ | |/ /
| * | Remove many redundant #include from InnoDBMarko Mäkelä2018-11-191-1/+0
| | |
* | | Merge 10.2 into 10.3Marko Mäkelä2018-06-181-412/+168
|\ \ \ | |/ /
| * | MDEV-13103 Deal with page_compressed page corruptionMarko Mäkelä2018-06-141-415/+170
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fil_page_decompress(): Replaces fil_decompress_page(). Allow the caller detect errors. Remove duplicated code. Use the "safe" instead of "fast" variants of decompression routines. fil_page_compress(): Replaces fil_compress_page(). The length of the input buffer always was srv_page_size (innodb_page_size). Remove printouts, and remove the fil_space_t* parameter. buf_tmp_buffer_t::reserved: Make private; the accessors acquire() and release() will use atomic memory access. buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter. Do not acquire any mutex. Remove the allocation of the buffers. buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf(): Refactored away from buf_pool_reserve_tmp_slot(). buf_page_decrypt_after_read(): Make static, and simplify the logic. Use the encryption buffer also for decompressing. buf_page_io_complete(), buf_dblwr_process(): Check more failures. fil_space_encrypt(): Simplify the debug checks. fil_space_t::printed_compression_failure: Remove. fil_get_compression_alg_name(): Remove. fil_iterate(): Allocate a buffer for compression and decompression only once, instead of allocating and freeing it for every page that uses compression, during IMPORT TABLESPACE. Also, validate the page checksum before decryption, and reduce the scope of some variables. fil_page_is_index_page(), fil_page_is_lzo_compressed(): Remove (unused). AbstractCallback::operator()(): Remove the parameter 'offset'. The check for it in FetchIndexRootPages::operator() was basically redundant and dead code since the previous refactoring.
* | | Replace univ_page_size and UNIV_PAGE_SIZEMarko Mäkelä2018-04-281-20/+20
| | | | | | | | | | | | | | | | | | Try to use one variable (srv_page_size) for innodb_page_size. Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.
* | | Fix most -Wsign-conversion in InnoDBMarko Mäkelä2018-04-281-1/+1
| | | | | | | | | | | | Change innodb_buffer_pool_size, innodb_fill_factor to unsigned.
* | | Fix a lot of compiler warnings found by -WunusedMonty2018-04-261-3/+3
| | |
* | | MDEV-15983 Reduce fil_system.mutex contention furtherMarko Mäkelä2018-04-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fil_space_t::n_pending_ops, n_pending_ios: Use a combination of fil_system.mutex and atomic memory access for protection. fil_space_t::release(): Replaces fil_space_release(). Does not acquire fil_system.mutex. fil_space_t::release_for_io(): Replaces fil_space_release_for_io(). Does not acquire fil_system.mutex.
* | | Merge bb-10.2-ext into 10.3Marko Mäkelä2017-10-041-2/+10
|\ \ \ | |/ /
| * | Merge 10.1 into 10.2Marko Mäkelä2017-10-021-2/+10
| |\ \ | | |/
| | * MDEV-13932: fil0pagecompress.cc fails to compile with lzo 2.10Jan Lindström2017-09-291-2/+10
| | | | | | | | | | | | Patch contibuted by Gordon Fisher. Thank you for your contribution!
* | | Merge bb-10.2-ext into 10.3Marko Mäkelä2017-09-071-2/+4
|\ \ \ | |/ /
| * | Merge 10.1 into 10.2Marko Mäkelä2017-09-061-2/+4
| |\ \ | | |/
| | * Merge 10.0 into 10.1Marko Mäkelä2017-09-061-1/+2
| | |
* | | Merge bb-10.2-ext into 10.3Marko Mäkelä2017-06-191-25/+31
|\ \ \ | |/ /
| * | Merge 10.1 into 10.2Marko Mäkelä2017-05-231-25/+32
| |\ \ | | |/
| | * MDEV-12615: InnoDB page compression method snappy mostly does not compress pagesJan Lindström2017-05-201-37/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Snappy compression method require that output buffer used for compression is bigger than input buffer. Similarly lzo require additional work memory buffer. Increase the allocated buffer accordingly. buf_tmp_buffer_t: removed unnecessary lzo_mem, crypt_buf_free and comp_buf_free. buf_pool_reserve_tmp_slot: use alligned_alloc and if snappy available allocate size based on snappy_max_compressed_length and if lzo is available increase buffer by LZO1X_1_15_MEM_COMPRESS. fil_compress_page: Remove unneeded lzo mem (we use same buffer) and if output buffer is not yet allocated allocate based similarly as above. Decompression does not require additional work area. Modify test to use same test as other compression method tests.
| * | Merge 10.1 into 10.2Marko Mäkelä2017-05-221-1/+0
| |\ \ | | |/
| | * MDEV-12593: InnoDB page compression should use lz4_compress_default ifJan Lindström2017-05-181-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | available lz4.cmake: Check if shared or static lz4 library has LZ4_compress_default function and if it has define HAVE_LZ4_COMPRESS_DEFAULT. fil_compress_page: If HAVE_LZ4_COMPRESS_DEFAULT is defined use LZ4_compress_default function for compression if not use LZ4_compress_limitedOutput function. Introduced a innodb-page-compression.inc file for page compression tests that will also search .ibd file to verify that pages are compressed (i.e. used search string is not found). Modified page compression tests to use this file. Note that snappy method is not included because of MDEV-12615 InnoDB page compression method snappy mostly does not compress pages that will be fixed on different commit.
* | | Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3Alexander Barkov2017-05-071-5/+4
|\ \ \ | |/ /
| * | MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0Marko Mäkelä2017-04-281-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes a regression caused by MDEV-12428. When we introduced a variant of fil_space_acquire() that could increment space->n_pending_ops after space->stop_new_ops was set, the logic of fil_check_pending_operations() was broken. fil_space_t::n_pending_ios: A new field to track read or write access from the buffer pool routines immediately before a block write or after a block read in the file system. fil_space_acquire_for_io(), fil_space_release_for_io(): Similar to fil_space_acquire_silent() and fil_space_release(), but modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops. fil_space_free_low(): Wait for space->n_pending_ios to reach 0, to avoid accessing freed data in a concurrent thread. Future calls to fil_space_acquire_for_io() will not find this tablespace, because it will already have been detached from fil_system. Adjust a number of places accordingly, and remove some redundant tablespace lookups. FIXME: buf_page_check_corrupt() should take a tablespace from fil_space_acquire_for_io() as a parameter. This will be done in the 10.1 version of this patch and merged from there. That depends on MDEV-12253, which has not been merged from 10.1 yet.
| * | Merge 10.1 into 10.2Marko Mäkelä2017-04-281-3/+1
| |\ \ | | |/
| | * Fix a compilation errorMarko Mäkelä2017-04-211-26/+41
| | |
| | * MDEV-12545 Reduce the amount of fil_space_t lookupsMarko Mäkelä2017-04-211-33/+30
| | | | | | | | | | | | | | | | | | buf_flush_write_block_low(): Acquire the tablespace reference once, and pass it to lower-level functions. This is only a start; further calls may be removed later.
| | * Fix some InnoDB type mismatch introduced in 10.1Marko Mäkelä2017-04-211-2/+3
| | | | | | | | | | | | On 64-bit Windows, sizeof(ulint)!=sizeof(ulong).
* | | Merge remote-tracking branch 'origin/10.2' into 10.3Alexander Barkov2017-04-221-42/+30
|\ \ \ | |/ /
| * | Fix a crash when page_compression fails during IMPORT TABLESPACEMarko Mäkelä2017-04-211-2/+2
| | | | | | | | | | | | fil_compress_page(): Check for space==NULL.
| * | MDEV-12545 Reduce the amount of fil_space_t lookupsMarko Mäkelä2017-04-211-41/+26
| | | | | | | | | | | | | | | | | | | | | | | | buf_flush_write_block_low(): Acquire the tablespace reference once, and pass it to lower-level functions. This is only a start; further calls may be removed. fil_decompress_page(): Remove unsafe use of fil_space_get_by_id().
| * | Fix some InnoDB type mismatchMarko Mäkelä2017-04-211-2/+3
| | | | | | | | | | | | On 64-bit Windows, sizeof(ulint)!=sizeof(ulong).
| * | MDEV-12488 Remove type mismatch in InnoDB printf-like callsMarko Mäkelä2017-04-211-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Alias the InnoDB ulint and lint data types to size_t and ssize_t, which are the standard names for the machine-word-width data types. Correspondingly, define ULINTPF as "%zu" and introduce ULINTPFx as "%zx". In this way, better compiler warnings for type mismatch are possible. Furthermore, use PRIu64 for that 64-bit format, and define the feature macro __STDC_FORMAT_MACROS to enable it on Red Hat systems. Fix some errors in error messages, and replace some error messages with assertions. Most notably, an IMPORT TABLESPACE error message in InnoDB was displaying the number of columns instead of the mismatching flags.
* | | Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3Alexander Barkov2017-04-031-7/+4
|\ \ \ | |/ /
View Lorry Controller Status