summaryrefslogtreecommitdiff
path: root/extra/mariabackup
Commit message (Collapse)AuthorAgeFilesLines
...
* | MDEV-18611: mariabackup silently ended during xtrabackup_copy_logfile()Marko Mäkelä2019-02-191-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | log_group_read_log_seg(): Always return false when returning before reading end_lsn. xtrabackup_copy_logfile(): On error, indicate whether a corrupt log record was encountered. Only xtrabackup_copy_logfile() in Mariabackup cared about the return value of the function. InnoDB crash recovery was not affected by this bug.
* | MDEV-18204 - fixupVladislav Vaintroub2019-02-191-11/+11
| |
* | MDEV-18204 Fix rocksdb incremental backupVladislav Vaintroub2019-02-181-0/+13
| | | | | | | | | | Fix incremental prepare to copy #rocksdb subdirectory from the incremental dir.
* | MDEV-18575 remove innodb-encrypt-log parameter from mariabackupVladislav Vaintroub2019-02-141-5/+0
| | | | | | | | | | | | | | | | | | The variable is obsolete. In mariabackup --backup, encryption plugin loading code sets the value this parameter to the same as in server. In mariabackup --prepare, no new redo log is generated, and xtrabackup_logfile is removed after it anyway.
* | Merge 10.1 into 10.2Marko Mäkelä2019-02-021-18/+16
|\ \ | |/
| * mariabackup : Remove unused parameter innodb_buffer_pool_sizeVladislav Vaintroub2019-01-301-15/+0
| |
| * MDEV-18347: mariabackup doesn't read all server option groups from ↵Geoff Montee2019-01-241-4/+16
| | | | | | | | configuration files
* | MDEV-18380 : adjust max_statement_time in mariabackupVladislav Vaintroub2019-02-011-1/+1
| |
* | MDEV-18415 mariabackup.mdev-14447 test case fails with Table 'test.t' ↵Thirunarayanan Balathandayuthapani2019-02-011-4/+11
| | | | | | | | | | | | | | doesn't exist in engine - Added retry logic if validation of first page fails with checksum mismatch.
* | MDEV-18356 Renamed backup-encrypted option introduced in ↵Vladislav Vaintroub2019-01-233-8/+8
| | | | | | | | | | | | 7158edcba3af3766e9329f9927ce4adfd2a40bf8 Rename it because it caused parser warning whenever --backup was used.
* | mariabackup : use die() macro for fatal exit with error message.Vladislav Vaintroub2019-01-169-90/+63
| |
* | MDEV-18212 mariabackup: Make output format uniform whenever possibleVladislav Vaintroub2019-01-1522-476/+407
| |
* | Merge 10.1 into 10.2Marko Mäkelä2019-01-141-0/+2
|\ \ | |/
| * xb_process_datadir(): Fix resource leaksFaramosCZ2019-01-141-0/+2
| | | | | | | | Closes #983, #984
* | MDEV-18201 : mariabackup- fix processing of rename/create sequence in prepareVladislav Vaintroub2019-01-101-5/+8
| | | | | | | | | | | | | | | | | | Fix one more bug in "DDL redo" phase in prepare If table was renamed, and then new table was created with the old name, prepare can be confused, and .ibd can end up with wrong name. Fix the order of how DDL fixup is applied , once again - ".new" files should be processed after renames.
* | MDEV-18185 - mariabackup - fix specific case of table rename handing in prepare.Vladislav Vaintroub2019-01-091-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | If, during backup 1) Innodb table is dropped (after being copied to backup) and then 2) Before backup finished, another Innodb table is renamed, and new name is the name of the dropped table in 1) then, --prepare fails with assertion, as DDL fixup code in prepare did not handle this specific case. The fix is to process drops before renames, in prepare DDL-"redo" phase.
* | Fix a merge error in the parent commitMarko Mäkelä2019-01-041-2/+2
| | | | | | | | | | Fix an inadvertently inverted condition that caused galera.galera_sst_mariabackup_table_options test failure.
* | Merge 10.1 into 10.2Marko Mäkelä2019-01-033-9/+28
|\ \ | |/
| * MDEV-18129 Backup fails for encrypted tables: mariabackup: Database page ↵Marko Mäkelä2019-01-033-8/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | corruption detected at page 1 If an encrypted table is created during backup, then mariabackup --backup could wrongly fail. This caused a failure of the test mariabackup.huge_lsn once on buildbot. This is due to the way how InnoDB creates .ibd files. It would first write a dummy page 0 with no encryption information. Due to this, xb_fil_cur_open() could wrongly interpret that the table is not encrypted. Subsequently, page_is_corrupted() would compare the computed page checksum to the wrong checksum. (There are both "before" and "after" checksums for encrypted pages.) To work around this problem, we introduce a Boolean option --backup-encrypted that is enabled by default. With this option, Mariabackup will assume that a nonzero key_version implies that the page is encrypted. We need this option in order to be able to copy encrypted tables from MariaDB 10.1 or 10.2, because unencrypted pages that were originally created before MySQL 5.1.48 could contain nonzero garbage in the fields that were repurposed for encryption. Later, MDEV-18128 would clean up the way how .ibd files are created, to remove the need for this option. page_is_corrupted(): Add missing const qualifiers, and do not check space->crypt_data unless --skip-backup-encrypted has been specified. xb_fil_cur_read(): After a failed page read, output a page dump.
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-291-4/+10
|\ \ | |/
| * MDEV-18105 Mariabackup fails to copy encrypted InnoDB system tablespace if ↵Marko Mäkelä2018-12-291-3/+10
| | | | | | | | | | | | | | | | | | | | LSN>4G This is a regression caused by commit 8c43f963882a9d5ac4e4289c8dd3dbcaeb40a0ce that was part of the MDEV-12112 fixes. page_is_corrupted(): Never interpret page_no=0 as encrypted.
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-291-0/+9
|\ \ | |/
| * Silence LeakSanitizer by default in mariabackup, so that phanthom "leaks"Vladislav Vaintroub2018-12-291-0/+9
| | | | | | | | | | | | | | | | | | | | | | would not hide more interesting information, like invalid memory accesses. some "leaks" are expected - partly this is due to weird options parsing, that runs twice, and does not free memory after the first run. - also we do not mind to exit() whenever it makes sense, without full cleanup.
* | Add test for partial backup for partitioned table.Vladislav Vaintroub2018-12-211-8/+15
| |
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-213-96/+121
|\ \ | |/
| * Cleanup recent mariabackup validation patches.Vladislav Vaintroub2018-12-203-73/+93
| | | | | | | | | | | | | | | | | | | | | | - Refactor code to isolate page validation in page_is_corrupted() function. - Introduce --extended-validation parameter(default OFF) for mariabackup --backup to enable decryption of encrypted uncompressed pages during backup. - mariabackup would still always check checksum on encrypted data, it is needed to detect partially written pages.
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-191-14/+36
|\ \ | |/
| * MDEV-18025: Improve test case and consistency checksMarko Mäkelä2018-12-191-8/+30
| | | | | | | | | | | | | | | | | | | | | | | | Write a test case that computes valid crc32 checksums for an encrypted page, but zeroes out the payload area, so that the checksum after decryption fails. xb_fil_cur_read(): Validate the page number before trying any checksum calculation or decrypting or decompression. Also, skip zero-filled pages. For page_compressed pages, ensure that the FIL_PAGE_TYPE was changed. Also, reject FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED if no decryption was attempted.
* | MDEV-16268 : sporadic checksum mismatch when opening system tablespace in backupVladislav Vaintroub2018-12-181-2/+9
| | | | | | | | | | Attempt to fix by retrying srv_sys_space.open_or_create() if it reports corruption.
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-181-1/+24
|\ \ | |/
| * MDEV-18025 Mariabackup fails to detect corrupted page_compressed=1 tablesThirunarayanan Balathandayuthapani2018-12-181-4/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: ======= Mariabackup seems to fail to verify the pages of compressed tables. The reason is that both fil_space_verify_crypt_checksum() and buf_page_is_corrupted() will skip the validation for compressed pages. Fix: ==== Mariabackup should call fil_page_decompress() for compressed and encrypted compressed page. After that, call buf_page_is_corrupted() to check the page corruption.
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-171-3/+8
|\ \ | |/
| * Follow-up to MDEV-12112: corruption in encrypted table may be overlookedMarko Mäkelä2018-12-171-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The initial fix only covered a part of Mariabackup. This fix hardens InnoDB and XtraDB in a similar way, in order to reduce the probability of mistaking a corrupted encrypted page for a valid unencrypted one. This is based on work by Thirunarayanan Balathandayuthapani. fil_space_verify_crypt_checksum(): Assert that key_version!=0. Let the callers guarantee that. Now that we have this assertion, we also know that buf_page_is_zeroes() cannot hold. Also, remove all diagnostic output and related parameters, and let the relevant callers emit such messages. Last but not least, validate the post-encryption checksum according to the innodb_checksum_algorithm (only accepting one checksum for the strict variants), and no longer try to validate the page as if it was unencrypted. buf_page_is_zeroes(): Move to the compilation unit of the only callers, and declare static. xb_fil_cur_read(), buf_page_check_corrupt(): Add a condition before calling fil_space_verify_crypt_checksum(). This is a non-functional change. buf_dblwr_process(): Validate the page only as encrypted or unencrypted, but not both.
* | MDEV-14975 : fix last commit's typo.Vladislav Vaintroub2018-12-151-1/+1
| |
* | MDEV-14975 mariabackup starts with unprivileged user.Vladislav Vaintroub2018-12-142-5/+173
| | | | | | | | | | | | | | | | | | | | ported privilege checking from xtrabackup. Now, mariabackup would terminate early if either RELOAD or PROCESS privilege is not held, not at the very end of backup The behavior can be disabled with nre setting --check-privileges=0. Also , --no-lock does not need all of these privileges, since it skips FTWRL and SHOW ENGINE STATUS INNODB.
* | Merge 10.1 into 10.2Marko Mäkelä2018-12-141-21/+49
|\ \ | |/
| * MDEV-12112 corruption in encrypted table may be overlookedMarko Mäkelä2018-12-143-39/+61
| | | | | | | | | | | | | | | | | | | | | | After validating the post-encryption checksum on an encrypted page, Mariabackup should decrypt the page and validate the pre-encryption checksum as well. This should reduce the probability of accepting invalid pages as valid ones. This is a backport and refactoring of a patch that was originally written by Thirunarayanan Balathandayuthapani for the 10.2 branch.
| * Merge branch '10.0' into 10.1Sergei Golubchik2018-10-301-1/+1
| |
* | Remove some unnecessary InnoDB #includeMarko Mäkelä2018-11-291-1/+0
| |
* | MDEV-17849 Undo tablespace truncation recovery fails to shrink fileMarko Mäkelä2018-11-272-56/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fil_space_t::add(): Replaces fil_node_create(), fil_node_create_low(). Let the caller pass fil_node_t::handle, to avoid having to close and re-open files. fil_node_t::read_page0(): Refactored from fil_node_open_file(). Read the first page of a data file. fil_node_open_file(): Open the file only once. srv_undo_tablespace_open(): Set the file handle for the opened undo tablespace. This should ensure that ut_ad(file->is_open()) no longer fails in recv_add_trim(). xtrabackup_backup_func(): Remove some dead code. xb_fil_cur_open(): Open files only if needed. Undo tablespaces should already have been opened.
* | Remove the redundant variable fil_n_file_openedMarko Mäkelä2018-11-271-2/+0
| |
* | MDEV-17195 Speedup lock-ddl-per-table, if there is a large number of tablesVladislav Vaintroub2018-10-311-29/+30
| | | | | | | | | | | | | | Query INFORMATION_SCHEMA.INNODB_SYS_TABLES only once, and cache results. As a small cleanup, remove mdl_lock_con_mutex, the MDL handling connection is never accessed by multiple threads at the same time.
* | MDEV-14267: correct FSF addressDaniel Black2018-10-305-9/+9
| |
* | MDEV-13564 Mariabackup does not work with TRUNCATEMarko Mäkelä2018-09-251-6/+2
|\ \ | | | | | | | | | | | | | | | | | | Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP.
| * | MDEV-14717: Prevent crash-downgrade to earlier MariaDB 10.2Marko Mäkelä2018-09-071-7/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A crash-downgrade of a RENAME (or TRUNCATE or table-rebuilding ALTER TABLE or OPTIMIZE TABLE) operation to an earlier 10.2 version would trigger a debug assertion failure during rollback, in trx_roll_pop_top_rec_of_trx(). In a non-debug build, the TRX_UNDO_RENAME_TABLE record would be misinterpreted as an update_undo log record, and typically the file name would be interpreted as DB_TRX_ID,DB_ROLL_PTR,PRIMARY KEY. If a matching record would be found, row_undo_mod() would hit ut_error in switch (node->rec_type). Typically, ut_a(table2 == NULL) would fail when opening the table from SQL. Because of this, we prevent a crash-downgrade to earlier MariaDB 10.2 versions by changing the InnoDB redo log format identifier to the 10.3 identifier, and by introducing a subformat identifier so that 10.2 can continue to refuse crash-downgrade from 10.3 or later. After a clean shutdown, a downgrade to MariaDB 10.2.13 or later would still be possible thanks to MDEV-14909. A downgrade to older 10.2 versions is only possible after removing the log files (not recommended). LOG_HEADER_FORMAT_CURRENT: Change to 103 (originally the 10.3 format). log_group_t: Add subformat. For 10.2, we will use subformat 1, and will refuse crash recovery from any other subformat of the 10.3 format, that is, a genuine 10.3 redo log. recv_find_max_checkpoint(): Allow startup after clean shutdown from a future LOG_HEADER_FORMAT_10_4 (unencrypted only). We cannot handle the encrypted 10.4 redo log block format, which was introduced in MDEV-12041. Allow crash recovery from the original 10.2 format as well as the new format. In Mariabackup --backup, do not allow any startup from 10.3 or 10.4 redo logs. recv_recovery_from_checkpoint_start(): Skip redo log apply for clean 10.3 redo log, but not for the new 10.2 redo log (10.3 format, subformat 1). srv_prepare_to_delete_redo_log_files(): On format or subformat mismatch, set srv_log_file_size = 0, so that we will display the correct message. innobase_start_or_create_for_mysql(): Check for format or subformat mismatch. xtrabackup_backup_func(): Remove debug assertions that were made redundant by the code changes in recv_find_max_checkpoint().
| * | MDEV-13564 Mariabackup does not work with TRUNCATEMarko Mäkelä2018-09-071-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib*.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.
* | | Merge branch '10.1' into 10.2Sergei Golubchik2018-09-241-2/+33
|\ \ \ | | |/ | |/|
| * | MDEV-12956 provide default datadir for mariabackup --copy-backVladislav Vaintroub2018-09-111-2/+33
| | | | | | | | | | | | | | | | | | | | | | | | On Unix, it is compiled-in datadir value. On Windows, the directory is ..\data, relative to directory mariabackup.exe server uses the same logic to determine datadir.
* | | Fix mariabackup leaks (except my_load_defaults)Marko Mäkelä2018-09-213-22/+27
| | |
* | | Windows : Fix application verifier errors.Vladislav Vaintroub2018-09-202-8/+8
| | | | | | | | | | | | | | | | | | Make different threads that are running data_copy_thread_func() use the same pthread_mutex for locking (not the copies of the same pthread_mutex_t).
View Lorry Controller Status