summaryrefslogtreecommitdiff
path: root/mysql-test/suite/mariabackup
Commit message (Collapse)AuthorAgeFilesLines
* Fix test result.Vladislav Vaintroub2018-11-151-2/+0
| | | | | | in 10.3, AWS encryption key was loaded during recovery, by the innodb purge thread, therefore no note "loaded key" was written by the foreground thread.
* Merge branch '10.2' into 10.3Sergei Golubchik2018-09-284-1/+18
|\
| * Amend fix for MDEV-17236Vladislav Vaintroub2018-09-201-1/+1
| | | | | | | | | | | | | | | | | | | | Simplify, and make it work with system tablespace outside of innodb data home. Also, do not reread TRX_SYS page in endless loop, if it appears to be corrupted. Use finite number of attempts.
| * MDEV-17236 mariabackup incorrectly tries to open ibdata1Thirunarayanan Balathandayuthapani2018-09-201-0/+1
| | | | | | | | | | | | | | | | | | if custom undo tablespace is defined - In case of multiple undo tablespace, mariabackup have to open system tablespace to find the list of undo tablespace present in TRX_SYS page. For opening system tablespace, mariabackup should fetch the file name from already initialized system tablespace object.
| * MDEV-17192 Backup with -no-lock should fail, if DDL is detected at the end ↵Vladislav Vaintroub2018-09-142-0/+16
| | | | | | | | of backup
* | MDEV-13564 Mariabackup does not work with TRUNCATEMarko Mäkelä2018-09-071-1/+0
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a merge from 10.2, but the 10.2 version of this will not be pushed into 10.2 yet, because the 10.2 version would include backports of MDEV-14717 and MDEV-14585, which would introduce a crash recovery regression: Tables could be lost on table-rebuilding DDL operations, such as ALTER TABLE, OPTIMIZE TABLE or this new backup-friendly TRUNCATE TABLE. The test innodb.truncate_crash occasionally loses the table due to the following bug: MDEV-17158 log_write_up_to() sometimes fails
| * | MDEV-13564 Mariabackup does not work with TRUNCATEMarko Mäkelä2018-09-071-1/+0
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib*.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.
* | Merge 10.2 into 10.3Marko Mäkelä2018-09-063-0/+23
|\ \ | |/
| * MDEV-17149 mariabackup hangs if innodb is not startedVladislav Vaintroub2018-09-063-0/+23
| | | | | | | | Fix exit condition for the log copying thread.
* | Fixed wrong result file that happend because of a previous mergeMonty2018-08-311-0/+1
| |
* | Merge 10.2 into 10.3Marko Mäkelä2018-08-285-36/+25
|\ \ | |/
| * MDEV-13564: Refuse MLOG_TRUNCATE in mariabackupMarko Mäkelä2018-08-162-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The MySQL 5.7 TRUNCATE TABLE is inherently incompatible with hot backup, because it is creating and deleting a separate log file, and it is not writing redo log for all changes of the InnoDB data dictionary tables. Refuse to create a corrupted backup if the unsafe form of TRUNCATE was executed. Note: Undo log tablespace truncation cannot be detected easily. Also it is incompatible with backup, for similar reasons. xtrabackup_backup_func(): "Subscribe to" the log events before the first invocation of xtrabackup_copy_logfile(). recv_parse_or_apply_log_rec_body(): If the function pointer log_truncate is set, invoke it to report MLOG_TRUNCATE.
| * Re-enable the test mariabackup.unsupported_redoMarko Mäkelä2018-08-163-36/+2
| | | | | | | | | | | | Remove the logic for skipping the test if a log checkpoint occurred, and the logic for tolerating failures. Thanks to MDEV-16791, MLOG_INDEX_LOAD is supposed to always work.
* | Merge 10.2 into 10.3Marko Mäkelä2018-08-1619-9/+449
|\ \ | |/
| * MDEV-16791 mariabackup : Support DDL commands during backupVladislav Vaintroub2018-08-1418-9/+447
| |
| * adjust result fileVladislav Vaintroub2018-08-131-0/+2
| |
* | Merge 10.2 into 10.3Marko Mäkelä2018-07-262-0/+84
|\ \ | |/
| * MDEV-16809 Allow full redo logging for ALTER TABLEMarko Mäkelä2018-07-262-0/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Introduce the configuration option innodb_log_optimize_ddl for controlling whether native index creation or table-rebuild in InnoDB should keep optimizing the redo log (and writing MLOG_INDEX_LOAD records to ensure that concurrent backup would fail). By default, we have innodb_log_optimize_ddl=ON, that is, the default behaviour that was introduced in MariaDB 10.2.2 (with the merge of InnoDB from MySQL 5.7) will be unchanged. BtrBulk::m_trx: Replaces m_trx_id. We must be able to check for KILL QUERY even if !m_flush_observer (innodb_log_optimize_ddl=OFF). page_cur_insert_rec_write_log(): Declare globally, so that this can be called from PageBulk::insert(). row_merge_insert_index_tuples(): Remove the unused parameter trx_id. row_merge_build_indexes(): Enable or disable redo logging based on the innodb_log_optimize_ddl parameter. PageBulk::init(), PageBulk::insert(), PageBulk::finish(): Write redo log records if needed. For ROW_FORMAT=COMPRESSED, redo log will be written in PageBulk::compress() unless we called m_mtr.set_log_mode(MTR_LOG_NO_REDO).
* | Merge 10.2 into 10.3Marko Mäkelä2018-07-062-0/+2
|\ \ | |/
| * MDEV-14188 mariabackup.incremental_encrypted wrong resultMarko Mäkelä2018-07-062-0/+2
| | | | | | | | | | | | Add an explicit redo log flush. In this test innodb_flush_log_at_trx_commit was 2 by default. It is also possible that this failure occurs because of MDEV-15740.
* | Merge 10.2 into 10.3Marko Mäkelä2018-07-033-5/+1
|\ \ | |/
| * MDEV-16571 - some backup tests sometimes with missing data after restore.Vladislav Vaintroub2018-07-023-5/+1
| | | | | | | | | | | | | | | | Marko mentions, it could be caused by MDEV-15740 where InnoDB does not flush redo log as often as it should, with innodb_flush_log_at_trx_commit=1 The workaround is to use innodb_flush_log_at_trx_commit=2, which, according to MDEV-15740 is more durable.
* | Merge branch '10.2' into 10.3Sergei Golubchik2018-06-304-4/+20
|\ \ | |/
| * MDEV-16519 : mariabackup should fail if MDL could not be acquired with ↵Vladislav Vaintroub2018-06-222-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | lock-ddl-per-table There is a tiny chance for race condition during MDL acquisition. If table is renamed just prior to SELECT 1 FROM <table_name> LIMIT 0 then this query would fail, yet mariabackup --backup does not handle it as fatal error and continues, only to fail later during file copy. The fix is to die on error, of MDL lock query fails.
| * MDEV-16496 Mariabackup: Implement --verbose option to instrument InnoDB log ↵Marko Mäkelä2018-06-152-4/+4
| | | | | | | | | | | | | | | | | | | | apply srv_print_verbose_log: Introduce the value 2 to refer to mariabackup --verbose. recv_recover_page(), recv_parse_log_recs(): Add output for mariabackup --verbose.
* | Merge 10.2 into 10.3Marko Mäkelä2018-06-1810-0/+146
|\ \ | |/
| * MDEV-13122 Backup myrocksdb with mariabackup.Vladislav Vaintroub2018-06-0710-0/+146
| |
* | Merge 10.2 into 10.3Marko Mäkelä2018-05-295-2/+4
|\ \ | |/
| * Allow tests to work with cmake -DPLUGIN_PARTITION=NOMarko Mäkelä2018-05-295-2/+4
| |
* | Merge branch '10.2' into 10.3Sergei Golubchik2018-05-112-0/+25
|\ \ | |/
| * Merge branch '10.1' into 10.2Sergei Golubchik2018-05-102-0/+25
| |\
| | * MDEV-16105: Mariabackup does not support SSLVladislav Vaintroub2018-05-082-0/+25
| | | | | | | | | | | | The reason is the missing HAVE_OPENSSL define for mariabackup.
* | | Merge 10.2 into 10.3Marko Mäkelä2018-04-233-0/+42
|\ \ \ | |/ /
| * | Merge 10.1 into 10.2Marko Mäkelä2018-04-213-0/+42
| |\ \ | | |/
| | * MDEV-15780 : Mariabackup with absolute paths in innodb_data_file_pathVladislav Vaintroub2018-04-123-0/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | System tablespace can be specified with absolute path, if innodb_data_home_dir is empty. This was not handled well by mariabackup 1. In backup phase, empty innodb_data_home_dir variable from server was not recognized. full paths were stored in backup-my.ini, even if it stored all files locally. thus prepare phase would not find the system tablespace files. 2. In copy-back phase, copy would not be done to the absolute destination path, as path would be stripped with trim_dotslash This patch fixes the above defects.
* | | Merge 10.2 into 10.3Marko Mäkelä2018-03-292-15/+37
|\ \ \ | |/ /
| * | MDEV-15682 mariabackup.unsupported_redo fails in buildbot with wrong result codeMarko Mäkelä2018-03-292-15/+37
| | | | | | | | | | | | | | | | | | | | | Skip the test mariabackup.unsupported_redo if a checkpoint occurred before mariabackup --backup completed. Remove the slow shutdowns and restarts which were attempting to prevent the checkpoints from occurring.
* | | Merge branch '10.2' into 10.3Sergei Golubchik2018-03-285-0/+162
|\ \ \ | |/ /
| * | Merge 10.1 into 10.2Marko Mäkelä2018-03-223-0/+40
| |\ \ | | |/
| | * MDEV-13561 Mariabackup is incompatible with retroactively created ↵Thirunarayanan Balathandayuthapani2018-03-223-0/+40
| | | | | | | | | | | | | | | | | | | | | innodb_undo_tablespaces - Mariabackup supports starting undo tablespace id which is greater than 1.
| * | MDEV-15384 buf_flush_LRU_list_batch() always reports n->flushed=0, n->evicted=0Thirunarayanan Balathandayuthapani2018-03-162-6/+18
| | | | | | | | | | | | | | | | | | | | | MDEV-14545 Backup fails due to MLOG_INDEX_LOAD record - Changed the unsupported_redo test case to avoid checkpoint - Inserting more rows in purge_secondary test case to display evict monitor.
| * | MDEV-14545 Backup fails due to MLOG_INDEX_LOAD recordThirunarayanan Balathandayuthapani2018-03-142-32/+4
| | | | | | | | | | | | | | | - Fixing the windows failure of unsupported_redo test case. mariabackup --tables-exclude option only restricts ibd file.
| * | MDEV-14545 Backup fails due to MLOG_INDEX_LOAD recordThirunarayanan Balathandayuthapani2018-03-132-0/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: ======= Mariabackup exits during prepare phase if it encounters MLOG_INDEX_LOAD redo log record. MLOG_INDEX_LOAD record informs Mariabackup that the backup cannot be completed based on the redo log scan, because some information is purposely omitted due to bulk index creation in ALTER TABLE. Solution: ======== Detect the MLOG_INDEX_LOAD redo record during backup phase and exit the mariabackup with the proper error message.
* | | MDEV-15158 On commit, do not write to the TRX_SYS pageMarko Mäkelä2018-02-202-0/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is based on a prototype by Thirunarayanan Balathandayuthapani <thiru@mariadb.com>. Binlog and Galera write-set replication information was written into TRX_SYS page on each commit. Instead of writing to the TRX_SYS during normal operation, InnoDB can make use of rollback segment header pages, which are already being written to during a commit. The following list of fields in rollback segment header page are added: TRX_RSEG_BINLOG_OFFSET TRX_RSEG_BINLOG_NAME (NUL-terminated; empty name = not present) TRX_RSEG_WSREP_XID_FORMAT (0=not present; 1=present) TRX_RSEG_WSREP_XID_GTRID TRX_RSEG_WSREP_XID_BQUAL TRX_RSEG_WSREP_XID_DATA trx_sys_t: Introduce the fields recovered_binlog_filename, recovered_binlog_offset, recovered_wsrep_xid. To facilitate upgrade from older mysql or mariaDB versions, we will read the information in TRX_SYS page. It will be overridden by the information that we find in rollback segment header pages. Mariabackup --prepare will read the metadata from the rollback segment header pages via trx_rseg_array_init(). It will still not read any undo log pages or recover any transactions.
* | | Merge bb-10.2-ext into 10.3Marko Mäkelä2018-02-192-0/+11
|\ \ \ | |/ /
| * | MDEV-15071 backup does not store xtrabackup_info in the --extra-lsndir ↵Vladislav Vaintroub2018-02-152-0/+11
| | | | | | | | | | | | directory
* | | Merge bb-10.2-ext into 10.3Marko Mäkelä2018-02-159-9/+47
|\ \ \ | |/ /
| * | MDEV-15270 Mariabackup should not try to use doublewrite buffermariadb-10.2.13Marko Mäkelä2018-02-121-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When Mariabackup gets a bad read of the first page of the system tablespace file, it would inappropriately try to apply the doublewrite buffer and write changes back to the data file (to the source file)! This is very wrong and must be prevented. The correct action would be to retry reading the system tablespace as well as any other files whose first page was read incorrectly. Fixing this was not attempted. xb_load_tablespaces(): Shorten a bogus message to be more relevant. The message can be displayed by --backup or --prepare. xtrabackup_backup_func(), os_file_write_func(): Add a missing space to a message. Datafile::restore_from_doublewrite(): Do not even attempt the operation in Mariabackup. recv_init_crash_recovery_spaces(): Do not attempt to restore the doublewrite buffer in Mariabackup (--prepare or --export), because all pages should have been copied correctly in --backup already, and because --backup should ignore the doublewrite buffer. SysTablespace::read_lsn_and_check_flags(): Do not attempt to initialize the doublewrite buffer in Mariabackup. innodb_make_page_dirty(): Correct the bounds check. Datafile::read_first_page(): Correct the name of the parameter.
| * | Merge branch 'github/10.1' into 10.2Sergei Golubchik2018-02-062-0/+42
| |\ \ | | |/
| | * MDEV-13499: Backing up table that "doesn't exist in engine" cause crash in ↵Jan Lindström2018-01-292-0/+33
| | | | | | | | | | | | | | | | | | | | | mariabackup when using encryption Problem was that there is intentional crah when .ibd file does not found. In mariabackup case we should avoid this crash.
View Lorry Controller Status