summaryrefslogtreecommitdiff
path: root/extra
Commit message (Collapse)AuthorAgeFilesLines
* MDEV-25321: mariabackup failed if password is passed via environment variablebb-10.2-MDEV-25321Julius Goryavsky2021-04-011-0/+7
| | | | | | | | | | | | The mariabackup interface currently supports passing a password through an explicit command line variable, but does not support passing a password through the MYSQL_PWD environment variable. At the same time, the Galera SST script for mariabackup uses the environment variable to pass the password, which leads (in some cases) to an unsuccessful launch of mariabackup and to the inability to start the cluster. This patch fixes this issue. It does not need a separate test, as the problem is visible in general testing on buildbot.
* MDEV-24197: Add "innodb_force_recovery" for "mariabackup --prepare"Srinidhi Kaushik2021-04-013-8/+68
| | | | | | | | | | | | | | | | | | | During the prepare phase of restoring backups, "mariabackup" does not seem to allow (or recognize) the option "innodb_force_recovery" for the embedded InnoDB server instance that it starts. If page corruption observed during page recovery, the prepare step fails. While this is indeed the correct behavior ideally, allowing this option to be set in case of emergencies might be useful when the current backup is the only copy available. Some error messages during "--prepare" suggest to set "innodb_force_recovery" to 1: [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption. For backwards compatibility, "mariabackup --innobackupex --apply-log" should also have this option. Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
* MDEV-25221 Do not remove source file, if copy_file() fails in mariabackup ↵Vladislav Vaintroub2021-03-319-5/+27
| | | | | | --move-back Remove an incompletely copied destination file.
* MDEV-22929 fixup. Print "completed OK!" if page corruption and ↵Vladislav Vaintroub2021-03-051-4/+3
| | | | | | | --log-innodb-page-corruption Since we do not stop at corrupted page error, there is no reason to log a backup error.
* MDEV-22810 mariabackup does not honor open_files_limit from option during ↵Vlad Lesin2020-12-161-0/+4
| | | | | | | backup prepare open_files_limit option was processed only for --backup, but not for --prepare.
* MDEV-24340 Unique final message of InnoDB during shutdownbb-10.2-MDEV-24340Marko Mäkelä2020-12-041-4/+1
| | | | | | | | | | | innobase_space_shutdown(): Remove. We want this step to be executed before the message "InnoDB: Shutdown completed; log sequence number " is output by innodb_shutdown(). It used to be executed after that step. innodb_shutdown(): Duplicate the code that used to live in innobase_space_shutdown(). innobase_init_abort(): Merge with innobase_space_shutdown().
* MDEV-22929 MariaBackup option to report and/or continue when corruption is ↵Vlad Lesin2020-12-0110-127/+489
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | encountered The new option --log-innodb-page-corruption is introduced. When this option is set, backup is not interrupted if innodb corrupted page is detected. Instead it logs all found corrupted pages in innodb_corrupted_pages file in backup directory and finishes with error. For incremental backup corrupted pages are also copied to .delta file, because we can't do LSN check for such pages during backup, innodb_corrupted_pages will also be created in incremental backup directory. During --prepare, corrupted pages list is read from the file just after redo log is applied, and each page from the list is checked if it is allocated in it's tablespace or not. If it is not allocated, then it is zeroed out, flushed to the tablespace and removed from the list. If all pages are removed from the list, then --prepare is finished successfully and innodb_corrupted_pages file is removed from backup directory. Otherwise --prepare is finished with error message and innodb_corrupted_pages contains the list of the pages, which are detected as corrupted during backup, and are allocated in their tablespaces, what means backup directory contains corrupted innodb pages, and backup can not be considered as consistent. For incremental --prepare corrupted pages from .delta files are applied to the base backup, innodb_corrupted_pages is read from both base in incremental directories, and the same action is proceded for corrupted pages list as for full --prepare. innodb_corrupted_pages file is modified or removed only in base directory. If DDL happens during backup, it is also processed at the end of backup to have correct tablespace names in innodb_corrupted_pages.
* MDEV-24026: InnoDB: Failing assertion: os_total_large_mem_allocated >= size ↵Vlad Lesin2020-10-293-2/+15
| | | | | | | | upon incremental backup mariabackup deallocated uninitialized write_filt_ctxt.u.wf_incremental_ctxt in xtrabackup_copy_datafile() when some table should be skipped due to parsed DDL redo log record.
* MDEV-20755 InnoDB: Database page corruption on disk or a failed file read of ↵Vlad Lesin2020-10-231-51/+102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | tablespace upon prepare of mariabackup incremental backup The problem: When incremental backup is taken, delta files are created for innodb tables which are marked as new tables during innodb ddl tracking. When such tablespace is tried to be opened during prepare in xb_delta_open_matching_space(), it is "created", i.e. xb_space_create_file() is invoked, instead of opening, even if a tablespace with the same name exists in the base backup directory. xb_space_create_file() writes page 0 header the tablespace. This header does not contain crypt data, as mariabackup does not have any information about crypt data in delta file metadata for tablespaces. After delta file is applied, recovery process is started. As the sequence of recovery for different pages is not defined, there can be the situation when crypt data redo log event is executed after some other page is read for recovery. When some page is read for recovery, it's decrypted using crypt data stored in tablespace header in page 0, if there is no crypt data, the page is not decryped and does not pass corruption test. This causes error for incremental backup --prepare for encrypted tablespaces. The error is not stable because crypt data redo log event updates crypt data on page 0, and recovery for different pages can be executed in undefined order. The fix: When delta file is created, the corresponding write filter copies only the pages which LSN is greater then some incremental LSN. When new file is created during incremental backup, the LSN of all it's pages must be greater then incremental LSN, so there is no need to create delta for such table, we can just copy it completely. The fix is to copy the whole file which was tracked during incremental backup with innodb ddl tracker, and copy it to base directory during --prepare instead of delta applying. There is also DBUG_EXECUTE_IF() in innodb code to avoid writing redo log record for crypt data updating on page 0 to make the test case stable. Note: The issue is not reproducible in 10.5 as optimized DDL's are deprecated in 10.5. But the fix is still useful because it allows to decrease data copy size during backup, as delta file contains some extra info. The test case should be removed for 10.5 as it will always pass.
* MDEV-21951: mariabackup SST fail if data-directory have lost+found directoryJulius Goryavsky2020-10-201-1/+1
| | | | | | | | | | | | | To fix this, it is necessary to add an option to exclude the database with the name "lost+found" from processing (the database name will be checked by the check_if_skip_database_by_path() or by the check_if_skip_database() function, and as a result "lost+found" will be skipped). In addition, it is necessary to slightly modify the verification logic in the check_if_skip_database() function. Also added a new test galera_sst_mariabackup_lost_found.test
* MDEV-23982: Mariabackup hangs on backupMarko Mäkelä2020-10-191-1/+1
| | | | | | | | MDEV-13318 introduced a condition to Mariabackup that can cause it to hang if the server goes idle after writing a log block that has no payload after the 12-byte header. Normal recovery in log0recv.cc would allow blocks with exactly 12 bytes of length, and only reject blocks where the length field is shorter than that.
* MDEV-23711 fixup: GCC -Og -Wmaybe-uninitializedMarko Mäkelä2020-09-211-1/+1
|
* MDEV-23711 make mariabackup innodb redo log read error message more clearVlad Lesin2020-09-211-3/+16
| | | | | | | | | | | | | | | | | | | | | log_group_read_log_seg() returns error when: 1) Calculated log block number does not correspond to read log block number. This can be caused by: a) Garbage or an incompletely written log block. We can exclude this case by checking log block checksum if it's enabled(see innodb-log-checksums, encrypted log block contains checksum always). b) The log block is overwritten. In this case checksum will be correct and read log block number will be greater then requested one. 2) When log block length is wrong. In this case recv_sys->found_corrupt_log is set. 3) When redo log block checksum is wrong. In this case innodb code writes messages to error log with the following prefix: "Invalid log block checksum." The fix processes all the cases above.
* MDEV-19264 Better support MariaDB GTID for Mariabackup's --slave-info optionVlad Lesin2020-09-141-1/+4
| | | | | | Parse SHOW SLAVE STATUS output for the "Using_Gtid" column. If the value is "No", then old log file and position is backed up, otherwise gtid_slave_pos is backed up.
* MDEV-23475 InnoDB performance regression for write-heavy workloadsMarko Mäkelä2020-08-191-0/+6
| | | | | | | | | | | | | | | | | In commit fe39d02f51b96536dccca7ff89faf05e13548877 (MDEV-20638) we removed some wake-up signaling of the master thread that should have been there, to ensure a steady log checkpointing workload. Common sense suggests that the commit omitted some necessary calls to srv_inc_activity_count(). But, an attempt to add the call to trx_flush_log_if_needed_low() as well as to reinstate the function innobase_active_small() did not restore the performance for the case where sync_binlog=1 is set. Therefore, we will revert the entire commit in MariaDB Server 10.2. In MariaDB Server 10.5, adding a srv_inc_activity_count() call to trx_flush_log_if_needed_low() did restore the performance, so we will not revert MDEV-20638 across all versions.
* MDEV-23474 InnoDB fails to restart after SET GLOBAL innodb_log_checksums=OFFMarko Mäkelä2020-08-181-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | Regretfully, the parameter innodb_log_checksums was introduced in MySQL 5.7.9 (the first GA release of that series) by mysql/mysql-server@af0acedd885eb7103e319f79d25fda7386ef1506 which partly replaced a parameter that had been introduced in 5.7.8 mysql/mysql-server@22ba38218e1d76c24f69b5a5595ad3bf5933acb0 as innodb_log_checksum_algorithm. Given that the CRC-32C operations are accelerated on many processor implementations (AMD64 with SSE4.2; since MDEV-22669 also on IA-32 with SSE4.2, POWER 8 and later, ARMv8 with some extensions) and by lookup tables when only generic SISD instructions are available, there should be no valid reason to disable checksums. In MariaDB 10.5.2, as a preparation for MDEV-12353, MDEV-19543 deprecated and ignored the parameter innodb_log_checksums altogether. This should imply that after a clean shutdown with innodb_log_checksums=OFF one cannot upgrade to MariaDB Server 10.5 at all. Due to these problems, let us deprecate the parameter innodb_log_checksums and honor it only during server startup. The command SET GLOBAL innodb_log_checksums will always set the parameter to ON.
* Fix GCC 10.2.0 -Og -Wmaybe-uninitializedMarko Mäkelä2020-08-111-1/+1
| | | | | | For some reason, GCC emits more -Wmaybe-uninitialized warnings when using the flag -Og than when using -O2. Many of the warnings look genuine.
* MDEV-20638 Remove the deadcode from srv_master_thread() and ↵Thirunarayanan Balathandayuthapani2020-07-231-6/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | srv_active_wake_master_thread_low() - Due to commit fe95cb2e40d73adbbe88efa0e7f48c055ccd042f (MDEV-16125), InnoDB master thread does not need to call srv_resume_thread() and therefore there is no need to wake up the thread. Due to the above patch, InnoDB should remove the following dead code. srv_check_activity(): Makes the parameter as in,out and returns the recent activity value innobase_active_small(): Removed srv_active_wake_master_thread(): Removed srv_wake_master_thread(): Removed srv_active_wake_master_thread_low(): Removed Simplify srv_master_thread() and remove switch cases, added the assert. Replace srv_wake_master_thread() with srv_inc_activity_count() INNOBASE_WAKE_INTERVAL: Removed
* Merge 10.1 into 10.2bb-10.2-mergeMarko Mäkelä2020-07-011-1/+1
|\
| * mtr: use env for perlDaniel Black2020-06-231-1/+1
| | | | | | | | | | | | | | | | On FreeBSD, perl isn't in /usr/bin, its in /usr/local/bin or elsewhere in the path. Like storage/{maria/unittest/,}ma_test_* , we use /usr/bin/env to find perl and run it.
| * mbstream: Remove duplicate definition of datasink_bufferMarko Mäkelä2020-04-151-1/+0
| | | | | | | | | | | | | | | | | | For some reason, when mbstream is compiled with GCC 10, ld will fail to link it due to a multiple definition of datasink_buffer. The code appears to have been like that since the very beginning. Let us remove the duplicate dummy definition.
* | MDEV-22545: my_vsnprintf behaves not as in C standardOleksandr Byelkin2020-05-241-0/+1
| | | | | | | | Added parameter %T for string which should be visibly truncated.
* | MDEV-22554: "mariabackup --prepare" exits with code 0 even though innodbVlad Lesin2020-05-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | error is logged The fix is to set flag in ib::error::~error() and check it in mariabackup. ib::error::error() is replaced with ib::warn::warn() in AIO::linux_create_io_ctx() because of two reasons: 1) if we leave it as is, then mariabackup MTR tests will fail with --mem option, because Linux AIO can not be used on tmpfs, 2) when Linux AIO can not be initialized, InnoDB falls back to simulated AIO, so such sutiation is not fatal error, it should be treated as warning.
* | MDEV-20230: mariabackup --ftwrl-wait-timeout never times out on explicitbb-10.2-MDEV-20230-ftwrl-wait-timeoutVlad Lesin2020-04-271-55/+69
| | | | | | | | | | | | | | | | | | | | | | | | lock --ftwrl-wait-timeout does not finish mariabackup execution when acquired backup lock can't be grabbed for the certain amount of time, it just waits for a long queries finishing before acquiring the lock to avoid unnecessary locking. This commit extends --ftwrl-wait-timeout so, that mariabackup execution is finished if it waits for backup lock during certain amount of time.
* | MDEV-19347: Mariabackup does not honor ignore_db_dirs from serverbb-10.2-MDEV-19347-ignore_db_dirsVlad Lesin2020-04-213-177/+208
| | | | | | | | | | | | | | | | | | | | | | config. The solution is to read the system variable value on startup and to fill databases_exclude_hash. xb_load_list_string() became non-static and was reformatted. The system variable value is read and processed in get_mysql_vars(), which was also reformatted.
* | Fixed memory leaks in resolve_stack_dumpMonty2020-04-181-15/+30
| | | | | | | | | | | | - Remove memory leaks reported by safemalloc - Changed that all 0x strings are converted. This is needed to easily be able to resolve safemalloc backtraces
* | MDEV-21168: Active XA transactions stop slave from working after backupVlad Lesin2020-04-071-357/+468
| | | | | | | | | | | | | | | | | | was restored. Optionally rollback prepared XA's on "mariabackup --prepare". The fix MUST NOT be ported on 10.5+, as MDEV-742 fix solves the issue for slaves.
* | fix use-after-freeEugene Kosov2020-03-101-2/+2
| |
* | Merge branch '10.1' into 10.2Oleksandr Byelkin2020-01-243-7/+9
|\ \ | |/
| * MDEV-17601: MariaDB Galera does not expect 'mbstream' as streamfmtJulius Goryavsky2020-01-213-7/+9
| | | | | | | | | | | | | | | | | | | | | | Setting "streamfmt=mbstream" in the "[sst]" section causes SST to fail because the format automatically switches to 'tar' by default (insead of mbstream). To fix this, we need to add mbstream to the list of valid values for the format, making it synonymous with xbstream. This must be done both in the SST script and when parsing the options of the corresponding utilities.
* | MDEV-19176 Reduce the memory usage during recoveryThirunarayanan Balathandayuthapani2019-12-231-1/+3
| | | | | | | | | | | | | | | | | | - Moved the recv_sys->heap memory condition inside recv_parse_log_recs(). So that, InnoDB can mark the status as STORE_NO earlier. - InnoDB uses one third of buffer pool chunk size for reading the redo log records. In that case, we can avoid the scenario where buffer ran out of memory issue during recovery.
* | MDEV-21255: Deadlock of parallel slave and mariabackup (with failed logVlad Lesin2019-12-121-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | copy thread) mariabackup hangs waiting until innodb redo log thread read log till certain LSN, and it waits under FTWRL. If there is redo log read error in the thread, it is finished, and main thread knows nothing about it, what leads to hanging. As it hangs under FTWRL, slave threads on server side can be blocked due to MDL lock conflict. The fix is to finish mariabackup with error message on innodb redo log read failure.
* | MDEV-11345 Compile english error messages into mysqld executable.Vladislav Vaintroub2019-12-111-32/+6
| | | | | | | | | | Simplify loading messages into mariabackup. Do the same as server does We're forcing english, so there is no attempt to load errmsg.sys
* | MDEV-18310: Aria engine: Undo phase failed with "Got error 121 whenVlad Lesin2019-11-291-11/+20
| | | | | | | | | | | | | | | | executing undo undo_key_delete" upon startup on datadir restored from incremental backup aria_log* files were not copied on --prepare --incremental-dir step from incremental to destination backup directory.
* | Merge 10.1 to 10.2Marko Mäkelä2019-11-062-2/+2
|\ \ | |/
| * Fix GCC 9.2.1 -Wstringop-truncationMarko Mäkelä2019-11-043-3/+4
| | | | | | | | | | | | | | | | | | dict_table_rename_in_cache(): Use strcpy() instead of strncpy(), because they are known to be equivalent in this case (the length of old_name was already validated). mariabackup: Invoke strncpy() with one less than the buffer size, and explicitly add NUL as the last byte of the buffer.
* | MDEV-19335 Remove buf_page_t::encryptedThirunarayanan Balathandayuthapani2019-10-091-3/+1
| | | | | | | | | | The field buf_page_t::encrypted was added in MDEV-8588. It was made mostly redundant in MDEV-12699. Remove the field.
* | MDEV-20703: mariabackup creates binlog files in server binlog directory on ↵Vlad Lesin2019-10-011-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | --prepare --export step When "--export" mariabackup option is used, mariabackup starts the server in bootstrap mode to generate *.cfg files for the certain innodb tables. The started instance of the server reads options from the file, pointed out in "--defaults-file" mariabackup option. If the server uses the same config file as mariabackup, and binlog is switched on in that config file, then "mariabackup --prepare --export" will create binary log files in the server's binary log directory, what can cause issues. The fix is to add "--skip-log-bin" in mysld options when the server is started to generate *.cfg files.
* | MDEV-18438 Don't stream xtrabackup_info of extra-lsndirSimon Lipp2019-09-194-10/+41
| |
* | MDEV-19740: Silence a bogus "may be uninitialized" warningMarko Mäkelä2019-08-151-8/+3
| |
* | Merge 10.1 into 10.2Marko Mäkelä2019-08-121-1/+2
|\ \ | |/
| * MDEV-17638 Improve error message about corruption of encrypted pagebb-10.1-MDEV-17638-improve-warningEugene Kosov2019-08-011-1/+2
| | | | | | | | Help user distinguish between space ID and page number.
* | MDEV-20060: Failing assertion: srv_log_file_size <= 512ULL << 30 while ↵Vlad Lesin2019-08-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | preparing backup The general reason why innodb redo log file is limited by 512G is that log_block_convert_lsn_to_no() returns value limited by 1G. But there is no need to have unique log block numbers in log group. The fix removes 512G limit and limits log group size by (uint32_t maximum value) * (minimum page size), which, in turns, can be removed if fil_io() is no longer used for innodb redo log io.
* | Merge 10.1 into 10.2Eugene Kosov2019-07-091-1/+1
|\ \ | |/
| * imporve clang buildEugene Kosov2019-06-253-4/+2
| | | | | | | | | | | | | | | | | | cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Debug Maintainer mode makes all warnings errors. This patch fix warnings. Mostly about deprecated `register` keyword. Too much warnings came from Mroonga and I gave up on it.
* | MDEV-19781 Add page id matching check in innochecksum toolThirunarayanan Balathandayuthapani2019-06-281-13/+88
| | | | | | | | | | | | | | | | Added the condition in innochecksum tool to check page id mismatch. This could catch the write corruption caused by InnoDB. Added the debug insert inside fil_io() to check whether it writes the page to wrong offset.
* | MDEV-19865: AddressSanitizer error in open_or_create_log_file() in mariabackupVlad Lesin2019-06-261-1/+1
| | | | | | | | Decrease array on stack in open_or_create_log_file() to avoid stack overflow.
* | Fixed compiler warningMonty2019-06-031-1/+5
| | | | | | | | | | | | Wrong compiler warning from GCC: ‘snprintf’ output 2 or more bytes (assuming 4001) into a destination of size 4000
* | MDEV-14192: mariabackup.incremental_backup failed in buildbot with Failing ↵Vlad Lesin2019-05-241-33/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | assertion: byte_offset % OS_FILE_LOG_BLOCK_SIZE == 0 In some cases it's possible that InnoDB redo log file header is re-written so, that checkpoint lsn and checkpoint lsn offset are updated, but checkpoint number stays the same. The fix is to re-read redo log header if at least one of those three parametes is changed at backup start. Repeat the logic of log_group_checkpoint() on choosing InnoDB checkpoint info field on backup start. This does not influence backup correctness, but simplifies bugs analysis.
* | Merge 10.1 into 10.2Marko Mäkelä2019-05-13251-264/+264
|\ \ | |/
View Lorry Controller Status