Implement Multi Part AOF mechanism to avoid AOFRW overheads. (#9788)

Implement Multi-Part AOF mechanism to avoid overheads during AOFRW.
Introducing a folder with multiple AOF files tracked by a manifest file.

The main issues with the the original AOFRW mechanism are:
* buffering of commands that are processed during rewrite (consuming a lot of RAM)
* freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it.
* double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files)

The main modifications of this PR:
1. Remove the AOF rewrite buffer and related code.
2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type,
  it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only
  one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the
  incremental commands since the last AOFRW.
3. Use a AOF manifest file to record and manage these AOF files mentioned above.
4. The original configuration of `appendfilename` will be the base part of the new file name, for example:
  `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof`
5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename`
6. Remove the `aof_rewrite_buffer_length` field in info.
7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs.
  It also gives users the opportunity to preserve the history AOFs. just for testing use now.
8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now),
  we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be
  delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit
  period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately.
9. Support upgrade (load) data from old version redis.
10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and
  manifest file will be placed in this directory.
11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if
  `aof-load-truncated` is enabled.

Co-authored-by: Oran Agra <oran@redislabs.com>

1 files changed, 33 insertions, 10 deletions

diff --git a/redis.conf b/redis.conf
index 9d0f0dfb7..e996c95a0 100644
--- a/redis.conf
+++ b/redis.conf
@@ -1330,10 +1330,39 @@ disable-thp yes
 
 appendonly no
 
-# The name of the append only file (default: "appendonly.aof")
+# The base name of the append only file.
+#
+# Redis 7 and newer use a set of append-only files to persist the dataset
+# and changes appleid to it. There are two basic types of files in use:
+#
+# - Base files, which are a snapshot representing the complete state of the
+#   dataset at the time the file was created. Base files can be either in
+#   the form of RDB (binary serialized) or AOF (textual commands).
+# - Incremenetal files, which contain additional commands that were applied
+#   to the dataset following the previous file.
+#
+# In addition, manifest files are used to track the files and the order in
+# which they were created and should be applied.
+#
+# Append-only file names are created by Redis following a specific pattern.
+# The file name's prefix is based on the 'appendfilename' configuration
+# parameter, followed by additional information about the sequence and type.
+#
+# For example, if appendfilename is set to appendonly.aof, the following file
+# names could be derived:
+#
+# - appendonly.aof.1.base.rdb as a base file.
+# - appendonly.aof.1.incr.aof, appendonly.aof.2.incr.aof as incremental files.
+# - appendonly.aof.manifest as a manifest file.
 
 appendfilename "appendonly.aof"
 
+# For convenience, Redis stores all persistent append-only files in a dedicated
+# directory. The name of the directory is determined by the appenddirname
+# configuration parameter.
+
+appenddirname "appendonlydir"
+
 # The fsync() call tells the Operating System to actually write data on disk
 # instead of waiting for more data in the output buffer. Some OS will really flush
 # data on disk, some other OS will just try to do it ASAP.
@@ -1426,15 +1455,9 @@ auto-aof-rewrite-min-size 64mb
 # will be found.
 aof-load-truncated yes
 
-# When rewriting the AOF file, Redis is able to use an RDB preamble in the
-# AOF file for faster rewrites and recoveries. When this option is turned
-# on the rewritten AOF file is composed of two different stanzas:
-#
-#   [RDB file][AOF tail]
-#
-# When loading, Redis recognizes that the AOF file starts with the "REDIS"
-# string and loads the prefixed RDB file, then continues loading the AOF
-# tail.
+# Redis can create append-only base files in either RDB or AOF formats. Using
+# the RDB format is always faster and more efficient, and disabling it is only
+# supported for backward compatibility purposes.
 aof-use-rdb-preamble yes
 
 # Redis supports recording timestamp annotations in the AOF to support restoring