Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323)

For diskless replication in swapdb mode, considering we already spend replica memory
having a backup of current db to restore in case of failure, we can have the following benefits
by instead swapping database only in case we succeeded in transferring db from master:

- Avoid `LOADING` response during failed and successful synchronization for cases where the
  replica is already up and running with data.
- Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
  time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
- This could be implemented also for disk replication with similar benefits if consumers are willing
  to spend the extra memory usage.

General notes:
- The concept of `backupDb` becomes `tempDb` for clarity.
- Async loading mode will only kick in if the replica is syncing from a master that has the same
  repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline. 
- New property in INFO: `async_loading` to differentiate from the blocking loading
- Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
  and the tempDb that is passed around.
- Because this is affecting replicas only, we assume that if they are not readonly and write commands
  during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
  here anyways to avoid complications.

Considerations for review:
- We have many cases where server.loading flag is used and even though I tried my best, there may
  be cases where async_loading should be checked as well and cases where it shouldn't (would require
  very good understanding of whole code)
- Several places that had different behavior depending on the loading flag where actually meant to just
  handle commands coming from the AOF client differently than ones coming from real clients, changed
  to check CLIENT_ID_AOF instead.

**Additional for Release Notes**
- Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
  contribute on triggering next database SAVE
- New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
- Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
  Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
  ABORTED and COMPLETED.
- New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
  to allow modules to declare they support the diskless replication with async loading (when absent, we fall
  back to disk-based loading).

Co-authored-by: Eduardo Semprebon <edus@saxobank.com>
Co-authored-by: Oran Agra <oran@redislabs.com>

1 files changed, 7 insertions, 3 deletions

diff --git a/redis.conf b/redis.conf
index bcb5fa022..04e8bb117 100644
--- a/redis.conf
+++ b/redis.conf
@@ -605,9 +605,13 @@ repl-diskless-sync-delay 5
 #
 # "disabled"    - Don't use diskless load (store the rdb file to the disk first)
 # "on-empty-db" - Use diskless load only when it is completely safe.
-# "swapdb"      - Keep a copy of the current db contents in RAM while parsing
-#                 the data directly from the socket. note that this requires
-#                 sufficient memory, if you don't have it, you risk an OOM kill.
+# "swapdb"      - Keep current db contents in RAM while parsing the data directly
+#                 from the socket. Replicas in this mode can keep serving current
+#                 data set while replication is in progress, except for cases where
+#                 they can't recognize master as having a data set from same
+#                 replication history.
+#                 Note that this requires sufficient memory, if you don't have it,
+#                 you risk an OOM kill.
 repl-diskless-load disabled
 
 # Replicas send PINGs to server in a predefined interval. It's possible to