Implementing the WAITAOF command (issue #10505) (#11713)

Implementing the WAITAOF functionality which would allow the user to
block until a specified number of Redises have fsynced all previous write
commands to the AOF.

Syntax: `WAITAOF <num_local> <num_replicas> <timeout>`
Response: Array containing two elements: num_local, num_replicas
num_local is always either 0 or 1 representing the local AOF on the master.
num_replicas is the number of replicas that acknowledged the a replication
offset of the last write being fsynced to the AOF.

Returns an error when called on replicas, or when called with non-zero
num_local on a master with AOF disabled, in all other cases the response
just contains number of fsync copies.

Main changes:
* Added code to keep track of replication offsets that are confirmed to have
  been fsynced to disk.
* Keep advancing master_repl_offset even when replication is disabled (and
  there's no replication backlog, only if there's an AOF enabled).
  This way we can use this command and it's mechanisms even when replication
  is disabled.
* Extend REPLCONF ACK to `REPLCONF ACK <ofs> FACK <ofs>`, the FACK
  will be appended only if there's an AOF on the replica, and already ignored on
  old masters (thus backwards compatible)
* WAIT now no longer wait for the replication offset after your last command, but
  rather the replication offset after your last write (or read command that caused
  propagation, e.g. lazy expiry).

Unrelated changes:
* WAIT command respects CLIENT_DENY_BLOCKING (not just CLIENT_MULTI)

Implementation details:
* Add an atomic var named `fsynced_reploff_pending` that's updated
  (usually by the bio thread) and later copied to the main `fsynced_reploff`
  variable (only if the AOF base file exists).
  I.e. during the initial AOF rewrite it will not be used as the fsynced offset
  since the AOF base is still missing.
* Replace close+fsync bio job with new BIO_CLOSE_AOF (AOF specific)
  job that will also update fsync offset the field.
* Handle all AOF jobs (BIO_CLOSE_AOF, BIO_AOF_FSYNC) in the same bio
  worker thread, to impose ordering on their execution. This solves a
  race condition where a job could set `fsynced_reploff_pending` to a higher
  value than another pending fsync job, resulting in indicating an offset
  for which parts of the data have not yet actually been fsynced.
  Imposing an ordering on the jobs guarantees that fsync jobs are executed
  in increasing order of replication offset.
* Drain bio jobs when switching `appendfsync` to "always"
  This should prevent a write race between updates to `fsynced_reploff_pending`
  in the main thread (`flushAppendOnlyFile` when set to ALWAYS fsync), and
  those done in the bio thread.
* Drain the pending fsync when starting over a new AOF to avoid race conditions
  with the previous AOF offsets overriding the new one (e.g. after switching to
  replicate from a new master).
* Make sure to update the fsynced offset at the end of the initial AOF rewrite.
  a must in case there are no additional writes that trigger a periodic fsync,
  specifically for a replica that does a full sync.

Limitations:
It is possible to write a module and a Lua script that propagate to the AOF and doesn't
propagate to the replication stream. see REDISMODULE_ARGV_NO_REPLICAS and luaRedisSetReplCommand.
These features are incompatible with the WAITAOF command, and can result
in two bad cases. The scenario is that the user executes command that only
propagates to AOF, and then immediately
issues a WAITAOF, and there's no further writes on the replication stream after that.
1. if the the last thing that happened on the replication stream is a PING
  (which increased the replication offset but won't trigger an fsync on the replica),
  then the client would hang forever (will wait for an fack that the replica will never
  send sine it doesn't trigger any fsyncs).
2. if the last thing that happened is a write command that got propagated properly,
  then WAITAOF will be released immediately, without waiting for an fsync (since
  the offset didn't change)

Refactoring:
* Plumbing to allow bio worker to handle multiple job types
  This introduces infrastructure necessary to allow BIO workers to
  not have a 1-1 mapping of worker to job-type. This allows in the
  future to assign multiple job types to a single worker, either as
  a performance/resource optimization, or as a way of enforcing
  ordering between specific classes of jobs.

Co-authored-by: Oran Agra <oran@redislabs.com>

1 files changed, 231 insertions, 0 deletions

diff --git a/tests/unit/wait.tcl b/tests/unit/wait.tcl
index 0ab36a0e6..7e040fbc0 100644
--- a/tests/unit/wait.tcl
+++ b/tests/unit/wait.tcl
@@ -69,3 +69,234 @@ start_server {} {
         assert {[$master wait 1 1000] == 1}
     }
 }}
+
+
+tags {"wait aof network external:skip"} {
+    start_server {overrides {appendonly {yes} auto-aof-rewrite-percentage {0}}} {
+        set master [srv 0 client]
+
+        test {WAITAOF local copy before fsync} {
+            r config set appendfsync no
+            $master incr foo
+            assert_equal [$master waitaof 1 0 50] {0 0} ;# exits on timeout
+            r config set appendfsync everysec
+        }
+
+        test {WAITAOF local copy everysec} {
+            $master incr foo
+            assert_equal [$master waitaof 1 0 0] {1 0}
+        }
+
+        test {WAITAOF local copy with appendfsync always} {
+            r config set appendfsync always
+            $master incr foo
+            assert_equal [$master waitaof 1 0 0] {1 0}
+            r config set appendfsync everysec
+        }
+
+        test {WAITAOF local wait and then stop aof} {
+            set rd [redis_deferring_client]
+            $rd incr foo
+            $rd read
+            $rd waitaof 1 0 0
+            wait_for_blocked_client
+            r config set appendonly no ;# this should release the blocked client as an error
+            assert_error {ERR WAITAOF cannot be used when appendonly is disabled} {$rd read}
+            $rd close
+        }
+
+        test {WAITAOF local on server with aof disabled} {
+            $master incr foo
+            assert_error {ERR WAITAOF cannot be used when appendonly is disabled} {$master waitaof 1 0 0}
+        }
+
+        $master config set appendonly yes
+        waitForBgrewriteaof $master
+
+        start_server {overrides {appendonly {yes} auto-aof-rewrite-percentage {0}}} {
+            set master_host [srv -1 host]
+            set master_port [srv -1 port]
+            set replica [srv 0 client]
+            set replica_host [srv 0 host]
+            set replica_port [srv 0 port]
+            set replica_pid [srv 0 pid]
+
+            # make sure the master always fsyncs first (easier to test)
+            $master config set appendfsync always
+            $replica config set appendfsync no
+
+            test {WAITAOF on demoted master gets unblocked with an error} {
+                set rd [redis_deferring_client]
+                $rd incr foo
+                $rd read
+                $rd waitaof 0 1 0
+                wait_for_blocked_client
+                $replica replicaof $master_host $master_port
+                assert_error {UNBLOCKED force unblock from blocking operation,*} {$rd read}
+                $rd close
+            }
+
+            wait_for_ofs_sync $master $replica
+
+            test {WAITAOF replica copy before fsync} {
+                $master incr foo
+                assert_equal [$master waitaof 0 1 50] {1 0} ;# exits on timeout
+            }
+            $replica config set appendfsync everysec
+
+            test {WAITAOF replica copy everysec} {
+                $master incr foo
+                assert_equal [$master waitaof 0 1 0] {1 1}
+            }
+
+            test {WAITAOF replica copy appendfsync always} {
+                $replica config set appendfsync always
+                $master incr foo
+                assert_equal [$master waitaof 0 1 0] {1 1}
+                $replica config set appendfsync everysec
+            }
+
+            test {WAITAOF replica copy if replica is blocked} {
+                exec kill -SIGSTOP $replica_pid
+                $master incr foo
+                assert_equal [$master waitaof 0 1 50] {1 0} ;# exits on timeout
+                exec kill -SIGCONT $replica_pid
+                assert_equal [$master waitaof 0 1 0] {1 1}
+            }
+
+            test {WAITAOF on promoted replica} {
+                $replica replicaof no one
+                $replica incr foo
+                assert_equal [$replica waitaof 1 0 0] {1 0}
+            }
+
+            test {WAITAOF master that loses a replica and backlog is dropped} {
+                $master config set repl-backlog-ttl 1
+                after 2000 ;# wait for backlog to expire
+                $master incr foo
+                assert_equal [$master waitaof 1 0 0] {1 0}
+            }
+
+            test {WAITAOF master without backlog, wait is released when the replica finishes full-sync} {
+                set rd [redis_deferring_client -1]
+                $rd incr foo
+                $rd read
+                $rd waitaof 0 1 0
+                wait_for_blocked_client -1
+                $replica replicaof $master_host $master_port
+                assert_equal [$rd read] {1 1}
+                $rd close
+            }
+
+            test {WAITAOF master isn't configured to do AOF} {
+                $master config set appendonly no
+                $master incr foo
+                assert_equal [$master waitaof 0 1 0] {0 1}
+            }
+
+            test {WAITAOF replica isn't configured to do AOF} {
+                $master config set appendonly yes
+                waitForBgrewriteaof $master
+                $replica config set appendonly no
+                $master incr foo
+                assert_equal [$master waitaof 1 0 0] {1 0}
+            }
+
+            test {WAITAOF both local and replica got AOF enabled at runtime} {
+                $replica config set appendonly yes
+                waitForBgrewriteaof $replica
+                $master incr foo
+                assert_equal [$master waitaof 1 1 0] {1 1}
+            }
+
+            test {WAITAOF master sends PING after last write} {
+                $master config set repl-ping-replica-period 1
+                $master incr foo
+                after 1200 ;# wait for PING
+                $master get foo
+                assert_equal [$master waitaof 1 1 0] {1 1}
+                $master config set repl-ping-replica-period 10
+            }
+
+            test {WAITAOF master client didn't send any write command} {
+                $master config set repl-ping-replica-period 1
+                set client [redis_client -1]
+                after 1200 ;# wait for PING
+                assert_equal [$master waitaof 1 1 0] {1 1}
+                $client close
+                $master config set repl-ping-replica-period 10
+            }
+
+            test {WAITAOF master client didn't send any command} {
+                $master config set repl-ping-replica-period 1
+                set client [redis [srv -1 "host"] [srv -1 "port"] 0 $::tls]
+                after 1200 ;# wait for PING
+                assert_equal [$master waitaof 1 1 0] {1 1}
+                $client close
+                $master config set repl-ping-replica-period 10
+            }
+
+            foreach fsync {no everysec always} {
+                test "WAITAOF when replica switches between masters, fsync: $fsync" {
+                    # test a case where a replica is moved from one master to the other
+                    # between two replication streams with different offsets that should
+                    # not be mixed. done to smoke-test race conditions with bio thread.
+                    start_server {overrides {appendonly {yes} auto-aof-rewrite-percentage {0}}} {
+                        start_server {overrides {appendonly {yes} auto-aof-rewrite-percentage {0}}} {
+                            set master2 [srv -1 client]
+                            set master2_host [srv -1 host]
+                            set master2_port [srv -1 port]
+                            set replica2 [srv 0 client]
+                            set replica2_host [srv 0 host]
+                            set replica2_port [srv 0 port]
+                            set replica2_pid [srv 0 pid]
+
+                            $replica2 replicaof $master2_host $master2_port
+                            wait_for_ofs_sync $master2 $replica2
+
+                            $master config set appendfsync $fsync
+                            $master2 config set appendfsync $fsync
+                            $replica config set appendfsync $fsync
+                            $replica2 config set appendfsync $fsync
+                            if {$fsync eq "no"} {
+                                after 2000 ;# wait for any previous fsync to finish
+                                # can't afford "no" on the masters
+                                $master config set appendfsync always
+                                $master2 config set appendfsync always
+                            } elseif {$fsync eq "everysec"} {
+                                after 990 ;# hoping to hit a race
+                            }
+
+                            # add some writes and block a client on each master
+                            set rd [redis_deferring_client -3]
+                            set rd2 [redis_deferring_client -1]
+                            $rd set boo 11
+                            $rd2 set boo 22
+                            $rd read
+                            $rd2 read
+                            $rd waitaof 1 1 0
+                            $rd2 waitaof 1 1 0
+
+                            if {$fsync eq "no"} {
+                                # since appendfsync is disabled in the replicas, the client
+                                # will get released only with full sync
+                                wait_for_blocked_client -1
+                                wait_for_blocked_client -3
+                            }
+                            # switch between the two replicas
+                            $replica2 replicaof $master_host $master_port
+                            $replica replicaof $master2_host $master2_port
+                            assert_equal [$rd read] {1 1}
+                            assert_equal [$rd2 read] {1 1}
+                            $rd close
+                            $rd2 close
+
+                            assert_equal [$replica get boo] 22
+                            assert_equal [$replica2 get boo] 11
+                        }
+                    }
+                }
+            }
+        }
+    }
+}