ovsdb: relay: Add support for transaction forwarding.

Current version of ovsdb relay allows to scale out read-only
access to the primary database.  However, many clients are not
read-only but read-mostly.  For example, ovn-controller.

In order to scale out database access for this case ovsdb-server
need to process transactions that are not read-only.  Relay is not
allowed to do that, i.e. not allowed to modify the database, but it
can act like a proxy and forward transactions that includes database
modifications to the primary server and forward replies back to a
client.  At the same time it may serve read-only transactions and
monitor requests by itself greatly reducing the load on primary
server.

This configuration will slightly increase transaction latency, but
it's not very important for read-mostly use cases.

Implementation details:
With this change instead of creating a trigger to commit the
transaction, ovsdb-server will create a trigger for transaction
forwarding.  Later, ovsdb_relay_run() will send all new transactions
to the relay source.  Once transaction reply received from the
relay source, ovsdb-relay module will update the state of the
transaction forwarding with the reply.  After that, trigger_run()
will complete the trigger and jsonrpc_server_run() will send the
reply back to the client.  Since transaction reply from the relay
source will be received after all the updates, client will receive
all the updates before receiving the transaction reply as it is in
a normal scenario with other database models.

Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
Acked-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>

1 files changed, 25 insertions, 16 deletions

diff --git a/ovsdb/trigger.h b/ovsdb/trigger.h
index 79af7f6be..d060c72e5 100644
--- a/ovsdb/trigger.h
+++ b/ovsdb/trigger.h
@@ -22,26 +22,34 @@ struct ovsdb;
 
 /* Triggers have the following states:
  *
- *    - Initialized (reply == NULL, progress == NULL): Executing the trigger
- *      can keep it in the initialized state, if it has a "wait" condition that
- *      isn't met.  Executing the trigger can also yield an error, in which
- *      case it transitions to "complete".  Otherwise, execution yields a
- *      transaction, which the database attempts to commit.  If the transaction
- *      completes immediately and synchronously, then the trigger transitions
- *      to the "complete" state.  If the transaction requires some time to
- *      complete, it transitions to the "committing" state.
+ *    - Initialized (reply == NULL, progress == NULL, txn_forward == NULL):
+ *      Executing the trigger can keep it in the initialized state, if it has a
+ *      "wait" condition that isn't met.  Executing the trigger can also yield
+ *      an error, in which case it transitions to "complete".  Otherwise,
+ *      execution yields a transaction, which the database attempts to commit.
+ *      If the transaction completes immediately and synchronously, then the
+ *      trigger transitions to the "complete" state.  If the transaction
+ *      requires some time to complete, it transitions to the "committing"
+ *      state.  If the transaction can not be completed locally due to
+ *      read-only restrictions and transaction forwarding is enabled, starts
+ *      forwarding and transitions to the "forwarding" state.
  *
- *    - Committing (reply != NULL, progress != NULL): The transaction is
- *      committing.  If it succeeds, or if it fails permanently, then the
- *      trigger transitions to "complete".  If it fails temporarily
- *      (e.g. because someone else committed to cluster-based storage before we
- *      did), then we transition back to "initialized" to try again.
+ *    - Committing (reply != NULL, progress != NULL, txn_forward == NULL):
+ *      The transaction is committing.  If it succeeds, or if it fails
+ *      permanently, then the trigger transitions to "complete".  If it fails
+ *      temporarily (e.g. because someone else committed to cluster-based
+ *      storage before we did), then we transition back to "initialized" to
+ *      try again.
  *
- *    - Complete (reply != NULL, progress == NULL): The transaction is done
- *      and either succeeded or failed.
+ *    - Forwarding (reply == NULL, progress == NULL, txn_forward != NULL):
+ *      Transaction is forwarded.  Either it succeeds or it fails, the trigger
+ *      transitions to "complete".
+ *
+ *    - Complete (reply != NULL, progress == NULL, txn_forward == NULL):
+ *      The transaction is done and either succeeded or failed.
  */
 struct ovsdb_trigger {
-    /* In "initialized" or "committing" state, in db->triggers.
+    /* In "initialized", "committing" or "forwarding" state, in db->triggers.
      * In "complete", in session->completions. */
     struct ovs_list node;
     struct ovsdb_session *session; /* Session that owns this trigger. */
@@ -49,6 +57,7 @@ struct ovsdb_trigger {
     struct jsonrpc_msg *request; /* Database request. */
     struct jsonrpc_msg *reply;   /* Result (null if none yet). */
     struct ovsdb_txn_progress *progress;
+    struct ovsdb_txn_forward *txn_forward; /* Tracks transaction forwarding. */
     long long int created;      /* Time created. */
     long long int timeout_msec; /* Max wait duration. */
     bool read_only;             /* Database is in read only mode. */