Using same config with primary for replicas

Change-Id: Icadc95ea54e4509dc148f8e84f2eaac5840509f3

4 files changed, 137 insertions, 82 deletions

diff --git a/doc/source/user/backup-db.rst b/doc/source/user/backup-db.rst
index 35bb8782..9a1a10e5 100644
--- a/doc/source/user/backup-db.rst
+++ b/doc/source/user/backup-db.rst
@@ -143,7 +143,7 @@ instance from the backup.
 
    .. code-block:: console
 
-      $ openstack database instance create guest2 10 --size 2 --nic net-id=$network_id --backup BACKUP_ID
+      $ openstack database instance create guest2 --flavor 10 --size 2 --nic net-id=$network_id --backup BACKUP_ID
       +-------------------+----------------------------------------------+
       |      Property     |                Value                         |
       +-------------------+----------------------------------------------+
diff --git a/doc/source/user/create-db.rst b/doc/source/user/create-db.rst
index d43ee650..b8446b84 100644
--- a/doc/source/user/create-db.rst
+++ b/doc/source/user/create-db.rst
@@ -107,7 +107,7 @@ Create and access a database
    .. code-block:: console
 
       $ openstack database instance create mysql_instance_1 \
-          6 \
+          --flavor 6 \
           --size 5 \
           --nic net-id=8799cf10-01ef-40e2-b04e-06da7cfa5668 \
           --databases test --users userA:password \
diff --git a/doc/source/user/set-up-replication.rst b/doc/source/user/set-up-replication.rst
index 6ec3c16b..6528ea87 100644
--- a/doc/source/user/set-up-replication.rst
+++ b/doc/source/user/set-up-replication.rst
@@ -2,109 +2,164 @@
 Set up database replication
 ===========================
 
-You can create a replica of an existing database instance. When you make
-subsequent changes to the original instance, the system automatically
-applies those changes to the replica.
+You can create replicas of an existing database instance(the primary) to
+improve the performance and scale of read-intensive workloads. Read workloads
+can be isolated to the replicas, while write workloads can be directed to the
+primary. When you make subsequent changes to the primary, the system
+automatically applies those changes to the replicas. Because replicas are
+read-only, they don't directly reduce write-capacity burdens on the primary.
+This feature isn't targeted at write-intensive workloads.
 
--  Replicas are read-only.
+- Not all the datastores support replication feature in Trove.
 
--  When you create a replica, do not specify the ``--users`` or
-   ``--databases`` options.
+- A replica is created by using the same server configuration as the primary,
+  e.g. flavor, data volume, datastore, etc. After a replica is created, several
+  settings can be changed independently from the primary server, e.g. the data
+  volume size.
 
--  You can choose a smaller volume or flavor for a replica than for the
-   original, but the replica's volume must be big enough to hold the
-   data snapshot from the original.
+- Currently, There is no automated failover between primary and replicas.
 
-This example shows you how to replicate a MySQL database instance.
+- Trove can only create a new replica. Adding an already existing instance to
+  the replication group is not supported.
+
+- Creating a replica of a replica is not supported.
+
+- When deleting replication instances, replicas need to be removed before the
+  primary.
 
 Set up replication
-~~~~~~~~~~~~~~~~~~
+------------------
 
-#. **Get the instance ID**
+#. Create a replica
 
-   Get the ID of the original instance you want to replicate:
+   First, make sure you have an instance (ID:
+   cebbf187-e223-46dd-8802-6dc04e895d0a) up and running in HEALTHY status,
+   create a replica:
 
    .. code-block:: console
 
-      $ openstack database instance list
-      +-----------+------------+-----------+-------------------+--------+-----------+------+
-      |     id    |  name      | datastore | datastore_version | status | flavor_id | size |
-      +-----------+------------+-----------+-------------------+--------+-----------+------+
-      | 97b...ae6 | base_1     |   mysql   |     mysql-5.5     | ACTIVE |     10    |  2   |
-      +-----------+------------+-----------+-------------------+--------+-----------+------+
+      $ openstack database instance create test-mysql-replica-1 \
+          --nic net-id=$netid \
+          --replica_of cebbf187-e223-46dd-8802-6dc04e895d0a
+
+#. Wait for the replica instance successfully created, verify status of the
+   replication servers.
+
+   .. code-block:: console
 
-#. **Create the replica**
+      $ odbi list
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+      | ID                                   | Name                 | Datastore | Datastore Version | Status  | Addresses                                      | Flavor ID | Size | Region    | Role    |
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+      | 71f30a72-4e47-4505-9e7f-ffd8933a331c | test-mysql-replica-1 | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.155', 'type': 'private'}] | d2        |    2 | RegionOne | replica |
+      | cebbf187-e223-46dd-8802-6dc04e895d0a | test-mysql           | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.43', 'type': 'private'}]  | d2        |    2 | RegionOne | primary |
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
 
-   Create a new instance that will be a replica of the original
-   instance. You do this by passing in the ``--replica_of`` option with
-   the :command:`openstack database instance create` command. This example creates a replica
-   called ``replica_1``. ``replica_1`` is a replica of the original instance,
-   ``base_1``:
+#. Verify replication status.
+
+   Replication can be verified by making some modifications to the primary and
+   ensuring that the modifications also propagate back to the replica. We will
+   create a database called "newdb" on the primary and check it's automatically
+   created on the replica.
+
+   First, get the existing databases of primary and replica, they should be the
+   same:
 
    .. code-block:: console
 
-      $ openstack database instance create replica_1 6 --size=5 --nic net-id=$netid \
-        --datastore_version mysql-5.5 \
-        --datastore mysql --replica_of ID_OF_ORIGINAL_INSTANCE
+      $ openstack database db list cebbf187-e223-46dd-8802-6dc04e895d0a # The primary
+      +--------+
+      | Name   |
+      +--------+
+      | testdb |
+      +--------+
+      $ openstack database db list 71f30a72-4e47-4505-9e7f-ffd8933a331c # The replica
+      +--------+
+      | Name   |
+      +--------+
+      | testdb |
+      +--------+
+
+   Create a new database on the primary:
+
+   .. code-block:: console
 
-#. **Verify replication status**
+      $ openstack database db create cebbf187-e223-46dd-8802-6dc04e895d0a newdb
 
-   Pass in ``replica_1``'s instance ID with the :command:`openstack database instance show` command
-   to verify that the newly created ``replica_1`` instance is a replica
-   of the original ``base_1``. Note that the ``replica_of`` property is
-   set to the ID of ``base_1``.
+   Check the new database is also created on the replica:
 
    .. code-block:: console
 
-      $ openstack database instance show INSTANCE_ID_OF_REPLICA_1
-      +-------------------+--------------------------------------+
-      | Property          | Value                                |
-      +-------------------+--------------------------------------+
-      | created           | 2014-09-16T11:16:49                  |
-      | datastore         | mysql                                |
-      | datastore_version | mysql-5.5                            |
-      | flavor            | 6                                    |
-      | id                | 49c6eff6-ef91-4eff-91c0-efbda7e83c38 |
-      | name              | replica_1                            |
-      | replica_of        | 97b4b853-80f6-414f-ba6f-c6f455a79ae6 |
-      | status            | BUILD                                |
-      | updated           | 2014-09-16T11:16:49                  |
-      | volume            | 5                                    |
-      +-------------------+--------------------------------------+
-
-   Now pass in ``base_1``'s instance ID with the :command:`openstack database instance show` command
-   to list the replica(s) associated with the original instance. Note
-   that the ``replicas`` property is set to the ID of ``replica_1``. If
-   there are multiple replicas, they appear as a comma-separated list.
+      $ openstack database db list 71f30a72-4e47-4505-9e7f-ffd8933a331c
+      +--------+
+      | Name   |
+      +--------+
+      | newdb  |
+      | testdb |
+      +--------+
+
+Failover
+--------
+
+Since replication is asynchronous, there is lag between the primary and the
+replica. The amount of lag can be influenced by a number of factors like how
+heavy the workload running on the primary server is and the latency between
+data centers. In most cases, replica lag ranges between a few seconds to a
+couple minutes.
+
+#. Before performing failover, we will create one more replica:
+
+   .. code-block:: console
+
+      $ openstack database instance create test-mysql-replica-2 \
+          --nic net-id=$netid \
+          --replica_of cebbf187-e223-46dd-8802-6dc04e895d0a
+
+   Now we have 3 instances running in a replication group:
 
    .. code-block:: console
 
-      $ openstack database instance show INSTANCE_ID_OF_BASE_1
-      +-------------------+--------------------------------------+
-      | Property          | Value                                |
-      +-------------------+--------------------------------------+
-      | created           | 2014-09-16T11:04:56                  |
-      | datastore         | mysql                                |
-      | datastore_version | mysql-5.5                            |
-      | flavor            | 6                                    |
-      | id                | 97b4b853-80f6-414f-ba6f-c6f455a79ae6 |
-      | ip                | 172.16.200.2                         |
-      | name              | base_1                               |
-      | replicas          | 49c6eff6-ef91-4eff-91c0-efbda7e83c38 |
-      | status            | ACTIVE                               |
-      | updated           | 2014-09-16T11:05:06                  |
-      | volume            | 5                                    |
-      | volume_used       | 0.11                                 |
-      +-------------------+--------------------------------------+
-
-#. **Detach the replica**
-
-   If the original instance goes down, you can detach the replica. The
-   replica becomes a standalone database instance. You can then take the
-   new standalone instance and create a new replica of that instance.
-
-   You detach a replica using the :command:`openstack database instance detach replica` command:
+      $ odbi list
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+      | ID                                   | Name                 | Datastore | Datastore Version | Status  | Addresses                                      | Flavor ID | Size | Region    | Role    |
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+      | 71f30a72-4e47-4505-9e7f-ffd8933a331c | test-mysql-replica-1 | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.155', 'type': 'private'}] | d2        |    2 | RegionOne | replica |
+      | a85ece86-9f62-4aa8-bb15-eba604cd2a01 | test-mysql-replica-2 | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.243', 'type': 'private'}] | d2        |    2 | RegionOne | replica |
+      | cebbf187-e223-46dd-8802-6dc04e895d0a | test-mysql           | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.43', 'type': 'private'}]  | d2        |    2 | RegionOne | primary |
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+
+#. Failover(promote) "test-mysql-replica-1" to primary.
 
    .. code-block:: console
 
-      $ openstack database instance detach replica INSTANCE_ID_OF_REPLICA
+      $ openstack database instance promote 71f30a72-4e47-4505-9e7f-ffd8933a331c
+
+   Wait for Trove setting up the new replication, the status of the 3 instances become "PROMOTE" then "HEALTHY".
+
+   .. code-block:: console
+
+      $ openstack database instance list
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+      | ID                                   | Name                 | Datastore | Datastore Version | Status  | Addresses                                      | Flavor ID | Size | Region    | Role    |
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+      | 71f30a72-4e47-4505-9e7f-ffd8933a331c | test-mysql-replica-1 | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.155', 'type': 'private'}] | d2        |    2 | RegionOne | primary |
+      | a85ece86-9f62-4aa8-bb15-eba604cd2a01 | test-mysql-replica-2 | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.243', 'type': 'private'}] | d2        |    2 | RegionOne | replica |
+      | cebbf187-e223-46dd-8802-6dc04e895d0a | test-mysql           | mysql     | 5.7.29            | HEALTHY | [{'address': '10.1.0.43', 'type': 'private'}]  | d2        |    2 | RegionOne | replica |
+      +--------------------------------------+----------------------+-----------+-------------------+---------+------------------------------------------------+-----------+------+-----------+---------+
+
+#. Point your application to the (former) replica.
+
+   Each server has a unique connection string. Update your application to point
+   to the (former) replica instead of the primary.
+
+Other supported operations
+--------------------------
+
+* Remove a failed primary. This essentially is used to eject an already failed
+  primary in order to establish a new one between the replicas. Command:
+  ``openstack database instance eject <primary_ID>``
+
+* Change replica to a standalone database server. The detached replica becomes
+  a standalone server that accepts both reads and writes. The standalone server
+  can't be made into a replica again.. Command:
+  ``openstack database instance detach <replica_ID>``
\ No newline at end of file
diff --git a/doc/source/user/upgrade-datastore.rst b/doc/source/user/upgrade-datastore.rst
index 3947b37c..4fb8dccc 100644
--- a/doc/source/user/upgrade-datastore.rst
+++ b/doc/source/user/upgrade-datastore.rst
@@ -49,7 +49,7 @@ Upgrading datastore
    .. code-block:: console
 
       $ openstack database instance create test-mysql-upgrade \
-        d2 \
+        --flavor d2 \
         --size 1 \
         --nic net-id=$netid \
         --datastore mysql --datastore_version 5.7.29 \