Merge branch '9228-geo-how-to-reconcile-differences-with-geo-checksum-mismatches-docs' into 'master'

Document how to reconcile differences with Geo checksum mismatches

Closes gitlab-ee#9228

See merge request gitlab-org/gitlab-ce!28183

(cherry picked from commit dbc0f8c1d3275b8c07102c68055e7c58c335ebb8)

ecfadcfb Document how to reconcile differences with Geo checksum mismatches
8d3c42c3 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
71757a82 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
e51488cb Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
507cd1b9 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
97a3f165 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
450a3798 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
65b98874 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
ea8e7bd8 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
7473d40e Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
6b269518 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
6d4489e2 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
dcc949a4 Apply suggestion to doc/administration/geo/disaster_recovery/background_verification.md
911fc185 Fix fenced code blocks

3 files changed, 36 insertions, 0 deletions

diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md
index 7d2fd51f834..c7299b6e196 100644
--- a/doc/administration/geo/disaster_recovery/background_verification.md
+++ b/doc/administration/geo/disaster_recovery/background_verification.md
@@ -157,6 +157,42 @@ For wikis:
     sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production
     ```
 
+## Reconcile differences with checksum mismatches
+
+If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch:
+
+1. Navigate to the **Admin Area > Projects** dashboard on the **primary** node, find the
+   project that you want to check the checksum differences and click on the
+   **Edit** button:
+   ![Projects dashboard](img/checksum-differences-admin-projects.png)
+
+1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:
+   ![Project admin page](img/checksum-differences-admin-project-page.png)
+
+1. Navigate to the project's repository directory on both **primary** and **secondary** nodes. For an installation from source, the path is usually `/home/git/repositories`. For Omnibus installs, the path is usually `/var/opt/gitlab/git-data/repositories`. Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
+
+    ```sh
+    cd /var/opt/gitlab/git-data/repositories
+    ```
+
+1. Run the following command on the **primary** node, redirecting the output to a file:
+
+   ```sh
+   git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-node-refs
+   ```
+
+1. Run the following command on the **secondary** node, redirecting the output to a file:
+
+   ```sh
+   git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-node-refs
+   ```
+
+1. Copy the files from the previous steps on the same system, and do a diff between the contents:
+
+   ```sh
+   diff primary-node-refs secondary-node-refs
+   ```
+
 ## Current limitations
 
 Until [issue #5064][ee-5064] is completed, background verification doesn't cover
diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png
new file mode 100644
index 00000000000..fd51523104b
--- /dev/null
+++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png
diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png
new file mode 100644
index 00000000000..b2a6da69d3d
--- /dev/null
+++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png