Add latest changes from gitlab-org/gitlab@13-3-stable-ee

1 files changed, 71 insertions, 4 deletions

diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 44f1d075a5e..1883f6659e6 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -331,6 +331,9 @@ The different supported drivers are:
 | swift      | OpenStack Swift Object Storage      |
 | oss        | Aliyun OSS                          |
 
+NOTE: **Note:**
+Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket.
+
 Read more about the individual driver's configuration options in the
 [Docker Registry docs](https://docs.docker.com/registry/configuration/#storage).
 
@@ -446,27 +449,62 @@ to be in read-only mode for a while. During this time,
 you can pull from the Container Registry, but you cannot push.
 
 1. Optional: To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime).
-1. Copy initial data to your S3 bucket, for example with the AWS CLI [`cp`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html)
+1. This example uses the `aws` CLI. If you haven't configured the
+   CLI before, you have to configure your credentials by running `sudo aws configure`.
+   Because a non-admin user likely can't access the Container Registry folder,
+   ensure you use `sudo`. To check your credential configuration, run
+   [`ls`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html) to list
+   all buckets.
+
+   ```shell
+   sudo aws --endpoint-url https://your-object-storage-backend.com s3 ls
+   ```
+
+   If you are using AWS as your back end, you do not need the [`--endpoint-url`](https://docs.aws.amazon.com/cli/latest/reference/#options).
+1. Copy initial data to your S3 bucket, for example with the `aws` CLI
+   [`cp`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html)
    or [`sync`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html)
    command. Make sure to keep the `docker` folder as the top-level folder inside the bucket.
 
    ```shell
-   aws s3 sync registry s3://mybucket
+   sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket
    ```
 
+   TIP: **Tip:**
+   If you have a lot of data, you may be able to improve performance by
+   [running parallel sync operations](https://aws.amazon.com/premiumsupport/knowledge-center/s3-improve-transfer-sync-command/).
+
 1. To perform the final data sync,
    [put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and
    [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. Sync any changes since the initial data load to your S3 bucket and delete files that exist in the destination bucket but not in the source:
 
    ```shell
-   aws s3 sync registry s3://mybucket --delete
+   sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket --delete --dryrun
    ```
 
+   After verifying the command is going to perform as expected, remove the
+   [`--dryrun`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)
+   flag and run the command.
+
    DANGER: **Danger:**
-   The `--delete` flag will delete files that exist in the destination but not in the source.
+   The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)
+   flag will delete files that exist in the destination but not in the source.
    Make sure not to swap the source and destination, or you will delete all data in the Registry.
 
+1. Verify all Container Registry files have been uploaded to object storage
+   by looking at the file count returned by these two commands:
+
+   ```shell
+   sudo find registry -type f | wc -l
+   ```
+
+   ```shell
+   sudo aws --endpoint-url https://your-object-storage-backend.com s3 ls s3://mybucket --recursive | wc -l
+   ```
+
+   The output of these commands should match, except for the content in the
+   `_uploads` directories and sub-directories.
 1. Configure your registry to [use the S3 bucket for storage](#use-object-storage).
 1. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
@@ -693,6 +731,35 @@ notifications:
       backoff: 1000
 ```
 
+## Run the Cleanup policy now
+
+To reduce the amount of [Container Registry disk space used by a given project](../troubleshooting/gitlab_rails_cheat_sheet.md#registry-disk-space-usage-by-project),
+administrators can clean up image tags
+and [run garbage collection](#container-registry-garbage-collection).
+
+To remove image tags by running the cleanup policy, run the following commands in the
+[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md):
+
+```ruby
+# Numeric ID of the project whose container registry should be cleaned up
+P = <project_id>
+
+# Numeric ID of a developer, maintainer or owner in that project
+U = <user_id>
+
+# Get required details / objects
+user    = User.find_by_id(U)
+project = Project.find_by_id(P)
+repo    = ContainerRepository.find_by(project_id: P)
+policy  = ContainerExpirationPolicy.find_by(project_id: P)
+
+# Start the tag cleanup
+Projects::ContainerRepository::CleanupTagsService.new(project, user, policy.attributes.except("created_at", "updated_at")).execute(repo)
+```
+
+NOTE: **Note:**
+You can also [run cleanup on a schedule](../../user/packages/container_registry/index.md#cleanup-policy).
+
 ## Container Registry garbage collection
 
 NOTE: **Note:**