Add latest changes from gitlab-org/gitlab@14-8-stable-eev14.8.0-rc42

3 files changed, 252 insertions, 105 deletions

diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md
index c77fc5a0f4b..b28f0cbfb35 100644
--- a/doc/user/packages/container_registry/index.md
+++ b/doc/user/packages/container_registry/index.md
@@ -489,11 +489,13 @@ Container Registry.
 ## Limitations
 
 - Moving or renaming existing Container Registry repositories is not supported
-once you have pushed images, because the images are stored in a path that matches
-the repository path. To move or rename a repository with a
-Container Registry, you must delete all existing images.
+  once you have pushed images, because the images are stored in a path that matches
+  the repository path. To move or rename a repository with a
+  Container Registry, you must delete all existing images.
+  Community suggestions to work around this limitation have been shared in
+  [issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround).
 - Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag
-are not deleted by the cleanup policy.
+  are not deleted by the cleanup policy.
 
 ## Disable the Container Registry for a project
 
@@ -581,103 +583,27 @@ For information on how to update your images, see the [Docker help](https://docs
 
 ### `Blob unknown to registry` error when pushing a manifest list
 
-When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) to the GitLab Container Registry, you may receive the error `manifest blob unknown: blob unknown to registry`. [This issue](https://gitlab.com/gitlab-org/gitlab/-/issues/209008) occurs when the individual child manifests referenced in the manifest list were not pushed to the same repository.
+When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list)
+to the GitLab Container Registry, you may receive the error
+`manifest blob unknown: blob unknown to registry`. This is likely caused by having multiple images
+with different architectures, spread out over several repositories instead of the same repository.
 
-For example, you may have two individual images, one for `amd64` and another for `arm64v8`, and you want to build a multi-arch image with them. The `amd64` and `arm64v8` images must be pushed to the same repository where you want to push the multi-arch image.
+For example, you may have two images, each representing an architecture:
 
-As a workaround, you should include the architecture in the tag name of individual images. For example, use `mygroup/myapp:1.0.0-amd64` instead of using sub repositories, like `mygroup/myapp/amd64:1.0.0`. You can then tag the manifest list with `mygroup/myapp:1.0.0`.
+- The `amd64` platform
+- The `arm64v8` platform
 
-### The cleanup policy doesn't delete any tags
+To build a multi-arch image with these images, you must push them to the same repository as the
+multi-arch image.
 
-There can be different reasons behind this:
-
-- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and
-  it does not. This occurs when the cleanup policy is saved without editing the value in the
-  **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless
-  `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted.
-  This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the
-  cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates
-  that all tags should be removed.
-
-- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you
-  might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814),
-  with `error authorizing context: invalid token` in the logs.
-
-  To fix this, there are two workarounds:
-
-  - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources).
-    This limits the cleanup execution in time, and avoids the expired token error.
-
-  - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5
-    minutes. You can set a custom value by running
-    `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails
-    console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the
-    value currently in use for GitLab.com. Be aware that by extending this value you increase the
-    time required to revoke permissions.
-
-If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list
-of the tags that you want to delete, and then use that list to delete the tags. To do this, follow
-these steps:
-
-1. Run the following shell script. The command just before the `for` loop ensures that
-   `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all
-   the tags' names will be in the `list_o_tags.out` file:
-
-   ```shell
-   # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination)
-   echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done
-   ```
-
-   If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date:
-
-   ```shell
-   output = File.open( "/tmp/list_o_tags.out","w" )
-   Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag|
-     output << tag.name + "\n" if tag.created_at < 1.month.ago
-   end;nil
-   output.close
-   ```
-
-   This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month.
-
-1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example
-   `sed` commands for this. Note that these commands are simply examples. You may change them to
-   best suit your needs:
-
-   ```shell
-   # Remove the `latest` tag from the file
-   sed -i '/latest/d' list_o_tags.out
-
-   # Remove the first N tags from the file
-   sed -i '1,Nd' list_o_tags.out
-
-   # Remove the tags starting with `Av` from the file
-   sed -i '/^Av/d' list_o_tags.out
-
-   # Remove the tags ending with `_v3` from the file
-   sed -i '/_v3$/d' list_o_tags.out
-   ```
-
-   If you are running macOS, you must add `.bak` to the commands. For example:
-
-   ```shell
-   sed -i .bak '/latest/d' list_o_tags.out
-   ```
-
-1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to
-   delete.
-
-1. Run this shell script to delete the tags in the `list_o_tags.out` file:
-
-   ```shell
-   # loop over list_o_tags.out to delete a single tag at a time
-   while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs
-   ```
+To address the `Blob unknown to registry` error, include the architecture in the tag name of
+individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`.
+You can then tag the manifest list with `mygroup/myapp:1.0.0`.
 
 ### Troubleshoot as a GitLab server administrator
 
 Troubleshooting the GitLab Container Registry, most of the times, requires
-you to log in to GitLab server with the Administrator role.
+you to log in to GitLab server with administrator access.
 
 [Read how to troubleshoot the Container Registry](../../../administration/packages/container_registry.md#troubleshooting).
 
diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md
new file mode 100644
index 00000000000..5f678a661f8
--- /dev/null
+++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md
@@ -0,0 +1,133 @@
+---
+stage: Package
+group: Package
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Reduce Container Registry data transfers **(FREE)**
+
+Depending on the frequency with which images or tags are downloaded from the Container Registry,
+data transfers can exceed the GitLab limit. This page offers several recommendations and tips for
+reducing the amount of data you transfer with the Container Registry.
+are downloaded from the Container Registry, data transfers can exceed the GitLab limit. This page
+offers several recommendations and tips for reducing the amount of data you transfer with the
+Container Registry.
+
+## Check data transfer use
+
+Transfer usage is not available within the GitLab UI. [GitLab-#350905](https://gitlab.com/gitlab-org/gitlab/-/issues/350905)
+is the epic tracking the work to surface this information. In the interim, GitLab team members may reach out to customers that have
+significantly exceeded their transfer limits to better understand their use case and offer ways to reduce data transfer
+usage.
+
+## Determine image size
+
+Use these tools and techniques to determine your image's size:
+
+- [Skopeo](https://github.com/containers/skopeo):
+  use Skopeo's `inspect` command to examine layer count and sizes through API calls. You can
+  therefore inspect this data prior to running `docker pull IMAGE`.
+
+- Docker in CI: examine and record the image size when using GitLab CI prior to pushing an image
+  with Docker. For example:
+
+  ```yaml
+  docker inspect "$CI_REGISTRY_IMAGE:$IMAGE_TAG" \
+        | awk '/"Size": ([0-9]+)[,]?/{ printf "Final Image Size: %d\n", $2 }'
+  ```
+
+- [Dive](https://github.com/wagoodman/dive)
+  is a tool for exploring a Docker image, layer contents, and discovering ways to reduce its size.
+
+## Reduce image size
+
+### Use a smaller base image
+
+Consider using a smaller base image, such as [Alpine Linux](https://alpinelinux.org/).
+An Alpine image is around 5MB, which is several times smaller than popular base images such as
+[Debian](https://hub.docker.com/_/debian).
+If your application is distributed as a self-contained static binary, such as for Go applications,
+you can also consider using Docker's [scratch](https://hub.docker.com/_/scratch/)
+base image.
+
+If you need to use a specific base image OS, look for `-slim` or `-minimal` variants, as this helps
+to reduce the image size.
+
+Also be mindful about the operating system packages you install on top of your base image. These can
+add up to hundreds of megabytes. Try keeping the number of installed packages to the bare minimum.
+
+[Multi-stage builds](#use-multi-stage-builds) can be a powerful ally in cleaning up transient build
+dependencies.
+
+You may also consider using tools such as these:
+
+- [DockerSlim](https://github.com/docker-slim/docker-slim)
+  provides a set of commands to reduce the size of your container images.
+- [Distroless](https://github.com/GoogleContainerTools/distroless) images contain only your
+  application and its runtime dependencies. They don't contain package managers, shells, or any
+  other programs you would expect to find in a standard Linux distribution.
+
+### Minimize layers
+
+Every instruction in your Dockerfile leads to a new layer, which records the file system changes
+applied during such an instruction. In general, more or larger layers lead to larger images. Try to
+minimize the number of layers to install the packages in the Dockerfile. Otherwise, this may cause
+each step in the build process to increase the image size.
+
+There are multiple strategies to reduce the number and size of layers. For example, instead of using
+a `RUN` command per operating system package that you want to install (which would lead to a layer
+per package), you can install all the packages on a single `RUN` command to reduce the number of
+steps in the build process and reduce the size of the image.
+
+Another useful strategy is to ensure that you remove all transient build dependencies and disable or
+empty the operating system package manager cache before and after installing a package.
+
+When building your images, make sure you only copy the relevant files. For Docker, using a
+[`.dockerignore`](https://docs.docker.com/engine/reference/builder/#dockerignore-file)
+helps ensure that the build process ignores irrelevant files.
+
+You can use other third-party tools to minify your images, such as [DockerSlim](https://github.com/docker-slim/docker-slim).
+Be aware that if used improperly, such tools may remove dependencies that your application needs to
+operate under certain conditions. Therefore, it's preferable to strive for smaller images during the
+build process instead of trying to minify images afterward.
+
+### Use multi-stage builds
+
+With [multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/),
+you use multiple `FROM` statements in your Dockerfile. Each `FROM` instruction can use a different
+base, and each begins a new build stage. You can selectively copy artifacts from one stage to
+another, leaving behind everything you don't want in the final image. This is especially useful when
+you need to install build dependencies, but you don't need them to be present in your final image.
+
+## Use an image pull policy
+
+When using the `docker` or `docker+machine` executors, you can set a [`pull_policy`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy)
+parameter in your runner `config.toml` that defines how the runner works when pulling Docker images.
+To avoid transferring data when using large and rarely updated images, consider using the
+`if-not-present` pull policy when pulling images from remote registries.
+
+## Use Docker layer caching
+
+When running `docker build`, each command in `Dockerfile` results in a layer. These layers are kept
+as a cache and can be reused if there haven't been any changes. You can specify a tagged image to be
+used as a cache source for the `docker build` command by using the `--cache-from` argument. Multiple
+images can be specified as a cache source by using multiple `--cache-from` arguments. This can speed
+up your builds and reduce the amount of data transferred. For more information, see the
+[documentation on Docker layer caching](../../../ci/docker/using_docker_build.md#make-docker-in-docker-builds-faster-with-docker-layer-caching).
+
+## Move to GitLab Premium or Ultimate
+
+GitLab data transfer limits are set at the tier level. If you need a higher limit, consider
+upgrading to [GitLab Premium or Ultimate](https://about.gitlab.com/upgrade/).
+
+## Purchase additional data transfer
+
+Read more about managing your [data transfer limits](../../../subscriptions/gitlab_com/#purchase-more-storage-and-transfer).
+
+## Related issues
+
+- You may want to rebuild your image when the base Docker image is updated. However, the
+  [pipeline subscription limit is too low](https://gitlab.com/gitlab-org/gitlab/-/issues/225278)
+  to leverage this feature. As a workaround, you can rebuild daily or multiple times per day.
+  [GitLab-#225278](https://gitlab.com/gitlab-org/gitlab/-/issues/225278)
+  proposes raising the limit to help with this workflow.
diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md
index e2242a85b75..a2a22f138e3 100644
--- a/doc/user/packages/container_registry/reduce_container_registry_storage.md
+++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md
@@ -16,8 +16,9 @@ to automatically manage your container registry usage.
 
 ## Check Container Registry Storage Use
 
-The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, which includes Container Registry,
-however, the storage is not being calculated.
+The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages,
+which doesn't include the Container Registry. To track work on this, see the epic
+[Storage management for the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/7226).
 
 ## Cleanup policy
 
@@ -252,21 +253,108 @@ It is recommended you only enable container cleanup
 policies for projects that were created before GitLab 12.8 if you are confident the number of tags
 being cleaned up is minimal.
 
-## Related topics
+## More Container Registry storage reduction options
 
-- [Delete images](index.md#delete-images)
-- [Delete registry repository](../../../api/container_registry.md#delete-registry-repository)
-- [Delete a registry repository tag](../../../api/container_registry.md#delete-a-registry-repository-tag)
-- [Delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk)
-- [Delete a package](../package_registry/index.md#delete-a-package)
+Here are some other options to reduce your project's use of Container Registry storage:
 
-## Troubleshooting cleanup policies
+- Use the [GitLab UI](index.md#delete-images)
+  to delete individual image tags or the entire repository containing all the tags.
+- Use the API to [delete individual image tags](../../../api/container_registry.md#delete-a-registry-repository-tag).
+- Use the API to [delete the entire container registry repository containing all the tags](../../../api/container_registry.md#delete-registry-repository).
+- Use the API to [delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk).
 
-If you see the following message:
+## Troubleshooting cleanup policies
 
-"Something went wrong while updating the cleanup policy."
+### `Something went wrong while updating the cleanup policy.`
 
-Check the regex patterns to ensure they are valid.
+If you see this error message, check the regex patterns to ensure they are valid.
 
 GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/).
 View some common [regex pattern examples](#regex-pattern-examples).
+
+### The cleanup policy doesn't delete any tags
+
+There can be different reasons behind this:
+
+- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and
+  it does not. This occurs when the cleanup policy is saved without editing the value in the
+  **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless
+  `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted.
+  This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the
+  cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates
+  that all tags should be removed.
+
+- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you
+  might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814),
+  with `error authorizing context: invalid token` in the logs.
+
+  To fix this, there are two workarounds:
+
+  - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources).
+    This limits the cleanup execution in time, and avoids the expired token error.
+
+  - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5
+    minutes. You can set a custom value by running
+    `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails
+    console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the
+    value currently in use for GitLab.com. Be aware that by extending this value you increase the
+    time required to revoke permissions.
+
+If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list
+of the tags that you want to delete, and then use that list to delete the tags. To do this, follow
+these steps:
+
+1. Run the following shell script. The command just before the `for` loop ensures that
+   `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all
+   the tags' names will be in the `list_o_tags.out` file:
+
+   ```shell
+   # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination)
+   echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done
+   ```
+
+   If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date:
+
+   ```shell
+   output = File.open( "/tmp/list_o_tags.out","w" )
+   Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag|
+     output << tag.name + "\n" if tag.created_at < 1.month.ago
+   end;nil
+   output.close
+   ```
+
+   This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month.
+
+1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example
+   `sed` commands for this. Note that these commands are simply examples. You may change them to
+   best suit your needs:
+
+   ```shell
+   # Remove the `latest` tag from the file
+   sed -i '/latest/d' list_o_tags.out
+
+   # Remove the first N tags from the file
+   sed -i '1,Nd' list_o_tags.out
+
+   # Remove the tags starting with `Av` from the file
+   sed -i '/^Av/d' list_o_tags.out
+
+   # Remove the tags ending with `_v3` from the file
+   sed -i '/_v3$/d' list_o_tags.out
+   ```
+
+   If you are running macOS, you must add `.bak` to the commands. For example:
+
+   ```shell
+   sed -i .bak '/latest/d' list_o_tags.out
+   ```
+
+1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to
+   delete.
+
+1. Run this shell script to delete the tags in the `list_o_tags.out` file:
+
+   ```shell
+   # loop over list_o_tags.out to delete a single tag at a time
+   while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs
+   ```