diff options
Diffstat (limited to 'doc/user/packages/container_registry')
3 files changed, 252 insertions, 105 deletions
diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index c77fc5a0f4b..b28f0cbfb35 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -489,11 +489,13 @@ Container Registry. ## Limitations - Moving or renaming existing Container Registry repositories is not supported -once you have pushed images, because the images are stored in a path that matches -the repository path. To move or rename a repository with a -Container Registry, you must delete all existing images. + once you have pushed images, because the images are stored in a path that matches + the repository path. To move or rename a repository with a + Container Registry, you must delete all existing images. + Community suggestions to work around this limitation have been shared in + [issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround). - Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag -are not deleted by the cleanup policy. + are not deleted by the cleanup policy. ## Disable the Container Registry for a project @@ -581,103 +583,27 @@ For information on how to update your images, see the [Docker help](https://docs ### `Blob unknown to registry` error when pushing a manifest list -When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) to the GitLab Container Registry, you may receive the error `manifest blob unknown: blob unknown to registry`. [This issue](https://gitlab.com/gitlab-org/gitlab/-/issues/209008) occurs when the individual child manifests referenced in the manifest list were not pushed to the same repository. +When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) +to the GitLab Container Registry, you may receive the error +`manifest blob unknown: blob unknown to registry`. This is likely caused by having multiple images +with different architectures, spread out over several repositories instead of the same repository. -For example, you may have two individual images, one for `amd64` and another for `arm64v8`, and you want to build a multi-arch image with them. The `amd64` and `arm64v8` images must be pushed to the same repository where you want to push the multi-arch image. +For example, you may have two images, each representing an architecture: -As a workaround, you should include the architecture in the tag name of individual images. For example, use `mygroup/myapp:1.0.0-amd64` instead of using sub repositories, like `mygroup/myapp/amd64:1.0.0`. You can then tag the manifest list with `mygroup/myapp:1.0.0`. +- The `amd64` platform +- The `arm64v8` platform -### The cleanup policy doesn't delete any tags +To build a multi-arch image with these images, you must push them to the same repository as the +multi-arch image. -There can be different reasons behind this: - -- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and - it does not. This occurs when the cleanup policy is saved without editing the value in the - **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless - `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted. - This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the - cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates - that all tags should be removed. - -- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you - might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814), - with `error authorizing context: invalid token` in the logs. - - To fix this, there are two workarounds: - - - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources). - This limits the cleanup execution in time, and avoids the expired token error. - - - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5 - minutes. You can set a custom value by running - `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails - console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the - value currently in use for GitLab.com. Be aware that by extending this value you increase the - time required to revoke permissions. - -If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list -of the tags that you want to delete, and then use that list to delete the tags. To do this, follow -these steps: - -1. Run the following shell script. The command just before the `for` loop ensures that - `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all - the tags' names will be in the `list_o_tags.out` file: - - ```shell - # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination) - echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done - ``` - - If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date: - - ```shell - output = File.open( "/tmp/list_o_tags.out","w" ) - Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag| - output << tag.name + "\n" if tag.created_at < 1.month.ago - end;nil - output.close - ``` - - This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month. - -1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example - `sed` commands for this. Note that these commands are simply examples. You may change them to - best suit your needs: - - ```shell - # Remove the `latest` tag from the file - sed -i '/latest/d' list_o_tags.out - - # Remove the first N tags from the file - sed -i '1,Nd' list_o_tags.out - - # Remove the tags starting with `Av` from the file - sed -i '/^Av/d' list_o_tags.out - - # Remove the tags ending with `_v3` from the file - sed -i '/_v3$/d' list_o_tags.out - ``` - - If you are running macOS, you must add `.bak` to the commands. For example: - - ```shell - sed -i .bak '/latest/d' list_o_tags.out - ``` - -1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to - delete. - -1. Run this shell script to delete the tags in the `list_o_tags.out` file: - - ```shell - # loop over list_o_tags.out to delete a single tag at a time - while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs - ``` +To address the `Blob unknown to registry` error, include the architecture in the tag name of +individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`. +You can then tag the manifest list with `mygroup/myapp:1.0.0`. ### Troubleshoot as a GitLab server administrator Troubleshooting the GitLab Container Registry, most of the times, requires -you to log in to GitLab server with the Administrator role. +you to log in to GitLab server with administrator access. [Read how to troubleshoot the Container Registry](../../../administration/packages/container_registry.md#troubleshooting). diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md new file mode 100644 index 00000000000..5f678a661f8 --- /dev/null +++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md @@ -0,0 +1,133 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Reduce Container Registry data transfers **(FREE)** + +Depending on the frequency with which images or tags are downloaded from the Container Registry, +data transfers can exceed the GitLab limit. This page offers several recommendations and tips for +reducing the amount of data you transfer with the Container Registry. +are downloaded from the Container Registry, data transfers can exceed the GitLab limit. This page +offers several recommendations and tips for reducing the amount of data you transfer with the +Container Registry. + +## Check data transfer use + +Transfer usage is not available within the GitLab UI. [GitLab-#350905](https://gitlab.com/gitlab-org/gitlab/-/issues/350905) +is the epic tracking the work to surface this information. In the interim, GitLab team members may reach out to customers that have +significantly exceeded their transfer limits to better understand their use case and offer ways to reduce data transfer +usage. + +## Determine image size + +Use these tools and techniques to determine your image's size: + +- [Skopeo](https://github.com/containers/skopeo): + use Skopeo's `inspect` command to examine layer count and sizes through API calls. You can + therefore inspect this data prior to running `docker pull IMAGE`. + +- Docker in CI: examine and record the image size when using GitLab CI prior to pushing an image + with Docker. For example: + + ```yaml + docker inspect "$CI_REGISTRY_IMAGE:$IMAGE_TAG" \ + | awk '/"Size": ([0-9]+)[,]?/{ printf "Final Image Size: %d\n", $2 }' + ``` + +- [Dive](https://github.com/wagoodman/dive) + is a tool for exploring a Docker image, layer contents, and discovering ways to reduce its size. + +## Reduce image size + +### Use a smaller base image + +Consider using a smaller base image, such as [Alpine Linux](https://alpinelinux.org/). +An Alpine image is around 5MB, which is several times smaller than popular base images such as +[Debian](https://hub.docker.com/_/debian). +If your application is distributed as a self-contained static binary, such as for Go applications, +you can also consider using Docker's [scratch](https://hub.docker.com/_/scratch/) +base image. + +If you need to use a specific base image OS, look for `-slim` or `-minimal` variants, as this helps +to reduce the image size. + +Also be mindful about the operating system packages you install on top of your base image. These can +add up to hundreds of megabytes. Try keeping the number of installed packages to the bare minimum. + +[Multi-stage builds](#use-multi-stage-builds) can be a powerful ally in cleaning up transient build +dependencies. + +You may also consider using tools such as these: + +- [DockerSlim](https://github.com/docker-slim/docker-slim) + provides a set of commands to reduce the size of your container images. +- [Distroless](https://github.com/GoogleContainerTools/distroless) images contain only your + application and its runtime dependencies. They don't contain package managers, shells, or any + other programs you would expect to find in a standard Linux distribution. + +### Minimize layers + +Every instruction in your Dockerfile leads to a new layer, which records the file system changes +applied during such an instruction. In general, more or larger layers lead to larger images. Try to +minimize the number of layers to install the packages in the Dockerfile. Otherwise, this may cause +each step in the build process to increase the image size. + +There are multiple strategies to reduce the number and size of layers. For example, instead of using +a `RUN` command per operating system package that you want to install (which would lead to a layer +per package), you can install all the packages on a single `RUN` command to reduce the number of +steps in the build process and reduce the size of the image. + +Another useful strategy is to ensure that you remove all transient build dependencies and disable or +empty the operating system package manager cache before and after installing a package. + +When building your images, make sure you only copy the relevant files. For Docker, using a +[`.dockerignore`](https://docs.docker.com/engine/reference/builder/#dockerignore-file) +helps ensure that the build process ignores irrelevant files. + +You can use other third-party tools to minify your images, such as [DockerSlim](https://github.com/docker-slim/docker-slim). +Be aware that if used improperly, such tools may remove dependencies that your application needs to +operate under certain conditions. Therefore, it's preferable to strive for smaller images during the +build process instead of trying to minify images afterward. + +### Use multi-stage builds + +With [multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/), +you use multiple `FROM` statements in your Dockerfile. Each `FROM` instruction can use a different +base, and each begins a new build stage. You can selectively copy artifacts from one stage to +another, leaving behind everything you don't want in the final image. This is especially useful when +you need to install build dependencies, but you don't need them to be present in your final image. + +## Use an image pull policy + +When using the `docker` or `docker+machine` executors, you can set a [`pull_policy`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) +parameter in your runner `config.toml` that defines how the runner works when pulling Docker images. +To avoid transferring data when using large and rarely updated images, consider using the +`if-not-present` pull policy when pulling images from remote registries. + +## Use Docker layer caching + +When running `docker build`, each command in `Dockerfile` results in a layer. These layers are kept +as a cache and can be reused if there haven't been any changes. You can specify a tagged image to be +used as a cache source for the `docker build` command by using the `--cache-from` argument. Multiple +images can be specified as a cache source by using multiple `--cache-from` arguments. This can speed +up your builds and reduce the amount of data transferred. For more information, see the +[documentation on Docker layer caching](../../../ci/docker/using_docker_build.md#make-docker-in-docker-builds-faster-with-docker-layer-caching). + +## Move to GitLab Premium or Ultimate + +GitLab data transfer limits are set at the tier level. If you need a higher limit, consider +upgrading to [GitLab Premium or Ultimate](https://about.gitlab.com/upgrade/). + +## Purchase additional data transfer + +Read more about managing your [data transfer limits](../../../subscriptions/gitlab_com/#purchase-more-storage-and-transfer). + +## Related issues + +- You may want to rebuild your image when the base Docker image is updated. However, the + [pipeline subscription limit is too low](https://gitlab.com/gitlab-org/gitlab/-/issues/225278) + to leverage this feature. As a workaround, you can rebuild daily or multiple times per day. + [GitLab-#225278](https://gitlab.com/gitlab-org/gitlab/-/issues/225278) + proposes raising the limit to help with this workflow. diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index e2242a85b75..a2a22f138e3 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -16,8 +16,9 @@ to automatically manage your container registry usage. ## Check Container Registry Storage Use -The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, which includes Container Registry, -however, the storage is not being calculated. +The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, +which doesn't include the Container Registry. To track work on this, see the epic +[Storage management for the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/7226). ## Cleanup policy @@ -252,21 +253,108 @@ It is recommended you only enable container cleanup policies for projects that were created before GitLab 12.8 if you are confident the number of tags being cleaned up is minimal. -## Related topics +## More Container Registry storage reduction options -- [Delete images](index.md#delete-images) -- [Delete registry repository](../../../api/container_registry.md#delete-registry-repository) -- [Delete a registry repository tag](../../../api/container_registry.md#delete-a-registry-repository-tag) -- [Delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk) -- [Delete a package](../package_registry/index.md#delete-a-package) +Here are some other options to reduce your project's use of Container Registry storage: -## Troubleshooting cleanup policies +- Use the [GitLab UI](index.md#delete-images) + to delete individual image tags or the entire repository containing all the tags. +- Use the API to [delete individual image tags](../../../api/container_registry.md#delete-a-registry-repository-tag). +- Use the API to [delete the entire container registry repository containing all the tags](../../../api/container_registry.md#delete-registry-repository). +- Use the API to [delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk). -If you see the following message: +## Troubleshooting cleanup policies -"Something went wrong while updating the cleanup policy." +### `Something went wrong while updating the cleanup policy.` -Check the regex patterns to ensure they are valid. +If you see this error message, check the regex patterns to ensure they are valid. GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/). View some common [regex pattern examples](#regex-pattern-examples). + +### The cleanup policy doesn't delete any tags + +There can be different reasons behind this: + +- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and + it does not. This occurs when the cleanup policy is saved without editing the value in the + **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless + `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted. + This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the + cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates + that all tags should be removed. + +- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you + might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814), + with `error authorizing context: invalid token` in the logs. + + To fix this, there are two workarounds: + + - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources). + This limits the cleanup execution in time, and avoids the expired token error. + + - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5 + minutes. You can set a custom value by running + `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails + console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the + value currently in use for GitLab.com. Be aware that by extending this value you increase the + time required to revoke permissions. + +If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list +of the tags that you want to delete, and then use that list to delete the tags. To do this, follow +these steps: + +1. Run the following shell script. The command just before the `for` loop ensures that + `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all + the tags' names will be in the `list_o_tags.out` file: + + ```shell + # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination) + echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done + ``` + + If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date: + + ```shell + output = File.open( "/tmp/list_o_tags.out","w" ) + Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag| + output << tag.name + "\n" if tag.created_at < 1.month.ago + end;nil + output.close + ``` + + This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month. + +1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example + `sed` commands for this. Note that these commands are simply examples. You may change them to + best suit your needs: + + ```shell + # Remove the `latest` tag from the file + sed -i '/latest/d' list_o_tags.out + + # Remove the first N tags from the file + sed -i '1,Nd' list_o_tags.out + + # Remove the tags starting with `Av` from the file + sed -i '/^Av/d' list_o_tags.out + + # Remove the tags ending with `_v3` from the file + sed -i '/_v3$/d' list_o_tags.out + ``` + + If you are running macOS, you must add `.bak` to the commands. For example: + + ```shell + sed -i .bak '/latest/d' list_o_tags.out + ``` + +1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to + delete. + +1. Run this shell script to delete the tags in the `list_o_tags.out` file: + + ```shell + # loop over list_o_tags.out to delete a single tag at a time + while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs + ``` |