diff options
Diffstat (limited to 'doc/user/packages')
21 files changed, 1254 insertions, 1313 deletions
diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 3d3fe35fd65..dd6605c2f01 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -270,8 +270,9 @@ Prerequisites: ``` NOTE: -If you try to install the package you just created in this tutorial, the package -already exists on your local computer, so this command has no effect. +If you try installing the package you created in this tutorial, the install command +will have no effect because the package already exists. +Delete `~/.conan/data` to clean up the packages stored in the cache. ## Remove a Conan package diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 65e7918d827..4b4d6190dc2 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Container Registry **(FREE)** -> - The group-level Container Registry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23315) in GitLab 12.10. -> - Searching by image repository name was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31322) in GitLab 13.0. +> Searching by image repository name was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31322) in GitLab 13.0. NOTE: -If you pull container images from Docker Hub, you can also use the [GitLab Dependency Proxy](../dependency_proxy/index.md#use-the-dependency-proxy-for-docker-images) to avoid running into rate limits and speed up your pipelines. +If you pull container images from Docker Hub, you can use the [GitLab Dependency Proxy](../dependency_proxy/index.md#use-the-dependency-proxy-for-docker-images) +to avoid rate limits and speed up your pipelines. With the Docker Container Registry integrated into GitLab, every GitLab project can have its own space to store its Docker images. @@ -28,26 +28,26 @@ You can view the Container Registry for a project or group. 1. Go to your project or group. 1. Go to **Packages and registries > Container Registry**. -You can search, sort, filter, and [delete](#delete-images-from-within-gitlab) +You can search, sort, filter, and [delete](#delete-images-using-the-gitlab-ui) containers on this page. You can share a filtered view by copying the URL from your browser. Only members of the project or group can access a private project's Container Registry. +Images downloaded from a private registry may be [available to other users in a shared runner](https://docs.gitlab.com/runner/security/index.html#usage-of-private-docker-images-with-if-not-present-pull-policy). If a project is public, so is the Container Registry. ### View the tags of a specific image -You can view a list of tags associated with a given container image: +You can use the Container Registry **Tag Details** page to view a list of tags associated with a given container image: 1. Go to your project or group. 1. Go to **Packages and registries > Container Registry**. 1. Select the container image you are interested in. -This brings up the Container Registry **Tag Details** page. You can view details about each tag, -such as when it was published, how much storage it consumes, and the manifest and configuration -digests. +You can view details about each tag, such as when it was published, how much storage it consumes, +and the manifest and configuration digests. -You can search, sort (by tag name), filter, and [delete](#delete-images-from-within-gitlab) +You can search, sort (by tag name), filter, and [delete](#delete-images-using-the-gitlab-ui) tags on this page. You can share a filtered view by copying the URL from your browser. ## Use images from the Container Registry @@ -67,7 +67,7 @@ To download and run a container image hosted in the GitLab Container Registry: docker run [options] registry.example.com/group/project/image [arguments] ``` -[Authentication](#authenticate-with-the-container-registry) is needed to download images from private repository. +[Authentication](#authenticate-with-the-container-registry) is needed to download images from a private repository. For more information on running Docker containers, visit the [Docker documentation](https://docs.docker.com/get-started/). @@ -85,7 +85,7 @@ then your image must be named `gitlab.example.com/mynamespace/myproject` at a mi You can append additional names to the end of an image name, up to two levels deep. -For example, these are all valid image names for images within the project named `myproject`: +For example, these are all valid image names for images in the project named `myproject`: ```plaintext registry.example.com/mynamespace/myproject:some-tag @@ -192,12 +192,12 @@ You can configure your `.gitlab-ci.yml` file to build and push images to the Con - Before building, use `docker build --pull` to fetch changes to base images. It takes slightly longer, but it ensures your image is up-to-date. - Before each `docker run`, do an explicit `docker pull` to fetch - the image that was just built. This is especially important if you are + the image that was just built. This step is especially important if you are using multiple runners that cache images locally. If you use the Git SHA in your image tag, each job is unique and you should never have a stale image. However, it's still possible to have a - stale image if you re-build a given commit after a dependency has changed. + stale image if you rebuild a given commit after a dependency has changed. - Don't build directly to the `latest` tag because multiple jobs may be happening simultaneously. @@ -234,12 +234,12 @@ build: - docker push $IMAGE_TAG ``` -Here, `$CI_REGISTRY_IMAGE` would be resolved to the address of the registry tied -to this project. Since `$CI_COMMIT_REF_NAME` resolves to the branch or tag name, -and your branch name can contain forward slashes (for example, `feature/my-feature`), it is -safer to use `$CI_COMMIT_REF_SLUG` as the image tag. This is due to that image tags -cannot contain forward slashes. We also declare our own variable, `$IMAGE_TAG`, -combining the two to save us some typing in the `script` section. +In this example, `$CI_REGISTRY_IMAGE` resolves to the address of the registry tied +to this project. `$CI_COMMIT_REF_NAME` resolves to the branch or tag name, which +can contain forward slashes. Image tags can't contain forward slashes. Use +`$CI_COMMIT_REF_SLUG` as the image tag. You can declare the variable, `$IMAGE_TAG`, +combining `$CI_REGISTRY_IMAGE` and `$CI_REGISTRY_IMAGE` to save some typing in the +`script` section. Here's a more elaborate example that splits up the tasks into 4 pipeline stages, including two tests that run in parallel. The `build` is stored in the container @@ -385,17 +385,18 @@ unreferenced, administrators must run [garbage collection](../../../administrati On GitLab.com, the latest version of the Container Registry includes an automatic online garbage collector. For more information, see [this blog post](https://about.gitlab.com/blog/2021/10/25/gitlab-com-container-registry-update/). -This is an instance-wide feature, rolling out gradually to a subset of the user base, so some new image repositories created -from GitLab 14.5 onwards are served by this new version of the Container Registry. In this new -version of the Container Registry, layers that aren't referenced by any image manifest, and image -manifests that have no tags and aren't referenced by another manifest (such as multi-architecture -images), are automatically scheduled for deletion after 24 hours if left unreferenced. +The automatic online garbage collector is an instance-wide feature, rolling out gradually to a subset +of the user base. Some new image repositories created from GitLab 14.5 onward are served by this +new version of the Container Registry. In this new version of the Container Registry, layers that aren't +referenced by any image manifest, and image manifests that have no tags and aren't referenced by another +manifest (such as multi-architecture images), are automatically scheduled for deletion after 24 hours if +left unreferenced. -### Delete images from within GitLab +### Delete images using the GitLab UI -To delete images from within GitLab: +To delete images using the GitLab UI: -1. Navigate to your project's or group's **Packages and registries > Container Registry**. +1. Go to your project's or group's **Packages and registries > Container Registry**. 1. From the **Container Registry** page, you can select what you want to delete, by either: @@ -419,7 +420,7 @@ information, see the following endpoints: ### Delete images using GitLab CI/CD WARNING: -GitLab CI/CD doesn't provide a built-in way to remove your images, but this example +GitLab CI/CD doesn't provide a built-in way to remove your images. This example uses a third-party tool called [reg](https://github.com/genuinetools/reg) that talks to the GitLab Registry API. You are responsible for your own actions. For assistance with this tool, see @@ -455,21 +456,18 @@ build_image: - main delete_image: - image: docker:20.10.16 + before_script: + - curl --fail --show-error --location "https://github.com/genuinetools/reg/releases/download/v$REG_VERSION/reg-linux-amd64" --output ./reg + - echo "$REG_SHA256 ./reg" | sha256sum -c - + - chmod a+x ./reg + image: curlimages/curl:7.86.0 + script: + - ./reg rm -d --auth-url $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $IMAGE_TAG stage: clean - services: - - docker:20.10.16-dind variables: IMAGE_TAG: $CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG REG_SHA256: ade837fc5224acd8c34732bf54a94f579b47851cc6a7fd5899a98386b782e228 REG_VERSION: 0.16.1 - before_script: - - apk add --no-cache curl - - curl --fail --show-error --location "https://github.com/genuinetools/reg/releases/download/v$REG_VERSION/reg-linux-amd64" --output /usr/local/bin/reg - - echo "$REG_SHA256 /usr/local/bin/reg" | sha256sum -c - - - chmod a+x /usr/local/bin/reg - script: - - /usr/local/bin/reg rm -d --auth-url $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $IMAGE_TAG only: - branches except: @@ -487,16 +485,14 @@ defined in the `delete_image` job. You can create a per-project [cleanup policy](reduce_container_registry_storage.md#cleanup-policy) to ensure older tags and images are regularly removed from the Container Registry. -## Limitations +## Known issues -- Moving or renaming existing Container Registry repositories is not supported - once you have pushed images, because the images are stored in a path that matches - the repository path. To move or rename a repository with a - Container Registry, you must delete all existing images. - Community suggestions to work around this limitation have been shared in - [issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround). -- Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag - are not deleted by the cleanup policy. +Moving or renaming existing Container Registry repositories is not supported +after you have pushed images. The images are stored in a path that matches +the repository path. To move or rename a repository with a +Container Registry, you must delete all existing images. +Community suggestions to work around this known issue have been shared in +[issue 18383](https://gitlab.com/gitlab-org/gitlab/-/issues/18383#possible-workaround). ## Disable the Container Registry for a project @@ -530,7 +526,7 @@ for more details about the permissions that this setting grants to users. is internal or private, the Container Registry is also internal or private. - **Only Project Members**: The Container Registry is visible only to project members with - Reporter role or higher. This is similar to the behavior of a private project with Container + Reporter role or higher. This visibility is similar to the behavior of a private project with Container Registry visibility set to **Everyone With Access**. 1. Select **Save changes**. @@ -541,7 +537,7 @@ The ability to view the Container Registry and pull images is controlled by the visibility permissions. You can change this through the [visibility setting on the UI](#change-visibility-of-the-container-registry) or the [API](../../../api/container_registry.md#change-the-visibility-of-the-container-registry). [Other permissions](../../permissions.md) -such as updating the Container Registry, pushing or deleting images, and so on are not affected by +such as updating the Container Registry and pushing or deleting images are not affected by this setting. However, disabling the Container Registry disables all Container Registry operations. | | | Anonymous<br/>(Everyone on internet) | Guest | Reporter, Developer, Maintainer, Owner | @@ -553,188 +549,3 @@ this setting. However, disabling the Container Registry disables all Container R | Private project with Container Registry visibility <br/> set to **Everyone With Access** (UI) or `enabled` (API) | View Container Registry <br/> and pull images | No | No | Yes | | Private project with Container Registry visibility <br/> set to **Only Project Members** (UI) or `private` (API) | View Container Registry <br/> and pull images | No | No | Yes | | Any project with Container Registry `disabled` | All operations on Container Registry | No | No | No | - -## Troubleshooting the GitLab Container Registry - -### Migrating OCI container images to GitLab Container Registry - -Migrating built container images to the GitLab registry is not a current feature. However, an [epic](https://gitlab.com/groups/gitlab-org/-/epics/5210) is open to track the work on this feature. - -Some third-party tools can help migrate container images, for example, [skopeo](https://github.com/containers/skopeo), which can [copy container images](https://github.com/containers/skopeo#copying-images) between various storage mechanisms. You can use skopeo to copy from container registries, container storage backends, local directories, and local OCI-layout directories to the GitLab Container Registry. - -### Docker connection error - -A Docker connection error can occur when there are special characters in either the group, -project or branch name. Special characters can include: - -- Leading underscore -- Trailing hyphen/dash - -To get around this, you can [change the group path](../../group/manage.md#change-a-groups-path), -[change the project path](../../project/settings/index.md#rename-a-repository) or change the branch -name. - -You may also get a `404 Not Found` or `Unknown Manifest` message if you are using -a Docker Engine version earlier than 17.12. Later versions of Docker Engine use -[the v2 API](https://docs.docker.com/registry/spec/manifest-v2-2/). - -The images in your GitLab Container Registry must also use the Docker v2 API. -For information on how to update your images, see the [Docker help](https://docs.docker.com/registry/spec/deprecated-schema-v1). - -### `Blob unknown to registry` error when pushing a manifest list - -When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) -to the GitLab Container Registry, you may receive the error -`manifest blob unknown: blob unknown to registry`. This is likely caused by having multiple images -with different architectures, spread out over several repositories instead of the same repository. - -For example, you may have two images, each representing an architecture: - -- The `amd64` platform -- The `arm64v8` platform - -To build a multi-arch image with these images, you must push them to the same repository as the -multi-arch image. - -To address the `Blob unknown to registry` error, include the architecture in the tag name of -individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`. -You can then tag the manifest list with `mygroup/myapp:1.0.0`. - -### Troubleshoot as a GitLab server administrator - -Troubleshooting the GitLab Container Registry, most of the times, requires -you to sign in to GitLab server with administrator access. - -[Read how to troubleshoot the Container Registry](../../../administration/packages/container_registry.md#troubleshooting). - -### Unable to change path or transfer a project - -If you try to change a project's path or transfer a project to a new namespace, -you may receive one of the following errors: - -- "Project cannot be transferred, because tags are present in its container registry." -- "Namespace cannot be moved because at least one project has tags in container registry." - -This issue occurs when the project has images in the Container Registry. -You must delete or move these images before you can change the path or transfer -the project. - -The following procedure uses these sample project names: - -- For the current project: `gitlab.example.com/org/build/sample_project/cr:v2.9.1` -- For the new project: `gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1` - -Use your own URLs to complete the following steps: - -1. Download the Docker images on your computer: - - ```shell - docker login gitlab.example.com - docker pull gitlab.example.com/org/build/sample_project/cr:v2.9.1 - ``` - - NOTE: - For container registry authentication, use either a - [personal access token](../../profile/personal_access_tokens.md) or a - [deploy token](../../project/deploy_tokens/index.md). - -1. Rename the images to match the new project name: - - ```shell - docker tag gitlab.example.com/org/build/sample_project/cr:v2.9.1 gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 - ``` - -1. Delete the images in the old project by using the [UI](#delete-images) or [API](../../../api/packages.md#delete-a-project-package). - There may be a delay while the images are queued and deleted. -1. Change the path or transfer the project by going to **Settings > General** - and expanding **Advanced**. -1. Restore the images: - - ```shell - docker push gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 - ``` - -Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/18383) for details. - -### Tags on S3 backend remain after successful deletion requests - -With S3 as your storage backend, tags may remain even though: - -- In the UI, you see that the tags are scheduled for deletion. -- In the API, you get an HTTP `200` response. -- The registry log shows a successful `Delete` request. - -An example `DELETE` request in the registry log: - -```shell -{"content_type":"","correlation_id":"01FQGNSKVMHQEAVE21KYTJN2P4","duration_ms":62,"host":"localhost:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:47498","remote_ip":"127.0.0.1","status":202,"system":"http","time":"2021-12-22T08:58:15Z","ttfb_ms":62,"uri":"/v2/<path to repo>/tags/reference/<tag_name>","user_agent":"GitLab/<version>","written_bytes":0} -``` - -There may be some errors not properly cached. Follow these steps to investigate further: - -1. In your configuration file, set the registry's log level to `debug`, and the S3 driver's log - level to `logdebugwithhttpbody`. For Omnibus, make these edits in the `gitlab.rb` file: - - ```shell - # Change the registry['log_level'] to debug - registry['log_level'] = 'debug' - - # Set log level for registry log from storage side - registry['storage'] = { - 's3' => { - 'bucket' => 'your-s3-bucket', - 'region' => 'your-s3-region' - }, - - 'loglevel' = "logdebugwithhttpbody" - } - ``` - - Then save and reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. Attempt to delete one or more tags using the GitLab UI or API. - -1. Inspect the registry logs and look for a response from S3. Although the response could be - `200 OK`, the body might have the error `AccessDenied`. This indicates a permission problem from - the S3 side. - -1. Ensure your S3 configuration has the `deleteObject` permission scope. Here's an - [example role for an S3 bucket](../../../administration/object_storage.md#iam-permissions). - Once adjusted, trigger another tag deletion. You should be able to successfully delete tags. - -Follow [this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/551) for details. - -### Tags temporarily cannot be marked for deletion - -GitLab is [migrating to the next generation of the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/5523). -During the migration, you may encounter difficulty deleting tags. -If you encounter an error, it's likely that your image repository is in the process of being migrated. -Wait a few minutes and try again. - -### `unauthorized: authentication required` when pushing large images - -When pushing large images, you might get an error like the following: - -```shell -docker push gitlab.example.com/myproject/docs:latest -The push refers to a repository [gitlab.example.com/myproject/docs] -630816f32edb: Preparing -530d5553aec8: Preparing -... -4b0bab9ff599: Waiting -d1c800db26c7: Waiting -42755cf4ee95: Waiting -unauthorized: authentication required -``` - -On self-managed GitLab instances, by default, tokens for the Container Registry expire every five minutes. -When pushing larger images, or images that take longer than five minutes to push, -you might encounter this error. On GitLab.com, the expiration time is 15 minutes. - -If you are using self-managed GitLab, you can ask an administrator to -[increase the token duration](../../../administration/packages/container_registry.md#increase-token-duration) -if necessary. diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md index 74cbcba2ffc..0ce9635e05a 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md +++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md @@ -44,7 +44,7 @@ Consider using a smaller base image, such as [Alpine Linux](https://alpinelinux. An Alpine image is around 5MB, which is several times smaller than popular base images such as [Debian](https://hub.docker.com/_/debian). If your application is distributed as a self-contained static binary, such as for Go applications, -you can also consider using Docker's [scratch](https://hub.docker.com/_/scratch/) +you can also consider using the Docker [scratch](https://hub.docker.com/_/scratch/) base image. If you need to use a specific base image OS, look for `-slim` or `-minimal` variants, as this helps diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 23d835ddf5f..cbf9af633ac 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -4,17 +4,18 @@ group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Reduce Container Registry Storage **(FREE)** +# Reduce Container Registry storage **(FREE)** -Container registries become large over time without cleanup. When a large number of images or tags are added: +Container registries can grow in size over time if you don't manage your registry usage. For example, +if you add a large number of images or tags: -- Fetching the list of available tags or images becomes slower. +- Retrieving the list of available tags or images becomes slower. - They take up a large amount of storage space on the server. -We recommend deleting unnecessary images and tags and setting up a [cleanup policy](#cleanup-policy) +You should delete unnecessary images and tags and set up a [cleanup policy](#cleanup-policy) to automatically manage your container registry usage. -## Check Container Registry Storage Use +## Check Container Registry storage use The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages. This page includes the Container Registry usage, which is only available on GitLab.com. @@ -23,10 +24,16 @@ metadata database. Support for improvements is proposed in epic [5523](https://g You cannot use the Container Registry in self-managed instances, but epic [5521](https://gitlab.com/groups/gitlab-org/-/epics/5521) proposes to change this behavior. Image layers stored in the Container Registry are deduplicated at the root namespace level. -If you tag the same image more than once in the same repository or across distinct -repositories under the same root namespace, it is only counted once. -If an image layer is shared across multiple images, in the same -container repository, project, group, or across different repositories, it is only counted once. + +An image is only counted once if: + +- You tag the same image more than once in the same repository. +- You tag the same image across distinct repositories under the same root namespace. + +An image layer is only counted once if: + +- You share the image layer across multiple images in the same container repository, project, or group. +- You share the image layer across different repositories. Only layers that are referenced by tagged images are accounted for. Untagged images and any layers referenced exclusively by them are subject to [online garbage collection](index.md#delete-images). @@ -50,7 +57,7 @@ To delete the underlying layers and images that aren't associated with any tags, ### Enable the cleanup policy -Cleanup policies can be run on all projects, with these exceptions: +You can run cleanup policies on all projects with these exceptions: - For self-managed GitLab instances, the project must have been created in GitLab 12.8 or later. However, an administrator can enable the cleanup policy @@ -63,7 +70,7 @@ Cleanup policies can be run on all projects, with these exceptions: ApplicationSetting.last.update(container_expiration_policies_enable_historic_entries: true) ``` - Enabling cleanup policies on all project can impact performance, especially if you + Enabling cleanup policies on all projects can impact performance, especially if you are using an [external registry](#use-with-external-container-registries). WARNING: @@ -72,34 +79,34 @@ GitLab.com that don't have a container image. ### How the cleanup policy works -The cleanup policy collects all tags in the Container Registry and excludes tags -until only the tags to be deleted remain. +The cleanup policy collects all tags in the Container Registry and excludes tags until the only +tags you want to delete remain. The cleanup policy searches for images based on the tag name. Support for full path matching is tracked in issue [281071](https://gitlab.com/gitlab-org/gitlab/-/issues/281071). The cleanup policy: 1. Collects all tags for a given repository in a list. -1. Excludes the tag named `latest` from the list. -1. Evaluates the `name_regex` (tags to expire), excluding non-matching names from the list. -1. Excludes from the list any tags matching the `name_regex_keep` value (tags to preserve). +1. Excludes the tag named `latest`. +1. Evaluates the `name_regex` (tags to expire), excluding non-matching names. +1. Excludes any tags matching the `name_regex_keep` value (tags to preserve). 1. Excludes any tags that do not have a manifest (not part of the options in the UI). 1. Orders the remaining tags by `created_date`. -1. Excludes from the list the N tags based on the `keep_n` value (Number of tags to retain). -1. Excludes from the list the tags more recent than the `older_than` value (Expiration interval). -1. Finally, the remaining tags in the list are deleted from the Container Registry. +1. Excludes the N tags based on the `keep_n` value (Number of tags to retain). +1. Excludes the tags more recent than the `older_than` value (Expiration interval). +1. Deletes the remaining tags in the list from the Container Registry. WARNING: On GitLab.com, the execution time for the cleanup policy is limited. Some tags may remain in the Container Registry after the policy runs. The next time the policy runs, the remaining tags are included. -It may take multiple runs for all tags to be deleted. +It may take multiple runs to delete all tags. WARNING: GitLab self-managed installations support third-party container registries that comply with the [Docker Registry HTTP API V2](https://docs.docker.com/registry/spec/api/) -specification. However, this specification does not include a tag delete operation. Therefore, when -interacting with third-party container registries, GitLab uses a workaround to delete tags. See the -[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/15737) +specification. However, this specification does not include a tag delete operation. Therefore, GitLab uses a +workaround to delete tags when interacting with third-party container registries. Refer to +issue [15737](https://gitlab.com/gitlab-org/gitlab/-/issues/15737) for more information. Due to possible implementation variations, this workaround is not guaranteed to work with all third-party registries in the same predictable way. If you use the GitLab Container Registry, this workaround is not required because we implemented a special tag delete operation. In @@ -115,18 +122,18 @@ To create a cleanup policy in the UI: 1. Expand the **Clean up image tags** section. 1. Complete the fields. - | Field | Description | - |---------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| - | **Toggle** | Turn the policy on or off. | - | **Run cleanup** | How often the policy should run. | - | **Keep the most recent** | How many tags to _always_ keep for each image. | - | **Keep tags matching** | The regex pattern that determines which tags to preserve. The `latest` tag is always preserved. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | - | **Remove tags older than** | Remove only tags older than X days. | - | **Remove tags matching** | The regex pattern that determines which tags to remove. This value cannot be blank. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | + | Field | Description | + |----------------------------|-------------------------------------------------| + | **Toggle** | Turn the policy on or off. | + | **Run cleanup** | How often the policy should run. | + | **Keep the most recent** | How many tags to _always_ keep for each image. | + | **Keep tags matching** | A regex pattern that determines which tags to preserve. The `latest` tag is always preserved. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | + | **Remove tags older than** | Remove only tags older than X days. | + | **Remove tags matching** | A regex pattern that determines which tags to remove. This value cannot be blank. For all tags, use `.*`. See other [regex pattern examples](#regex-pattern-examples). | 1. Select **Save**. -Depending on the interval you chose, the policy is scheduled to run. +The policy runs on the scheduled interval you selected. NOTE: If you edit the policy and select **Save** again, the interval is reset. @@ -135,7 +142,8 @@ If you edit the policy and select **Save** again, the interval is reset. Cleanup policies use regex patterns to determine which tags should be preserved or removed, both in the UI and the API. -Regex patterns are automatically surrounded with `\A` and `\Z` anchors. Do not include any `\A`, `\Z`, `^` or `$` token in the regex patterns as they are not necessary. +Regex patterns are automatically surrounded with `\A` and `\Z` anchors. Therefore, you do not need to include any +`\A`, `\Z`, `^` or `$` tokens in the regex patterns. Here are some examples of regex patterns you can use: @@ -180,17 +188,17 @@ Here are some examples of regex patterns you can use: Cleanup policies are executed as a background process. This process is complex, and depending on the number of tags to delete, the process can take time to finish. -To prevent server resource starvation, the following application settings are available: +You can use the following application settings to prevent server resource starvation: - `container_registry_expiration_policies_worker_capacity`: the maximum number of cleanup workers - running concurrently. This must be greater than or equal to `0`. We recommend starting with a low - number and increasing it after monitoring the resources used by the background workers. To remove + running concurrently. This value must be greater than or equal to `0`. You should start with a low + number and increase it after monitoring the resources used by the background workers. To remove all workers and not execute the cleanup policies, set this to `0`. The default value is `4`. - `container_registry_delete_tags_service_timeout`: the maximum time (in seconds) that the cleanup process can take to delete a batch of tags. The default value is `250`. - `container_registry_cleanup_tags_service_max_list_size`: the maximum number of tags that can be - deleted in a single execution. Additional tags must be deleted in another execution. We recommend - starting with a low number and increasing it after monitoring that container images are properly + deleted in a single execution. Additional tags must be deleted in another execution. You should + start with a low number and increase it after verifying that container images are properly deleted. The default value is `200`. - `container_registry_expiration_policies_caching`: enable or disable tag creation timestamp caching during execution of policies. Cached timestamps are stored in [Redis](../../../development/architecture.md#redis). @@ -213,7 +221,8 @@ You can set, update, and disable the cleanup policies using the GitLab API. Examples: -- Select all tags, keep at least 1 tag per image, clean up any tag older than 14 days, run once a month, preserve any images with the name `main` and the policy is enabled: +- Select all tags, keep at least 1 tag per image, clean up any tag older than 14 days, run once a month, preserve +any images with the name `main`, and the policy is enabled: ```shell curl --request PUT --header 'Content-Type: application/json;charset=UTF-8' --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -251,14 +260,14 @@ See the API documentation for further details: [Edit project API](../../../api/p When using an [external container registry](../../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint), running a cleanup policy on a project may have some performance risks. -If a project runs a policy to remove thousands of tags +If a project runs a policy to remove thousands of tags, the GitLab background jobs may get backed up or fail completely. -For projects created before GitLab 12.8, we recommend you enable container cleanup policies +For projects created before GitLab 12.8, you should enable container cleanup policies only if the number of tags being cleaned up is minimal. ## More Container Registry storage reduction options -Here are some other options to reduce your project's use of Container Registry storage: +Here are some other options you can use to reduce the Container Registry storage used by your project: - Use the [GitLab UI](index.md#delete-images) to delete individual image tags or the entire repository containing all the tags. @@ -330,6 +339,10 @@ the tags. To create the list and delete the tags: 1. Remove any tags that you want to keep from the `list_o_tags.out` file. For example, you can use `sed` to parse the file and remove the tags. + ::Tabs + + :::TabTitle Linux + ```shell # Remove the `latest` tag from the file sed -i '/latest/d' list_o_tags.out @@ -344,12 +357,24 @@ the tags. To create the list and delete the tags: sed -i '/_v3$/d' list_o_tags.out ``` - If you are running macOS, you must add `.bak` to the commands. For example: + :::TabTitle macOS ```shell + # Remove the `latest` tag from the file sed -i .bak '/latest/d' list_o_tags.out + + # Remove the first N tags from the file + sed -i .bak '1,Nd' list_o_tags.out + + # Remove the tags starting with `Av` from the file + sed -i .bak '/^Av/d' list_o_tags.out + + # Remove the tags ending with `_v3` from the file + sed -i .bak '/_v3$/d' list_o_tags.out ``` + ::EndTabs + 1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to delete. diff --git a/doc/user/packages/container_registry/troubleshoot_container_registry.md b/doc/user/packages/container_registry/troubleshoot_container_registry.md new file mode 100644 index 00000000000..eac7e0fcacd --- /dev/null +++ b/doc/user/packages/container_registry/troubleshoot_container_registry.md @@ -0,0 +1,129 @@ +--- +stage: Package +group: Container Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting the GitLab Container Registry + +You must sign in to GitLab with administrator rights to troubleshoot most issues with the GitLab Container Registry. + +You can find [additional troubleshooting information](../../../administration/packages/container_registry.md#troubleshooting) in the GitLab Container Registry administration documentation. + +## Migrating OCI container images to GitLab Container Registry + +Migrating container images to the GitLab registry is not supported, but [epic](https://gitlab.com/groups/gitlab-org/-/epics/5210) proposes to change this behavior. + +You can use third-party tools to migrate container images. For example, [skopeo](https://github.com/containers/skopeo), can [copy container images](https://github.com/containers/skopeo#copying-images) between various storage mechanisms. You can use skopeo to copy from container registries, container storage backends, local directories, and local OCI-layout directories to the GitLab Container Registry. + +## Docker connection error + +A Docker connection error can occur when there are special characters in either the group, +project, or branch name. Special characters include: + +- A leading underscore. +- A trailing hyphen or dash. + +To resolve this error, you can change the [group path](../../group/manage.md#change-a-groups-path), +the [project path](../../project/settings/index.md#rename-a-repository) or the branch name. + +You may get a `404 Not Found` or `Unknown Manifest` error message if you use +Docker Engine 17.11 or earlier. Current versions of Docker Engine use +the [v2 API](https://docs.docker.com/registry/spec/manifest-v2-2/). + +The images in your GitLab Container Registry must use the Docker v2 API. +For information on how to update version 1 images to version 2, see the [Docker documentation](https://docs.docker.com/registry/spec/deprecated-schema-v1). + +## `Blob unknown to registry` error when pushing a manifest list + +When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) +to the GitLab Container Registry, you may receive the error +`manifest blob unknown: blob unknown to registry`. This error is likely caused by having multiple images +with different architectures spread out over several repositories instead of the same repository. + +For example, you may have two images, each representing an architecture: + +- The `amd64` platform. +- The `arm64v8` platform. + +To build a multi-arch image with these images, you must push them to the same repository as the +multi-arch image. + +To address the `Blob unknown to registry` error, include the architecture in the tag name of +individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`. +You can then tag the manifest list with `mygroup/myapp:1.0.0`. + +## Unable to change project path or transfer a project + +If you try to change a project path or transfer a project to a new namespace, +you may receive one of the following errors: + +- Project cannot be transferred because tags are present in its container registry. +- Namespace cannot be moved because at least one project has tags in the container registry. + +This error occurs when the project has images in the Container Registry. +You must delete or move these images before you change the path or transfer +the project. + +The following procedure uses these sample project names: + +- For the current project: `gitlab.example.com/org/build/sample_project/cr:v2.9.1`. +- For the new project: `gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1`. + +1. Download the Docker images on your computer: + + ```shell + docker login gitlab.example.com + docker pull gitlab.example.com/org/build/sample_project/cr:v2.9.1 + ``` + + NOTE: + Use either a [personal access token](../../profile/personal_access_tokens.md) or a + [deploy token](../../project/deploy_tokens/index.md) to authenticate your user account. + +1. Rename the images to match the new project name: + + ```shell + docker tag gitlab.example.com/org/build/sample_project/cr:v2.9.1 gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 + ``` + +1. Delete the images in the old project by using the [UI](index.md#delete-images) or [API](../../../api/packages.md#delete-a-project-package). + There may be a delay while the images are queued and deleted. +1. Change the path or transfer the project: + + 1. On the top bar, select **Main menu > Projects** and find your project. + 1. On the left sidebar, select **Settings > General**. + 1. Expand the **Advanced** section. + 1. In the **Change path** text box, edit the path. + 1. Select **Change path**. + +1. Restore the images: + + ```shell + docker push gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 + ``` + +See this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/18383) for details. + +## `unauthorized: authentication required` when pushing large images + +When pushing large images, you may see an authentication error like the following: + +```shell +docker push gitlab.example.com/myproject/docs:latest +The push refers to a repository [gitlab.example.com/myproject/docs] +630816f32edb: Preparing +530d5553aec8: Preparing +... +4b0bab9ff599: Waiting +d1c800db26c7: Waiting +42755cf4ee95: Waiting +unauthorized: authentication required +``` + +This error happens when your authentication token expires before the image push is complete. By default, tokens for +the Container Registry on self-managed GitLab instances expire every five minutes. On GitLab.com, the token expiration +time is set to 15 minutes. + +If you are using self-managed GitLab, you can ask an administrator to +[increase the token duration](../../../administration/packages/container_registry.md#increase-token-duration). diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 563f35f2f4f..9b49f946984 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -118,7 +118,7 @@ API or the UI. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293755) in GitLab 13.12. > - [Required permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/350682) changed from developer to maintainer in GitLab 15.0. -To prevent users from publishing duplicate generic packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) +To prevent users from publishing duplicate generic packages, you can use the [GraphQL API](../../../api/graphql/reference/index.md#packagesettings) or the UI. In the UI: diff --git a/doc/user/packages/gradle_repository/index.md b/doc/user/packages/gradle_repository/index.md new file mode 100644 index 00000000000..4247c13297d --- /dev/null +++ b/doc/user/packages/gradle_repository/index.md @@ -0,0 +1,372 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Maven packages in the Package Registry **(FREE)** + +Publish [Maven](https://maven.apache.org) artifacts in your project's Package Registry using Gradle. +Then, install the packages whenever you need to use them as a dependency. + +For documentation of the specific API endpoints that the Maven package manager +client uses, see the [Maven API documentation](../../../api/packages/maven.md). + +Learn how to build a [Gradle](../workflows/build_packages.md#gradle) package. + +## Publish to the GitLab Package Registry + +### Tokens + +You need a token to publish a package. Different tokens are available depending on what you're trying to +achieve. For more information, review the [guidance on tokens](../package_registry/index.md#authenticate-with-the-registry). + +- If your organization uses two-factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you publish a package via CI/CD pipelines, you must use a CI job token. + +Create a token and save it to use later in the process. + +## Authenticate to the Package Registry with Gradle + +### Authenticate with a personal access token or deploy token in Gradle + +In [your `GRADLE_USER_HOME` directory](https://docs.gradle.org/current/userguide/directory_layout.html#dir:gradle_user_home), +create a file `gradle.properties` with the following content: + +```properties +gitLabPrivateToken=REPLACE_WITH_YOUR_TOKEN +``` + +Your token name depends on which token you use. + +| Token type | Token name | +| --------------------- | --------------- | +| Personal access token | `Private-Token` | +| Deploy token | `Deploy-Token` | + +Add a `repositories` section to your +[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) +file: + +```groovy +repositories { + maven { + url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" + name "GitLab" + credentials(HttpHeaderCredentials) { + name = 'REPLACE_WITH_TOKEN_NAME' + value = gitLabPrivateToken + } + authentication { + header(HttpHeaderAuthentication) + } + } +} +``` + +Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: + +```kotlin +repositories { + maven { + url = uri("https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven") + name = "GitLab" + credentials(HttpHeaderCredentials::class) { + name = "REPLACE_WITH_TOKEN_NAME" + value = findProperty("gitLabPrivateToken") as String? + } + authentication { + create("header", HttpHeaderAuthentication::class) + } + } +} +``` + +### Authenticate with a CI job token in Gradle + +To authenticate with a CI job token, add a `repositories` section to your +[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) +file: + +```groovy +repositories { + maven { + url "${CI_API_V4_URL}/groups/<group>/-/packages/maven" + name "GitLab" + credentials(HttpHeaderCredentials) { + name = 'Job-Token' + value = System.getenv("CI_JOB_TOKEN") + } + authentication { + header(HttpHeaderAuthentication) + } + } +} +``` + +Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: + +```kotlin +repositories { + maven { + url = uri("$CI_API_V4_URL/groups/<group>/-/packages/maven") + name = "GitLab" + credentials(HttpHeaderCredentials::class) { + name = "Job-Token" + value = System.getenv("CI_JOB_TOKEN") + } + authentication { + create("header", HttpHeaderAuthentication::class) + } + } +} +``` + +### Naming convention + +You can use one of three API endpoints to install a Maven package. You must publish a package to a project, but note which endpoint +you use to install the package. The option you choose determines the settings you add to your `pom.xml` file for publishing. + +The three endpoints are: + +- **Project-level**: Use when you have a few Maven packages that are not in the same GitLab group. +- **Group-level**: Use when installing packages from many different projects in the same GitLab group. GitLab does not guarantee the uniqueness of package names in the group. You can have two projects with the same package name and package version. As a result, GitLab serves whichever one is more recent. +- **Instance-level**: Use when installing many packages from different GitLab groups or in their own namespace. + +**Only packages with the same path as the project** are exposed by the instance-level endpoint. + +| Project | Package | Instance-level endpoint available | +| ------------------- | -------------------------------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | + +#### Endpoint URLs + +| Endpoint | Endpoint URL | Additional information | +| -------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| Project | `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<project_id>` with your project ID found on your project's homepage. | +| Group | `https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<group_id>` with your group ID found on your group's homepage. | +| Instance | `https:///gitlab.example.com/api/v4/packages/maven` | Replace `gitlab.example.com` with your domain name. | + +In all cases, to publish a package, you need: + +- A project-specific URL in the `distributionManagement` section. +- A `repository` and `distributionManagement` section. + +### Edit the Groovy DSL or Kotlin DSL + +The Gradle Groovy DSL `repositories` section should look like this: + +```groovy +repositories { + maven { + url "<your_endpoint_url>" + name "GitLab" + } +} +``` + +In Kotlin DSL: + +```kotlin +repositories { + maven { + url = uri("<your_endpoint_url>") + name = "GitLab" + } +} +``` + +- Replace `<your_endpoint_url>` with the [endpoint](#endpoint-urls) you chose. + +## Publish using Gradle + +Your token name depends on which token you use. + +| Token type | Token name | +| --------------------- | --------------- | +| Personal access token | `Private-Token` | +| Deploy token | `Deploy-Token` | + +To publish a package by using Gradle: + +1. Add the Gradle plugin [`maven-publish`](https://docs.gradle.org/current/userguide/publishing_maven.html) to the plugins section: + + In Groovy DSL: + + ```groovy + plugins { + id 'java' + id 'maven-publish' + } + ``` + + In Kotlin DSL: + + ```kotlin + plugins { + java + `maven-publish` + } + ``` + +1. Add a `publishing` section: + + In Groovy DSL: + + ```groovy + publishing { + publications { + library(MavenPublication) { + from components.java + } + } + repositories { + maven { + url "https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven" + credentials(HttpHeaderCredentials) { + name = "REPLACE_WITH_TOKEN_NAME" + value = gitLabPrivateToken // the variable resides in $GRADLE_USER_HOME/gradle.properties + } + authentication { + header(HttpHeaderAuthentication) + } + } + } + } + ``` + + In Kotlin DSL: + + ```kotlin + publishing { + publications { + create<MavenPublication>("library") { + from(components["java"]) + } + } + repositories { + maven { + url = uri("https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven") + credentials(HttpHeaderCredentials::class) { + name = "REPLACE_WITH_TOKEN_NAME" + value = + findProperty("gitLabPrivateToken") as String? // the variable resides in $GRADLE_USER_HOME/gradle.properties + } + authentication { + create("header", HttpHeaderAuthentication::class) + } + } + } + } + ``` + +1. Replace `PROJECT_ID` with your project ID, which you can find on your project's home page. + +1. Run the publish task: + + ```shell + gradle publish + ``` + +Go to your project's **Packages and registries** page and view the published packages. + +## Install a package + +To install a package from the GitLab Package Registry, you must configure +the [remote and authenticate](#authenticate-to-the-package-registry-with-gradle). +After configuring the remote and authenticate, you can install a package from a project, group, or namespace. + +If multiple packages have the same name and version, when you install +a package, the most recently-published package is retrieved. + +Add a [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html) to `build.gradle` in the dependencies section: + +```groovy +dependencies { + implementation 'com.mycompany.mydepartment:my-project:1.0-SNAPSHOT' +} +``` + +Or to `build.gradle.kts` if you are using Kotlin DSL: + +```kotlin +dependencies { + implementation("com.mycompany.mydepartment:my-project:1.0-SNAPSHOT") +} +``` + +## Helpful hints + +For the complete list of helpful hints, see the [Maven documentation](../maven_repository/index.md#helpful-hints). + +### Create Maven packages with GitLab CI/CD by using Gradle + +You can create a package each time the `main` branch +is updated. + +1. Authenticate with [a CI job token in Gradle](#authenticate-with-a-ci-job-token-in-gradle). + +1. Add a `deploy` job to your `.gitlab-ci.yml` file: + + ```yaml + deploy: + image: gradle:6.5-jdk11 + script: + - 'gradle publish' + only: + - main + ``` + +1. Commit files to your repository. + +When the pipeline is successful, the Maven package is created. + +### Publishing a package with the same name or version + +When you publish a package with the same name and version as an existing package, the new package +files are added to the existing package. You can still use the UI or API to access and view the +existing package's older assets. + +Consider using the Packages API or the UI to delete older package versions. + +### Do not allow duplicate Maven packages + +To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. + +In the UI: + +1. For your group, go to **Settings > Packages and registries**. +1. Expand the **Package Registry** section. +1. Turn on the **Do not allow duplicates** toggle. +1. Optional. To allow some duplicate packages, in the **Exceptions** box, enter a regex pattern that matches the names and/or versions of packages you want to allow. + +Your changes are automatically saved. + +### Request forwarding to Maven Central + +FLAG: +By default, this feature is not available for self-managed. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. +This feature is not available for SaaS users. + +When a Maven package is not found in the Package Registry, the request is forwarded +to [Maven Central](https://search.maven.org/). + +When the feature flag is enabled, administrators can disable this behavior in the +[Continuous Integration settings](../../admin_area/settings/continuous_integration.md). + +There are many ways to configure your Maven project to request packages +in Maven Central from GitLab. Maven repositories are queried in a +[specific order](https://maven.apache.org/guides/mini/guide-multiple-repositories.html#repository-order). +By default, maven-central is usually checked first through the +[Super POM](https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Super_POM), so +GitLab needs to be configured to be queried before maven-central. + +[Using GitLab as a mirror of the central proxy](../maven_repository/index.md#setting-gitlab-as-a-mirror-for-the-central-proxy) is one +way to force GitLab to be queried in place of maven-central. + +Maven forwarding is restricted to only the project level and +group level [endpoints](#naming-convention). The instance-level endpoint +has naming restrictions that prevent it from being used for packages that don't follow that convention and also +introduces too much security risk for supply-chain style attacks. diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index bba68494c2d..785ef344c8e 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -72,7 +72,7 @@ Once built, a chart can be uploaded to the desired channel with `curl` or `helm ### Release channels You can publish Helm charts to channels in GitLab. Channels are a method you can use to differentiate Helm chart repositories. -For example, you can use `stable` and `devel` as channels to allow users to add the `stable` repo while `devel` charts are isolated. +For example, you can use `stable` and `devel` as channels to allow users to add the `stable` repository while `devel` charts are isolated. ## Use CI/CD to publish a Helm package diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 2aa71e111fb..c6c2f238564 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -4,9 +4,7 @@ group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Maven packages in the Package Repository **(FREE)** - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +# Maven packages in the Package Registry **(FREE)** Publish [Maven](https://maven.apache.org) artifacts in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. @@ -14,75 +12,29 @@ Then, install the packages whenever you need to use them as a dependency. For documentation of the specific API endpoints that the Maven package manager client uses, see the [Maven API documentation](../../../api/packages/maven.md). -Learn how to build a [Maven](../workflows/build_packages.md#maven) or [Gradle](../workflows/build_packages.md#gradle) package. - -## Authenticate to the Package Registry with Maven - -To authenticate to the Package Registry, you need one of the following: +Learn how to build a [Maven](../workflows/build_packages.md#maven) package. -- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -- A [deploy token](../../project/deploy_tokens/index.md) with the scope set to `read_package_registry`, `write_package_registry`, or both. -- A [CI_JOB_TOKEN](#authenticate-with-a-ci-job-token-in-maven). +## Publish to the GitLab Package Registry -### Authenticate with a personal access token in Maven - -To use a personal access token, add this section to your -[`settings.xml`](https://maven.apache.org/settings.html) file. +### Authenticate to the Package Registry -The `name` must be `Private-Token`. - -```xml -<settings> - <servers> - <server> - <id>gitlab-maven</id> - <configuration> - <httpHeaders> - <property> - <name>Private-Token</name> - <value>REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN</value> - </property> - </httpHeaders> - </configuration> - </server> - </servers> -</settings> -``` +You need an token to publish a package. There are different tokens available depending on what you're trying to achieve. For more information, review the [guidance on tokens](../package_registry/index.md#authenticate-with-the-registry). -### Authenticate with a deploy token in Maven +Create a token and save it to use later in the process. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213566) deploy token authentication in GitLab 13.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +### Edit the `settings.xml` -To use a deploy token, add this section to your +Add the following section to your [`settings.xml`](https://maven.apache.org/settings.html) file. -The `name` must be `Deploy-Token`. - -```xml -<settings> - <servers> - <server> - <id>gitlab-maven</id> - <configuration> - <httpHeaders> - <property> - <name>Deploy-Token</name> - <value>REPLACE_WITH_YOUR_DEPLOY_TOKEN</value> - </property> - </httpHeaders> - </configuration> - </server> - </servers> -</settings> -``` - -### Authenticate with a CI job token in Maven - -To authenticate with a CI job token, add this section to your -[`settings.xml`](https://maven.apache.org/settings.html) file. +NOTE: +The `<name>` field must be named to match the token you chose. -The `name` must be `Job-Token`. +| Token type | Name must be | Token | +| --------------------- | --------------- | ---------------------------------------------------------------------- | +| Personal access token | `Private-Token` | Paste token as-is, or define an environment variable to hold the token | +| Deploy token | `Deploy-Token` | Paste token as-is, or define an environment variable to hold the token | +| CI Job token | `Job-Token` | `${CI_JOB_TOKEN}` | ```xml <settings> @@ -92,8 +44,8 @@ The `name` must be `Job-Token`. <configuration> <httpHeaders> <property> - <name>Job-Token</name> - <value>${CI_JOB_TOKEN}</value> + <name>REPLACE_WITH_NAME</name> + <value>REPLACE_WITH_TOKEN</value> </property> </httpHeaders> </configuration> @@ -102,361 +54,70 @@ The `name` must be `Job-Token`. </settings> ``` -Read more about [how to create Maven packages using GitLab CI/CD](#create-maven-packages-with-gitlab-cicd). - -## Authenticate to the Package Registry with Gradle - -To authenticate to the Package Registry, you need either a personal access token or deploy token. - -- If you use a [personal access token](../../../user/profile/personal_access_tokens.md), set the scope to `api`. -- If you use a [deploy token](../../project/deploy_tokens/index.md), set the scope to `read_package_registry`, `write_package_registry`, or both. - -### Authenticate with a personal access token in Gradle - -In [your `GRADLE_USER_HOME` directory](https://docs.gradle.org/current/userguide/directory_layout.html#dir:gradle_user_home), -create a file `gradle.properties` with the following content: - -```properties -gitLabPrivateToken=REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN -``` - -Add a `repositories` section to your -[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) -file: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" - name "GitLab" - credentials(HttpHeaderCredentials) { - name = 'Private-Token' - value = gitLabPrivateToken - } - authentication { - header(HttpHeaderAuthentication) - } - } -} -``` - -Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven") - name = "GitLab" - credentials(HttpHeaderCredentials::class) { - name = "Private-Token" - value = findProperty("gitLabPrivateToken") as String? - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } -} -``` +### Naming convention -### Authenticate with a deploy token in Gradle - -To authenticate with a deploy token, add a `repositories` section to your -[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) -file: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" - name "GitLab" - credentials(HttpHeaderCredentials) { - name = 'Deploy-Token' - value = '<deploy-token>' - } - authentication { - header(HttpHeaderAuthentication) - } - } -} -``` +You can use one of three endpoints to install a Maven package. You must publish a package to a project, but the endpoint you choose determines the settings you add to your `pom.xml` file for publishing. -Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven") - name = "GitLab" - credentials(HttpHeaderCredentials::class) { - name = "Deploy-Token" - value = "<deploy-token>" - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } -} -``` +The three endpoints are: -### Authenticate with a CI job token in Gradle - -To authenticate with a CI job token, add a `repositories` section to your -[`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) -file: - -```groovy -repositories { - maven { - url "${CI_API_V4_URL}/groups/<group>/-/packages/maven" - name "GitLab" - credentials(HttpHeaderCredentials) { - name = 'Job-Token' - value = System.getenv("CI_JOB_TOKEN") - } - authentication { - header(HttpHeaderAuthentication) - } - } -} -``` +- **Project-level**: Use when you have a few Maven packages and they are not in the same GitLab group. +- **Group-level**: Use when you want to install packages from many different projects in the same GitLab group. GitLab does not guarantee the uniqueness of package names within the group. You can have two projects with the same package name and package version. As a result, GitLab serves whichever one is more recent. +- **Instance-level**: Use when you have many packages in different GitLab groups or in their own namespace. -Or add it to your `build.gradle.kts` file if you are using Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("$CI_API_V4_URL/groups/<group>/-/packages/maven") - name = "GitLab" - credentials(HttpHeaderCredentials::class) { - name = "Job-Token" - value = System.getenv("CI_JOB_TOKEN") - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } -} -``` +**Only packages that have the same path as the project** are exposed by the instance-level endpoint. -## Use the GitLab endpoint for Maven packages +| Project | Package | Instance-level endpoint available | +| ------------------- | -------------------------------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | -To use the GitLab endpoint for Maven packages, choose an option: +#### Endpoint URLs -- **Project-level**: To publish Maven packages to a project, use a project-level endpoint. - To install Maven packages, use a project-level endpoint when you have few Maven packages - and they are not in the same GitLab group. -- **Group-level**: Use a group-level endpoint when you want to install packages from - many different projects in the same GitLab group. -- **Instance-level**: Use an instance-level endpoint when you want to install many - packages from different GitLab groups or in their own namespace. +| Endpoint | Endpoint URL for `pom.xml` | Additional information | +| -------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | +| Project | `https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<project_id>` with your project ID, found on your project's homepage. | +| Group | `https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/maven` | Replace `gitlab.example.com` with your domain name. Replace `<group_id>` with your group ID, found on your group's homepage. | +| Instance | `https:///gitlab.example.com/api/v4/packages/maven` | Replace `gitlab.example.com` with your domain name. | -The option you choose determines the settings you add to your `pom.xml` file. +### Edit the `pom.xml` for publishing -In all cases, to publish a package, you need: +No matter which endpoint you choose, you must have: - A project-specific URL in the `distributionManagement` section. - A `repository` and `distributionManagement` section. -### Project-level Maven endpoint - -The relevant `repository` section of your `pom.xml` -in Maven should look like this: +The relevant `repository` section of your `pom.xml` in Maven should look like this: ```xml <repositories> <repository> <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> + <url><your_endpoint_url></url> </repository> </repositories> <distributionManagement> <repository> <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> + <url>https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven</url> </repository> <snapshotRepository> <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> + <url>https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven</url> </snapshotRepository> </distributionManagement> ``` -The corresponding section in Gradle Groovy DSL would be: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven" - name "GitLab" - } -} -``` - -In Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven") - name = "GitLab" - } -} -``` - -- The `id` is what you [defined in `settings.xml`](#authenticate-to-the-package-registry-with-maven). -- The `PROJECT_ID` is your project ID, which you can view on your project's home page. -- Replace `gitlab.example.com` with your domain name. -- For retrieving artifacts, use either the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the project - (like `group%2Fproject`) or the project's ID (like `42`). However, only the - project's ID can be used for publishing. - -### Group-level Maven endpoint - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -If you rely on many packages, it might be inefficient to include the `repository` section -with a unique URL for each package. Instead, you can use the group-level endpoint for -all the Maven packages stored within one GitLab group. Only packages you have access to -are available for download. - -The group-level endpoint works with any package names, so you -have more flexibility in naming, compared to the [instance-level endpoint](#instance-level-maven-endpoint). -However, GitLab does not guarantee the uniqueness of package names within -the group. You can have two projects with the same package name and package -version. As a result, GitLab serves whichever one is more recent. - -This example shows the relevant `repository` section of your `pom.xml` file. -You still need a project-specific URL for publishing a package in -the `distributionManagement` section: - -```xml -<repositories> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/groups/GROUP_ID/-/packages/maven</url> - </repository> -</repositories> -<distributionManagement> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </repository> - <snapshotRepository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </snapshotRepository> -</distributionManagement> -``` - -For Gradle, the corresponding `repositories` section in Groovy DSL would look like: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/groups/GROUP_ID/-/packages/maven" - name "GitLab" - } -} -``` - -In Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/groups/GROUP_ID/-/packages/maven") - name = "GitLab" - } -} -``` - -- For the `id`, use what you [defined in `settings.xml`](#authenticate-to-the-package-registry-with-maven). -- For `GROUP_ID`, use your group ID, which you can view on your group's home page. -- For `PROJECT_ID`, use your project ID, which you can view on your project's home page. +- The `id` is what you [defined in `settings.xml`](#edit-the-settingsxml). +- The `<your_endpoint_url>` depends on which [endpoint](#endpoint-urls) you choose. - Replace `gitlab.example.com` with your domain name. -- For retrieving artifacts, use either the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the group - (like `group%2Fsubgroup`) or the group's ID (like `12`). - -### Instance-level Maven endpoint - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -If you rely on many packages, it might be inefficient to include the `repository` section -with a unique URL for each package. Instead, you can use the instance-level endpoint for -all Maven packages stored in GitLab. All packages you have access to are available -for download. - -**Only packages that have the same path as the project** are exposed by -the instance-level endpoint. - -| Project | Package | Instance-level endpoint available | -| ------------------- | -------------------------------- | --------------------------------- | -| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | -| `gitlab-org/gitlab` | `foo/bar/1.0-SNAPSHOT` | No | -| `gitlab-org/gitlab` | `gitlab-org/gitlab/1.0-SNAPSHOT` | Yes | - -This example shows how relevant `repository` section of your `pom.xml`. -You still need a project-specific URL in the `distributionManagement` section. - -```xml -<repositories> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/packages/maven</url> - </repository> -</repositories> -<distributionManagement> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </repository> - <snapshotRepository> - <id>gitlab-maven</id> - <url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven</url> - </snapshotRepository> -</distributionManagement> -``` - -The corresponding repositories section in Gradle Groovy DSL would look like: - -```groovy -repositories { - maven { - url "https://gitlab.example.com/api/v4/packages/maven" - name "GitLab" - } -} -``` - -In Kotlin DSL: - -```kotlin -repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/packages/maven") - name = "GitLab" - } -} -``` - -- The `id` is what you [defined in `settings.xml`](#authenticate-to-the-package-registry-with-maven). -- The `PROJECT_ID` is your project ID, which you can view on your project's home page. -- Replace `gitlab.example.com` with your domain name. -- For retrieving artifacts, use either the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the project - (like `group%2Fproject`) or the project's ID (like `42`). However, only the - project's ID can be used for publishing. ## Publish a package -After you have set up the [remote and authentication](#authenticate-to-the-package-registry-with-maven) -and [configured your project](#use-the-gitlab-endpoint-for-maven-packages), +After you have set up the [authentication](#authenticate-to-the-package-registry) +and [chosen an endpoint for publishing](#naming-convention), publish a Maven package to your project. -### Publish by using Maven - To publish a package by using Maven: ```shell @@ -474,122 +135,13 @@ If the deploy is successful, the build success message should be displayed: The message should also show that the package was published to the correct location: ```shell -Uploading to gitlab-maven: https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/mydepartment/my-project/1.0-SNAPSHOT/my-project-1.0-20200128.120857-1.jar +Uploading to gitlab-maven: https://example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/mydepartment/my-project/1.0-SNAPSHOT/my-project-1.0-20200128.120857-1.jar ``` -### Publish by using Gradle - -To publish a package by using Gradle: - -1. Add the Gradle plugin [`maven-publish`](https://docs.gradle.org/current/userguide/publishing_maven.html) to the plugins section: - - In Groovy DSL: - - ```groovy - plugins { - id 'java' - id 'maven-publish' - } - ``` - - In Kotlin DSL: - - ```kotlin - plugins { - java - `maven-publish` - } - ``` - -1. Add a `publishing` section: - - In Groovy DSL: - - ```groovy - publishing { - publications { - library(MavenPublication) { - from components.java - } - } - repositories { - maven { - url "https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven" - credentials(HttpHeaderCredentials) { - name = "Private-Token" - value = gitLabPrivateToken // the variable resides in $GRADLE_USER_HOME/gradle.properties - } - authentication { - header(HttpHeaderAuthentication) - } - } - } - } - ``` - - In Kotlin DSL: - - ```kotlin - publishing { - publications { - create<MavenPublication>("library") { - from(components["java"]) - } - } - repositories { - maven { - url = uri("https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven") - credentials(HttpHeaderCredentials::class) { - name = "Private-Token" - value = - findProperty("gitLabPrivateToken") as String? // the variable resides in $GRADLE_USER_HOME/gradle.properties - } - authentication { - create("header", HttpHeaderAuthentication::class) - } - } - } - } - ``` - -1. Replace `PROJECT_ID` with your project ID, which can be found on your project's home page. - -1. Run the publish task: - - ```shell - gradle publish - ``` - -Now navigate to your project's **Packages and registries** page and view the published artifacts. - -### Publishing a package with the same name or version - -When you publish a package with the same name and version as an existing package, the new package -files are added to the existing package. You can still use the UI or API to access and view the -existing package's older files. - -To delete these older package versions, consider using the Packages API or the UI. - -#### Do not allow duplicate Maven packages - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296895) in GitLab 13.9. -> - [Required permissions](https://gitlab.com/gitlab-org/gitlab/-/issues/350682) changed from developer to maintainer in GitLab 15.0. - -To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. - -In the UI: - -1. For your group, go to **Settings > Packages and registries**. -1. Expand the **Package Registry** section. -1. Turn on the **Reject duplicates** toggle. -1. Optional. To allow some duplicate packages, in the **Exceptions** box, enter a regex pattern that matches the names and/or versions of packages you want to allow. - -Your changes are automatically saved. - ## Install a package To install a package from the GitLab Package Registry, you must configure -the [remote and authenticate](#authenticate-to-the-package-registry-with-maven). +the [remote and authenticate](#authenticate-to-the-package-registry). When this is completed, you can install a package from a project, group, or namespace. @@ -633,8 +185,8 @@ You can install packages by using the Maven `dependency:get` [command](https://m mvn dependency:get -Dartifact=com.nickkipling.app:nick-test-app:1.1-SNAPSHOT -DremoteRepositories=gitlab-maven::::<gitlab endpoint url> -s <path to settings.xml> ``` - - `<gitlab endpoint url>` is the URL of the GitLab [endpoint](#use-the-gitlab-endpoint-for-maven-packages). - - `<path to settings.xml>` is the path to the `settings.xml` file that contains the [authentication details](#authenticate-to-the-package-registry-with-maven). + - `<gitlab endpoint url>` is the URL of the GitLab [endpoint](#endpoint-urls). + - `<path to settings.xml>` is the path to the `settings.xml` file that contains the [authentication details](#edit-the-settingsxml). NOTE: The repository IDs in the command(`gitlab-maven`) and the `settings.xml` file must match. @@ -645,34 +197,34 @@ The message should show that the package is downloading from the Package Registr Downloading from gitlab-maven: http://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven/com/mycompany/mydepartment/my-project/1.0-SNAPSHOT/my-project-1.0-20200128.120857-1.pom ``` -NOTE: -In the GitLab UI, on the Package Registry page for Maven, you can view and copy these commands. +## Helpful hints -### Use Gradle +### Publishing a package with the same name or version -Add a [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html) to `build.gradle` in the dependencies section: +When you publish a package with the same name and version as an existing package, the new package +files are added to the existing package. You can still use the UI or API to access and view the +existing package's older assets. -```groovy -dependencies { - implementation 'com.mycompany.mydepartment:my-project:1.0-SNAPSHOT' -} -``` +To delete older package versions, consider using the Packages API or the UI. -Or to `build.gradle.kts` if you are using Kotlin DSL: +### Do not allow duplicate Maven packages -```kotlin -dependencies { - implementation("com.mycompany.mydepartment:my-project:1.0-SNAPSHOT") -} -``` +To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. -### Request forwarding to Maven Central +In the UI: + +1. For your group, go to **Settings > Packages and registries**. +1. Expand the **Package Registry** section. +1. Turn on the **Do not allow duplicates** toggle. +1. Optional. To allow some duplicate packages, in the **Exceptions** box, enter a regex pattern that matches the names and/or versions of packages you want to allow. + +Your changes are automatically saved. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362657) behind a [feature flag](../../feature_flags.md), disabled by default in GitLab 15.4 +### Request forwarding to Maven Central FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. -On GitLab.com, this feature is not available. +By default this feature is not available for self-managed. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `maven_central_request_forwarding`. +This feature is not available for SaaS users. When a Maven package is not found in the Package Registry, the request is forwarded to [Maven Central](https://search.maven.org/). @@ -690,8 +242,8 @@ GitLab needs to be configured to be queried before maven-central. [Using GitLab as a mirror of the central proxy](#setting-gitlab-as-a-mirror-for-the-central-proxy) is one way to force GitLab to be queried in place of maven-central. -Maven forwarding is restricted to only the [project level](#project-level-maven-endpoint) and -[group level](#group-level-maven-endpoint) endpoints. The [instance level endpoint](#instance-level-maven-endpoint) +Maven forwarding is restricted to only the project level and +group level [endpoints](#naming-convention). The instance level endpoint has naming restrictions that prevent it from being used for packages that don't follow that convention and also introduces too much security risk for supply-chain style attacks. @@ -710,7 +262,7 @@ section to your `settings.xml`: <httpHeaders> <property> <name>Private-Token</name> - <value>{personal_access_token}</value> + <value><personal_access_token></value> </property> </httpHeaders> </configuration> @@ -720,25 +272,19 @@ section to your `settings.xml`: <mirror> <id>central-proxy</id> <name>GitLab proxy of central repo</name> - <url>https://gitlab.example.com/api/v4/projects/{project_id}/packages/maven</url> + <url>https://gitlab.example.com/api/v4/projects/<project_id>/packages/maven</url> <mirrorOf>central</mirrorOf> </mirror> </mirrors> </settings> ``` -## Remove a package - -For your project, go to **Packages and registries > Package Registry**. - -To remove a package, select the red trash icon or, from the package details, the **Delete** button. - -## Create Maven packages with GitLab CI/CD +### Create Maven packages with GitLab CI/CD After you have configured your repository to use the Package Repository for Maven, you can configure GitLab CI/CD to build new packages automatically. -### Create Maven packages with GitLab CI/CD by using Maven +### Create Maven packages with GitLab CI/CD using Maven You can create a new package each time the `main` branch is updated. @@ -808,37 +354,51 @@ user's home location. In this example: - The user is `root`, because the job runs in a Docker container. - Maven uses the configured CI/CD variables. -### Create Maven packages with GitLab CI/CD by using Gradle +### Version validation -You can create a package each time the `main` branch -is updated. +The version string is validated by using the following regex. -1. Authenticate with [a CI job token in Gradle](#authenticate-with-a-ci-job-token-in-gradle). +```ruby +\A(?!.*\.\.)[\w+.-]+\z +``` -1. Add a `deploy` job to your `.gitlab-ci.yml` file: +You can experiment with the regex and try your version strings on [this regular expression editor](https://rubular.com/r/rrLQqUXjfKEoL6). - ```yaml - deploy: - image: gradle:6.5-jdk11 - script: - - 'gradle publish' - only: - - main - ``` +### Useful Maven command-line options -1. Commit files to your repository. +There are some [Maven command-line options](https://maven.apache.org/ref/current/maven-embedder/cli.html) +that you can use when performing tasks with GitLab CI/CD. -When the pipeline is successful, the package is created. +- File transfer progress can make the CI logs hard to read. + Option `-ntp,--no-transfer-progress` was added in + [3.6.1](https://maven.apache.org/docs/3.6.1/release-notes.html#User_visible_Changes). + Alternatively, look at `-B,--batch-mode` + [or lower level logging changes.](https://stackoverflow.com/questions/21638697/disable-maven-download-progress-indication) -### Version validation +- Specify where to find the `pom.xml` file (`-f,--file`): -The version string is validated by using the following regex. + ```yaml + package: + script: + - 'mvn --no-transfer-progress -f helloworld/pom.xml package' + ``` -```ruby -\A(?!.*\.\.)[\w+.-]+\z -``` +- Specify where to find the user settings (`-s,--settings`) instead of + [the default location](https://maven.apache.org/settings.html). There's also a `-gs,--global-settings` option: + + ```yaml + package: + script: + - 'mvn -s settings/ci.xml package' + ``` -You can play around with the regex and try your version strings on [this regular expression editor](https://rubular.com/r/rrLQqUXjfKEoL6). +### Supported CLI commands + +The GitLab Maven repository supports the following Maven CLI commands: + +- `mvn deploy`: Publish your package to the Package Registry. +- `mvn install`: Install packages specified in your Maven project. +- `mvn dependency:get`: Install a specific package. ## Troubleshooting @@ -870,34 +430,6 @@ mvn deploy \ WARNING: When you set these options, all network requests are logged and a large amount of output is generated. -### Useful Maven command-line options - -There are some [Maven command-line options](https://maven.apache.org/ref/current/maven-embedder/cli.html) -that you can use when performing tasks with GitLab CI/CD. - -- File transfer progress can make the CI logs hard to read. - Option `-ntp,--no-transfer-progress` was added in - [3.6.1](https://maven.apache.org/docs/3.6.1/release-notes.html#User_visible_Changes). - Alternatively, look at `-B,--batch-mode` - [or lower level logging changes.](https://stackoverflow.com/questions/21638697/disable-maven-download-progress-indication) - -- Specify where to find the `pom.xml` file (`-f,--file`): - - ```yaml - package: - script: - - 'mvn --no-transfer-progress -f helloworld/pom.xml package' - ``` - -- Specify where to find the user settings (`-s,--settings`) instead of - [the default location](https://maven.apache.org/settings.html). There's also a `-gs,--global-settings` option: - - ```yaml - package: - script: - - 'mvn -s settings/ci.xml package' - ``` - ### Verify your Maven settings If you encounter issues within CI/CD that relate to the `settings.xml` file, try adding @@ -916,11 +448,3 @@ package: - 'mvn help:system' - 'mvn package' ``` - -## Supported CLI commands - -The GitLab Maven repository supports the following Maven CLI commands: - -- `mvn deploy`: Publish your package to the Package Registry. -- `mvn install`: Install packages specified in your Maven project. -- `mvn dependency:get`: Install a specific package. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 5d2efc52ba9..c62999100c1 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -6,250 +6,97 @@ info: To determine the technical writer assigned to the Stage/Group associated w # npm packages in the Package Registry **(FREE)** -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -Publish npm packages in your project's Package Registry. Then install the -packages whenever you need to use them as a dependency. - -Only [scoped](https://docs.npmjs.com/misc/scope/) packages are supported. - -For documentation of the specific API endpoints that the npm package manager -client uses, see the [npm API documentation](../../../api/packages/npm.md). - -WARNING: -Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can -be committed to a repository. +For documentation of the specific API endpoints that the npm package manager client uses, see the [npm API documentation](../../../api/packages/npm.md). Learn how to build an [npm](../workflows/build_packages.md#npm) or [yarn](../workflows/build_packages.md#yarn) package. -## Use the GitLab endpoint for npm packages - -To use the GitLab endpoint for npm packages, choose an option: +Watch a [video demo](https://youtu.be/yvLxtkvsFDA) of how to publish npm packages to the GitLab Package Registry. -- **Project-level**: Use when you have few npm packages and they are not in - the same GitLab group. The [package naming convention](#package-naming-convention) is not enforced at this level. - Instead, you should use a [scope](https://docs.npmjs.com/cli/v6/using-npm/scope/) for your package. - When you use a scope, the registry URL is [updated](#authenticate-to-the-package-registry) only for that scope. -- **Instance-level**: Use when you have many npm packages in different - GitLab groups or in their own namespace. Be sure to comply with the [package naming convention](#package-naming-convention). +## Publish to GitLab Package Registry -Some features such as [publishing](#publish-an-npm-package) a package is only available on the project-level endpoint. +### Authentication to the Package Registry -## Authenticate to the Package Registry +You need an token to publish a package. There are different tokens available depending on what you're trying to achieve. For more information, review the [guidance on tokens](../../../user/packages/package_registry/index.md#authenticate-with-the-registry). -You must authenticate with the Package Registry when the project -is private. Public projects do not require authentication. +- If your organization uses two factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you are publishing a package via CI/CD pipelines, you must use a CI job token. -To authenticate, use one of the following: +Create a token and save it to use later in the process. -- A [personal access token](../../../user/profile/personal_access_tokens.md) - (required for two-factor authentication (2FA)), with the scope set to `api`. -- A [deploy token](../../project/deploy_tokens/index.md), with the scope set to `read_package_registry`, `write_package_registry`, or both. -- It's not recommended, but you can use [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). - Standard OAuth tokens cannot authenticate to the GitLab npm Registry. You must use a personal access token with OAuth headers. -- A [CI job token](#authenticate-with-a-ci-job-token). -- Your npm package name must be in the format of [`@scope/package-name`](#package-naming-convention). - It must match exactly, including the case. +### Naming convention -### Authenticate with a personal access token or deploy token +Depending on how the package will be installed, you may need to adhere to the naming convention. -To authenticate with the Package Registry, you need a [personal access token](../../profile/personal_access_tokens.md) or [deploy token](../../project/deploy_tokens/index.md). +You can use one of two API endpoints to install packages: -#### Project-level npm endpoint +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have few npm packages and they are not in the same GitLab group. -To use the [project-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, set your npm configuration: +If you plan to install a package through the [project level](#install-from-the-project-level), then you do not have to adhere to the naming convention. -```shell -# Set URL for your scoped packages. -# For example package with name `@foo/bar` will use this URL for download -npm config set @foo:registry https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/ - -# Add the token for the scoped packages URL. Replace <your_project_id> -# with the project where your package is located. -npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" -``` - -- `<your_project_id>` is your project ID, found on the project's home page. -- `<your_token>` is your personal access token or deploy token. -- Replace `gitlab.example.com` with your domain name. +If you plan to install a package through the [instance level](#install-from-the-instance-level), then you must name your package with a [scope](https://docs.npmjs.com/misc/scope/). Scoped packages begin with a `@` have the format of `@owner/package-name`. You can set up the scope for your package in the `.npmrc` file and by using the `publishConfig` option in the `package.json`. -You should now be able to publish and install npm packages in your project. +- The value used for the `@scope` is the root of the project that will host the packages and not the root + of the project with the source code of the package itself. The scope should be lowercase. +- The package name can be anything you want -If you encounter an error with [Yarn](https://classic.yarnpkg.com/en/), view -[troubleshooting steps](#troubleshooting). +| Project URL | Package Registry in | Scope | Full package name | +| ------------------------------------------------------- | ------------------- | --------- | ---------------------- | +| `https://gitlab.com/my-org/engineering-group/analytics` | Analytics | `@my-org` | `@my-org/package-name` | -#### Instance-level npm endpoint - -NOTE: -Note: Using `CI_JOB_TOKEN` to install npm packages with dependencies in another project will give you 404 errors. You can use a [personal access token](../../profile/personal_access_tokens.md) as a workaround. [GitLab-#352962](https://gitlab.com/gitlab-org/gitlab/-/issues/352962) proposes a fix to this bug. - -To use the [instance-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, set your npm configuration: +Make sure that the name of your package in the `package.json` file matches this convention: ```shell -# Set URL for your scoped packages. -# For example package with name `@foo/bar` will use this URL for download -npm config set @foo:registry https://gitlab.example.com/api/v4/packages/npm/ - -# Add the token for the scoped packages URL. This will allow you to download -# `@foo/` packages from private projects. -npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" -``` - -- `<your_token>` is your personal access token or deploy token. -- Replace `gitlab.example.com` with your domain name. - -You should now be able to install npm packages in your project. - -If you encounter an error with [Yarn](https://classic.yarnpkg.com/en/), view -[troubleshooting steps](#troubleshooting). - -### Authenticate with a CI job token - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9104) in GitLab 12.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. - -If you're using npm with GitLab CI/CD, a CI job token can be used instead of a personal access token or deploy token. -The token inherits the permissions of the user that generates the pipeline. - -#### Project-level npm endpoint - -To use the [project-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, add a corresponding section to your `.npmrc` file: - -```ini -@foo:registry=https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/npm/ -//gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN} +"name": "@my-org/package-name" ``` -#### Instance-level npm endpoint - -To use the [instance-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, add a corresponding section to your `.npmrc` file: - -```ini -@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken=${CI_JOB_TOKEN} -``` +## Publishing a package via the command line -#### Use variables to avoid hard-coding auth token values +### Authenticating via the `.npmrc` -To avoid hard-coding the `authToken` value, you may use a variable in its place: +Create or edit the `.npmrc` file in the same directory as your `package.json`. Include the following lines in the `.npmrc` file: ```shell -npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "${NPM_TOKEN}" -npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "${NPM_TOKEN}" +@scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ +//your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken="${NPM_TOKEN}" ``` -Then, you can run `npm publish` either locally or by using GitLab CI/CD. +- Replace `@scope` with the [root level group](#naming-convention) of the project you're publishing to the package to. +- Replace `your_domain_name` with your domain name, for example, `gitlab.com`. +- Replace `your_project_id` is your project ID, found on the project's home page. +- `"${NPM_TOKEN}"` will be associated with the token you created later in the process. -- **Locally:** Export `NPM_TOKEN` before publishing: - - ```shell - NPM_TOKEN=<your_token> npm publish - ``` - -- **GitLab CI/CD:** Set an `NPM_TOKEN` [CI/CD variable](../../../ci/variables/index.md) - under your project's **Settings > CI/CD > Variables**. +WARNING: +Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can +be committed to a repository. -## Working with private registries +### Publishing a package via the command line -When working with private repositories, you may want to configure additional settings to ensure a secure communication channel: +Associate your [token](#authentication-to-the-package-registry) with the `"${NPM_TOKEN}"` in the `.npmrc`. Replace `your_token` with a deploy token, group access token, project access token, or personal access token. ```shell -# Force npm to always require authentication when accessing the registry, even for GET requests. -npm config set always-auth true -``` - -## Package naming convention - -When you use the [instance-level endpoint](#use-the-gitlab-endpoint-for-npm-packages), only the packages with names in the format of `@scope/package-name` are available. - -- The `@scope` is the root namespace of the GitLab project. To follow npm's convention, it should be - lowercase. However, the GitLab package registry allows for uppercase. Before GitLab 13.10, the - `@scope` had to be a case-sensitive match of the GitLab project's root namespace. This was - problematic because the npm public registry does not allow uppercase letters. GitLab 13.10 relaxes - this requirement and translates uppercase in the GitLab `@scope` to lowercase for npm. For - example, a package `@MyScope/package-name` in GitLab becomes `@myscope/package-name` for npm. -- The `package-name` can be whatever you want. - -NOTE: -The value used for the `@scope` is the root of the project that will end up hosting the packages and not the root -of the project with the source code of the package itself. For example, assume your package source code is located -at `source-code-group/package-code` and deployed to a package registry inside `registries-group/registry-project`. -In this case, the `@scope` needs to be `@registries-group` and not `@source-code-group`. - -For example, if your project is `https://gitlab.example.com/my-org/engineering-group/team-amazing/analytics`, -the root namespace is `my-org`. When you publish a package, it must have `my-org` as the scope. - -| Project | Package | Supported | -| ------------------- | -------------------- | --------- | -| `my-org/bar` | `@my-org/bar` | Yes | -| `my-org/bar/baz` | `@my-org/baz` | Yes | -| `My-Org/Bar/baz` | `@my-org/Baz` | Yes | -| `My-Org/Bar/baz` | `@My-Org/Baz` | Yes | -| `my-org/bar/buz` | `@my-org/anything` | Yes | -| `gitlab-org/gitlab` | `@gitlab-org/gitlab` | Yes | -| `gitlab-org/gitlab` | `@foo/bar` | No | - -In GitLab, this regex validates all package names from all package managers: - -```plaintext -/\A\@?(([\w\-\.\+]*)\/)*([\w\-\.]+)@?(([\w\-\.\+]*)\/)*([\w\-\.]*)\z/ +NPM_TOKEN=your_token npm publish ``` -This regex allows almost all of the characters that npm allows, with a few exceptions (for example, `~` is not allowed). - -The regex also allows for capital letters, while npm does not. +Your package should now publish to the Package Registry. -## Limitations +## Publishing a package via a CI/CD pipeline -When you update the path of a user or group, or transfer a subgroup or project, -you must remove any npm packages first. You cannot update the root namespace -of a project with npm packages. Make sure you update your `.npmrc` files to follow -the naming convention and run `npm publish` if necessary. +### Authenticating via the `.npmrc` -## Publish an npm package - -Prerequisites: - -- [Authenticate](#authenticate-to-the-package-registry) to the Package Registry. -- Set a [project-level npm endpoint](#use-the-gitlab-endpoint-for-npm-packages). - -To upload an npm package to your project, run this command: +Create or edit the `.npmrc` file in the same directory as your `package.json` in a GitLab project. Include the following lines in the `.npmrc` file: ```shell -npm publish +@scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ +//your_domain_name/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN} ``` -To view the package, go to your project's **Packages and registries**. +- Replace `@scope` with the [root level group](#naming-convention) of the project you're publishing to the package to. +- The `${CI_PROJECT_ID}` and `${CI_JOB_TOKEN}` are [predefined variables](../../../ci/variables/predefined_variables.md) that are available in the pipeline and do not need to be replaced. -You can also define `"publishConfig"` for your project in `package.json`. For example: +### Publishing a package via a CI/CD pipeline -```json -{ - "publishConfig": { - "@foo:registry": " https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/" - } -} -``` - -This forces the package to publish only to the specified registry. - -If you try to publish a package [with a name that already exists](#publishing-packages-with-the-same-name-or-version) within -a given scope, you get a `403 Forbidden!` error. - -## Publish an npm package by using CI/CD - -Prerequisites: - -- [Authenticate](#authenticate-to-the-package-registry) to the Package Registry. -- Set a [project-level npm endpoint](#use-the-gitlab-endpoint-for-npm-packages). -- Your npm package name must be in the format of [`@scope/package-name`](#package-naming-convention). - It must match exactly, including the case. This is different than the - npm naming convention, but it is required to work with the GitLab Package Registry. - -To work with npm commands within [GitLab CI/CD](../../../ci/index.md), you can use -`CI_JOB_TOKEN` in place of the personal access token or deploy token in your commands. - -An example `.gitlab-ci.yml` file for publishing npm packages: +In the GitLab project that houses your `.npmrc` and `package.json`, edit or create a `.gitlab-ci.yml` file. For example: ```yaml image: node:latest @@ -262,143 +109,105 @@ deploy: script: - echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">.npmrc - npm publish - environment: production ``` -See the -[Publish npm packages to the GitLab Package Registry using semantic-release](../../../ci/examples/semantic-release.md) -step-by-step guide and demo project for a complete example. - -## Configure the GitLab npm registry with Yarn 2 - -You can get started with Yarn 2 by following the [Yarn documentation](https://yarnpkg.com/getting-started/install/). +Your package should now publish to the Package Registry when the pipeline runs. -To publish and install with the project-level npm endpoint, set the following configuration in -`.yarnrc.yml`: +## Install a package -```yaml -npmScopes: - foo: - npmRegistryServer: 'https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/' - npmPublishRegistry: 'https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/' - -npmRegistries: - //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/: - npmAlwaysAuth: true - npmAuthToken: '<your_token>' -``` +If multiple packages have the same name and version, when you install a package, the most recently-published package is retrieved. -For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.yml`: +You can install a package from a GitLab project or instance: -```yaml -npmScopes: - foo: - npmRegistryServer: 'https://gitlab.example.com/api/v4/packages/npm/' - -npmRegistries: - //gitlab.example.com/api/v4/packages/npm/: - npmAlwaysAuth: true - npmAuthToken: '<your_token>' -``` +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have few npm packages and they are not in the same GitLab group. -In this configuration: +### Install from the instance level -- Replace `<your_token>` with your personal access token or deploy token. -- Replace `<your_project_id>` with your project's ID, which you can find on the project's home page. -- Replace `gitlab.example.com` with your domain name. -- Your scope is `foo`, without `@`. +WARNING: +In order to install a package from the instance level, the package must have been published following the scoped [naming convention](#naming-convention). -## Publishing packages with the same name or version +1. Authenticate to the Package Registry -You cannot publish a package if a package of the same name and version already exists. -You must delete the existing package first. + If you would like to install a package from a private project, you will need to authenticate to the Package Registry. Skip this step if the project is not private. -This rule has a different impact depending on the package name: + ```shell + npm config set -- //your_domain_name/api/v4/packages/npm/:_authToken=your_token + ``` -- For packages following the [naming convention](#package-naming-convention), you can't publish a - package with a duplicate name and version to the root namespace. -- For packages not following the [naming convention](#package-naming-convention), you can't publish - a package with a duplicate name and version to the project you target with the upload. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. -This aligns with npmjs.org's behavior. However, npmjs.org does not ever let you publish -the same version more than once, even if it has been deleted. +1. Set the registry -## `package.json` limitations + ```shell + npm config set @scope:registry https://your_domain_name.com/api/v4/packages/npm/ + ``` -You can't publish a package if its `package.json` file exceeds 20,000 characters. + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. -## Install a package +1. Install the package -npm packages are commonly-installed by using the `npm` or `yarn` commands -in a JavaScript project. You can install a package from the scope of a project or instance. + ```shell + npm install @scope/my-package + ``` -If multiple packages have the same name and version, when you install a package, the most recently-published package is retrieved. +### Install from the project level -1. Set the URL for scoped packages. +1. Authenticate to the Package Registry - For [instance-level endpoints](#use-the-gitlab-endpoint-for-npm-packages) run: + If you would like to install a package from a private project, you will need to authenticate to the Package Registry. Skip this step if the project is not private. ```shell - npm config set @foo:registry https://gitlab.example.com/api/v4/packages/npm/ + npm config set -- //your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken=your_token ``` - - Replace `@foo` with your scope. - - Replace `gitlab.example.com` with your domain name. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. - For [project-level endpoints](#use-the-gitlab-endpoint-for-npm-packages) run: +1. Set the registry ```shell - npm config set @foo:registry https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/ + npm config set @scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ ``` - - Replace `@foo` with your scope. - - Replace `gitlab.example.com` with your domain name. - - Replace `<your_project_id>` with your project ID, found on the project's home page. + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. -1. Ensure [authentication](#authenticate-to-the-package-registry) is configured. - -1. To install a package in your project, run: +1. Install the package ```shell - npm install @my-scope/my-package + npm install @scope/my-package ``` - Or if you're using Yarn: +## Helpful hints - ```shell - yarn add @my-scope/my-package - ``` +### Package forwarding to npmjs.com -In [GitLab 12.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/55344), -when an npm package is not found in the Package Registry, the request is forwarded to [npmjs.com](https://www.npmjs.com/). +When an npm package is not found in the Package Registry, the request is forwarded to [npmjs.com](https://www.npmjs.com/). Administrators can disable this behavior in the [Continuous Integration settings](../../admin_area/settings/continuous_integration.md). +Group owners can disable this behavior in the group Packages and Registries settings. + ### Install npm packages from other organizations You can route package requests to organizations and users outside of GitLab. -To do this, add lines to your `.npmrc` file. Replace `my-org` with the namespace or group that owns your project's repository, +To do this, add lines to your `.npmrc` file. Replace `@my-other-org` with the namespace or group that owns your project's repository, and use your organization's URL. The name is case-sensitive and must match the name of your group or namespace exactly. -Use environment variables to set up your tokens: `export MY_TOKEN="<your token>"`. - ```shell -@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=${MY_TOKEN} - -@my-other-org:registry=https://gitlab.example.com/api/v4/packages/npm/ -//gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=${MY_TOKEN} +@scope:registry=https://my_domain_name.com/api/v4/packages/npm/ +@my-other-org:registry=https://my_domain_name.example.com/api/v4/packages/npm/ ``` ### npm metadata -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11867) in GitLab 12.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/330929) in GitLab 14.5. - The GitLab Package Registry exposes the following attributes to the npm client. These are similar to the [abbreviated metadata format](https://github.com/npm/registry/blob/9e368cf6aaca608da5b2c378c0d53f475298b916/docs/responses/package-metadata.md#abbreviated-metadata-format): @@ -417,10 +226,7 @@ These are similar to the [abbreviated metadata format](https://github.com/npm/re - `engines` - `_hasShrinkwrap` -## Add npm distribution tags - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9425) in GitLab 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +### Add npm distribution tags You can add [distribution tags](https://docs.npmjs.com/cli/dist-tag/) to newly-published packages. Tags are optional and can be assigned to only one package at a time. @@ -443,87 +249,46 @@ View [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/258835) for deta Due to a bug in npm 6.9.0, deleting distribution tags fails. Make sure your npm version is 6.9.1 or later. -## Troubleshooting - -When troubleshooting npm issues, first run the same command with the `--verbose` flag to confirm -what registry you are hitting. - -To improve performance, npm caches files related to a package. Note that npm doesn't remove data by -itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with -this command: - -```shell -npm cache clean --force -``` - -### Error running Yarn with the Package Registry for npm registry - -If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get -an error message like: - -```shell -yarn install v1.15.2 -warning package.json: No license field -info No lockfile found. -warning XXX: No license field -[1/4] 🔍 Resolving packages... -[2/4] 🚚 Fetching packages... -error An unexpected error occurred: "https://gitlab.example.com/api/v4/projects/XXX/packages/npm/XXX/XXX/-/XXX/XXX-X.X.X.tgz: Request failed \"404 Not Found\"". -info If you think this is a bug, please open a bug report with the information provided in "/Users/XXX/gitlab-migration/module-util/yarn-error.log". -info Visit https://classic.yarnpkg.com/en/docs/cli/install for documentation about this command -``` - -In this case, try adding this to your `.npmrc` file (and replace `<your_token>` -with your personal access token or deploy token): +### Supported CLI commands -```plaintext -//gitlab.example.com/api/v4/projects/:_authToken=<your_token> -``` +The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI +(`yarn`): -You can also use `yarn config` instead of `npm config` when setting your auth-token dynamically: +- `npm install`: Install npm packages. +- `npm publish`: Publish an npm package to the registry. +- `npm dist-tag add`: Add a dist-tag to an npm package. +- `npm dist-tag ls`: List dist-tags for a package. +- `npm dist-tag rm`: Delete a dist-tag. +- `npm ci`: Install npm packages directly from your `package-lock.json` file. +- `npm view`: Show package metadata. -```shell -yarn config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" -yarn config set '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" -``` +## Troubleshooting ### `npm publish` targets default npm registry (`registry.npmjs.org`) Ensure that your package scope is set consistently in your `package.json` and `.npmrc` files. -For example, if your project name in GitLab is `foo/my-package`, then your `package.json` file +For example, if your project name in GitLab is `@scope/my-package`, then your `package.json` file should look like: ```json { - "name": "@foo/my-package", - "version": "1.0.0", - "description": "Example package for GitLab npm registry" + "name": "@scope/my-package" } ``` And the `.npmrc` file should look like: -```ini -//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_token> -//gitlab.example.com/api/v4/packages/npm/:_authToken=<your_token> -@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ -``` - -### `npm install` returns `Error: Failed to replace env in config: ${npm_TOKEN}` - -You do not need a token to run `npm install` unless your project is private. The token is only required to publish. If the `.npmrc` file was checked in with a reference to `$npm_TOKEN`, you can remove it. If you prefer to leave the reference in, you must set a value prior to running `npm install` or set the value by using [GitLab CI/CD variables](../../../ci/variables/index.md): - ```shell -NPM_TOKEN=<your_token> npm install +@scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ +//your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken="${NPM_TOKEN}" ``` ### `npm install` returns `npm ERR! 403 Forbidden` If you get this error, ensure that: -- The Package Registry is enabled in your project settings. Although the Package Registry is enabled - by default, it's possible to [disable it](../package_registry/index.md#disable-the-package-registry). +- The Package Registry is enabled in your project settings. Although the Package Registry is enabled by default, it's possible to [disable it](../package_registry/index.md#disable-the-package-registry). - Your token is not expired and has appropriate permissions. - A package with the same name or version doesn't already exist within the given scope. - The scoped packages URL includes a trailing slash: @@ -534,30 +299,25 @@ If you get this error, ensure that: If you get this error, one of the following problems could be causing it. -#### Package name does not meet the naming convention +### Package name does not meet the naming convention -Your package name may not meet the -[`@scope/package-name` package naming convention](#package-naming-convention). +Your package name may not meet the [`@scope/package-name` package naming convention](#naming-convention). -Ensure the name meets the convention exactly, including the case. -Then try to publish again. +Ensure the name meets the convention exactly, including the case. Then try to publish again. -#### Package already exists +### Package already exists -Your package has already been published to another project in the same -root namespace and therefore cannot be published again using the same name. +Your package has already been published to another project in the same root namespace and therefore cannot be published again using the same name. -This is also true even if the prior published package shares the same name, -but not the version. +This is also true even if the prior published package shares the same name, but not the version. -#### Package JSON file is too large +### Package JSON file is too large -Make sure that your `package.json` file does not [exceed `20,000` characters](#packagejson-limitations). +Make sure that your `package.json` file does not exceed `20,000` characters. ### `npm publish` returns `npm ERR! 500 Internal Server Error - PUT` -This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab -13.3.x and later. The error in the logs will appear as: +This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab 13.3.x and later. The error in the logs will appear as: ```plaintext >NoMethodError - undefined method `preferred_language' for #<Rack::Response @@ -572,22 +332,7 @@ This might be accompanied by another error: This is usually a permissions issue with either: - `'packages_storage_path'` default `/var/opt/gitlab/gitlab-rails/shared/packages/`. -- The remote bucket if [object storage](../../../administration/packages/index.md#using-object-storage) +- The remote bucket if [object storage](../../../administration/packages/index.md#use-object-storage) is used. In the latter case, ensure the bucket exists and GitLab has write access to it. - -## Supported CLI commands - -The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI -(`yarn`): - -- `npm install`: Install npm packages. -- `npm publish`: Publish an npm package to the registry. -- `npm dist-tag add`: Add a dist-tag to an npm package. -- `npm dist-tag ls`: List dist-tags for a package. -- `npm dist-tag rm`: Delete a dist-tag. -- `npm ci`: Install npm packages directly from your `package-lock.json` file. -- `npm view`: Show package metadata. -- `yarn add`: Install an npm package. -- `yarn update`: Update your dependencies. diff --git a/doc/user/packages/nuget_repository/img/visual_studio_adding_nuget_source.png b/doc/user/packages/nuget_repository/img/visual_studio_adding_nuget_source.png Binary files differdeleted file mode 100644 index 7397403f4bf..00000000000 --- a/doc/user/packages/nuget_repository/img/visual_studio_adding_nuget_source.png +++ /dev/null diff --git a/doc/user/packages/nuget_repository/img/visual_studio_nuget_source_added.png b/doc/user/packages/nuget_repository/img/visual_studio_nuget_source_added.png Binary files differdeleted file mode 100644 index 8c14a14e304..00000000000 --- a/doc/user/packages/nuget_repository/img/visual_studio_nuget_source_added.png +++ /dev/null diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 956202bb990..540db463f0a 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -118,27 +118,25 @@ A project-level endpoint is also required to install NuGet packages from a proje To use the [project-level](#use-the-gitlab-endpoint-for-nuget-packages) NuGet endpoint, add the Package Registry as a source with Visual Studio: 1. Open [Visual Studio](https://visualstudio.microsoft.com/vs/). -1. In Windows, select **File > Options**. On macOS, select **Visual Studio > Preferences**. +1. In Windows, select **Tools > Options**. On macOS, select **Visual Studio > Preferences**. 1. In the **NuGet** section, select **Sources** to view a list of all your NuGet sources. 1. Select **Add**. 1. Complete the following fields: - **Name**: Name for the source. - - **Location**: `https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/nuget/index.json`, + - **Source**: `https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/nuget/index.json`, where `<your_project_id>` is your project ID, and `gitlab.example.com` is your domain name. - - **Username**: Your GitLab username or deploy token username. - - **Password**: Your personal access token or deploy token. - -  1. Select **Save**. +1. When you access the package, you must enter your **Username** and **Password**: -The source is displayed in your list. + - **Username**: Your GitLab username or deploy token username. + - **Password**: Your personal access token or deploy token. - +The source is displayed in your list. -If you get a warning, ensure that the **Location**, **Username**, and +If you get a warning, ensure that the **Source**, **Username**, and **Password** are correct. #### Group-level endpoint @@ -148,27 +146,25 @@ To install a package from a group, use a group-level endpoint. To use the [group-level](#use-the-gitlab-endpoint-for-nuget-packages) NuGet endpoint, add the Package Registry as a source with Visual Studio: 1. Open [Visual Studio](https://visualstudio.microsoft.com/vs/). -1. In Windows, select **File > Options**. On macOS, select **Visual Studio > Preferences**. +1. In Windows, select **Tools > Options**. On macOS, select **Visual Studio > Preferences**. 1. In the **NuGet** section, select **Sources** to view a list of all your NuGet sources. 1. Select **Add**. 1. Complete the following fields: - **Name**: Name for the source. - - **Location**: `https://gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/nuget/index.json`, + - **Source**: `https://gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/nuget/index.json`, where `<your_group_id>` is your group ID, and `gitlab.example.com` is your domain name. - - **Username**: Your GitLab username or deploy token username. - - **Password**: Your personal access token or deploy token. - -  1. Select **Save**. +1. When you access the package, you must enter your **Username** and **Password**. -The source is displayed in your list. + - **Username**: Your GitLab username or deploy token username. + - **Password**: Your personal access token or deploy token. - +The source is displayed in your list. -If you get a warning, ensure that the **Location**, **Username**, and +If you get a warning, ensure that the **Source**, **Username**, and **Password** are correct. ### Add a source with the .NET CLI diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 1aeb98fd48a..caa305999c5 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. With the GitLab Package Registry, you can use GitLab as a private or public registry for a variety -of [supported package managers](#supported-package-managers). +of [supported package managers](supported_package_managers.md). You can publish and share packages, which can be consumed as a dependency in downstream projects. ## Package workflows @@ -60,6 +60,9 @@ For most package types, the following credential types are valid: allows access to packages in the project running the job for the users running the pipeline. Access to other external projects can be configured. +- If your organization uses two factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you are publishing a package via CI/CD pipelines, you must use a CI job token. + NOTE: If you have not activated the "Packages" feature for your project at **Settings > General > Project features**, you will receive a 403 Forbidden response. Accessing package registry via deploy token is not available when external authorization is enabled. @@ -78,7 +81,7 @@ Learn more about using the GitLab Package Registry with CI/CD: - [Conan](../conan_repository/index.md#publish-a-conan-package-by-using-cicd) - [Generic](../generic_packages/index.md#publish-a-generic-package-by-using-cicd) - [Maven](../maven_repository/index.md#create-maven-packages-with-gitlab-cicd) -- [npm](../npm_registry/index.md#publish-an-npm-package-by-using-cicd) +- [npm](../npm_registry/index.md#publishing-a-package-via-a-cicd-pipeline) - [NuGet](../nuget_repository/index.md#publish-a-nuget-package-by-using-cicd) - [PyPI](../pypi_repository/index.md#authenticate-with-a-ci-job-token) - [RubyGems](../rubygems_registry/index.md#authenticate-with-a-ci-job-token) @@ -117,50 +120,40 @@ The **Packages and registries > Package Registry** entry is removed from the sid [Project-level permissions](../../permissions.md) determine actions such as downloading, pushing, or deleting packages. -The visibility of the Package Registry is independent of the repository and can't be controlled from +The visibility of the Package Registry is independent of the repository and can be controlled from your project's settings. For example, if you have a public project and set the repository visibility -to **Only Project Members**, the Package Registry is then public. However, disabling the Package +to **Only Project Members**, the Package Registry is then public. Disabling the Package Registry disables all Package Registry operations. -[GitLab-#329253](https://gitlab.com/gitlab-org/gitlab/-/issues/329253) -proposes adding the ability to control Package Registry visibility from the UI. - -| | | Anonymous<br/>(everyone on internet) | Guest | Reporter, Developer, Maintainer, Owner | -| -------------------- | --------------------- | --------- | ----- | ------------------------------------------ | -| Public project with Package Registry enabled | View Package Registry <br/> and pull packages | Yes | Yes | Yes | -| Internal project with Package Registry enabled | View Package Registry <br/> and pull packages | No | Yes | Yes | -| Private project with Package Registry enabled | View Package Registry <br/> and pull packages | No | No | Yes | -| Any project with Package Registry disabled | All operations on Package Registry | No | No | No | - -## Supported package managers +| Project visibility | Action | [Role](../../permissions.md#roles) required | +|--------------------|-----------------------|---------------------------------------------------------| +| Public | View Package Registry | `n/a`, everyone on the internet can perform this action | +| Public | Publish a package | Developer or higher | +| Public | Pull a package | `n/a`, everyone on the internet can perform this action | +| Internal | View Package Registry | Guest or higher | +| Internal | Publish a package | Developer or higher | +| Internal | Pull a package | Guest or higher(1) | +| Private | View Package Registry | Reporter or higher | +| Private | Publish a package | Developer or higher | +| Private | Pull a package | Reporter or higher(1) | -WARNING: -Not all package manager formats are ready for production use. To view each format's status, see the -table's **Status** column. +### Allow anyone to pull from Package Registry -The Package Registry supports the following formats: +> Introduced in GitLab 15.7 [with a flag](../../../administration/feature_flags.md) named `package_registry_access_level`. Enabled by default. -| Package type | GitLab version | Status | -| ------------ | -------------- |------- | -| [Maven](../maven_repository/index.md) | 11.3+ | GA | -| [npm](../npm_registry/index.md) | 11.7+ | GA | -| [NuGet](../nuget_repository/index.md) | 12.8+ | GA | -| [PyPI](../pypi_repository/index.md) | 12.10+ | GA | -| [Generic packages](../generic_packages/index.md) | 13.5+ | GA | -| [Composer](../composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | -| [Conan](../conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | -| [Helm](../helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | -| [Debian](../debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | -| [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | -| [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | +FLAG: +On self-managed GitLab, by default this feature is available. To disable it, +ask an administrator to [disable the feature flag](../../../administration/feature_flags.md) named `package_registry_access_level`. -[Status](../../../policy/alpha-beta-support.md): +If you want to allow anyone (everyone on the internet) to pull from the Package Registry, no matter what the project visibility is, you can use the additional toggle `Allow anyone to pull from Package Registry` that appears when the project visibility is Private or Internal. -- Alpha: behind a feature flag and not officially supported. -- Beta: several known issues that may prevent expected use. -- GA (Generally Available): ready for production use at any scale. +Several known issues exist when you allow anyone to pull from the Package Registry: -You can also use the [API](../../../api/packages.md) to administer the Package Registry. +- Project-level endpoints are supported. Group-level and instance-level endpoints are not supported. Support for group-level endpoints is proposed in [issue 383537](https://gitlab.com/gitlab-org/gitlab/-/issues/383537). +- It does not work with the [Composer](../composer_repository/index.md#install-a-composer-package), because Composer only has a group endpoint. +- It does not work with the [Debian](../debian_repository/index.md#install-a-package) repository. Support for the Debian repository is proposed in [issue 385258](https://gitlab.com/gitlab-org/gitlab/-/issues/385258). +- It does not work with the [Ruby gems](../rubygems_registry/index.md#install-a-ruby-gem) repository. Support for the Ruby gems repository is proposed in [issue 385259](https://gitlab.com/gitlab-org/gitlab/-/issues/385259). +- It will work with Conan, but using [`conan search`](../conan_repository/index.md#search-for-conan-packages-in-the-package-registry) does not work. ## Accepting contributions @@ -170,8 +163,8 @@ guides you through the process. <!-- vale gitlab.Spelling = NO --> -| Format | Status | -| ------ | ------ | +| Format | Status | +| --------- | ------------------------------------------------------------- | | Chef | [#36889](https://gitlab.com/gitlab-org/gitlab/-/issues/36889) | | CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | | Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | diff --git a/doc/user/packages/package_registry/supported_hash_types.md b/doc/user/packages/package_registry/supported_hash_types.md new file mode 100644 index 00000000000..6d7dbf87468 --- /dev/null +++ b/doc/user/packages/package_registry/supported_hash_types.md @@ -0,0 +1,25 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Supported hash types **(FREE)** + +Hash values are used to ensure you are using the correct package. You can view these values in the user interface or with the [API](../../../api/packages.md). + +The Package Registry supports the following hash types: + +| Package type | Supported hashes | +|--------------------------------------------------|----------------------------------| +| [Maven](../maven_repository/index.md) | MD5, SHA1 | +| [npm](../npm_registry/index.md) | SHA1 | +| [NuGet](../nuget_repository/index.md) | not applicable | +| [PyPI](../pypi_repository/index.md) | MD5, SHA256 | +| [Generic packages](../generic_packages/index.md) | SHA256 | +| [Composer](../composer_repository/index.md) | not applicable | +| [Conan](../conan_repository/index.md) | MD5, SHA1 | +| [Helm](../helm_repository/index.md) | not applicable | +| [Debian](../debian_repository/index.md) | MD5, SHA1, SHA256 | +| [Go](../go_proxy/index.md) | MD5, SHA1, SHA256 | +| [Ruby gems](../rubygems_registry/index.md) | MD5, SHA1, SHA256 (gemspec only) | diff --git a/doc/user/packages/package_registry/supported_package_managers.md b/doc/user/packages/package_registry/supported_package_managers.md new file mode 100644 index 00000000000..75b8c95a0fa --- /dev/null +++ b/doc/user/packages/package_registry/supported_package_managers.md @@ -0,0 +1,34 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Supported package managers **(FREE)** + +WARNING: +Not all package manager formats are ready for production use. + +The Package Registry supports the following package manager types: + +| Package type | GitLab version | Status | +| ------------------------------------------------ | -------------- | ---------------------------------------------------------- | +| [Maven](../maven_repository/index.md) | 11.3+ | GA | +| [npm](../npm_registry/index.md) | 11.7+ | GA | +| [NuGet](../nuget_repository/index.md) | 12.8+ | GA | +| [PyPI](../pypi_repository/index.md) | 12.10+ | GA | +| [Generic packages](../generic_packages/index.md) | 13.5+ | GA | +| [Composer](../composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | +| [Conan](../conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | +| [Helm](../helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | +| [Debian](../debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | +| [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | +| [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | + +[Status](../../../policy/alpha-beta-support.md): + +- Alpha: behind a feature flag and not officially supported. +- Beta: several known issues that may prevent expected use. +- GA (Generally Available): ready for production use at any scale. + +You can also use the [API](../../../api/packages.md) to administer the Package Registry. diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index f6ed9654882..0e2fc7ca7da 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -283,6 +283,32 @@ characters are removed. A `pip install` request for `my.package` looks for packages that match any of the three characters, such as `my-package`, `my_package`, and `my....package`. +## Using `requirements.txt` + +If you want pip to access your private registry, add the `--extra-index-url` parameter along with the URL for your registry to your `requirements.txt` file. + +```plaintext +--extra-index-url https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple +package-name==1.0.0 +``` + +If this is a private registry, you can authenticate in a couple of ways. For example: + +- Using your `requirements.txt` file: + +```plaintext +--extra-index-url https://__token__:<your_personal_token>@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple +package-name==1.0.0 +``` + +- Using a `~/.netrc` file: + +```plaintext +machine gitlab.example.com +login __token__ +password <your_personal_token> +``` + ## Troubleshooting To improve performance, the pip command caches files related to a package. Note that pip doesn't remove data by @@ -293,6 +319,18 @@ this command: pip cache purge ``` +### Multiple `index-url` or `extra-index-url` parameters + +You can define multiple `index-url` and `extra-index-url` parameters. + +If you use the same domain name (such as `gitlab.example.com`) multiple times with different authentication +tokens, `pip` may not be able to find your packages. This problem is due to how `pip` +[registers and stores your tokens](https://github.com/pypa/pip/pull/10904#issuecomment-1126690115) during commands executions. + +To workaround this issue, you can use a [group deploy token](../../project/deploy_tokens/index.md) with the +scope `read_package_registry` from a common parent group for all projects or groups targeted by the +`index-url` and `extra-index-url` values. + ## Supported CLI commands The GitLab PyPI repository supports the following CLI commands: diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 2b99ff807ec..9b09d846034 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -50,7 +50,7 @@ error `{"error":"404 Not Found"}`. Example request using a personal access token: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" \ +curl --fail-with-body --header "PRIVATE-TOKEN: <your_access_token>" \ --upload-file path/to/file.tgz \ "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/terraform/modules/my-module/my-system/0.0.1/file" ``` @@ -66,7 +66,7 @@ Example response: Example request using a deploy token: ```shell -curl --header "DEPLOY-TOKEN: <deploy_token>" \ +curl --fail-with-body --header "DEPLOY-TOKEN: <deploy_token>" \ --upload-file path/to/file.tgz \ "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/terraform/modules/my-module/my-system/0.0.1/file" ``` @@ -127,14 +127,14 @@ upload: script: - TERRAFORM_MODULE_NAME=$(echo "${TERRAFORM_MODULE_NAME}" | tr " _" -) # module-name must not have spaces or underscores, so translate them to hyphens - tar -vczf ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz -C ${TERRAFORM_MODULE_DIR} --exclude=./.git . - - 'curl --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" + - 'curl --fail-with-body --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file' rules: - if: $CI_COMMIT_TAG ``` -To trigger this upload job, add a Git tag to your commit. Ensure the tag follows the [Semantic Versioning Specification](https://semver.org/) that Terraform requires. The `rules:if: $CI_COMMIT_TAG` ensures that only tagged commits to your repo trigger the module upload job. +To trigger this upload job, add a Git tag to your commit. Ensure the tag follows the [Semantic Versioning Specification](https://semver.org/) that Terraform requires. The `rules:if: $CI_COMMIT_TAG` ensures that only tagged commits to your repository trigger the module upload job. For other ways to control jobs in your CI/CD pipeline, refer to the [`.gitlab-ci.yml`](../../../ci/yaml/index.md) keyword reference. ## Example projects diff --git a/doc/user/packages/workflows/build_packages.md b/doc/user/packages/workflows/build_packages.md index ec971195ea9..eab1e4392e3 100644 --- a/doc/user/packages/workflows/build_packages.md +++ b/doc/user/packages/workflows/build_packages.md @@ -302,7 +302,7 @@ The npm version is shown in the output: ``` 1. Enter responses to the questions. Ensure the **package name** follows - the [naming convention](../npm_registry/index.md#package-naming-convention) and is scoped to the project or group where the registry exists. + the [naming convention](../npm_registry/index.md#naming-convention) and is scoped to the project or group where the registry exists. ## Yarn @@ -334,7 +334,7 @@ The Yarn version is shown in the output: ``` 1. Enter responses to the questions. Ensure the **package name** follows - the [naming convention](../npm_registry/index.md#package-naming-convention) and is scoped to the + the [naming convention](../npm_registry/index.md#naming-convention) and is scoped to the project or group where the registry exists. A `package.json` file is created. diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index df4b087f6d5..371ab83a4fb 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -50,15 +50,15 @@ If you're using npm, create an `.npmrc` file. Add the appropriate URL for publis packages to your project. Finally, add a section to your `package.json` file. Follow the instructions in the -[GitLab Package Registry npm documentation](../npm_registry/index.md#authenticate-to-the-package-registry). After +[GitLab Package Registry npm documentation](../npm_registry/index.md#authentication-to-the-package-registry). After you do this, you can publish your npm package to your project using `npm publish`, as described in the -[publishing packages](../npm_registry/index.md#publish-an-npm-package) section. +[publishing packages](../npm_registry/index.md#publish-to-gitlab-package-registry) section. ### Maven If you are using Maven, you update your `pom.xml` file with distribution sections. These updates include the -appropriate URL for your project, as described in the [GitLab Maven Repository documentation](../maven_repository/index.md#project-level-maven-endpoint). -Then, you need to add a `settings.xml` file and [include your access token](../maven_repository/index.md#authenticate-with-a-personal-access-token-in-maven). +appropriate URL for your project, as described in the [GitLab Maven Repository documentation](../maven_repository/index.md#naming-convention). +Then, you need to add a `settings.xml` file and [include your access token](../maven_repository/index.md#authenticate-to-the-package-registry). Now you can [publish Maven packages](../maven_repository/index.md#publish-a-package) to your project. ### Conan diff --git a/doc/user/packages/yarn_repository/index.md b/doc/user/packages/yarn_repository/index.md new file mode 100644 index 00000000000..7e2f45019cd --- /dev/null +++ b/doc/user/packages/yarn_repository/index.md @@ -0,0 +1,248 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Publish packages with Yarn + +Publish npm packages in your project's Package Registry using Yarn. Then install the +packages whenever you need to use them as a dependency. + +Learn how to build a [yarn](../workflows/build_packages.md#yarn) package. + +You can get started with Yarn 2 by following the [Yarn documentation](https://yarnpkg.com/getting-started/install/). + +## Publish to GitLab Package Registry + +### Authentication to the Package Registry + +You need a token to publish a package. Different tokens are available depending on what you're trying to +achieve. For more information, review the [guidance on tokens](../../../user/packages/package_registry/index.md#authenticate-with-the-registry). + +- If your organization uses two-factor authentication (2FA), you must use a personal access token with the scope set to `api`. +- If you publish a package via CI/CD pipelines, you must use a CI job token. + +Create a token and save it to use later in the process. + +### Naming convention + +Depending on how you install the package, you may need to adhere to the naming convention. + +You can use one of two API endpoints to install packages: + +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have a few npm packages, and they are not in the same GitLab group. + +If you plan to install a package through the [project level](#install-from-the-project-level), you do not have to +adhere to the naming convention. + +If you plan to install a package through the [instance level](#install-from-the-instance-level), then you must name +your package with a [scope](https://docs.npmjs.com/misc/scope/). Scoped packages begin with a `@` and have the +`@owner/package-name` format. You can set up the scope for your package in the `.yarnrc.yml` file and by using the +`publishConfig` option in the `package.json`. + +- The value used for the `@scope` is the root of the project that hosts the packages and not the root + of the project with the package's source code. The scope should be lowercase. +- The package name can be anything you want + +| Project URL | Package Registry in | Scope | Full package name | +| ------------------------------------------------------- | ------------------- | --------- | ---------------------- | +| `https://gitlab.com/my-org/engineering-group/analytics` | Analytics | `@my-org` | `@my-org/package-name` | + +### Configuring `.yarnrc.yml` to publish from the project level + +To publish with the project-level npm endpoint, set the following configuration in +`.yarnrc.yml`: + +```yaml +npmScopes: + foo: + npmRegistryServer: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' + npmPublishRegistry: 'https://<your_domain>/api/v4/projects/<your_project_id>/packages/npm/' + +npmRegistries: + //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/: + npmAlwaysAuth: true + npmAuthToken: '<your_token>' +``` + +In this configuration: + +- Replace `<your_domain>` with your domain name. +- Replace `<your_project_id>` with your project's ID, which you can find on the project's home page. +- Replace `<your_token>` with a deploy token, group access token, project access token, or personal access token. + +### Configuring `.yarnrc.yml` to publish from the instance level + +For the instance-level npm endpoint, use this Yarn 2 configuration in `.yarnrc.yml`: + +```yaml +npmScopes: + <scope>: + npmRegistryServer: 'https://<your_domain>/api/v4/packages/npm/' + +npmRegistries: + //gitlab.example.com/api/v4/packages/npm/: + npmAlwaysAuth: true + npmAuthToken: '<your_token>' +``` + +In this configuration: + +- Replace `<your_domain>` with your domain name. +- Your scope is `<scope>`, without `@`. +- Replace `<your_token>` with a deploy token, group access token, project access token, or personal access token. + +### Publishing a package via the command line + +Publish a package: + +```shell +npm publish +``` + +Your package should now publish to the Package Registry. + +### Publishing via a CI/CD pipeline + +In the GitLab project that houses your `yarnrc.yml`, edit or create a `.gitlab-ci.yml` file. For example: + +```yaml +image: node:latest + +stages: + - deploy + +deploy: + stage: deploy + script: + - npm publish +``` + +Your package should now publish to the Package Registry when the pipeline runs. + +## Install a package + +If multiple packages have the same name and version, the most recently-published package is retrieved when you install a package. + +You can install a package from a GitLab project or instance: + +- **Instance-level**: Use when you have many npm packages in different GitLab groups or in their own namespace. +- **Project-level**: Use when you have a few npm packages, and they are not in the same GitLab group. + +### Install from the instance level + +WARNING: +You must use packages published with the scoped [naming convention](#naming-convention) when you install a package from the instance level. + +1. Authenticate to the Package Registry + + If you install a package from a private project, you must authenticate to the Package Registry. Skip this step if the project is not private. + + ```shell + npm config set -- //your_domain_name/api/v4/packages/npm/:_authToken=your_token + ``` + + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. + +1. Set the registry + + ```shell + npm config set @scope:registry https://your_domain_name.com/api/v4/packages/npm/ + ``` + + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. + +1. Install the package + + ```shell + yarn add @scope/my-package + ``` + +### Install from the project level + +1. Authenticate to the Package Registry + + If you install a package from a private project, you must authenticate to the Package Registry. Skip this step if the project is not private. + + ```shell + npm config set -- //your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken=your_token + ``` + + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. + - Replace `your_token` with a deploy token, group access token, project access token, or personal access token. + +1. Set the registry + + ```shell + npm config set @scope:registry=https://your_domain_name/api/v4/projects/your_project_id/packages/npm/ + ``` + + - Replace `@scope` with the [root level group](#naming-convention) of the project you're installing to the package from. + - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. + - Replace `your_project_id` is your project ID, found on the project's home page. + +1. Install the package + + ```shell + yarn add @scope/my-package + ``` + +## Helpful hints + +For full helpful hints information, refer to the [npm documentation](../npm_registry/index.md#helpful-hints). + +### Supported CLI commands + +The GitLab npm repository supports the following commands for the npm CLI (`npm`) and yarn CLI +(`yarn`): + +- `npm install`: Install npm packages. +- `npm publish`: Publish an npm package to the registry. +- `npm dist-tag add`: Add a dist-tag to an npm package. +- `npm dist-tag ls`: List dist-tags for a package. +- `npm dist-tag rm`: Delete a dist-tag. +- `npm ci`: Install npm packages directly from your `package-lock.json` file. +- `npm view`: Show package metadata. +- `yarn add`: Install an npm package. +- `yarn update`: Update your dependencies. + +## Troubleshooting + +For full troubleshooting information, refer to the [npm documentation](../npm_registry/index.md#troubleshooting). + +### Error running Yarn with the Package Registry for the npm registry + +If you are using [Yarn](https://classic.yarnpkg.com/en/) with the npm registry, you may get +an error message like: + +```shell +yarn install v1.15.2 +warning package.json: No license field +info No lockfile found. +warning XXX: No license field +[1/4] 🔍 Resolving packages... +[2/4] 🚚 Fetching packages... +error An unexpected error occurred: "https://gitlab.example.com/api/v4/projects/XXX/packages/npm/XXX/XXX/-/XXX/XXX-X.X.X.tgz: Request failed \"404 Not Found\"". +info If you think this is a bug, please open a bug report with the information provided in "/Users/XXX/gitlab-migration/module-util/yarn-error.log". +info Visit https://classic.yarnpkg.com/en/docs/cli/install for documentation about this command +``` + +In this case, try adding this to your `.npmrc` file (and replace `<your_token>` +with your personal access token or deploy token): + +```plaintext +//gitlab.example.com/api/v4/projects/:_authToken=<your_token> +``` + +You can also use `yarn config` instead of `npm config` when setting your auth-token dynamically: + +```shell +yarn config set '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" +yarn config set '//gitlab.example.com/api/v4/packages/npm/:_authToken' "<your_token>" +``` |
