diff options
Diffstat (limited to 'doc/user/packages')
21 files changed, 180 insertions, 76 deletions
diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 49c54ec191e..b990cf1f09b 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -308,7 +308,7 @@ used to access them: ### Caching -To improve performance, Composer caches files related to a package. Note that Composer doesn't remove data by +To improve performance, Composer caches files related to a package. Composer doesn't remove data by itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with this command: diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index dd6605c2f01..05daa525893 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -121,7 +121,9 @@ To authenticate to the Package Registry, you need one of the following: NOTE: Packages from private and internal projects are hidden if you are not -authenticated. If you try to search or download a package from a private or internal project without authenticating, you will receive the error `unable to find the package in remote` in the Conan client. +authenticated. If you try to search or download a package from a private or internal +project without authenticating, you receive the error `unable to find the package in remote` +in the Conan client. ### Add your credentials to the GitLab remote @@ -271,7 +273,7 @@ Prerequisites: NOTE: If you try installing the package you created in this tutorial, the install command -will have no effect because the package already exists. +has no effect because the package already exists. Delete `~/.conan/data` to clean up the packages stored in the cache. ## Remove a Conan package diff --git a/doc/user/packages/container_registry/authenticate_with_container_registry.md b/doc/user/packages/container_registry/authenticate_with_container_registry.md index cdc7cbe947b..f48ba7bbf52 100644 --- a/doc/user/packages/container_registry/authenticate_with_container_registry.md +++ b/doc/user/packages/container_registry/authenticate_with_container_registry.md @@ -6,6 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Authenticate with the Container Registry **(FREE)** +<!--- start_remove The following content will be removed on remove_date: '2023-11-22' --> +WARNING: +[External authorization](../../admin_area/settings/external_authorization.md) will be enabled by default in GitLab 16.0. External authorization prevents personal access tokens and deploy tokens from accessing container and package registries and affects all users who use these tokens to access the registries. You can disable external authorization if you want to use personal access tokens and deploy tokens with the container or package registries. +<!--- end_remove --> + To authenticate with the Container Registry, you can use a: - [Personal access token](../../profile/personal_access_tokens.md). diff --git a/doc/user/packages/container_registry/build_and_push_images.md b/doc/user/packages/container_registry/build_and_push_images.md index bbb82300488..aa86b87660b 100644 --- a/doc/user/packages/container_registry/build_and_push_images.md +++ b/doc/user/packages/container_registry/build_and_push_images.md @@ -144,7 +144,7 @@ In this example, `$CI_REGISTRY_IMAGE` resolves to the address of the registry ti to this project. `$CI_COMMIT_REF_NAME` resolves to the branch or tag name, which can contain forward slashes. Image tags can't contain forward slashes. Use `$CI_COMMIT_REF_SLUG` as the image tag. You can declare the variable, `$IMAGE_TAG`, -combining `$CI_REGISTRY_IMAGE` and `$CI_REGISTRY_IMAGE` to save some typing in the +combining `$CI_REGISTRY_IMAGE` and `$CI_COMMIT_REF_NAME` to save some typing in the `script` section. This example splits the tasks into 4 pipeline stages, including two tests that run in parallel. The `build` is stored in the container diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index c3790c252cc..4d7b25e2d77 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -72,7 +72,7 @@ NOTE: You must [authenticate with the container registry](authenticate_with_container_registry.md) to download container images from a private repository. -For more information on running container images, visit the [Docker documentation](https://docs.docker.com/get-started/). +For more information on running container images, see the [Docker documentation](https://docs.docker.com/get-started/). ## Naming convention for your container images @@ -165,3 +165,9 @@ this setting. However, disabling the Container Registry disables all Container R | Private project with Container Registry visibility <br/> set to **Everyone With Access** (UI) or `enabled` (API) | View Container Registry <br/> and pull images | No | No | Yes | | Private project with Container Registry visibility <br/> set to **Only Project Members** (UI) or `private` (API) | View Container Registry <br/> and pull images | No | No | Yes | | Any project with Container Registry `disabled` | All operations on Container Registry | No | No | No | + +## Supported image types + +The Container Registry supports [Docker V2](https://docs.docker.com/registry/spec/manifest-v2-2/) and [Open Container Initiative (OCI)](https://github.com/opencontainers/image-spec/blob/main/spec.md) image formats. + +OCI support means that you can host OCI-based image formats in the registry, such as [Helm 3+ chart packages](https://helm.sh/docs/topics/registries/). There is no distinction between image formats in the GitLab [API](../../../api/container_registry.md) and the UI. [Issue 38047](https://gitlab.com/gitlab-org/gitlab/-/issues/38047) addresses this distinction, starting with Helm. diff --git a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md index 0ce9635e05a..110f3ff908c 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md +++ b/doc/user/packages/container_registry/reduce_container_registry_data_transfer.md @@ -22,7 +22,7 @@ usage. Use these tools and techniques to determine your image's size: - [Skopeo](https://github.com/containers/skopeo): - use Skopeo's `inspect` command to examine layer count and sizes through API calls. You can + use the Skopeo `inspect` command to examine layer count and sizes through API calls. You can therefore inspect this data prior to running `docker pull IMAGE`. - Docker in CI: examine and record the image size when using GitLab CI prior to pushing an image @@ -41,7 +41,7 @@ Use these tools and techniques to determine your image's size: ### Use a smaller base image Consider using a smaller base image, such as [Alpine Linux](https://alpinelinux.org/). -An Alpine image is around 5MB, which is several times smaller than popular base images such as +An Alpine image is around 5 MB, which is several times smaller than popular base images such as [Debian](https://hub.docker.com/_/debian). If your application is distributed as a self-contained static binary, such as for Go applications, you can also consider using the Docker [scratch](https://hub.docker.com/_/scratch/) diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 15948569cb8..9229ea61821 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -320,7 +320,7 @@ the tags. To create the list and delete the tags: the tags' names are written to the `list_o_tags.out` file: ```shell - # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination) + # Get a list of all tags in a certain container repository while considering [pagination](../../../api/rest/index.md#pagination) echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done ``` diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index 7db2a341743..7ec20e3d036 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -15,9 +15,6 @@ The Debian package registry for GitLab is under development and isn't ready for limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6057) details the remaining work and timelines to make it production ready. -NOTE: -The Debian registry is not FIPS compliant and is disabled when [FIPS mode](../../../development/fips_compliance.md) is enabled. - Publish Debian packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -65,32 +62,52 @@ Feature.disable(:debian_group_packages) Creating a Debian package is documented [on the Debian Wiki](https://wiki.debian.org/Packaging). -## Authenticate to the Package Registry +## Authenticate to the Debian endpoints + +Authentication methods differs between [distributions APIs](#authenticate-to-the-debian-distributions-apis) +and [package repositories](#authenticate-to-the-debian-package-repositories). -To create a distribution, publish a package, or install a private package, you need one of the -following: +### Authenticate to the Debian distributions APIs -- [Personal access token](../../../api/index.md#personalprojectgroup-access-tokens) +To create, read, update, or delete a distribution, you need one of the following: + +- [Personal access token](../../../api/rest/index.md#personalprojectgroup-access-tokens), + using `--header "PRIVATE-TOKEN: <personal_access_token>"` +- [Deploy token](../../project/deploy_tokens/index.md) + using `--header "Deploy-Token: <deploy_token>"` - [CI/CD job token](../../../ci/jobs/ci_job_token.md) + using `--header "Job-Token: <job_token>"` + +### Authenticate to the Debian Package Repositories + +To publish a package, or install a private package, you need to use basic authentication, +with one of the following: + +- [Personal access token](../../../api/rest/index.md#personalprojectgroup-access-tokens), + using `<username>:<personal_access_token>` - [Deploy token](../../project/deploy_tokens/index.md) + using `<deploy_token_name>:<deploy_token>` +- [CI/CD job token](../../../ci/jobs/ci_job_token.md) + using `gitlab-ci-token:<job_token>` ## Create a Distribution On the project-level, Debian packages are published using *Debian Distributions*. To publish packages on the group level, create a distribution with the same `codename`. -To create a project-level distribution: +To create a project-level distribution using a personal access token: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions?codename=<codename>" +curl --request POST --header "PRIVATE-TOKEN: <personal_access_token>" \ + "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions?codename=<codename>" ``` -Example response with `codename=unstable`: +Example response with `codename=sid`: ```json { "id": 1, - "codename": "unstable", + "codename": "sid", "suite": null, "origin": null, "label": null, @@ -123,33 +140,50 @@ Once built, several files are created: - `.buildinfo` file: Used for Reproducible builds (optional) - `.changes` file: Upload metadata, and list of uploaded files (all the above) -To upload these files, you can use `dput-ng >= 1.32` (Debian bullseye): +To upload these files, you can use `dput-ng >= 1.32` (Debian bullseye). +`<username>` and `<password>` are defined +[as above](#authenticate-to-the-debian-package-repositories): ```shell cat <<EOF > dput.cf [gitlab] method = https -fqdn = <username>:<your_access_token>@gitlab.example.com +fqdn = <username>:<password>@gitlab.example.com incoming = /api/v4/projects/<project_id>/packages/debian EOF dput --config=dput.cf --unchecked --no-upload-log gitlab <your_package>.changes ``` +## Directly upload a package + +> Direct upload [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101838) in GitLab 15.9. + +When you don't have access to `.changes` file, you can directly upload a `.deb` by passing +distribution `codename` and target `component` as parameters with +your [credentials](#authenticate-to-the-debian-package-repositories). +For example, to upload to component `main` of distribution `sid` using a personal access token: + +```shell +curl --request PUT --user "<username>:<personal_access_token>" \ + "https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian/?distribution=sid&component=main" \ + --upload-file /path/to/your.deb +``` + ## Install a package To install a package: 1. Configure the repository: - If you are using a private project, add your [credentials](#authenticate-to-the-package-registry) to your apt configuration: + If you are using a private project, add your [credentials](#authenticate-to-the-debian-package-repositories) to your apt configuration: ```shell - echo 'machine gitlab.example.com login <username> password <your_access_token>' \ + echo 'machine gitlab.example.com login <username> password <password>' \ | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf ``` - Download your distribution key: + Download your distribution key using your [credentials](#authenticate-to-the-debian-distributions-apis): ```shell sudo mkdir -p /usr/local/share/keyrings @@ -182,14 +216,14 @@ To download a source package: 1. Configure the repository: - If you are using a private project, add your [credentials](#authenticate-to-the-package-registry) to your apt configuration: + If you are using a private project, add your [credentials](#authenticate-to-the-debian-package-repositories) to your apt configuration: ```shell - echo 'machine gitlab.example.com login <username> password <your_access_token>' \ + echo 'machine gitlab.example.com login <username> password <password>' \ | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf ``` - Download your distribution key: + Download your distribution key using your [credentials](#authenticate-to-the-debian-distributions-apis): ```shell sudo mkdir -p /usr/local/share/keyrings diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index a1a9d2a4915..8dc3c98795b 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -322,7 +322,7 @@ services: ### Issues when authenticating to the Dependency Proxy from CI/CD jobs -GitLab Runner will automatically authenticate to the Dependency Proxy. However, the underlying Docker engine is still subject to its [authorization resolving process](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/docs/configuration/advanced-configuration.md#precedence-of-docker-authorization-resolving). +GitLab Runner authenticates automatically to the Dependency Proxy. However, the underlying Docker engine is still subject to its [authorization resolving process](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/docs/configuration/advanced-configuration.md#precedence-of-docker-authorization-resolving). Misconfigurations in the authentication mechanism may cause `HTTP Basic: Access denied` and `403: Access forbidden` errors. diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 9b49f946984..932de0bcde6 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -13,13 +13,13 @@ Publish generic files, like release binaries, in your project's Package Registry ## Authenticate to the Package Registry -To authenticate to the Package Registry, you need either a [personal access token](../../../api/index.md#personalprojectgroup-access-tokens), +To authenticate to the Package Registry, you need either a [personal access token](../../../api/rest/index.md#personalprojectgroup-access-tokens), [CI/CD job token](../../../ci/jobs/ci_job_token.md), or [deploy token](../../project/deploy_tokens/index.md). In addition to the standard API authentication mechanisms, the generic package API allows authentication with HTTP Basic authentication for use with tools that do not support the other available mechanisms. The `user-id` is not checked and -may be any value, and the `password` must be either a [personal access token](../../../api/index.md#personalprojectgroup-access-tokens), +may be any value, and the `password` must be either a [personal access token](../../../api/rest/index.md#personalprojectgroup-access-tokens), a [CI/CD job token](../../../ci/jobs/ci_job_token.md), or a [deploy token](../../project/deploy_tokens/index.md). ## Publish a package file @@ -28,7 +28,7 @@ When you publish a package file, if the package does not exist, it is created. Prerequisites: -- You must [authenticate with the API](../../../api/index.md#authentication). +- You must [authenticate with the API](../../../api/rest/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. If authenticating with a personal access token or project access token, it must be configured with the `api` scope. @@ -44,7 +44,7 @@ PUT /projects/:id/packages/generic/:package_name/:package_version/:file_name?sta | Attribute | Type | Required | Description | | -------------------| --------------- | ---------| -------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/index.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/rest/index.md#namespaced-path-encoding). | | `package_name` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). | `package_version` | string | yes | The package version. The following regex validates this: `\A(\.?[\w\+-]+\.?)+\z`. You can test your version strings on [Rubular](https://rubular.com/r/aNCV0wG5K14uq8). | `file_name` | string | yes | The filename. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). @@ -139,7 +139,7 @@ If multiple packages have the same name, version, and filename, then the most re Prerequisites: -- You need to [authenticate with the API](../../../api/index.md#authentication). If authenticating with a deploy token, it must be configured with the `read_package_registry` and/or `write_package_registry` scope. +- You need to [authenticate with the API](../../../api/rest/index.md#authentication). If authenticating with a deploy token, it must be configured with the `read_package_registry` and/or `write_package_registry` scope. ```plaintext GET /projects/:id/packages/generic/:package_name/:package_version/:file_name @@ -147,7 +147,7 @@ GET /projects/:id/packages/generic/:package_name/:package_version/:file_name | Attribute | Type | Required | Description | | -------------------| --------------- | ---------| ------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/index.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/rest/index.md#namespaced-path-encoding). | | `package_name` | string | yes | The package name. | | `package_version` | string | yes | The package version. | | `file_name` | string | yes | The filename. | @@ -215,7 +215,7 @@ It also demonstrates how to manage a semantic version for the generic package: s ### Internal Server error on large file uploads to S3 -S3-compatible object storage [limits the size of a single PUT request to 5GB](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). If the `aws_signature_version` is set to `2` in the [object storage connection settings](../../../administration/object_storage.md), attempting to publish a package file larger than the 5GB limit can result in a `HTTP 500: Internal Server Error` response. +S3-compatible object storage [limits the size of a single PUT request to 5 GB](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). If the `aws_signature_version` is set to `2` in the [object storage connection settings](../../../administration/object_storage.md), attempting to publish a package file larger than the 5 GB limit can result in a `HTTP 500: Internal Server Error` response. If you are receiving `HTTP 500: Internal Server Error` responses when publishing large files to S3, set the `aws_signature_version` to `4`: diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index a147c3656b7..1a089cd82be 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -107,8 +107,8 @@ Open your [`~/.netrc`](https://everything.curl.dev/usingcurl/netrc) file and add the following text. Replace the variables in `< >` with your values. WARNING: -If you use an environment variable called `NETRC`, Go will use its value -as a filename and ignore `~/.netrc`. If you intend to use `~/.netrc` in +If you use an environment variable called `NETRC`, Go uses its value +as a filename and ignores `~/.netrc`. If you intend to use `~/.netrc` in the GitLab CI **do not use `NETRC` as an environment variable name**. ```plaintext diff --git a/doc/user/packages/harbor_container_registry/index.md b/doc/user/packages/harbor_container_registry/index.md index f159522d77d..1ad5e2c2f05 100644 --- a/doc/user/packages/harbor_container_registry/index.md +++ b/doc/user/packages/harbor_container_registry/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Harbor Registry **(FREE)** -You can integrate the [Harbor container registry](../../../user/project/integrations/harbor.md#harbor-container-registry-integration) into GitLab and use Harbor as the container registry for your GitLab project to store images. +You can integrate the [Harbor container registry](../../../user/project/integrations/harbor.md) into GitLab and use Harbor as the container registry for your GitLab project to store images. ## View the Harbor Registry diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index 785ef344c8e..58c8e17f48b 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -30,7 +30,7 @@ Read more in the Helm documentation about these topics: To authenticate to the Helm repository, you need either: -- A [personal access token](../../../api/index.md#personalprojectgroup-access-tokens) with the scope set to `api`. +- A [personal access token](../../../api/rest/index.md#personalprojectgroup-access-tokens) with the scope set to `api`. - A [deploy token](../../project/deploy_tokens/index.md) with the scope set to `read_package_registry`, `write_package_registry`, or both. - A [CI/CD job token](../../../ci/jobs/ci_job_token.md). @@ -54,7 +54,7 @@ Once built, a chart can be uploaded to the desired channel with `curl` or `helm - `<username>`: the GitLab username or the deploy token username. - `<access_token>`: the personal access token or the deploy token. - `<project_id>`: the project ID (like `42`) or the - [URL-encoded](../../../api/index.md#namespaced-path-encoding) path of the project (like `group%2Fproject`). + [URL-encoded](../../../api/rest/index.md#namespaced-path-encoding) path of the project (like `group%2Fproject`). - `<channel>`: the name of the channel (like `stable`). - With the [`helm cm-push`](https://github.com/chartmuseum/helm-push/#readme) plugin: diff --git a/doc/user/packages/infrastructure_registry/index.md b/doc/user/packages/infrastructure_registry/index.md index ac31b491a0e..d06c5e7fb1e 100644 --- a/doc/user/packages/infrastructure_registry/index.md +++ b/doc/user/packages/infrastructure_registry/index.md @@ -34,9 +34,8 @@ authenticate with the [`CI_JOB_TOKEN` predefined variable](../../../ci/variables CI/CD templates, which you can use to get started, are in [this repository](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). -Learn more about using CI/CD to build: - -- [Terraform modules](../terraform_module_registry/index.md#publish-a-terraform-module-by-using-cicd) +For more information about using CI/CD to build Terraform modules, +see [Publish a Terraform module by using CI/CD](../terraform_module_registry/index.md#publish-a-terraform-module-by-using-cicd). If you use CI/CD to build a package, you can find extended activity information when you view the package details: diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 2d1efd024a0..1899cdc213f 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -232,7 +232,7 @@ to [Maven Central](https://search.maven.org/). When the feature flag is enabled, administrators can disable this behavior in the [Continuous Integration settings](../../admin_area/settings/continuous_integration.md). -There are many ways to configure your Maven project so that it will request packages +There are many ways to configure your Maven project so that it requests packages in Maven Central from GitLab. Maven repositories are queried in a [specific order](https://maven.apache.org/guides/mini/guide-multiple-repositories.html#repository-order). By default, maven-central is usually checked first through the diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index c62999100c1..11e3d0e5131 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -25,7 +25,7 @@ Create a token and save it to use later in the process. ### Naming convention -Depending on how the package will be installed, you may need to adhere to the naming convention. +Depending on how the package is installed, you may need to adhere to the naming convention. You can use one of two API endpoints to install packages: @@ -36,7 +36,7 @@ If you plan to install a package through the [project level](#install-from-the-p If you plan to install a package through the [instance level](#install-from-the-instance-level), then you must name your package with a [scope](https://docs.npmjs.com/misc/scope/). Scoped packages begin with a `@` have the format of `@owner/package-name`. You can set up the scope for your package in the `.npmrc` file and by using the `publishConfig` option in the `package.json`. -- The value used for the `@scope` is the root of the project that will host the packages and not the root +- The value used for the `@scope` is the root of the project that is hosting the packages and not the root of the project with the source code of the package itself. The scope should be lowercase. - The package name can be anything you want @@ -64,7 +64,7 @@ Create or edit the `.npmrc` file in the same directory as your `package.json`. I - Replace `@scope` with the [root level group](#naming-convention) of the project you're publishing to the package to. - Replace `your_domain_name` with your domain name, for example, `gitlab.com`. - Replace `your_project_id` is your project ID, found on the project's home page. -- `"${NPM_TOKEN}"` will be associated with the token you created later in the process. +- `"${NPM_TOKEN}"` is associated with the token you created later in the process. WARNING: Never hardcode GitLab tokens (or any tokens) directly in `.npmrc` files or any other files that can @@ -125,11 +125,11 @@ You can install a package from a GitLab project or instance: ### Install from the instance level WARNING: -In order to install a package from the instance level, the package must have been published following the scoped [naming convention](#naming-convention). +To install a package from the instance level, the package must have been published following the scoped [naming convention](#naming-convention). 1. Authenticate to the Package Registry - If you would like to install a package from a private project, you will need to authenticate to the Package Registry. Skip this step if the project is not private. + If you would like to install a package from a private project, you would have to authenticate to the Package Registry. Skip this step if the project is not private. ```shell npm config set -- //your_domain_name/api/v4/packages/npm/:_authToken=your_token @@ -158,7 +158,7 @@ In order to install a package from the instance level, the package must have bee 1. Authenticate to the Package Registry - If you would like to install a package from a private project, you will need to authenticate to the Package Registry. Skip this step if the project is not private. + If you would like to install a package from a private project, you would have to authenticate to the Package Registry. Skip this step if the project is not private. ```shell npm config set -- //your_domain_name/api/v4/projects/your_project_id/packages/npm/:_authToken=your_token @@ -264,6 +264,22 @@ The GitLab npm repository supports the following commands for the npm CLI (`npm` ## Troubleshooting +### `404 Not Found` errors are happening on `npm install` or `yarn` + +Using `CI_JOB_TOKEN` to install npm packages with dependencies in another project gives you 404 Not Found errors. A fix for this problem is proposed in [issue 352962](https://gitlab.com/gitlab-org/gitlab/-/issues/352962). + +As a workaround, you can: + +1. Create a [personal access token](../../profile/personal_access_tokens.md). +1. Authenticate at both the instance level and project level for each package: + + ```ini + @foo:registry=https://gitlab.example.com/api/v4/packages/npm/ + //gitlab.example.com/api/v4/packages/npm/:_authToken=${MY_TOKEN} + //gitlab.example.com/api/v4/projects/<your_project_id_a>/packages/npm/:_authToken=${MY_TOKEN} + //gitlab.example.com/api/v4/projects/<your_project_id_b>/packages/npm/:_authToken=${MY_TOKEN} + ``` + ### `npm publish` targets default npm registry (`registry.npmjs.org`) Ensure that your package scope is set consistently in your `package.json` and `.npmrc` files. @@ -317,7 +333,7 @@ Make sure that your `package.json` file does not exceed `20,000` characters. ### `npm publish` returns `npm ERR! 500 Internal Server Error - PUT` -This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab 13.3.x and later. The error in the logs will appear as: +This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab 13.3.x and later. The error in the logs appears as: ```plaintext >NoMethodError - undefined method `preferred_language' for #<Rack::Response diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 540db463f0a..6710c147c87 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -283,7 +283,7 @@ To use the [group-level](#use-the-gitlab-endpoint-for-nuget-packages) Package Re Prerequisite: -- Set up the [source](#https://docs.gitlab.com/ee/user/packages/nuget_repository/#add-the-package-registry-as-a-source-for-nuget-packages) with a [project-level endpoint](#use-the-gitlab-endpoint-for-nuget-packages). +- Set up the [source](#add-the-package-registry-as-a-source-for-nuget-packages) with a [project-level endpoint](#use-the-gitlab-endpoint-for-nuget-packages). When publishing packages: @@ -461,9 +461,15 @@ for more details on what other GitLab CI patterns are demonstrated. ## Troubleshooting +### Clear NuGet cache + To improve performance, NuGet caches files related to a package. If you encounter issues, clear the cache with this command: ```shell nuget locals all -clear ``` + +### `Error publishing` or `Invalid Package: Failed metadata extraction error` messages when trying to publish NuGet packages in a Docker-based GitLab installation + +Webhook requests to local network addresses are blocked to prevent the exploitation of internal web services. If you get `Error publishing` or `Invalid Package` messages when you try to publish NuGet packages, change your network settings to [allow webhook and service requests to the local network](../../../security/webhooks.md#allow-webhook-and-service-requests-to-local-network). diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index ab5d652b731..cd60a229479 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -43,6 +43,11 @@ For information on how to create and upload a package, view the GitLab documenta ## Authenticate with the registry +<!--- start_remove The following content will be removed on remove_date: '2023-11-22' --> +WARNING: +[External authorization](../../admin_area/settings/external_authorization.md) will be enabled by default in GitLab 16.0. External authorization prevents personal access tokens and deploy tokens from accessing container and package registries and affects all users who use these tokens to access the registries. You can disable external authorization if you want to use personal access tokens and deploy tokens with the container or package registries. +<!--- end_remove --> + Authentication depends on the package manager being used. For more information, see the docs on the specific package format you want to use. @@ -59,12 +64,11 @@ For most package types, the following credential types are valid: - [Job token](../../../ci/jobs/ci_job_token.md): allows access to packages in the project running the job for the users running the pipeline. Access to other external projects can be configured. - - If your organization uses two factor authentication (2FA), you must use a personal access token with the scope set to `api`. - If you are publishing a package via CI/CD pipelines, you must use a CI job token. NOTE: -If you have not activated the "Packages" feature for your project at **Settings > General > Project features**, you will receive a 403 Forbidden response. +If you have not activated the "Package registry" feature for your project at **Settings > General > Visibility, project features, permissions**, you receive a 403 Forbidden response. Accessing package registry via deploy token is not available when external authorization is enabled. ## Use GitLab CI/CD to build packages @@ -75,7 +79,7 @@ authenticate with GitLab by using the `CI_JOB_TOKEN`. CI/CD templates, which you can use to get started, are in [this repository](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). -Learn more about using the GitLab Package Registry with CI/CD: +For more information about using the GitLab Package Registry with CI/CD, see: - [Composer](../composer_repository/index.md#publish-a-composer-package-by-using-cicd) - [Conan](../conan_repository/index.md#publish-a-conan-package-by-using-cicd) @@ -151,7 +155,7 @@ Several known issues exist when you allow anyone to pull from the Package Regist - Project-level endpoints are supported. Group-level and instance-level endpoints are not supported. Support for group-level endpoints is proposed in [issue 383537](https://gitlab.com/gitlab-org/gitlab/-/issues/383537). - It does not work with the [Composer](../composer_repository/index.md#install-a-composer-package), because Composer only has a group endpoint. -- It will work with Conan, but using [`conan search`](../conan_repository/index.md#search-for-conan-packages-in-the-package-registry) does not work. +- It works with Conan, but using [`conan search`](../conan_repository/index.md#search-for-conan-packages-in-the-package-registry) does not work. ## Accepting contributions diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 0e2fc7ca7da..e5b7f06f6ca 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -50,7 +50,7 @@ password = <your_personal_access_token> ``` The `<project_id>` is either the project's -[URL-encoded](../../../api/index.md#namespaced-path-encoding) +[URL-encoded](../../../api/rest/index.md#namespaced-path-encoding) path (for example, `group%2Fproject`), or the project's ID (for example `42`). ### Authenticate with a deploy token @@ -69,7 +69,7 @@ password = <deploy token> ``` The `<project_id>` is either the project's -[URL-encoded](../../../api/index.md#namespaced-path-encoding) +[URL-encoded](../../../api/rest/index.md#namespaced-path-encoding) path (for example, `group%2Fproject`), or the project's ID (for example `42`). ### Authenticate with a CI job token @@ -220,7 +220,7 @@ pip install --index-url https://<personal_access_token_name>:<personal_access_to - `<package_name>` is the package name. - `<personal_access_token_name>` is a personal access token name with the `read_api` scope. - `<personal_access_token>` is a personal access token with the `read_api` scope. -- `<project_id>` is either the project's [URL-encoded](../../../api/index.md#namespaced-path-encoding) +- `<project_id>` is either the project's [URL-encoded](../../../api/rest/index.md#namespaced-path-encoding) path (for example, `group%2Fproject`), or the project's ID (for example `42`). In these commands, you can use `--extra-index-url` instead of `--index-url`. However, using @@ -311,7 +311,7 @@ password <your_personal_token> ## Troubleshooting -To improve performance, the pip command caches files related to a package. Note that pip doesn't remove data by +To improve performance, the pip command caches files related to a package. Pip doesn't remove data by itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with this command: @@ -324,7 +324,7 @@ pip cache purge You can define multiple `index-url` and `extra-index-url` parameters. If you use the same domain name (such as `gitlab.example.com`) multiple times with different authentication -tokens, `pip` may not be able to find your packages. This problem is due to how `pip` +tokens, `pip` may not be able to find your packages. This problem is due to how `pip` [registers and stores your tokens](https://github.com/pypa/pip/pull/10904#issuecomment-1126690115) during commands executions. To workaround this issue, you can use a [group deploy token](../../project/deploy_tokens/index.md) with the diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md index 7710ad3db01..ae444880b6b 100644 --- a/doc/user/packages/rubygems_registry/index.md +++ b/doc/user/packages/rubygems_registry/index.md @@ -117,7 +117,7 @@ To push your gem, run a command like this one: gem push my_gem-0.0.1.gem --host <host> ``` -Note that `<host>` is the URL you used when setting up authentication. For example: +`<host>` is the URL you used when setting up authentication. For example: ```shell gem push my_gem-0.0.1.gem --host https://gitlab.example.com/api/v4/projects/1/packages/rubygems diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 9b09d846034..66a42f398b0 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -15,7 +15,7 @@ as a Terraform module registry. To authenticate to the Terraform module registry, you need either: -- A [personal access token](../../../api/index.md#personalprojectgroup-access-tokens) with at least `read_api` rights. +- A [personal access token](../../../api/rest/index.md#personalprojectgroup-access-tokens) with at least `read_api` rights. - A [CI/CD job token](../../../ci/jobs/ci_job_token.md). ## Publish a Terraform Module @@ -26,7 +26,7 @@ Prerequisites: - The package name and version [must be unique in the top-level namespace](../infrastructure_registry/index.md#how-module-resolution-works). - Your project and group names must not include a dot (`.`). For example, `source = "gitlab.example.com/my.group/project.name"`. -- You must [authenticate with the API](../../../api/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. +- You must [authenticate with the API](../../../api/rest/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. - The name of a module [must be unique within the scope of its group](../infrastructure_registry/index.md#how-module-resolution-works), otherwise an [error occurs](#troubleshooting). @@ -36,9 +36,9 @@ PUT /projects/:id/packages/terraform/modules/:module-name/:module-system/:module | Attribute | Type | Required | Description | | -------------------| --------------- | ---------| -------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/index.md#namespaced-path-encoding). | -| `module-name` | string | yes | The package name. **Supported syntax**: One to 64 ASCII characters, including lowercase letters (a-z), digits (0-9), and hyphens (`-`). -| `module-system` | string | yes | The package system. **Supported syntax**: One to 64 ASCII characters, including lowercase letters (a-z), digits (0-9), and hyphens (`-`). More information can be found in the [Terraform Module Registry Protocol documentation](https://www.terraform.io/internals/module-registry-protocol). +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/rest/index.md#namespaced-path-encoding). | +| `module-name` | string | yes | The package name. **Supported syntax**: One to 64 ASCII characters, including lowercase letters (a-z) and digits (0-9). The package name can't exceed 64 characters. +| `module-system` | string | yes | The package system. **Supported syntax**: One to 64 ASCII characters, including lowercase letters (a-z) and digits (0-9). The package system can't exceed 64 characters. More information can be found in the [Terraform Module Registry Protocol documentation](https://www.terraform.io/internals/module-registry-protocol). | `module-version` | string | yes | The package version. It must be valid according to the [Semantic Versioning Specification](https://semver.org/). Provide the file content in the request body. @@ -83,7 +83,7 @@ Example response: Prerequisites: -- You need to [authenticate with the API](../../../api/index.md#authentication). If authenticating with a personal access token, it must be configured with the `read_api` scope. +- You need to [authenticate with the API](../../../api/rest/index.md#authentication). If authenticating with a personal access token, it must be configured with the `read_api` scope. Authentication tokens (Job Token or Personal Access Token) can be provided for `terraform` in your `~/.terraformrc` file: @@ -107,6 +107,38 @@ Where `<namespace>` is the [namespace](../../../user/namespace/index.md) of the ## Publish a Terraform module by using CI/CD +> CI/CD template [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110493) in GitLab 15.9. + +### Use a CI/CD template (recommended) + +You can use the [`Terraform-Module.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform-Module.gitlab-ci.yml) +or the advanced [`Terraform/Module-Base.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Module-Base.gitlab-ci.yml) +CI/CD template to publish a Terraform module to the GitLab Terraform Registry: + +```yaml +include: + template: Terraform-Module.gitlab-ci.yml +``` + +The pipeline contains the following jobs: + +- `fmt` - Validate the formatting of the Terraform module. +- `kics-iac-sast` - Test the Terraform module for security issues. +- `deploy` - For tag pipelines only. Deploy the Terraform module to the GitLab Terraform Registry. + +#### Pipeline variables + +You can configure the pipeline with the following variables: + +| Variable | Default | Description | +|----------------------------|----------------------|-------------------------------------------------------------------------------------------------| +| `TERRAFORM_MODULE_DIR` | `${CI_PROJECT_DIR}` | The relative path to the root directory of the Terraform project. | +| `TERRAFORM_MODULE_NAME` | `${CI_PROJECT_NAME}` | The name of your Terraform module. Must not contain any spaces or underscores. | +| `TERRAFORM_MODULE_SYSTEM` | `local` | The system or provider of your Terraform module targets. For example, `local`, `aws`, `google`. | +| `TERRAFORM_MODULE_VERSION` | `${CI_COMMIT_TAG}` | The Terraform module version. You should follow the semantic versioning specification. | + +### Deploy manually via CI/CD + To work with Terraform modules in [GitLab CI/CD](../../../ci/index.md), you can use `CI_JOB_TOKEN` in place of the personal access token in your commands. @@ -114,21 +146,21 @@ For example, this job uploads a new module for the `local` [system provider](htt ```yaml stages: - - upload + - deploy upload: - stage: upload + stage: deploy image: curlimages/curl:latest variables: - TERRAFORM_MODULE_DIR: ${CI_PROJECT_DIR} # The path to your Terraform module - TERRAFORM_MODULE_NAME: ${CI_PROJECT_NAME} # The name of your Terraform module - TERRAFORM_MODULE_SYSTEM: local # The system or provider your Terraform module targets (ex. local, aws, google) - TERRAFORM_MODULE_VERSION: ${CI_COMMIT_TAG} # Tag commits with SemVer for the version of your Terraform module to be published + TERRAFORM_MODULE_DIR: ${CI_PROJECT_DIR} # The relative path to the root directory of the Terraform project. + TERRAFORM_MODULE_NAME: ${CI_PROJECT_NAME} # The name of your Terraform module, must not have any spaces or underscores (will be translated to hyphens). + TERRAFORM_MODULE_SYSTEM: local # The system or provider your Terraform module targets (ex. local, aws, google). + TERRAFORM_MODULE_VERSION: ${CI_COMMIT_TAG} # The version - it's recommended to follow SemVer for Terraform Module Versioning. script: - TERRAFORM_MODULE_NAME=$(echo "${TERRAFORM_MODULE_NAME}" | tr " _" -) # module-name must not have spaces or underscores, so translate them to hyphens - - tar -vczf ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz -C ${TERRAFORM_MODULE_DIR} --exclude=./.git . + - tar -vczf /tmp/${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz -C ${TERRAFORM_MODULE_DIR} --exclude=./.git . - 'curl --fail-with-body --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" - --upload-file ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz + --upload-file /tmp/${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file' rules: - if: $CI_COMMIT_TAG |