diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-21 07:08:36 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-21 07:08:36 +0000 |
commit | 48aff82709769b098321c738f3444b9bdaa694c6 (patch) | |
tree | e00c7c43e2d9b603a5a6af576b1685e400410dee /doc/user/packages | |
parent | 879f5329ee916a948223f8f43d77fba4da6cd028 (diff) | |
download | gitlab-ce-48aff82709769b098321c738f3444b9bdaa694c6.tar.gz |
Add latest changes from gitlab-org/gitlab@13-5-stable-eev13.5.0-rc42
Diffstat (limited to 'doc/user/packages')
23 files changed, 894 insertions, 634 deletions
diff --git a/doc/user/packages/composer_repository/img/project_id_v13_5.png b/doc/user/packages/composer_repository/img/project_id_v13_5.png Binary files differnew file mode 100644 index 00000000000..45861b6ff1b --- /dev/null +++ b/doc/user/packages/composer_repository/img/project_id_v13_5.png diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 89e02b4847c..d4fb662c3a6 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -4,172 +4,259 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Composer Repository +# Composer packages in the Package Registry > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Core in 13.3. -With the GitLab Composer Repository, every project can have its own space to store [Composer](https://getcomposer.org/) packages. +Publish [Composer](https://getcomposer.org/) packages in your project's Package Registry. +Then, install the packages whenever you need to use them as a dependency. -## Enabling the Composer Repository +Only Composer 1.x is supported. Consider contributing or even adding support for +[Composer 2.0 in the Package Registry](https://gitlab.com/gitlab-org/gitlab/-/issues/259840). -NOTE: **Note:** -This option is available only if your GitLab administrator has -[enabled support for the Package Registry](../../../administration/packages/index.md). +## Create a Composer package -When the Composer Repository is enabled, it is available for all new projects -by default. To enable it for existing projects, or if you want to disable it: +If you do not have a Composer package, create one and check it in to +a repository. This example shows a GitLab repository, but the repository +can be any public or private repository. -1. Navigate to your project's **Settings > General > Visibility, project features, permissions**. -1. Find the Packages feature and enable or disable it. -1. Click on **Save changes** for the changes to take effect. +1. Create a directory called `my-composer-package` and change to that directory: -You should then be able to see the **Packages & Registries** section on the left sidebar. + ```shell + mkdir my-composer-package && cd my-composer-package + ``` -## Getting started +1. Run [`composer init`](https://getcomposer.org/doc/03-cli.md#init) and answer the prompts. -This section covers creating a new example Composer package to publish. This is a -quickstart to test out the **GitLab Composer Registry**. + For namespace, enter your unique [namespace](../../../user/group/index.md#namespaces), like your GitLab username or group name. -To complete this section, you need a recent version of [Composer](https://getcomposer.org/). + A file called `composer.json` is created: -### Creating a package project + ```json + { + "name": "<namespace>/composer-test", + "description": "Library XY", + "type": "library", + "license": "GPL-3.0-only", + "authors": [ + { + "name": "John Doe", + "email": "john@example.com" + } + ], + "require": {} + } + ``` -Understanding how to create a full Composer project is outside the scope of this -guide, but you can create a small package to test out the registry. Start by -creating a new directory called `my-composer-package`: +1. Run Git commands to tag the changes and push them to your repository: -```shell -mkdir my-composer-package && cd my-composer-package -``` + ```shell + git init + git add composer.json + git commit -m 'Composer package test' + git tag v1.0.0 + git remote add origin git@gitlab.example.com:<namespace>/<project-name>.git + git push --set-upstream origin master + git push origin v1.0.0 + ``` -Create a new `composer.json` file inside this directory to set up the basic project: +The package is now in your GitLab Package Registry. -```shell -touch composer.json -``` +## Publish a Composer package by using the API -Inside `composer.json`, add the following code: +Publish a Composer package to the Package Registry, +so that anyone who can access the project can use the package as a dependency. -```json -{ - "name": "<namespace>/composer-test", - "type": "library", - "license": "GPL-3.0-only", - "version": "1.0.0" -} -``` +Prerequisites: -Replace `<namespace>` with a unique namespace like your GitLab username or group name. +- A package in a GitLab repository. +- A valid `composer.json` file. +- The Packages feature is enabled in a GitLab repository. +- The project ID, which is on the project's home page. +- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -After this basic package structure is created, we need to tag it in Git and push it to the repository. + NOTE: **Note:** + [Deploy tokens](./../../project/deploy_tokens/index.md) are + [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) for use with Composer. -```shell -git init -git add composer.json -git commit -m 'Composer package test' -git tag v1.0.0 -git remote add origin git@gitlab.com:<namespace>/<project-name>.git -git push --set-upstream origin master -git push origin v1.0.0 -``` +To publish the package: -### Publishing the package +- Send a `POST` request to the [Packages API](../../../api/packages.md). -Now that the basics of our project is completed, we can publish the package. -To publish the package, you need: + For example, you can use `curl`: -- A personal access token or `CI_JOB_TOKEN`. + ```shell + curl --data tag=<tag> "https://__token__:<personal-access-token>@gitlab.example.com/api/v4/projects/<project_id>/packages/composer" + ``` - ([Deploy tokens](./../../project/deploy_tokens/index.md) are not yet supported for use with Composer.) + - `<personal-access-token>` is your personal access token. + - `<project_id>` is your project ID. + - `<tag>` is the Git tag name of the version you want to publish. + To publish a branch, use `branch=<branch>` instead of `tag=<tag>`. -- Your project ID which can be found on the home page of your project. +You can view the published package by going to **Packages & Registries > Package Registry** and +selecting the **Composer** tab. -To publish the package hosted on GitLab, make a `POST` request to the GitLab package API. -A tool like `curl` can be used to make this request: +## Publish a Composer package by using CI/CD -You can generate a [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api` for repository authentication. For example: +You can publish a Composer package to the Package Registry as part of your CI/CD process. -```shell -curl --data tag=<tag> 'https://__token__:<personal-access-token>@gitlab.com/api/v4/projects/<project_id>/packages/composer' -``` +1. Specify a `CI_JOB_TOKEN` in your `.gitlab-ci.yml` file: -Where: + ```yaml + stages: + - deploy -- `<personal-access-token>` is your personal access token. -- `<project_id>` is your project ID. -- `<tag>` is the Git tag name of the version you want to publish. In this example it should be `v1.0.0`. Notice that instead of `tag=<tag>` you can also use `branch=<branch>` to publish branches. + deploy: + stage: deploy + script: + - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/packages/composer"' + ``` -If the above command succeeds, you now should be able to see the package under the **Packages & Registries** section of your project page. +1. Run the pipeline. -### Publishing the package with CI/CD +You can view the published package by going to **Packages & Registries > Package Registry** and selecting the **Composer** tab. -To work with Composer commands within [GitLab CI/CD](./../../../ci/README.md), you can -publish Composer packages by using `CI_JOB_TOKEN` in your `.gitlab-ci.yml` file: +### Use a CI/CD template -```yaml -stages: - - deploy +A more detailed Composer CI/CD file is also available as a `.gitlab-ci.yml` template: -deploy: - stage: deploy - script: - - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/packages/composer"' -``` +1. On the left sidebar, click **Project overview**. +1. Above the file list, click **Set up CI/CD**. If this button is not available, select **CI/CD Configuration** and then **Edit**. +1. From the **Apply a template** list, select **Composer**. -### Installing a package +CAUTION: **Warning:** +Do not save unless you want to overwrite the existing CI/CD file. -To install your package, you need: +## Install a Composer package -- A personal access token. You can generate a [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api` for repository authentication. -- Your group ID which can be found on the home page of your project's group. +Install a package from the Package Registry so you can use it as a dependency. -Add the GitLab Composer package repository to your existing project's `composer.json` file, along with the package name and version you want to install like so: +Prerequisites: -```json -{ - ... - "repositories": [ - { "type": "composer", "url": "https://gitlab.com/api/v4/group/<group_id>/-/packages/composer/packages.json" } - ], - "require": { - ... - "<package_name>": "<version>" - }, - ... -} -``` +- A package in the Package Registry. +- The group ID, which is on the group's home page. +- A [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to, at minimum, `read_api`. -Where: + NOTE: **Note:** + [Deploy tokens](./../../project/deploy_tokens/index.md) are + [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/240897) for use with Composer. -- `<group_id>` is the group ID found under your project's group page. -- `<package_name>` is your package name as defined in your package's `composer.json` file. -- `<version>` is your package version (`1.0.0` in this example). +To install a package: -You also need to create a `auth.json` file with your GitLab credentials: +1. Add the Package Registry URL to your project's `composer.json` file, along with the package name and version you want to install: -```json -{ - "gitlab-token": { - "gitlab.com": "<personal_access_token>" - } -} -``` + - Connect to the Package Registry for your group: -Where: + ```shell + composer config repositories.<group_id> composer https://gitlab.example.com/api/v4/group/<group_id>/-/packages/composer/ + ``` -- `<personal_access_token>` is your personal access token. + - Set the required package version: -With the `composer.json` and `auth.json` files configured, you can install the package by running `composer`: + ```shell + composer require <package_name>:<version> + ``` -```shell -composer update -``` + Result in the `composer.json` file: -If successful, you should be able to see the output indicating that the package has been successfully installed. + ```json + { + ... + "repositories": { + "<group_id>": { + "type": "composer", + "url": "https://gitlab.example.com/api/v4/group/<group_id>/-/packages/composer/" + }, + ... + }, + "require": { + ... + "<package_name>": "<version>" + }, + ... + } + ``` + + You can unset this with the command: + + ```shell + composer config --unset repositories.<group_id> + ``` + + - `<group_id>` is the group ID. + - `<package_name>` is the package name defined in your package's `composer.json` file. + - `<version>` is the package version. + +1. Create an `auth.json` file with your GitLab credentials: + + ```shell + composer config gitlab-token.<DOMAIN-NAME> <personal_access_token> + ``` + + Result in the `auth.json` file: + + ```json + { + ... + "gitlab-token": { + "<DOMAIN-NAME>": "<personal_access_token>", + ... + } + } + ``` + + You can unset this with the command: + + ```shell + composer config --unset --auth gitlab-token.<DOMAIN-NAME> + ``` + + - `<DOMAIN-NAME>` is the GitLab instance URL `gitlab.com` or `gitlab.example.com`. + - `<personal_access_token>` with the scope set to `read_api`. + +1. If you are on a GitLab self-managed instance, add `gitlab-domains` to `composer.json`. + + ```shell + composer config gitlab-domains gitlab01.example.com gitlab02.example.com + ``` + + Result in the `composer.json` file: + + ```json + { + ... + "repositories": [ + { "type": "composer", "url": "https://gitlab.example.com/api/v4/group/<group_id>/-/packages/composer/" } + ], + "config": { + ... + "gitlab-domains": ["gitlab01.example.com", "gitlab02.example.com"] + }, + "require": { + ... + "<package_name>": "<version>" + }, + ... + } + ``` + + You can unset this with the command: + + ```shell + composer config --unset gitlab-domains + ``` + + NOTE: **Note:** + On GitLab.com, Composer uses the GitLab token from `auth.json` as a private token by default. + Without the `gitlab-domains` definition in `composer.json`, Composer uses the GitLab token + as basic-auth, with the token as a username and a blank password. This results in a 401 error. + +Output indicates that the package has been successfully installed. CAUTION: **Important:** -Make sure to never commit the `auth.json` file to your repository. To install packages from a CI job, +Never commit the `auth.json` file to your repository. To install packages from a CI/CD job, consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages-with-satis.md#authentication) tool with your personal access token stored in a [GitLab CI/CD environment variable](../../../ci/variables/README.md) or in -[Hashicorp Vault](../../../ci/secrets/index.md). +[HashiCorp Vault](../../../ci/secrets/index.md). diff --git a/doc/user/packages/conan_repository/img/conan_package_view.png b/doc/user/packages/conan_repository/img/conan_package_view.png Binary files differdeleted file mode 100644 index 742fb4195da..00000000000 --- a/doc/user/packages/conan_repository/img/conan_package_view.png +++ /dev/null diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 7c3082e0f83..10c820a86be 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -4,153 +4,150 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Conan Repository +# Conan packages in the Package Registry > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8248) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Core in 13.3. -With the GitLab Conan Repository, every -project can have its own space to store Conan packages. +Publish Conan packages in your project’s Package Registry. Then install the +packages whenever you need to use them as a dependency. -![GitLab Conan Repository](img/conan_package_view.png) +To publish Conan packages to the Package Registry, add the +Package Registry as a remote and authenticate with it. -## Enabling the Conan Repository +Then you can run `conan` commands and publish your package to the Package Registry. -NOTE: **Note:** -This option is available only if your GitLab administrator has -[enabled support for the Conan Repository](../../../administration/packages/index.md). - -After the Conan Repository is enabled, it's available for all new projects -by default. To enable it for existing projects, or if you want to disable it: - -1. Navigate to your project's **Settings > General > Visibility, project features, permissions**. -1. Find the Packages feature and enable or disable it. -1. Click on **Save changes** for the changes to take effect. - -You should then be able to see the **Packages & Registries** section on the left sidebar. +## Build a Conan package -## Getting started +This section explains how to install Conan and build a package for your C/C++ project. -This section covers installing Conan and building a package for your C/C++ project. This is a quickstart if you're new -to Conan. If you already are using Conan and understand how to build your own packages, move on to the [next section](#adding-the-gitlab-package-registry-as-a-conan-remote). +If you already use Conan and know how to build your own packages, go to the [next section](#add-the-package-registry-as-a-conan-remote). -### Installing Conan +### Install Conan -Follow the instructions at [conan.io](https://conan.io/downloads.html) to download the Conan package manager to your local development environment. +Download the Conan package manager to your local development environment by following +the instructions at [conan.io](https://conan.io/downloads.html). -Once installation is complete, verify you can use Conan in your terminal by running +When installation is complete, verify you can use Conan in your terminal by running: ```shell conan --version ``` -You should see the Conan version printed in the output: +The Conan version is printed in the output: ```plaintext Conan version 1.20.5 ``` -### Installing CMake +### Install CMake + +When you develop with C++ and Conan, you have a range of compilers to choose from. +This example uses the CMake compiler. + +To install CMake: + +- For Mac, use [homebrew](https://brew.sh/) and run `brew install cmake`. +- For other operating systems, follow the instructions at [cmake.org](https://cmake.org/install/). -When developing with C++ and Conan, you have a wide range of options for compilers. This tutorial walks through using the cmake -compiler. In your terminal, run the command +When installation is complete, verify you can use CMake in your terminal by running: ```shell cmake --version ``` -You should see the cmake version printed in the output. If you see something else, you may have to install cmake. +The CMake version is printed in the output. -On a Mac, you can use [homebrew](https://brew.sh/) to install cmake by running `brew install cmake`. Otherwise, follow -instructions at [cmake.org](https://cmake.org/install/) for your operating system. +### Create a project -### Creating a project +To test the Package Registry, you need a C++ project. If you don't already have one, you can clone the +Conan [hello world starter project](https://github.com/conan-io/hello). -Understanding what is needed to create a valid and compilable C++ project is out of the scope of this guide, but if you're new to C++ and want to try out the GitLab -package registry, Conan.io has a great [hello world starter project](https://github.com/conan-io/hello) that you can clone to get started. +### Build a package -Clone the repository and it can be used for the rest of the tutorial if you don't have your own C++ project. +To build a package: -### Building a package +1. Open a terminal and navigate to your project's root folder. +1. Generate a new recipe by running `conan new` with a package name and version: -In your terminal, navigate to the root folder of your project. Generate a new recipe by running `conan new` and providing it with a -package name and version: + ```shell + conan new Hello/0.1 -t + ``` -```shell -conan new Hello/0.1 -t -``` - -Next, create a package for that recipe by running `conan create` providing the Conan user and channel: +1. Create a package for the recipe by running `conan create` with the Conan user and channel: -```shell -conan create . mycompany/beta -``` + ```shell + conan create . mycompany/beta + ``` -NOTE: **Note:** -If you are using the [instance level remote](#instance-level-remote), a specific [naming convention](#package-recipe-naming-convention-for-instance-level-remote) must be followed. + NOTE: **Note:** + If you use an [instance remote](#add-a-remote-for-your-instance), you must follow a specific [naming convention](#package-recipe-naming-convention-for-instance-remotes). -These two example commands generate a final package with the recipe `Hello/0.1@mycompany/beta`. +A package with the recipe `Hello/0.1@mycompany/beta` is created. -For more advanced details on creating and managing your packages, refer to the [Conan docs](https://docs.conan.io/en/latest/creating_packages.html). +For more details on creating and managing Conan packages, see the [Conan docs](https://docs.conan.io/en/latest/creating_packages.html). -You are now ready to upload your package to the GitLab registry. To get started, first you need to set GitLab as a remote. Then you need to add a Conan user for that remote to authenticate your requests. +## Add the Package Registry as a Conan remote -## Adding the GitLab Package Registry as a Conan remote +To run `conan` commands, you must add the Package Registry as a Conan remote for your project or instance. -You can add the GitLab Package Registry as a Conan remote at the project or instance level. - -### Project level remote +### Add a remote for your project > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) in GitLab 13.4. -The project level remote allows you to work with packages within a given project. -The advantage of using the project level remote is there are no restrictions to your -package name, however all GitLab Conan packages require a full recipe -with the user and channel (`package_name/version@user/channel`). +Set a remote so you can work with packages in a project without +having to specify the remote name in every command. + +When you set a remote for a project, there are no restrictions to your package names. +However, your commands must include the full recipe, including the user and channel, +for example, `package_name/version@user/channel`. To add the remote: -```shell -conan remote add gitlab https://gitlab.example.com/api/v4/projects/<project_id>/packages/conan -``` +1. In your terminal, run this command: -Once the remote is set, you can use the remote when running Conan commands by adding `--remote=gitlab` to the end of your commands. + ```shell + conan remote add gitlab https://gitlab.example.com/api/v4/projects/<project_id>/packages/conan + ``` -For example: +1. Use the remote by adding `--remote=gitlab` to the end of your Conan command. -```shell -conan search Hello* --all --remote=gitlab -``` + For example: -### Instance level remote + ```shell + conan search Hello* --all --remote=gitlab + ``` -The instance level remote allows you to use a single remote to access packages accross your entire -GitLab instance. However, when using this remote, there are certain -[package naming restrictions](#package-recipe-naming-convention-for-instance-level-remote) -that must be followed. +### Add a remote for your instance -Add a new remote to your Conan configuration: +Use a single remote to access packages across your entire GitLab instance. -```shell -conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan -``` +However, when using this remote, you must follow these +[package naming restrictions](#package-recipe-naming-convention-for-instance-remotes). + +To add the remote: -Once the remote is set, you can use the remote when running Conan commands by adding `--remote=gitlab` to the end of your commands. +1. In your terminal, run this command: -For example: + ```shell + conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan + ``` -```shell -conan search 'Hello*' --remote=gitlab -``` +1. Use the remote by adding `--remote=gitlab` to the end of your Conan command. -#### Package recipe naming convention for instance level remote + For example: -The standard Conan recipe convention looks like `package_name/version@user/channel`, -but if you're using the [instance level remote](#instance-level-remote), the recipe + ```shell + conan search 'Hello*' --remote=gitlab + ``` + +#### Package recipe naming convention for instance remotes + +The standard Conan recipe convention is `package_name/version@user/channel`, +but if you're using an [instance remote](#add-a-remote-for-your-instance), the recipe `user` must be the plus sign (`+`) separated project path. -The following table shows some example recipes you can give your package based on -the project name and path. +Example recipe names: | Project | Package | Supported | | ---------------------------------- | ----------------------------------------------- | --------- | @@ -159,179 +156,202 @@ the project name and path. | `gitlab-org/gitlab-ce` | `my-package/1.0.0@gitlab-org+gitlab-ce/stable` | Yes | | `gitlab-org/gitlab-ce` | `my-package/1.0.0@foo/stable` | No | -NOTE: **Note:** -[Project level remotes](#project-level-remote) allow for more flexible package names. +[Project remotes](#add-a-remote-for-your-project) have a more flexible naming convention. -## Authenticating to the GitLab Conan Repository +## Authenticate to the Package Registry -You need a personal access token or deploy token. +To authenticate to the Package Registry, you need either a personal access token or deploy token. -For repository authentication: +- If you use a [personal access token](../../../user/profile/personal_access_tokens.md), set the scope to `api`. +- If you use a [deploy token](./../../project/deploy_tokens/index.md), set the scope to `read_package_registry`, `write_package_registry`, or both. -- You can generate a [personal access token](../../../user/profile/personal_access_tokens.md) with the scope set to `api`. -- You can generate a [deploy token](./../../project/deploy_tokens/index.md) with the scope set to `read_package_registry`, `write_package_registry`, or both. +### Add your credentials to the GitLab remote -### Adding a Conan user to the GitLab remote +Associate your token with the GitLab remote, so that you don't have to explicitly +add a token to every Conan command. -Once you have a personal access token and have [set your Conan remote](#adding-the-gitlab-package-registry-as-a-conan-remote), you can associate the token with the remote so you don't have to explicitly add them to each Conan command you run: +Prerequisites: + +- You must have an authentication token. +- The Conan remote [must be set](#add-the-package-registry-as-a-conan-remote). + +In a terminal, run this command. In this example, the remote name is `gitlab`. Use the name of your remote. ```shell conan user <gitlab_username or deploy_token_username> -r gitlab -p <personal_access_token or deploy_token> ``` -NOTE: **Note:** -If you named your remote something other than `gitlab`, your remote name should be used in this command instead of `gitlab`. - -From now on, when you run commands using `--remote=gitlab`, your username and password is automatically included in the requests. - -NOTE: **Note:** -The personal access token is not stored locally at any moment. Conan uses JSON Web Tokens (JWT), so when you run this command, Conan requests an expirable token from GitLab using your token. The JWT does expire on a regular basis, so you need to re-enter your personal access token when that happens. +Now when you run commands with `--remote=gitlab`, your username and password are automatically included in the requests. -Alternatively, you could explicitly include your credentials in any given command. -For example: +Alternately, you can explicitly include your credentials in any given command. For example: ```shell CONAN_LOGIN_USERNAME=<gitlab_username or deploy_token_username> CONAN_PASSWORD=<personal_access_token or deploy_token> conan upload Hello/0.1@mycompany/beta --all --remote=gitlab ``` -### Setting a default remote to your project (optional) +NOTE: **Note:** +Your authentication with GitLab expires on a regular basis, +so occasionally you may need to re-enter your personal access token. + +### Set a default remote for your project (optional) + +If you want to interact with the GitLab Package Registry without having to specify a remote, +you can tell Conan to always use the Package Registry for your packages. -If you'd like Conan to always use GitLab as the registry for your package, you can tell Conan to always reference the GitLab remote for a given package recipe: +In a terminal, run this command. ```shell conan remote add_ref Hello/0.1@mycompany/beta gitlab ``` NOTE: **Note:** -The package recipe does include the version, so setting the default remote for `Hello/0.1@user/channel` will not work for `Hello/0.2@user/channel`. -This functionality is best suited for when you want to consume or install packages from the GitLab registry without having to specify a remote. +The package recipe includes the version, so the default remote for `Hello/0.1@user/channel` does not work for `Hello/0.2@user/channel`. -The rest of the example commands in this documentation assume that you've added a Conan user with your credentials to the `gitlab` remote and will not include the explicit credentials or remote option. With that said, be aware that any of the commands could be run without having added a user or default remote: +If you do not set a default user or remote, you can still include the user and remote in your commands: ```shell `CONAN_LOGIN_USERNAME=<gitlab_username or deploy_token_username> CONAN_PASSWORD=<personal_access_token or deploy_token> <conan command> --remote=gitlab ``` -## Uploading a package +## Publish a Conan package -First you need to [create your Conan package locally](https://docs.conan.io/en/latest/creating_packages/getting_started.html). +Publish a Conan package to the Package Registry, so that anyone who can access the project can use the package as a dependency. -NOTE: **Note:** -If you are using the [instance level remote](#instance-level-remote), a specific [naming convention](#package-recipe-naming-convention-for-instance-level-remote) must be followed. +Prerequisites: + +To publish a Conan package, you need: -Ensure you have a project created on GitLab and that the personal access token you're using has the correct permissions for write access to the container registry by selecting the `api` [scope](../../../user/profile/personal_access_tokens.md#limiting-scopes-of-a-personal-access-token). +- The Package Registry [set as a remote](#add-the-package-registry-as-a-conan-remote). +- [Authentication](#authenticate-to-the-package-registry) set up with the Package Registry. +- A local [Conan package](https://docs.conan.io/en/latest/creating_packages/getting_started.html). + - For an instance remote, the package must meet the [naming convention](#package-recipe-naming-convention-for-instance-remotes). +- A project ID, which is on the project's homepage. -You can upload your package to the GitLab Package Registry using the `conan upload` command: +To publish the package, use the `conan upload` command: ```shell conan upload Hello/0.1@mycompany/beta --all ``` -## Installing a package +## Publish a Conan package by using CI/CD -Conan packages are commonly installed as dependencies using the `conanfile.txt` file. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11678) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7. -In your project where you would like to install the Conan package as a dependency, open `conanfile.txt` or create -an empty file named `conanfile.txt` in the root of your project. +To work with Conan commands in [GitLab CI/CD](./../../../ci/README.md), you can use +`CI_JOB_TOKEN` in place of the personal access token in your commands. -Add the Conan recipe to the `[requires]` section of the file: +You can provide the `CONAN_LOGIN_USERNAME` and `CONAN_PASSWORD` with each +Conan command in your `.gitlab-ci.yml` file. For example: -```ini - [requires] - Hello/0.1@mycompany/beta +```yaml +image: conanio/gcc7 - [generators] - cmake +create_package: + stage: deploy + script: + - conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan + - conan new <package-name>/0.1 -t + - conan create . <group-name>+<project-name>/stable + - CONAN_LOGIN_USERNAME=ci_user CONAN_PASSWORD=${CI_JOB_TOKEN} conan upload <package-name>/0.1@<group-name>+<project-name>/stable --all --remote=gitlab ``` -Next, create a build directory from the root of your project and navigate to it: +Additional Conan images to use as the basis of your CI file are available +in the [Conan docs](https://docs.conan.io/en/latest/howtos/run_conan_in_docker.html#available-docker-images). -```shell -mkdir build && cd build -``` +## Install a Conan package -Now you can install the dependencies listed in `conanfile.txt`: +Install a Conan package from the Package Registry so you can use it as a dependency. -```shell -conan install .. -``` +Conan packages are often installed as dependencies by using the `conanfile.txt` file. + +Prerequisites: + +To install a Conan package, you need: + +- The Package Registry [set as a remote](#add-the-package-registry-as-a-conan-remote). +- [Authentication](#authenticate-to-the-package-registry) set up with the Package Registry. + +1. In the project where you want to install the package as a dependency, open `conanfile.txt`. + Or, in the root of your project, create a file called `conanfile.txt`. + +1. Add the Conan recipe to the `[requires]` section of the file: + + ```plaintext + [requires] + Hello/0.1@mycompany/beta + + [generators] + cmake + ``` + +1. At the root of your project, create a `build` directory and change to that directory: + + ```shell + mkdir build && cd build + ``` + +1. Install the dependencies listed in `conanfile.txt`: + + ```shell + conan install <options> + ``` NOTE: **Note:** -If you're trying to install the package you just created in this tutorial, not much will happen since that package -already exists on your local machine. +If you try to install the package you just created in this tutorial, the package +already exists on your local machine, so this command has no effect. -## Removing a package +## Remove a Conan package There are two ways to remove a Conan package from the GitLab Package Registry. -- **Using the Conan client in the command line:** +- From the command line, using the Conan client: ```shell conan remove Hello/0.2@user/channel --remote=gitlab ``` - You need to explicitly include the remote in this command, otherwise the package is only removed from your + You must explicitly include the remote in this command, otherwise the package is only removed from your local system cache. NOTE: **Note:** This command removes all recipe and binary package files from the Package Registry. -- **GitLab project interface**: in the packages view of your project page, you can delete packages by clicking the red trash icons. +- From the GitLab user interface: -## Searching the GitLab Package Registry for Conan packages + Go to your project's **Packages & Registries > Package Registry**. Remove the package by clicking the red trash icon. -The `conan search` command can be run searching by full or partial package name, or by exact recipe. +## Search for Conan packages in the Package Registry -To search using a partial name, use the wildcard symbol `*`, which should be placed at the end of your search (for example, `my-packa*`): +To search by full or partial package name, or by exact recipe, run the `conan search` command. -```shell -conan search Hello --all --remote=gitlab -conan search He* --all --remote=gitlab -conan search Hello/0.1@mycompany/beta --all --remote=gitlab -``` +- To search for all packages with a specific package name: + + ```shell + conan search Hello --remote=gitlab + ``` -The scope of your search includes all projects you have permission to access, this includes your private projects as well as all public projects. +- To search for a partial name, like all packages starting with `He`: -## Fetching Conan package information from the GitLab Package Registry + ```shell + conan search He* --remote=gitlab + ``` -The `conan info` command returns information about a given package: +The scope of your search includes all projects you have permission to access. This includes your private projects as well as all public projects. + +## Fetch Conan package information from the Package Registry + +The `conan info` command returns information about a package: ```shell conan info Hello/0.1@mycompany/beta ``` -## List of supported CLI commands +## Supported CLI commands The GitLab Conan repository supports the following Conan CLI commands: -- `conan upload`: Upload your recipe and package files to the GitLab Package Registry. -- `conan install`: Install a conan package from the GitLab Package Registry, this includes using the `conanfile.txt` file. -- `conan search`: Search the GitLab Package Registry for public packages, and private packages you have permission to view. -- `conan info`: View the information on a given package from the GitLab Package Registry. -- `conan remove`: Delete the package from the GitLab Package Registry. - -## Using GitLab CI with Conan packages - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11678) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7. - -To work with Conan commands within [GitLab CI/CD](./../../../ci/README.md), you can use -`CI_JOB_TOKEN` in place of the personal access token in your commands. - -It's easiest to provide the `CONAN_LOGIN_USERNAME` and `CONAN_PASSWORD` with each -Conan command in your `.gitlab-ci.yml` file. For example: - -```yaml -image: conanio/gcc7 - -create_package: - stage: deploy - script: - - conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan - - conan new <package-name>/0.1 -t - - conan create . <group-name>+<project-name>/stable - - CONAN_LOGIN_USERNAME=ci_user CONAN_PASSWORD=${CI_JOB_TOKEN} conan upload <package-name>/0.1@<group-name>+<project-name>/stable --all --remote=gitlab - -``` - -You can find additional Conan images to use as the base of your CI file -in the [Conan docs](https://docs.conan.io/en/latest/howtos/run_conan_in_docker.html#available-docker-images). +- `conan upload`: Upload your recipe and package files to the Package Registry. +- `conan install`: Install a Conan package from the Package Registry, this includes using the `conanfile.txt` file. +- `conan search`: Search the Package Registry for public packages, and private packages you have permission to view. +- `conan info`: View the information on a given package from the Package Registry. +- `conan remove`: Delete the package from the Package Registry. diff --git a/doc/user/packages/container_registry/img/container_registry_group_repositories_v13_1.png b/doc/user/packages/container_registry/img/container_registry_group_repositories_v13_1.png Binary files differdeleted file mode 100644 index bbbba44eb9b..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_group_repositories_v13_1.png +++ /dev/null diff --git a/doc/user/packages/container_registry/img/container_registry_hover_path_13_4.png b/doc/user/packages/container_registry/img/container_registry_hover_path_13_4.png Binary files differnew file mode 100644 index 00000000000..2d16c11fe61 --- /dev/null +++ b/doc/user/packages/container_registry/img/container_registry_hover_path_13_4.png diff --git a/doc/user/packages/container_registry/img/container_registry_repositories_v13_1.png b/doc/user/packages/container_registry/img/container_registry_repositories_v13_1.png Binary files differdeleted file mode 100644 index 13a6d1a4470..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_repositories_v13_1.png +++ /dev/null diff --git a/doc/user/packages/container_registry/img/container_registry_repositories_with_quickstart_v13_1.png b/doc/user/packages/container_registry/img/container_registry_repositories_with_quickstart_v13_1.png Binary files differdeleted file mode 100644 index 35a02182a77..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_repositories_with_quickstart_v13_1.png +++ /dev/null diff --git a/doc/user/packages/container_registry/img/container_registry_repository_details_v13.0.png b/doc/user/packages/container_registry/img/container_registry_repository_details_v13.0.png Binary files differdeleted file mode 100644 index 088e80221de..00000000000 --- a/doc/user/packages/container_registry/img/container_registry_repository_details_v13.0.png +++ /dev/null diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 077666bc036..baadd3c91a7 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -9,235 +9,186 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4040) in GitLab 8.8. > - Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker > versions earlier than 1.10. -> - Starting from GitLab 8.12, if you have 2FA enabled in your account, you need -> to pass a [personal access token](../../profile/personal_access_tokens.md) instead of your password in order to -> login to GitLab's Container Registry. -> - Multiple level image names support was added in GitLab 9.1. -> - The group level Container Registry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23315) in GitLab 12.10. +> - Starting in GitLab 8.12, if you have [two-factor authentication](../../profile/account/two_factor_authentication.md) enabled in your account, you need +> to pass a [personal access token](../../profile/personal_access_tokens.md) instead of your password to +> sign in to the Container Registry. +> - Support for multiple level image names was added in GitLab 9.1. +> - The group-level Container Registry was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23315) in GitLab 12.10. > - Searching by image repository name was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31322) in GitLab 13.0. -NOTE: **Note:** -This document is the user guide. To learn how to enable GitLab Container -Registry across your GitLab instance, visit the -[administrator documentation](../../../administration/packages/container_registry.md). - -With the Docker Container Registry integrated into GitLab, every project can +With the Docker Container Registry integrated into GitLab, every GitLab project can have its own space to store its Docker images. You can read more about Docker Registry at <https://docs.docker.com/registry/introduction/>. -![Container Registry repositories](img/container_registry_repositories_v13_1.png) - -## Enable the Container Registry for your project - -CAUTION: **Warning:** -The Container Registry follows the visibility settings of the project. If the project is public, so is the Container Registry. - -If you cannot find the **Packages & Registries > Container Registry** entry under your -project's sidebar, it is not enabled in your GitLab instance. Ask your -administrator to enable GitLab Container Registry following the -[administration documentation](../../../administration/packages/container_registry.md). +This document is the user guide. To learn how to enable the Container +Registry for your GitLab instance, visit the +[administrator documentation](../../../administration/packages/container_registry.md). -If you are using GitLab.com, this is enabled by default so you can start using -the Registry immediately. Currently there is a soft (10GB) size restriction for -Registry on GitLab.com, as part of the [repository size limit](../../project/repository/index.md). +## View the Container Registry -Once enabled for your GitLab instance, to enable Container Registry for your -project: +You can view the Container Registry for a project or group. -1. Go to your project's **Settings > General** page. -1. Expand the **Visibility, project features, permissions** section - and enable the **Container Registry** feature on your project. For new - projects this might be enabled by default. For existing projects - (prior GitLab 8.8), enable it explicitly. -1. Press **Save changes** for the changes to take effect. You should now be able - to see the **Packages & Registries > Container Registry** link in the sidebar. +1. Go to your project or group. +1. Go to **Packages & Registries > Container Registry**. -## Control Container Registry from within GitLab +You can search, sort, filter, and [delete](#delete-images-from-within-gitlab) containers on this page. -GitLab offers a simple Container Registry management panel. This management panel is available -for both projects and groups. +Only members of the project or group can access a private project's Container Registry. -### Control Container Registry for your project +If a project is public, so is the Container Registry. -Navigate to your project's **{package}** **Packages & Registries > Container Registry**. +## Use images from the Container Registry -![Container Registry project repositories](img/container_registry_repositories_with_quickstart_v13_1.png) +To download and run a container image hosted in the GitLab Container Registry: -This view allows you to: +1. Copy the link to your container image: + - Go to your project or group's **Packages & Registries > Container Registry** + and find the image you want. + - Next to the image name, click the **Copy** button. -- Show all the image repositories that belong to the project. -- Filter image repositories by their name. -- [Delete](#delete-images-from-within-gitlab) one or more image repository. -- Navigate to the image repository details page. -- Show a **Quick start** dropdown with the most common commands to log in, build and push. -- Show a banner if the optional [cleanup policy](#cleanup-policy) is enabled for this project. + ![Container Registry image URL](img/container_registry_hover_path_13_4.png) -### Control Container Registry for your group +1. Use `docker run` with the image link: -Navigate to your group's **{package}** **Packages & Registries > Container Registry**. + ```shell + docker run [options] registry.example.com/group/project/image [arguments] + ``` -![Container Registry group repositories](img/container_registry_group_repositories_v13_1.png) +For more information on running Docker containers, visit the +[Docker documentation](https://docs.docker.com/engine/userguide/intro/). -This view allows you to: +## Image naming convention -- Show all the image repositories of the projects that belong to this group. -- [Delete](#delete-images-from-within-gitlab) one or more image repositories. -- Navigate to a specific image repository details page. +Images follow this naming convention: -### Image Repository details page +```plaintext +<registry URL>/<namespace>/<project>/<image> +``` -Clicking on the name of any image repository navigates to the details. +If your project is `gitlab.example.com/mynamespace/myproject`, for example, +then your image must be named `gitlab.example.com/mynamespace/myproject/my-app` at a minimum. -![Container Registry project repository details](img/container_registry_repository_details_v13.0.png) +You can append additional names to the end of an image name, up to three levels deep. -NOTE: **Note:** -The following page has the same functionalities both in the **Group level container registry** -and in the **Project level container registry**. +For example, these are all valid image names for images within the project named `myproject`: -This view: +```plaintext +registry.example.com/mynamespace/myproject:some-tag +``` -- Shows all the image repository details. -- Shows all the tags of the image repository. -- Allows you to quickly copy the tag path (by clicking on the clipboard button near the tag name). -- Allows you to [delete one or more tags](#delete-images-from-within-gitlab). +```plaintext +registry.example.com/mynamespace/myproject/image:latest +``` -## Use images from GitLab Container Registry +```plaintext +registry.example.com/mynamespace/myproject/my/image:rc1 +``` -To download and run a container from images hosted in GitLab Container Registry, -use `docker run`: +## Build and push images by using Docker commands -```shell -docker run [options] registry.example.com/group/project/image [arguments] -``` +To build and push to the Container Registry, you can use Docker commands. -For more information on running Docker containers, visit the -[Docker documentation](https://docs.docker.com/engine/userguide/intro/). +### Authenticate with the Container Registry -## Authenticating to the GitLab Container Registry +Before you can build and push images, you must authenticate with the Container Registry. -If you visit the **Packages & Registries > Container Registry** link under your project's -menu, you can see the explicit instructions to login to the Container Registry -using your GitLab credentials. +To authenticate, you can use: -For example if the Registry's URL is `registry.example.com`, then you should be -able to login with: +- A [personal access token](../../profile/personal_access_tokens.md). +- A [deploy token](../../project/deploy_tokens/index.md). -```shell -docker login registry.example.com -``` +Both of these require the minimum scope to be: -NOTE: **Note:** -If you have [2 Factor Authentication](../../profile/account/two_factor_authentication.md) -enabled in your account, you need to pass a -[personal access token](../../profile/personal_access_tokens.md) instead -of your password in order to login to GitLab's Container Registry. +- For read (pull) access, `read_registry`. +- For write (push) access, `write_registry`. -Credentials must be provided for authorization to any non-public registry. Only project members can access private, -GitLab-hosted registries. +To authenticate, run the `docker` command. For example: -There are two ways to authenticate: + ```shell + docker login registry.example.com -u <username> -p <token> + ``` -- By using a [personal access token](../../profile/personal_access_tokens.md). -- By using a [deploy token](../../project/deploy_tokens/index.md). +### Build and push images by using Docker commands -The minimum scope needed for both of them is `read_registry`. +To build and push to the Container Registry: -Example of using a token: +1. Authenticate with the Container Registry. -```shell -docker login registry.example.com -u <username> -p <token> -``` +1. Run the command to build or push. For example, to build: -## Build and push images from your local machine + ```shell + docker build -t registry.example.com/group/project/image . + ``` -Building and publishing images should be a straightforward process. Just make -sure that you are using the Registry URL with the namespace and project name -that is hosted on GitLab: + Or to push: -```shell -docker build -t registry.example.com/group/project/image . -docker push registry.example.com/group/project/image -``` + ```shell + docker push registry.example.com/group/project/image + ``` -Your image is named after the following scheme: +You can also view these commands by going to your project's **Packages & Registries > Container Registry**. -```plaintext -<registry URL>/<namespace>/<project>/<image> -``` +## Build and push by using GitLab CI/CD -GitLab supports up to three levels of image repository names. -The following examples of image tags are valid: +Use [GitLab CI/CD](../../../ci/yaml/README.md) to build and push images to the +Container Registry. Use it to test, build, and deploy your project from the Docker +image you created. -```plaintext -registry.example.com/group/project:some-tag -registry.example.com/group/project/image:latest -registry.example.com/group/project/my/image:rc1 -``` +### Authenticate by using GitLab CI/CD -## Build and push images using GitLab CI/CD - -While you can build and push your images from your local machine, take -full advantage of the Container Registry by combining it with GitLab CI/CD. -You can then create workflows and automate any processes that involve testing, -building, and eventually deploying your project from the Docker image you -created. - -Before diving into the details, some things you should be aware of: - -- You must [authenticate to the container registry](#authenticating-to-the-container-registry-with-gitlab-cicd) - before running any commands. You can do this in the `before_script` if multiple - jobs depend on it. -- Using `docker build --pull` fetches any changes to base - images before building in case your cache is stale. It takes slightly - longer, but it means you don’t get stuck without security patches for base images. -- Doing an explicit `docker pull` before each `docker run` fetches - the latest image that was just built. This is especially important if you are - using multiple runners that cache images locally. Using the Git SHA in your - image tag makes this less necessary since each job is unique and you - shouldn't ever have a stale image. However, it's still possible to have a - stale image if you re-build a given commit after a dependency has changed. -- You don't want to build directly to `latest` tag in case there are multiple jobs - happening simultaneously. +Before you can build and push images by using GitLab CI/CD, you must authenticate with the Container Registry. -### Authenticating to the Container Registry with GitLab CI/CD +To use CI/CD to authenticate, you can use: -There are three ways to authenticate to the Container Registry via -[GitLab CI/CD](../../../ci/yaml/README.md): +- The `CI_REGISTRY_USER` variable. -- **Using the special `CI_REGISTRY_USER` variable**: The user specified by this variable is created for you in order to - push to the Registry connected to your project. Its password is automatically - set with the `CI_REGISTRY_PASSWORD` variable. This allows you to automate building and deploying - your Docker images and has read/write access to the Registry. This is ephemeral, - so it's only valid for one job. You can use the following example as-is: + This variable has read-write access to the Container Registry and is valid for + one job only. Its password is also automatically created and assigned to `CI_REGISTRY_PASSWORD`. ```shell docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY ``` -- **Using the GitLab Deploy Token**: You can create and use a - [special deploy token](../../project/deploy_tokens/index.md#gitlab-deploy-token) - with your projects. - Once created, you can use the special environment variables, and GitLab CI/CD - fills them in for you. You can use the following example as-is: +- A [CI job token](../../../ci/triggers/README.md#ci-job-token). ```shell - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY + docker login -u $CI_JOB_USER -p $CI_JOB_TOKEN $CI_REGISTRY ``` -- **Using a personal access token**: You can create and use a - [personal access token](../../profile/personal_access_tokens.md) - in case your project is private: +- A [deploy token](../../project/deploy_tokens/index.md#gitlab-deploy-token) with the minimum scope of: + - For read (pull) access, `read_registry`. + - For write (push) access, `write_registry`. - - For read (pull) access, the scope should be `read_registry`. - - For write (push) access, the scope should be `write_registry`. + ```shell + docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY + ``` - Replace the `<username>` and `<access_token>` in the following example: +- A [personal access token](../../profile/personal_access_tokens.md) with the minimum scope of: + - For read (pull) access, `read_registry`. + - For write (push) access, `write_registry`. ```shell docker login -u <username> -p <access_token> $CI_REGISTRY ``` +### Configure your `.gitlab-ci.yml` file + +You can configure your `.gitlab-ci.yml` file to build and push images to the Container Registry. + +- If multiple jobs require authentication, put the authentication command in the `before_script`. +- Before building, use `docker build --pull` to fetch changes to base images. It takes slightly + longer, but it ensures your image is up-to-date. +- Before each `docker run`, do an explicit `docker pull` to fetch + the image that was just built. This is especially important if you are + using multiple runners that cache images locally. + + If you use the Git SHA in your image tag, each job is unique and you + should never have a stale image. However, it's still possible to have a + stale image if you re-build a given commit after a dependency has changed. +- Don't build directly to the `latest` tag because multiple jobs may be + happening simultaneously. + ### Container Registry examples with GitLab CI/CD If you're using Docker-in-Docker on your runners, this is how your `.gitlab-ci.yml` @@ -359,15 +310,15 @@ in addition to the steps in the Below is an example of what your `.gitlab-ci.yml` should look like: ```yaml - build: - image: $CI_REGISTRY/group/project/docker:19.03.12 - services: - - name: $CI_REGISTRY/group/project/docker:19.03.12-dind - alias: docker - stage: build - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests +build: + image: $CI_REGISTRY/group/project/docker:19.03.12 + services: + - name: $CI_REGISTRY/group/project/docker:19.03.12-dind + alias: docker + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests ``` If you forget to set the service alias, the `docker:19.03.12` image is unable to find the @@ -394,7 +345,7 @@ the deleted images. To delete images from within GitLab: -1. Navigate to your project's or group's **{package}** **Packages & Registries > Container Registry**. +1. Navigate to your project's or group's **Packages & Registries > Container Registry**. 1. From the **Container Registry** page, you can select what you want to delete, by either: @@ -406,8 +357,6 @@ To delete images from within GitLab: 1. In the dialog box, click **Remove tag**. - ![Container Registry tags](img/container_registry_repository_details_v13.0.png) - ### Delete images using the API If you want to automate the process of deleting images, GitLab provides an API. For more @@ -512,9 +461,28 @@ Cleanup policies can be run on all projects, with these exceptions: for all projects (even those created before 12.8) in [GitLab application settings](../../../api/settings.md#change-application-settings) by setting `container_expiration_policies_enable_historic_entries` to true. + Alternatively, you can execute the following command in the [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ApplicationSetting.last.update(container_expiration_policies_enable_historic_entries: true) + ``` There are performance risks with enabling it for all projects, especially if you are using an [external registry](./index.md#use-with-external-container-registries). +- For self-managed GitLab instances, you can enable or disable the cleanup policy for a specific + project. + + To enable it: + + ```ruby + Feature.enable(:container_expiration_policies_historic_entry, Project.find(<project id>)) + ``` + + To disable it: + + ```ruby + Feature.disable(:container_expiration_policies_historic_entry, Project.find(<project id>)) + ``` ### How the cleanup policy works @@ -636,7 +604,7 @@ you can use the Container Registry to store Helm Charts. However, due to the way and stored by Docker, it is not possible for GitLab to parse this data and meet performance standards. [This epic](https://gitlab.com/groups/gitlab-org/-/epics/2313) updates the architecture of the Container Registry to support Helm Charts. -You can read more about the above challenges [here](https://gitlab.com/gitlab-org/gitlab/-/issues/38047#note_298842890). +[Read more about the above challenges](https://gitlab.com/gitlab-org/gitlab/-/issues/38047#note_298842890). ## Limitations @@ -647,6 +615,19 @@ Container Registry, you must delete all existing images. - Prior to GitLab 12.10, any tags that use the same image ID as the `latest` tag are not deleted by the cleanup policy. +## Disable the Container Registry for a project + +The Container Registry is enabled by default. + +You can, however, remove the Container Registry for a project: + +1. Go to your project's **Settings > General** page. +1. Expand the **Visibility, project features, permissions** section + and disable **Container Registry**. +1. Click **Save changes**. + +The **Packages & Registries > Container Registry** entry is removed from the project's sidebar. + ## Troubleshooting the GitLab Container Registry ### Docker connection error @@ -661,6 +642,21 @@ To get around this, you can [change the group path](../../group/index.md#changin [change the project path](../../project/settings/index.md#renaming-a-repository) or change the branch name. +You may also get a `404 Not Found` or `Unknown Manifest` message if you are using +a Docker Engine version earlier than 17.12. Later versions of Docker Engine use +[the v2 API](https://docs.docker.com/registry/spec/manifest-v2-2/). + +The images in your GitLab Container Registry must also use the Docker v2 API. +For information on how to update your images, see the [Docker help](https://docs.docker.com/registry/spec/deprecated-schema-v1). + +### `Blob unknown to registry` error when pushing a manifest list + +When [pushing a Docker manifest list](https://docs.docker.com/engine/reference/commandline/manifest/#create-and-push-a-manifest-list) to the GitLab Container Registry, you may receive the error `manifest blob unknown: blob unknown to registry`. This issue occurs when the individual child manifests referenced in the manifest list were not pushed to the same repository. + +For example, you may have two individual images, one for `amd64` and another for `arm64v8`, and you want to build a multi-arch image with them. The `amd64` and `arm64v8` images must be pushed to the same repository where you want to push the multi-arch image. + +As a workaround, you should include the architecture in the tag name of individual images. For example, use `mygroup/myapp:1.0.0-amd64` instead of using sub repositories, like `mygroup/myapp/amd64:1.0.0`. You can then tag the manifest list with `mygroup/myapp:1.0.0`. + ### Troubleshoot as a GitLab server admin Troubleshooting the GitLab Container Registry, most of the times, requires diff --git a/doc/user/packages/dependency_proxy/img/group_dependency_proxy.png b/doc/user/packages/dependency_proxy/img/group_dependency_proxy.png Binary files differdeleted file mode 100644 index e550d296d5a..00000000000 --- a/doc/user/packages/dependency_proxy/img/group_dependency_proxy.png +++ /dev/null diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index e3ee8909165..3fa21ef486b 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -8,80 +8,80 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. -NOTE: **Note:** -This is the user guide. In order to use the dependency proxy, an administrator -must first [configure it](../../../administration/packages/dependency_proxy.md). +The GitLab Dependency Proxy is a local proxy you can use for your frequently-accessed +upstream images. -For many organizations, it is desirable to have a local proxy for frequently used -upstream images/packages. In the case of CI/CD, the proxy is responsible for -receiving a request and returning the upstream image from a registry, acting -as a pull-through cache. +In the case of CI/CD, the Dependency Proxy receives a request and returns the +upstream image from a registry, acting as a pull-through cache. -The dependency proxy is available in the group level. To access it, navigate to -a group's **Packages & Registries > Dependency Proxy**. +## Prerequisites -![Dependency Proxy group page](img/group_dependency_proxy.png) +To use the Dependency Proxy: -## Supported dependency proxies +- Your group must be public. Authentication for private groups is [not supported yet](https://gitlab.com/gitlab-org/gitlab/-/issues/11582). -NOTE: **Note:** -For a list of the upcoming additions to the proxies, visit the -[direction page](https://about.gitlab.com/direction/package/dependency_proxy/#top-vision-items). +### Supported images and packages -The following dependency proxies are supported. +The following images and packages are supported. -| Dependency proxy | GitLab version | +| Image/Package | GitLab version | | ---------------- | -------------- | | Docker | 11.11+ | -## Using the Docker dependency proxy +For a list of planned additions, view the +[direction page](https://about.gitlab.com/direction/package/dependency_proxy/#top-vision-items). + +## View the Dependency Proxy + +To view the Dependency Proxy: + +- Go to your group's **Packages & Registries > Dependency Proxy**. + +The Dependency Proxy is not available for projects. + +## Use the Dependency Proxy for Docker images + +You can use GitLab as a source for your Docker images. -With the Docker dependency proxy, you can use GitLab as a source for a Docker image. -To get a Docker image into the dependency proxy: +Prerequisites: -1. Find the proxy URL on your group's page under **Packages & Registries > Dependency Proxy**, - for example `gitlab.com/groupname/dependency_proxy/containers`. -1. Trigger GitLab to pull the Docker image you want (e.g., `alpine:latest` or - `linuxserver/nextcloud:latest`) and store it in the proxy storage by using - one of the following ways: +- Your images must be stored on [Docker Hub](https://hub.docker.com/). +- Docker Hub must be available. Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/241639) + for progress on accessing images when Docker Hub is down. - - Manually pulling the Docker image: +To store a Docker image in Dependency Proxy storage: + +1. Go to your group's **Packages & Registries > Dependency Proxy**. +1. Copy the **Dependency Proxy URL**. +1. Use one of these commands. In these examples, the image is `alpine:latest`. + + - Add the URL to your [`.gitlab-ci.yml`](../../../ci/yaml/README.md#image) file: ```shell - docker pull gitlab.com/groupname/dependency_proxy/containers/alpine:latest + image: gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest ``` - - From a `Dockerfile`: + - Manually pull the Docker image: ```shell - FROM gitlab.com/groupname/dependency_proxy/containers/alpine:latest + docker pull gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest ``` - - In [`.gitlab-ci.yml`](../../../ci/yaml/README.md#image): + - Add the URL to a `Dockerfile`: ```shell - image: gitlab.com/groupname/dependency_proxy/containers/alpine:latest + FROM gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest ``` GitLab pulls the Docker image from Docker Hub and caches the blobs on the GitLab server. The next time you pull the same image, GitLab gets the latest -information about the image from Docker Hub but serves the existing blobs +information about the image from Docker Hub, but serves the existing blobs from the GitLab server. -The blobs are kept forever, and there is no hard limit on how much data can be -stored. - -## Clearing the cache +## Clear the Dependency Proxy cache -It is possible to use the GitLab API to purge the dependency proxy cache for a -given group to gain back disk space that may be taken up by image blobs that -are no longer needed. See the [dependency proxy API documentation](../../../api/dependency_proxy.md) -for more details. - -## Limitations - -The following limitations apply: +Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be +stored. -- Only [public groups are supported](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) (authentication is not supported yet). -- Only Docker Hub is supported. -- This feature requires Docker Hub being available. +To reclaim disk space used by image blobs that are no longer needed, use +the [Dependency Proxy API](../../../api/dependency_proxy.md). diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md new file mode 100644 index 00000000000..0c961b5c86f --- /dev/null +++ b/doc/user/packages/generic_packages/index.md @@ -0,0 +1,139 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# GitLab Generic Packages Repository **(CORE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4209) in GitLab 13.5. +> - It's [deployed behind a feature flag](../../../user/feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It's able to be enabled or disabled per-project. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-generic-packages-in-the-package-registry). + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +Publish generic files, like release binaries, in your project’s Package Registry. Then, install the packages whenever you need to use them as a dependency. + +## Authenticate to the Package Registry + +To authenticate to the Package Registry, you need either a [personal access token](../../../api/README.md#personalproject-access-tokens) +or [CI job token](../../../api/README.md#gitlab-ci-job-token). + +## Publish a package file + +When you publish a package file, if the package does not exist, it is created. + +Prerequisites: + +- You need to [authenticate with the API](../../../api/README.md#authentication). + +```plaintext +PUT /projects/:id/packages/generic/:package_name/:package_version/:file_name +``` + +| Attribute | Type | Required | Description | +| -------------------| --------------- | ---------| -------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/README.md#namespaced-path-encoding). | +| `package_name` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). +| `package_version` | string | yes | The package version. It can contain only numbers (`0-9`), and dots (`.`). Must be in the format of `X.Y.Z`, i.e. should match `/\A\d+\.\d+\.\d+\z/` regular expresion. +| `file_name` | string | yes | The file name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). + +Provide the file context in the request body. + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + --upload-file path/to/file.txt \ + https://gitlab.example.com/api/v4/projects/24/generic/my_package/0.0.1/file.txt +``` + +Example response: + +```json +{ + "message":"201 Created" +} +``` + +## Download package file + +Install a package file. + +Prerequisites: + +- You need to [authenticate with the API](../../../api/README.md#authentication). + +```plaintext +GET /projects/:id/packages/generic/:package_name/:package_version/:file_name +``` + +| Attribute | Type | Required | Description | +| -------------------| --------------- | ---------| ------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/README.md#namespaced-path-encoding). | +| `package_name` | string | yes | The package name. | +| `package_version` | string | yes | The package version. | +| `file_name` | string | yes | The file name. | + +The file context is served in the response body. The response content type will be `application/octet-stream`. + +Example request that uses a personal access token: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + https://gitlab.example.com/api/v4/projects/24/generic/my_package/0.0.1/file.txt +``` + +## Publish a generic package by using CI/CD + +To work with generic packages in [GitLab CI/CD](./../../../ci/README.md), you can use +`CI_JOB_TOKEN` in place of the personal access token in your commands. + +For example: + +```yaml +image: curlimages/curl:latest + +stages: + - upload + - download + +upload: + stage: upload + script: + - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file path/to/file.txt ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/0.0.1/file.txt' + +download: + stage: download + script: + - 'wget --header="JOB-TOKEN: $CI_JOB_TOKEN" ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/my_package/0.0.1/file.txt' +``` + +### Enable or disable generic packages in the Package Registry + +Support for generic packages is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +# For the instance +Feature.enable(:generic_packages) +# For a single project +Feature.enable(:generic_packages, Project.find(<project id>)) +``` + +To disable it: + +```ruby +# For the instance +Feature.disable(:generic_packages) +# For a single project +Feature.disable(:generic_packages, Project.find(<project id>)) +``` diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index edf1528a751..bd3b5b49ebd 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -50,7 +50,7 @@ Feature.disable(:go_proxy, Project.find(2)) ### Enable the Package Registry The Package Registry is enabled for new projects by default. If you cannot find -the **{package}** **Packages > List** entry under your project's sidebar, verify +the **Packages > List** entry under your project's sidebar, verify the following: 1. Your GitLab administrator has diff --git a/doc/user/packages/index.md b/doc/user/packages/index.md index 92d31c31986..53ba3d01b3e 100644 --- a/doc/user/packages/index.md +++ b/doc/user/packages/index.md @@ -23,6 +23,7 @@ The Package Registry supports the following formats: <tr><td><a href="https://docs.gitlab.com/ee/user/packages/npm_registry/index.html">NPM</a></td><td>11.7+</td></tr> <tr><td><a href="https://docs.gitlab.com/ee/user/packages/nuget_repository/index.html">NuGet</a></td><td>12.8+</td></tr> <tr><td><a href="https://docs.gitlab.com/ee/user/packages/pypi_repository/index.html">PyPI</a></td><td>12.10+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/generic_packages/index.html">Generic packages</a></td><td>13.5+</td></tr> </table> </div> </div> diff --git a/doc/user/packages/maven_repository/img/maven_package_view_v12_6.png b/doc/user/packages/maven_repository/img/maven_package_view_v12_6.png Binary files differdeleted file mode 100644 index 92cefc26660..00000000000 --- a/doc/user/packages/maven_repository/img/maven_package_view_v12_6.png +++ /dev/null diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 7329725a643..d4a8ff0cdb4 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -4,44 +4,24 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Maven Repository +# Maven packages in the Package Repository > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5811) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Core in 13.3. -With the GitLab [Maven](https://maven.apache.org) Repository, every -project can have its own space to store its Maven artifacts. +Publish [Maven](https://maven.apache.org) artifacts in your project’s Package Registry. +Then, install the packages whenever you need to use them as a dependency. -![GitLab Maven Repository](img/maven_package_view_v12_6.png) +## Build a Maven package -## Enabling the Maven Repository +This section explains how to install Maven and build a package. -NOTE: **Note:** -This option is available only if your GitLab administrator has -[enabled support for the Maven repository](../../../administration/packages/index.md). - -After the Packages feature is enabled, the Maven Repository is available for -all new projects by default. To enable it for existing projects, or if you want -to disable it: - -1. Navigate to your project's **Settings > General > Visibility, project features, permissions**. -1. Find the Packages feature and enable or disable it. -1. Click on **Save changes** for the changes to take effect. - -You should then be able to see the **Packages & Registries** section on the left sidebar. -Next, you must configure your project to authorize with the GitLab Maven -repository. +If you already use Maven and know how to build your own packages, go to the +[next section](#add-the-package-registry-as-a-maven-remote). -## Getting Started with Maven +Maven repositories work well with Gradle, too. To set up a Gradle project, see [get started with Gradle](#use-gradle-to-create-a-java-project). -This section covers installing Maven and building a package. This is a -quickstart to help if you're new to building Maven packages. If you're already -using Maven and understand how to build your own packages, move onto the -[next section](#adding-the-gitlab-package-registry-as-a-maven-remote). - -Maven repositories work well with Gradle, too. Move onto [getting started with Gradle](#getting-started-with-gradle) if you want to setup a Gradle project. - -### Installing Maven +### Install Maven The required minimum versions are: @@ -66,7 +46,7 @@ Default locale: en_GB, platform encoding: UTF-8 OS name: "mac os x", version: "10.15.2", arch: "x86_64", family: "mac" ``` -### Creating a project +### Create a project Understanding how to create a full Java project is outside the scope of this guide but you can follow the steps below to create a new project that can be @@ -105,14 +85,14 @@ your project has been set up successfully: You should see a new directory where you ran this command matching your `DartifactId` parameter (in this case it should be `my-project`). -## Getting started with Gradle +## Use Gradle to create a Java project + +This section explains how to install Gradle and initialize a Java project. -This section covers installing Gradle and initializing a Java project. This is a -quickstart to help if you're new to Gradle. If you're already -using Gradle and understand how to build your own packages, move onto the -[next section](#adding-the-gitlab-package-registry-as-a-maven-remote). +If you already use Gradle and know how to build your own packages, go to the +[next section](#add-the-package-registry-as-a-maven-remote). -### Installing Gradle +### Install Gradle Installation is needed only if you want to create a new Gradle project. Follow instructions at [gradle.org](https://gradle.org/install/) to download and install @@ -145,7 +125,7 @@ JVM: 11.0.5 (Oracle Corporation 11.0.5+10) OS: Windows 10 10.0 amd64 ``` -### Creating a project in Gradle +### Create a Java project Understanding how to create a full Java project in Gradle is outside the scope of this guide, but you can follow the steps below to create a new project that can be @@ -209,23 +189,23 @@ Project name (default: test): Enter a project name or hit enter to use the directory name as project name. -## Adding the GitLab Package Registry as a Maven remote +## Add the Package Registry as a Maven remote The next step is to add the GitLab Package Registry as a Maven remote. If a project is private or you want to upload Maven artifacts to GitLab, credentials must be provided for authorization too. Support is available -for [personal access tokens](#authenticating-with-a-personal-access-token), -[CI job tokens](#authenticating-with-a-ci-job-token), and +for [personal access tokens](#authenticate-with-a-personal-access-token), +[CI job tokens](#authenticate-with-a-ci-job-token), and [deploy tokens](../../project/deploy_tokens/index.md) only. Regular username/password credentials do not work. -### Authenticating with a personal access token +### Authenticate with a personal access token To authenticate with a [personal access token](../../profile/personal_access_tokens.md), set the scope to `api` when creating one, and add it to your Maven or Gradle configuration files. -#### Authenticating with a personal access token in Maven +#### Authenticate with a personal access token in Maven Add a corresponding section to your [`settings.xml`](https://maven.apache.org/settings.html) file: @@ -248,7 +228,7 @@ Add a corresponding section to your </settings> ``` -#### Authenticating with a personal access token in Gradle +#### Authenticate with a personal access token in Gradle Create a file `~/.gradle/gradle.properties` with the following content: @@ -278,12 +258,12 @@ repositories { You should now be able to upload Maven artifacts to your project. -### Authenticating with a CI job token +### Authenticate with a CI job token If you're using GitLab CI/CD, a CI job token can be used instead of a personal access token. -#### Authenticating with a CI job token in Maven +#### Authenticate with a CI job token in Maven To authenticate with a CI job token, add a corresponding section to your [`settings.xml`](https://maven.apache.org/settings.html) file: @@ -309,7 +289,7 @@ To authenticate with a CI job token, add a corresponding section to your You can read more on [how to create Maven packages using GitLab CI/CD](#creating-maven-packages-with-gitlab-cicd). -#### Authenticating with a CI job token in Gradle +#### Authenticate with a CI job token in Gradle To authenticate with a CI job token, add a repositories section to your [`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) @@ -331,7 +311,7 @@ repositories { } ``` -### Authenticating with a deploy token +### Authenticate with a deploy token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213566) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. @@ -339,7 +319,7 @@ To authenticate with a [deploy token](./../../project/deploy_tokens/index.md), set the scope to `api` when creating one, and add it to your Maven or Gradle configuration files. -#### Authenticating with a deploy token in Maven +#### Authenticate with a deploy token in Maven Add a corresponding section to your [`settings.xml`](https://maven.apache.org/settings.html) file: @@ -362,7 +342,7 @@ Add a corresponding section to your </settings> ``` -#### Authenticating with a deploy token in Gradle +#### Authenticate with a deploy token in Gradle To authenticate with a deploy token, add a repositories section to your [`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) @@ -441,7 +421,7 @@ repositories { ``` The `id` must be the same with what you -[defined in `settings.xml`](#adding-the-gitlab-package-registry-as-a-maven-remote). +[defined in `settings.xml`](#add-the-package-registry-as-a-maven-remote). Replace `PROJECT_ID` with your project ID which can be found on the home page of your project. @@ -452,7 +432,7 @@ domain name. NOTE: **Note:** For retrieving artifacts, you can use either the [URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project -(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +(such as `group%2Fproject`) or the project's ID (such as `42`). However, only the project's ID can be used for uploading. ### Group level Maven endpoint @@ -505,7 +485,7 @@ repositories { ``` The `id` must be the same with what you -[defined in `settings.xml`](#adding-the-gitlab-package-registry-as-a-maven-remote). +[defined in `settings.xml`](#add-the-package-registry-as-a-maven-remote). Replace `my-group` with your group name and `PROJECT_ID` with your project ID which can be found on the home page of your project. @@ -516,7 +496,7 @@ domain name. NOTE: **Note:** For retrieving artifacts, you can use either the [URL encoded](../../../api/README.md#namespaced-path-encoding) path of the group -(e.g., `group%2Fsubgroup`) or the group's ID (e.g., `12`). +(such as `group%2Fsubgroup`) or the group's ID (such as `12`). ### Instance level Maven endpoint @@ -571,7 +551,7 @@ repositories { ``` The `id` must be the same with what you -[defined in `settings.xml`](#adding-the-gitlab-package-registry-as-a-maven-remote). +[defined in `settings.xml`](#add-the-package-registry-as-a-maven-remote). Replace `PROJECT_ID` with your project ID which can be found on the home page of your project. @@ -582,12 +562,12 @@ domain name. NOTE: **Note:** For retrieving artifacts, you can use either the [URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project -(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +(such as `group%2Fproject`) or the project's ID (such as `42`). However, only the project's ID can be used for uploading. ## Uploading packages -Once you have set up the [remote and authentication](#adding-the-gitlab-package-registry-as-a-maven-remote) +Once you have set up the [remote and authentication](#add-the-package-registry-as-a-maven-remote) and [configured your project](#configuring-your-project-to-use-the-gitlab-maven-repository-url), test to upload a Maven artifact from a project of yours. @@ -661,7 +641,7 @@ artifacts or even delete them. ## Installing a package Installing a package from the GitLab Package Registry requires that you set up -the [remote and authentication](#adding-the-gitlab-package-registry-as-a-maven-remote) +the [remote and authentication](#add-the-package-registry-as-a-maven-remote) as above. Once this is completed, there are two ways to install a package. ### Install using Maven with `mvn install` @@ -708,7 +688,7 @@ Package details page, allowing for quick and easy installation. ### Install using Gradle -Add a [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html) to build.gradle in the dependencies section: +Add a [dependency](https://docs.gradle.org/current/userguide/declaring_dependencies.html) to `build.gradle` in the dependencies section: ```groovy dependencies { @@ -802,7 +782,7 @@ Docker container), and Maven uses the configured CI The example below shows how to create a new package each time the `master` branch is updated: -1. Make sure you use the Job-Token authentication as described in ["Authenticating with a CI job token in Gradle"](#authenticating-with-a-ci-job-token-in-gradle). +1. Make sure you use the Job-Token authentication as described in ["Authenticating with a CI job token in Gradle"](#authenticate-with-a-ci-job-token-in-gradle). 1. Add a `deploy` job to your `.gitlab-ci.yml` file: diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index 2a1c12c2afd..f15b31d8b67 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -286,6 +286,22 @@ By default, when an NPM package is not found in the GitLab NPM Registry, the req Administrators can disable this behavior in the [Continuous Integration settings](../../admin_area/settings/continuous_integration.md). +### Installing packages from other organizations + +You can route package requests to organizations and users outside of GitLab. + +To do this, add lines to your `.npmrc` file, replacing `my-org` with the namespace or group that owns your project's repository. The name is case-sensitive and must match the name of your group or namespace exactly. + +```shell +@foo:registry=https://gitlab.example.com/api/v4/packages/npm/ +//gitlab.com/api/v4/packages/npm/:_authToken= "<your_token>" +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" + +@my-other-org:registry=https://gitlab.example.com/api/v4/packages/npm/ +//gitlab.com/api/v4/packages/npm/:_authToken= "<your_token>" +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" +``` + ## Removing a package In the packages view of your project page, you can delete packages by clicking @@ -340,6 +356,13 @@ with your personal access token or deploy token): //gitlab.com/api/v4/projects/:_authToken=<your_token> ``` +You can also use `yarn config` instead of `npm config` when setting your auth-token dynamically: + +```shell +yarn config set '//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "<your_token>" +yarn config set '//gitlab.com/api/v4/packages/npm/:_authToken' "<your_token>" +``` + ### `npm publish` targets default NPM registry (`registry.npmjs.org`) Ensure that your package scope is set consistently in your `package.json` and `.npmrc` files. @@ -418,5 +441,8 @@ npm dist-tag rm @scope/package@version my-tag # Delete a tag from the package npm install @scope/package@my-tag # Install a specific tag ``` +NOTE: **Note:** +You cannot use your `CI_JOB_TOKEN` or deploy token with the `npm dist-tag` commands. View [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/258835) for details. + CAUTION: **Warning:** Due to a bug in NPM 6.9.0, deleting dist tags fails. Make sure your NPM version is greater than 6.9.1. diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 9fb50ce71fb..22e1a95649d 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -19,7 +19,7 @@ The GitLab NuGet Repository works with: ## Setting up your development environment -[NuGet CLI 5.2 or later](https://www.nuget.org/downloads) is required. Earlier versions have not been tested +[NuGet CLI 5.1 or later](https://www.nuget.org/downloads) is required. Earlier versions have not been tested against the GitLab NuGet Repository and might not work. If you have [Visual Studio](https://visualstudio.microsoft.com/vs/), NuGet CLI is probably already installed. @@ -34,7 +34,7 @@ nuget help You should see something similar to: ```plaintext -NuGet Version: 5.2.0.6090 +NuGet Version: 5.1.0.6013 usage: NuGet <command> [args] [options] Type 'NuGet help <command>' for help on a specific command. @@ -44,7 +44,7 @@ Available commands: ``` NOTE: **Note:** -GitLab currently only supports NuGet v3. Earlier versions are not supported. +GitLab currently only supports NuGet's protocol version 3. Earlier versions are not supported. ### macOS support @@ -154,7 +154,7 @@ To add the GitLab NuGet Repository as a source for .NET, create a file named `nu When uploading packages, note that: -- The maximum allowed size is 50 Megabytes. +- The Package Registry on GitLab.com can store up to 500 MB of content. This limit is [configurable for self-managed GitLab instances](../../../administration/instance_limits.md#package-registry-limits). - If you upload the same package with the same version multiple times, each consecutive upload is saved as a separate file. When installing a package, GitLab serves the most recent file. - When uploading packages to GitLab, they are not displayed in the packages UI of your project @@ -250,21 +250,21 @@ is updated: 1. Add a `deploy` job to your `.gitlab-ci.yml` file: ```yaml - image: mcr.microsoft.com/dotnet/core/sdk:3.1 - - stages: - - deploy - - deploy: - stage: deploy - script: - - dotnet restore -p:Configuration=Release - - dotnet build -c Release - - dotnet pack -c Release - - dotnet nuget add source "$CI_SERVER_URL/api/v4/projects/$CI_PROJECT_ID/packages/nuget/index.json" --name gitlab --username gitlab-ci-token --password $CI_JOB_TOKEN --store-password-in-clear-text - - dotnet nuget push "bin/Release/*.nupkg" --source gitlab - only: - - master + image: mcr.microsoft.com/dotnet/core/sdk:3.1 + + stages: + - deploy + + deploy: + stage: deploy + script: + - dotnet restore -p:Configuration=Release + - dotnet build -c Release + - dotnet pack -c Release + - dotnet nuget add source "$CI_SERVER_URL/api/v4/projects/$CI_PROJECT_ID/packages/nuget/index.json" --name gitlab --username gitlab-ci-token --password $CI_JOB_TOKEN --store-password-in-clear-text + - dotnet nuget push "bin/Release/*.nupkg" --source gitlab + only: + - master ``` 1. Commit the changes and push it to your GitLab repository to trigger a new CI build. diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index 9f954627b05..ae9ca5b2333 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -17,7 +17,7 @@ packages, which can be easily consumed as a dependency in downstream projects. You can view packages for your project or group. 1. Go to the project or group. -1. Go to **{package}** **Packages & Registries > Package Registry**. +1. Go to **Packages & Registries > Package Registry**. You can search, sort, and filter packages on this page. @@ -31,7 +31,7 @@ authenticate with GitLab by using the `CI_JOB_TOKEN`. CI/CD templates, which you can use to get started, are in [this repo](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). -Learn more about [using CI/CD to build Maven packages](../maven_repository/index.md#creating-maven-packages-with-gitlab-cicd), [NPM packages](../npm_registry/index.md#publishing-a-package-with-cicd), [Composer packages](../composer_repository/index.md#publishing-the-package-with-cicd), [NuGet Packages](../nuget_repository/index.md#publishing-a-nuget-package-with-cicd), [Conan Packages](../conan_repository/index.md#using-gitlab-ci-with-conan-packages), and [PyPI packages](../pypi_repository/index.md#using-gitlab-ci-with-pypi-packages). +Learn more about [using CI/CD to build Maven packages](../maven_repository/index.md#creating-maven-packages-with-gitlab-cicd), [NPM packages](../npm_registry/index.md#publishing-a-package-with-cicd), [Composer packages](../composer_repository/index.md#publish-a-composer-package-by-using-cicd), [NuGet Packages](../nuget_repository/index.md#publishing-a-nuget-package-with-cicd), [Conan Packages](../conan_repository/index.md#publish-a-conan-package-by-using-cicd), [PyPI packages](../pypi_repository/index.md#using-gitlab-ci-with-pypi-packages), and [generic packages](../generic_packages/index.md#publish-a-generic-package-by-using-cicd). If you use CI/CD to build a package, extended activity information is displayed when you view the package details: @@ -45,7 +45,7 @@ user who triggered it. To download a package: -1. Go to **{package}** **Packages & Registries > Package Registry**. +1. Go to **Packages & Registries > Package Registry**. 1. Click the name of the package you want to download. 1. In the **Activity** section, click the name of the package you want to download. @@ -60,7 +60,7 @@ You can delete packages by using [the API](../../../api/packages.md#delete-a-pro To delete a package in the UI, from your group or project: -1. Go to **{package}** **Packages & Registries > Package Registry**. +1. Go to **Packages & Registries > Package Registry**. 1. Find the name of the package you want to delete. 1. Click **Delete**. @@ -71,7 +71,7 @@ The package is permanently deleted. The Package Registry is automatically enabled. If you are using a self-managed instance of GitLab, your administrator can remove -the menu item, **{package}** **Packages & Registries**, from the GitLab sidebar. For more information, +the menu item, **Packages & Registries**, from the GitLab sidebar. For more information, see the [administration documentation](../../../administration/packages/index.md). You can also remove the Package Registry for your project specifically: @@ -81,7 +81,7 @@ You can also remove the Package Registry for your project specifically: **Packages** feature. 1. Click **Save changes**. -The **{package}** **Packages & Registries > Package Registry** entry is removed from the sidebar. +The **Packages & Registries > Package Registry** entry is removed from the sidebar. ## Package workflows diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 97f3f69d676..33c37ee6a6c 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -4,14 +4,14 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab PyPi Repository +# GitLab PyPI Repository > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208747) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.10. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Core in 13.3. -With the GitLab PyPi Repository, every project can have its own space to store PyPi packages. +With the GitLab PyPI Repository, every project can have its own space to store PyPI packages. -The GitLab PyPi Repository works with: +The GitLab PyPI Repository works with: - [pip](https://pypi.org/project/pip/) - [twine](https://pypi.org/project/twine/) @@ -20,13 +20,13 @@ The GitLab PyPi Repository works with: You need a recent version of [pip](https://pypi.org/project/pip/) and [twine](https://pypi.org/project/twine/). -## Enabling the PyPi Repository +## Enabling the PyPI Repository NOTE: **Note:** This option is available only if your GitLab administrator has [enabled support for the Package Registry](../../../administration/packages/index.md). -After the PyPi Repository is enabled, it is available for all new projects +After the PyPI Repository is enabled, it is available for all new projects by default. To enable it for existing projects, or if you want to disable it: 1. Navigate to your project's **Settings > General > Visibility, project features, permissions**. @@ -37,8 +37,8 @@ You should then be able to see the **Packages & Registries** section on the left ## Getting started -This section covers creating a new example PyPi package to upload. This is a -quickstart to test out the **GitLab PyPi Registry**. If you already understand how +This section covers creating a new example PyPI package to upload. This is a +quickstart to test out the **GitLab PyPI Registry**. If you already understand how to build and publish your own packages, move on to the [next section](#adding-the-gitlab-pypi-repository-as-a-source). ### Create a project @@ -111,6 +111,11 @@ touch setup.py This file contains all the information about our package. For more information about this file, see [creating setup.py](https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py). +GitLab identifies packages based on +[Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), +so ensure your package name meets these requirements. +See the [installation section](#install-packages) for more details. + For this guide, we don't need to extensively fill out this file, simply add the below to your `setup.py`: @@ -152,10 +157,10 @@ And confirm your output matches the below: mypypipackage-0.0.1-py3-none-any.whl mypypipackage-0.0.1.tar.gz ``` -Our package is now all set up and ready to be uploaded to the **GitLab PyPi +Our package is now all set up and ready to be uploaded to the **GitLab PyPI Package Registry**. Before we do so, we next need to set up authentication. -## Adding the GitLab PyPi Repository as a source +## Adding the GitLab PyPI Repository as a source ### Authenticating with a personal access token @@ -256,7 +261,7 @@ TWINE_PASSWORD=<personal_access_token or deploy_token> TWINE_USERNAME=<username ``` If you did not follow the guide above, then you need to ensure your package -has been properly built and you [created a PyPi package with `setuptools`](https://packaging.python.org/tutorials/packaging-projects/). +has been properly built and you [created a PyPI package with `setuptools`](https://packaging.python.org/tutorials/packaging-projects/). You can then upload your package using the following command: @@ -274,7 +279,7 @@ Where: Install the latest version of a package using the following command: ```shell -pip install --index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple --no-deps <package_name> +pip install --extra-index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple --no-deps <package_name> ``` Where: @@ -287,7 +292,7 @@ If you were following the guide above and want to test installing the `MyPyPiPackage` package, you can run the following: ```shell -pip install mypypipackage --no-deps --index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/simple +pip install mypypipackage --no-deps --extra-index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/simple ``` This should result in the following: @@ -300,6 +305,12 @@ Installing collected packages: mypypipackage Successfully installed mypypipackage-0.0.1 ``` +GitLab looks for packages using +[Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), +so the characters `-`, `_`, and `.` are all treated the same and repeated characters are removed. +A `pip install` request for `my.package` looks for packages that match any of +the three characters, such as `my-package`, `my_package`, and `my....package`. + ## Using GitLab CI with PyPI packages > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202012) in GitLab 13.4. diff --git a/doc/user/packages/workflows/monorepo.md b/doc/user/packages/workflows/monorepo.md index c87f181bf82..f20f3427ac5 100644 --- a/doc/user/packages/workflows/monorepo.md +++ b/doc/user/packages/workflows/monorepo.md @@ -7,17 +7,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Monorepo package management workflows Oftentimes, one project or Git repository may contain multiple different -subprojects or submodules that all get packaged and published individually. +sub-projects or submodules that all get packaged and published individually. ## Publishing different packages to the parent project The number and name of packages you can publish to one project is not limited. You can accomplish this by setting up different configuration files for each package. See the documentation for the package manager of your choice since -each will have its own specific files and instructions to follow to publish +each has its own specific files and instructions to follow to publish a given package. -Here, we will walk through how to do this with [NPM](../npm_registry/index.md). +Here, we take a walk through how to do this with [NPM](../npm_registry/index.md). Let us say we have a project structure like so: @@ -36,27 +36,27 @@ as well as `Foo`. Following the instructions in the [GitLab NPM registry documentation](../npm_registry/index.md), publishing `MyProject` consists of modifying the `package.json` file with a -`publishConfig` section, as well as either modifying your local NPM config with +`publishConfig` section, as well as either modifying your local NPM configuration with CLI commands like `npm config set`, or saving a `.npmrc` file in the root of the -project specifying these config settings. +project specifying these configuration settings. If you follow the instructions you can publish `MyProject` by running `npm publish` from the root directory. Publishing `Foo` is almost exactly the same, you simply have to follow the steps -while in the `Foo` directory. `Foo` will need its own `package.json` file, -which can be added manually or using `npm init`. And it will need its own +while in the `Foo` directory. `Foo` needs its own `package.json` file, +which can be added manually or using `npm init`. It also needs its own configuration settings. Since you are publishing to the same place, if you used `npm config set` to set the registry for the parent project, then no -additional setup is necessary. If you used a `.npmrc` file, you will need an +additional setup is necessary. If you used a `.npmrc` file, you need an additional `.npmrc` file in the `Foo` directory (be sure to add `.npmrc` files to the `.gitignore` file or use environment variables in place of your access tokens to prevent them from being exposed). It can be identical to the one you used in `MyProject`. You can now run `npm publish` from the `Foo` -directory and you will be able to publish `Foo` separately from `MyProject` +directory and you can publish `Foo` separately from `MyProject` A similar process could be followed for Conan packages, instead of dealing with -`.npmrc` and `package.json`, you will just be dealing with `conanfile.py` in +`.npmrc` and `package.json`, you just deal with `conanfile.py` in multiple locations within the project. ## Publishing to other projects @@ -64,9 +64,9 @@ multiple locations within the project. A package is associated with a project on GitLab, but the package does not need to be associated with the code in that project. Notice when configuring NPM or Maven, you only use the `Project ID` to set the registry URL that the -package will be uploaded to. If you set this to any project that you have -access to and update any other config similarly depending on the package type, -your packages will be published to that project. This means you can publish +package is to be uploaded to. If you set this to any project that you have +access to and update any other configuration similarly depending on the package type, +your packages are published to that project. This means you can publish multiple packages to one project, even if their code does not exist in the same place. See the [project registry workflow documentation](./project_registry.md) for more details. @@ -77,7 +77,7 @@ CI pipelines open an entire world of possibilities for dealing with the patterns described in the previous sections. A common desire would be to publish specific packages only if changes were made to those directories. -Using the example project above, this `gitlab-ci.yml` file will publish +Using the example project above, this `gitlab-ci.yml` file publishes `Foo` anytime changes are made to the `Foo` directory on the `master` branch, and publish `MyPackage` anytime changes are made to anywhere _except_ the `Foo` directory on the `master` branch. diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index 571cda09e69..24437e6dfac 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -26,8 +26,8 @@ point them at the same project on GitLab. There are a few reasons you might want to publish all your packages to one project on GitLab: 1. You want to publish your packages on GitLab, but to a project that is different from where your code is stored. -1. You would like to group packages together in ways that make sense for your usage (all NPM packages in one project, - all packages being used by a specific department in one project, all private packages in one project, etc.) +1. You would like to group packages together in ways that make sense for your usage (such as all NPM packages in one project, + all packages being used by a specific department in one project, or all private packages in one project) 1. You want to use one remote for all of your packages when installing them into other projects. 1. You would like to migrate your packages to a single place on GitLab from a third-party package registry and do not want to worry about setting up separate projects for each package. @@ -44,7 +44,7 @@ Let's take a look at how you might create a public place to hold all of your pub ### Create a project First, create a new project on GitLab. It does not have to have any code or content. Make note of the project ID -displayed on the project overview page, as you will need this later. +displayed on the project overview page for use later in this process. ### Create an access token @@ -67,24 +67,24 @@ If you are using NPM, this involves creating an `.npmrc` file and adding the app to your project using your project ID, then adding a section to your `package.json` file with a similar URL. Follow -the instructions in the [GitLab NPM Registry documentation](../npm_registry/index.md#authenticating-to-the-gitlab-npm-registry). Once -you do this, you will be able to push your NPM package to your project using `npm publish`, as described in the +the instructions in the [GitLab NPM Registry documentation](../npm_registry/index.md#authenticating-to-the-gitlab-npm-registry). After +you do this, you can push your NPM package to your project using `npm publish`, as described in the [uploading packages](../npm_registry/index.md#uploading-packages) section of the docs. #### Maven If you are using Maven, this involves updating your `pom.xml` file with distribution sections, including the appropriate URL for your project, as described in the [GitLab Maven Repository documentation](../maven_repository/index.md#project-level-maven-endpoint). -Then, you need to add a `settings.xml` file and [include your access token](../maven_repository/index.md#authenticating-with-a-personal-access-token). +Then, you need to add a `settings.xml` file and [include your access token](../maven_repository/index.md#authenticate-with-a-personal-access-token). Now you can [deploy Maven packages](../maven_repository/index.md#uploading-packages) to your project. #### Conan -For Conan, first you need to add GitLab as a Conan registry remote. Follow the instructions in the [GitLab Conan Repository docs](../conan_repository/index.md#adding-the-gitlab-package-registry-as-a-conan-remote) +For Conan, first you need to add GitLab as a Conan registry remote. Follow the instructions in the [GitLab Conan Repository docs](../conan_repository/index.md#add-the-package-registry-as-a-conan-remote) to do so. Then, create your package using the plus-separated (`+`) project path as your Conan user. For example, if your project is located at `https://gitlab.com/foo/bar/my-proj`, then you can [create your Conan package](../conan_repository/index.md) -using `conan create . foo+bar+my-proj/channel`, where `channel` is your package channel (`stable`, `beta`, etc.). Once your package -is created, you are ready to [upload your package](../conan_repository/index.md#uploading-a-package) depending on your final package recipe. For example: +using `conan create . foo+bar+my-proj/channel`, where `channel` is your package channel (such as `stable` or `beta`). After your package +is created, you are ready to [upload your package](../conan_repository/index.md#publish-a-conan-package) depending on your final package recipe. For example: ```shell CONAN_LOGIN_USERNAME=<gitlab-username> CONAN_PASSWORD=<personal_access_token> conan upload MyPackage/1.0.0@foo+bar+my-proj/channel --all --remote=gitlab |