diff options
Diffstat (limited to 'doc/ci')
40 files changed, 742 insertions, 309 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 440a79c7782..27bde2ac0f1 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -3,7 +3,7 @@ comments: false description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." --- -# GitLab Continuous Integration (GitLab CI/CD) +# GitLab CI/CD GitLab CI/CD is a tool built into GitLab for software development through the [continuous methodologies](introduction/index.md#introduction-to-cicd-methodologies): @@ -61,12 +61,12 @@ for all the attributes you can set and use. NOTE: **Note:** GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#extra-shared-runners-pipeline-minutes-quota). -## GitLab CI/CD configuration +## Configuration GitLab CI/CD supports numerous configuration options: -| Configuration | Description | -|:--- |:--- | +| Configuration | Description | +|:--------------|:-------------| | [Pipelines](pipelines.md) | Structure your CI/CD process through pipelines. | | [Environment variables](variables/README.md) | Reuse values based on a variable/value key pair. | | [Environments](environments.md) | Deploy your application to different environments (e.g., staging, production). | @@ -74,7 +74,7 @@ GitLab CI/CD supports numerous configuration options: | [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | | [Schedule pipelines](../user/project/pipelines/schedules.md) | Schedule pipelines to run as often as you need. | | [Custom path for `.gitlab-ci.yml`](../user/project/pipelines/settings.md#custom-ci-config-path) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | +| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| | [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | | [Pipelines triggers](triggers/README.md) | Trigger pipelines through the API. | | [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | @@ -85,33 +85,43 @@ GitLab CI/CD supports numerous configuration options: Note that certain operations can only be performed according to the [user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. -## GitLab CI/CD feature set +## Feature set -You can also use the vast GitLab CI/CD feature set to easily configure -it for specific purposes: +Use the vast GitLab CI/CD to easily configure it for specific purposes. +Its feature set is listed on the table below according to DevOps stages. | Feature | Description | -|:--- |:--- | -| [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | +|:--------|:------------| +| **Configure** || | [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | +| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | +|---+---| +| **Verify** || +| [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html) | Quickly determine the performance impact of pending code changes. | +| [CI services](services/README.md) | Link Docker containers with your base image.| +| [Code Quality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** | Analyze your source code quality. | +| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | +| [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | +| [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | +| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | +|---+---| +| **Release** || +| [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | | [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | | [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) **[PREMIUM]** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [CI services](services/README.md)| Link Docker containers with your base image. | -| [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]**| Check your Docker containers for known vulnerabilities. | -| [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]**| Analyze your dependencies for known vulnerabilities. | | [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | | [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) **[PREMIUM]** | Deploy your features behind Feature Flags. | -| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | | [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | -| [JUnit tests](junit_test_reports.md)| Identify script failures directly on merge requests. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | +|---+---| +| **Secure** || +| [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]** | Check your Docker containers for known vulnerabilities.| +| [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [License Management](https://docs.gitlab.com/ee/user/application_security/license_management/) **[ULTIMATE]** | Search your project dependencies for their licenses. | | [Security Test reports](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate) **[ULTIMATE]** | Check for app vulnerabilities. | -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -## GitLab CI/CD examples +## Examples GitLab provides examples of configuring GitLab CI/CD in the form of: @@ -120,7 +130,7 @@ GitLab provides examples of configuring GitLab CI/CD in the form of: - [`multi-project-pipelines`](https://gitlab.com/gitlab-examples/multi-project-pipelines) for examples of implementing multi-project pipelines. - [`review-apps-nginx`](https://gitlab.com/gitlab-examples/review-apps-nginx/) provides an example of using Review Apps. -## GitLab CI/CD administration **[CORE ONLY]** +## Administration **[CORE ONLY]** As a GitLab administrator, you can change the default behavior of GitLab CI/CD for: diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md new file mode 100644 index 00000000000..1c8e12d229c --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -0,0 +1,147 @@ +# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]** + +GitLab CI/CD can be used with Bitbucket Cloud by creating a +[CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) and connecting +your Git repository via URL. + +1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and + create the project. + +  + + GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. + +1. In GitLab create a + [Personal Access Token](../../user/profile/personal_access_tokens.md) + with `api` scope. This will be used to authenticate requests from the web + hook that will be created in Bitbucket to notify GitLab of new commits. + +1. In Bitbucket from **Settings > Webhooks** create a new web hook to notify + GitLab of new commits. + + The web hook URL should be set to the GitLab API to trigger pull mirroring, + using the Personal Access Token we just generated for authentication. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + + The web hook Trigger should be set to 'Repository Push'. + +  + + After saving, test the web hook by pushing a change to your Bitbucket + repository. + +1. In Bitbucket create an **App Password** from **Bitbucket Settings > App + Passwords** to authenticate the build status script setting commit build + statuses in Bitbucket. Repository write permissions are required. + +  + +1. In GitLab from **Settings > CI/CD > Environment variables** add variables to allow + communication with Bitbucket via the Bitbucket API. + + `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above + + `BITBUCKET_USERNAME`: the username of the Bitbucket account + + `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ + + `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ + +1. In Bitbucket add a script to push the pipeline status to Bitbucket. + + > Note: changes made in GitLab will be overwritten by any changes made + upstream in Bitbucket. + + Create a file `build_status` and insert the script below and run + `chmod +x build_status` in your terminal to make the script executable. + + ```bash + #!/usr/bin/env bash + + # Push GitLab CI/CD build status to Bitbucket Cloud + + if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then + echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" + exit 1 + fi + if [ -z "$BITBUCKET_USERNAME" ]; then + echo "ERROR: BITBUCKET_USERNAME is not set" + exit 1 + fi + if [ -z "$BITBUCKET_NAMESPACE" ]; then + echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" + BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE + fi + if [ -z "$BITBUCKET_REPOSITORY" ]; then + echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" + BITBUCKET_REPOSITORY=$CI_PROJECT_NAME + fi + + BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" + BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" + BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" + + case "$BUILD_STATUS" in + running) + BITBUCKET_STATE="INPROGRESS" + BITBUCKET_DESCRIPTION="The build is running!" + ;; + passed) + BITBUCKET_STATE="SUCCESSFUL" + BITBUCKET_DESCRIPTION="The build passed!" + ;; + failed) + BITBUCKET_STATE="FAILED" + BITBUCKET_DESCRIPTION="The build failed." + ;; + esac + + echo "Pushing status to $BITBUCKET_STATUS_API..." + curl --request POST $BITBUCKET_STATUS_API \ + --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ + --header "Content-Type:application/json" \ + --silent \ + --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": + \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" + ``` + +1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push + pipeline success and failures to Bitbucket. + + ``` + stages: + - test + - ci_status + + unit-tests: + script: + - echo "Success. Add your tests!" + + success: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=passed BUILD_KEY=push ./build_status + when: on_success + + failure: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=failed BUILD_KEY=push ./build_status + when: on_failure + ``` + +GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines +configured in `.gitlab-ci.yml` and push the status to Bitbucket. + +[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md new file mode 100644 index 00000000000..0e2acf957e0 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -0,0 +1,111 @@ +# Using GitLab CI/CD with a GitHub repository **[PREMIUM]** + +GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by +creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) to connect your GitHub repository to +GitLab. + +NOTE: **Note:** +To use **GitHub Enterprise** with **GitLab.com** you should use the +manual method. + +## Connect with GitHub integration + +If the [GitHub integration](../../integration/github.md) has been enabled by your GitLab +administrator: + +NOTE: **Note:** +Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps), +if you import more than 10 times, your oldest imported project's token will be +revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147) +for more information. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Once authenticated, you will be redirected to a list of your repositories to + connect. Click **Connect** to select the repository. + +  + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook +on GitHub to notify GitLab of new commits. + +## Connect with Personal Access Token + +NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com +repositories to GitLab. + +If you are not using the [GitHub integration](../../integration/github.md), you can +still perform a one-off authorization with GitHub to grant GitLab access your +repositories: + +1. Open https://github.com/settings/tokens/new to create a **Personal Access + Token**. This token with be used to access your repository and push commit + statuses to GitHub. + + The `repo` and `admin:repo_hook` should be enable to allow GitLab access to + your project, update commit statuses, and create a web hook to notify + GitLab of new commits. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Paste the token into the **Personal access token** field and click **List + Repositories**. Click **Connect** to select the repository. + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook +on GitHub to notify GitLab of new commits. + +## Connect manually + +If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled +for a different GitHub instance, you GitLab CI/CD can be manually enabled for +your repository. + +1. In GitHub open https://github.com/settings/tokens/new create a **Personal + Access Token.** GitLab will use this token to access your repository and + push commit statuses. + + Enter a **Token description** and update the scope to allow: + + `repo` so that GitLab can access your project and update commit statuses + +1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS + URL for your GitHub repository. If your project is private, use the personal + access token you just created for authentication. + + GitLab will automatically configure polling-based pull mirroring. + +1. Still in GitLab, enable the [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html) + from **Settings > Integrations.** + + Check the **Active** checkbox to enable the integration, paste your + personal access token and HTTPS repository URL into the form, and **Save.** + +1. Still in GitLab create a **Personal Access Token** with `API` scope to + authenticate the GitHub web hook notifying GitLab of new commits. + +1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of + new commits. + + The web hook URL should be set to the GitLab API to + [trigger pull mirroring](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter), + using the GitLab personal access token we just created. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + +  + +1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png Binary files differnew file mode 100644 index 00000000000..ac5be3c3058 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png Binary files differnew file mode 100644 index 00000000000..0a3476d9035 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png Binary files differnew file mode 100644 index 00000000000..f068688146b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png diff --git a/doc/ci/ci_cd_for_external_repos/img/external_repository.png b/doc/ci/ci_cd_for_external_repos/img/external_repository.png Binary files differnew file mode 100644 index 00000000000..b850d91f56b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/external_repository.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png Binary files differnew file mode 100644 index 00000000000..71a3a057a41 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png Binary files differnew file mode 100644 index 00000000000..3f2059504f5 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png Binary files differnew file mode 100644 index 00000000000..e8c17d664e1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png Binary files differnew file mode 100644 index 00000000000..73579dd3cf1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md new file mode 100644 index 00000000000..450ffe512dc --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -0,0 +1,31 @@ +# GitLab CI/CD for external repositories **[PREMIUM]** + +NOTE: **Note:** +This feature [is available for free](https://about.gitlab.com/2019/03/21/six-more-months-ci-cd-github/) to +GitLab.com users until September 22nd, 2019. + +>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6. + +GitLab CI/CD can be used with GitHub or any other Git server. +Instead of moving your entire project to GitLab, you can connect your +external repository to get the benefits of GitLab CI/CD. + +- [GitHub](github_integration.md) +- [Bitbucket Cloud](bitbucket_integration.md) + +Connecting an external repository will set up [repository mirroring][mirroring] +and create a lightweight project where issues, merge requests, wiki, and +snippets disabled. These features +[can be re-enabled later][settings]. + +1. From your GitLab dashboard click **New project** +1. Switch to the **CI/CD for external repo** tab +1. Choose **GitHub** or **Repo by URL** +1. The next steps are similar to the [import flow](../../user/project/import/index.md) + + + +[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642 +[eep]: https://about.gitlab.com/pricing/ +[mirroring]: ../../workflow/repository_mirroring.md +[settings]: ../../user/project/settings/index.md#sharing-and-permissions diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 6b4d4f1b9d4..d5e6fbe8113 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -669,7 +669,7 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/* Some GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) features can behave differently for each environment. For example, you can -[create a secret variable to be injected only into a production environment](https://docs.gitlab.com/ee/ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium). +[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables-premium). In most cases, these features use the _environment specs_ mechanism, which offers an efficient way to implement scoping within each environment group. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 7f686781e3c..6b9f8181d1d 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -18,30 +18,30 @@ Examples are available in several forms. As a collection of: The following table lists examples for different use cases: -| Use case | Resource | -|:-----------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------| -| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | -| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | -| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | -| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | -| Dependency scanning | [Dependency Scanning with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html). **[ULTIMATE]** | -| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | -| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | -| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | -| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | -| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | -| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | -| JUnit | [JUnit test reports](../junit_test_reports.md). | -| License management | [Dependencies license management with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/license_management.html) **[ULTIMATE]** | -| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | -| PHP | [Testing PHP projects](php.md). | -| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | -| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | -| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | -| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | -| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/sast.html) **[ULTIMATE]** | -| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | +| Use case | Resource | +|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------| +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | +| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | +| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | +| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | +| Dependency scanning | [Dependency Scanning with GitLab CI/CD](dependency_scanning.md). **[ULTIMATE]** | +| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | +| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | +| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | +| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | +| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | +| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | +| JUnit | [JUnit test reports](../junit_test_reports.md). | +| License management | [Dependencies license management with GitLab CI/CD](license_management.md) **[ULTIMATE]** | +| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | +| PHP | [Testing PHP projects](php.md). | +| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | +| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | +| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | +| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](sast.md) **[ULTIMATE]** | +| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | ### Contributing examples diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index b47038011de..442d0788d37 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -56,7 +56,7 @@ provide a list of URLs to test, please consult TIP: **Tip:** For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically extracted and shown right in the merge request widget. -[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html). +[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ## Performance testing on Review Apps diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index 36fdc29fe65..4d41e424f4a 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -1,119 +1,5 @@ -# Container Scanning with GitLab CI/CD +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +--- -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -It also requires the GitLab Runner 11.5 or later. -For earlier versions, use the [previous job definitions](#previous-job-definitions). - -You can check your Docker images (or more precisely the containers) for known -vulnerabilities by using [Clair](https://github.com/coreos/clair) and -[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools -for Vulnerability Static Analysis for containers. - -First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: - -```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - reports: - container_scanning: gl-container-scanning-report.json -``` - -The above example will create a `container_scanning` job in your CI/CD pipeline, pull -the image from the [Container Registry](../../user/project/container_registry.md) -(whose name is defined from the two `CI_APPLICATION_` variables) and scan it -for possible vulnerabilities. The report will be saved as a -[Container Scanning report artifact](../yaml/README.md#artifactsreportscontainer_scanning-ultimate) -that you can later download and analyze. -Due to implementation limitations we always take the latest Container Scanning artifact available. - -If you want to whitelist some specific vulnerabilities, you can do so by defining -them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), -in our case its named `clair-whitelist.yml`. - -TIP: **Tip:** -For [GitLab Ultimate][ee] users, this information will -be automatically extracted and shown right in the merge request widget. -[Learn more on Container Scanning in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). - -CAUTION: **Caution:** -Starting with GitLab 11.5, Container Scanning feature is licensed under the name `container_scanning`. -While the old name `sast_container` is still maintained, it has been deprecated with GitLab 11.5 and -may be removed in next major release, GitLab 12.0. You are advised to update your current `.gitlab-ci.yml` -configuration to reflect that change if you are using the `$GITLAB_FEATURES` environment variable. - -## Previous job definitions - -CAUTION: **Caution:** -Before GitLab 11.5, Container Scanning job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. -You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - paths: [gl-container-scanning-report.json] -``` - -Alternatively the job name could be `sast:container` -and the artifact name could be `gl-sast-container-report.json`. -These names have been deprecated with GitLab 11.0 -and may be removed in next major release, GitLab 12.0. - -[ee]: https://about.gitlab.com/pricing/ +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index ab0ca13d2cf..b676c661267 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -1,102 +1,5 @@ -# Dynamic Application Security Testing with GitLab CI/CD +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dast/index.html' +--- -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -It also requires the GitLab Runner 11.5 or later. -For earlier versions, use the [previous job definitions](#previous-job-definitions). - -[Dynamic Application Security Testing (DAST)](https://en.wikipedia.org/wiki/Dynamic_program_analysis) -is using the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) -to perform an analysis on your running web application. -Since it is based on [ZAP Baseline](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan) -DAST will perform passive scanning only; -it will not actively attack your application. - -It can be very useful combined with [Review Apps](../review_apps/index.md). - -## Example - -First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` - -The above example will create a `dast` job in your CI/CD pipeline which will run -the tests on the URL defined in the `website` variable (change it to use your -own) and scan it for possible vulnerabilities. The report will be saved as a -[DAST report artifact](../yaml/README.md#artifactsreportsdast-ultimate) -that you can later download and analyze. -Due to implementation limitations we always take the latest DAST artifact available. - -It's also possible to authenticate the user before performing DAST checks: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - login_url: "https://example.com/sign-in" - username: "john.doe@example.com" - password: "john-doe-password" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website - --auth-url $login_url - --auth-username $username - --auth-password $password || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` -See [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) -to learn more about authentication settings. - -TIP: **Tip:** -For [GitLab Ultimate][ee] users, this information will -be automatically extracted and shown right in the merge request widget. -[Learn more on DAST in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). - -## Previous job definitions - -CAUTION: **Caution:** -Before GitLab 11.5, DAST job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. -You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - paths: [gl-dast-report.json] -``` - -[ee]: https://about.gitlab.com/pricing/ +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dast/index.html). diff --git a/doc/ci/examples/dependency_scanning.md b/doc/ci/examples/dependency_scanning.md new file mode 100644 index 00000000000..3a8b53b425c --- /dev/null +++ b/doc/ci/examples/dependency_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). diff --git a/doc/ci/examples/license_management.md b/doc/ci/examples/license_management.md new file mode 100644 index 00000000000..08704425a75 --- /dev/null +++ b/doc/ci/examples/license_management.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/license_management/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/license_management/index.html). diff --git a/doc/ci/examples/sast.md b/doc/ci/examples/sast.md new file mode 100644 index 00000000000..688cc79d0f6 --- /dev/null +++ b/doc/ci/examples/sast.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/sast/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/sast/index.html). diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md index 70b269046e5..570898b2d23 100644 --- a/doc/ci/examples/sast_docker.md +++ b/doc/ci/examples/sast_docker.md @@ -1,5 +1,5 @@ --- -redirect_to: 'container_scanning.md' +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' --- -This document was moved to [another location](container_scanning.md). +This document was moved to [another location](../../user/application_security/container_scanning/index.html). diff --git a/doc/ci/img/metrics_reports.png b/doc/ci/img/metrics_reports.png Binary files differnew file mode 100644 index 00000000000..ffd9f6830a2 --- /dev/null +++ b/doc/ci/img/metrics_reports.png diff --git a/doc/ci/img/multi_pipeline_mini_graph.gif b/doc/ci/img/multi_pipeline_mini_graph.gif Binary files differnew file mode 100644 index 00000000000..de49ba5aa12 --- /dev/null +++ b/doc/ci/img/multi_pipeline_mini_graph.gif diff --git a/doc/ci/img/multi_project_pipeline_graph.png b/doc/ci/img/multi_project_pipeline_graph.png Binary files differnew file mode 100644 index 00000000000..723a455cb4a --- /dev/null +++ b/doc/ci/img/multi_project_pipeline_graph.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 2a4160f62b0..7109b2ec583 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -54,3 +54,8 @@ terminal will block the job from finishing for the duration configured in close the terminal window.  + +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md). + diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index d03c0b68daf..799217c9a08 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -71,11 +71,11 @@ merge request widget. NOTE: **Note:** If you also want the ability to browse JUnit output files, include the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword. +[`artifacts:paths`](yaml/README.md#artifactspaths) keyword. An example of this is shown in the Ruby example below. ### Ruby example -Use the following job in `.gitlab-ci.yml`: +Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the JUnit output file. ```yaml ## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec @@ -85,6 +85,8 @@ ruby: - bundle install - rspec spec/lib/ --format RspecJunitFormatter --out rspec.xml artifacts: + paths: + - rspec.xml reports: junit: rspec.xml ``` diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md new file mode 100644 index 00000000000..83a7094faaa --- /dev/null +++ b/doc/ci/metrics_reports.md @@ -0,0 +1,40 @@ +# Metrics Reports **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing) 11.10. +Requires GitLab Runner 11.10 and above. + +## Overview + +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](./junit_test_reports.md), [codequality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. + +You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. + + + +## Use cases + +Consider the following examples of data that can utilize Metrics Reports: + +1. Memory usage +1. Load testing results +1. Code complexity +1. Code coverage stats + +## How it works + +Metrics are read from the metrics report (default: `metrics.txt`). They are parsed and displayed in the MR widget. + +## How to set it up + +Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. + +For example: + +```yaml +metrics: + script: + - echo 'metric_name metric_value' > metrics.txt + artifacts: + reports: + metrics: metrics.txt +``` diff --git a/doc/ci/multi_project_pipeline_graphs.md b/doc/ci/multi_project_pipeline_graphs.md new file mode 100644 index 00000000000..af10e5b6126 --- /dev/null +++ b/doc/ci/multi_project_pipeline_graphs.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'multi_project_pipelines.md' +--- + +This document was moved to [another location](multi_project_pipelines.md). diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md new file mode 100644 index 00000000000..556059c01b6 --- /dev/null +++ b/doc/ci/multi_project_pipelines.md @@ -0,0 +1,152 @@ +# Multi-project pipelines **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) in +[GitLab Premium 9.3](https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). + +When you set up [GitLab CI/CD](README.md) across multiple projects, you can visualize +the entire pipeline, including all cross-project inter-dependencies. + +## Overview + +GitLab CI/CD is a powerful continuous integration tool that works not only per project, but also across projects. When you +configure GitLab CI for your project, you can visualize the stages +of your [jobs](pipelines.md#configuring-pipelines) on a [pipeline graph](pipelines.md#visualizing-pipelines). + + + +In the Merge Request Widget, multi-project pipeline mini-graphs are displayed, +and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other. + + + +Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those +adopting a [microservices architecture](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/). + +## Use cases + +Let's assume you deploy your web app from different projects in GitLab: + +- One for the free version, which has its own pipeline that builds and tests your app +- One for the paid version add-ons, which also pass through builds and tests +- One for the documentation, which also builds, tests, and deploys with an SSG + +With Multi-Project Pipelines, you can visualize the entire pipeline, including all stages of builds and tests for the three projects. + +## Triggering multi-project pipelines through API + +When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab +recognizes the source of the job token, and thus internally ties these pipelines +together, allowing you to visualize their relationships on pipeline graphs. + +These relationships are displayed in the pipeline graph by showing inbound and +outbound connections for upstream and downstream pipeline dependencies. + +## Creating multi-project pipelines from `.gitlab-ci.yml` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +### Triggering a downstream pipeline using a bridge job + +Before GitLab 11.8, it was necessary to implement a pipeline job that was +responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api) +in a different project. + +In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this +task easier, and avoid needing GitLab Runner for triggering cross-project +pipelines. The following illustrates configuring a bridge job: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +In the example above, as soon as `rspec` job succeeds in the `test` stage, +the `staging` bridge job is going to be started. The initial status of this +job will be `pending`. GitLab will create a downstream pipeline in the +`my/deployment` project and, as soon as the pipeline gets created, the +`staging` job will succeed. `my/deployment` is a full path to that project. + +The user that created the upstream pipeline needs to have access rights to the +downstream project (`my/deployment` in this case). If a downstream project can +not be found, or a user does not have access rights to create pipeline there, +the `staging` job is going to be marked as _failed_. + +CAUTION: **Caution:** +`staging` will succeed as soon as a downstream pipeline gets created. +GitLab does not support status attribution yet, however adding first-class +`trigger` configuration syntax is ground work for implementing +[status attribution](https://gitlab.com/gitlab-org/gitlab-ce/issues/39640). + +NOTE: **Note:** +Bridge jobs do not support every configuration entry that a user can use +in the case of regular jobs. Bridge jobs will not to be picked by a Runner, +thus there is no point in adding support for `script`, for example. If a user +tries to use unsupported configuration syntax, YAML validation will fail upon +pipeline creation. + +### Specifying a downstream pipeline branch + +It is possible to specify a branch name that a downstream pipeline will use: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` + +Use a `project` keyword to specify full path to a downstream project. Use +a `branch` keyword to specify a branch name. + +GitLab will use a commit that is currently on the HEAD of the branch when +creating a downstream pipeline. + +### Passing variables to a downstream pipeline + +Sometimes you might want to pass variables to a downstream pipeline. +You can do that using the `variables` keyword, just like you would when +defining a regular job. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +The `ENVIRONMENT` variable will be passed to every job defined in a downstream +pipeline. It will be available as an environment variable when GitLab Runner picks a job. + +### Limitations + +Because bridge jobs are a little different to regular jobs, it is not +possible to use exactly the same configuration syntax here, as one would +normally do when defining a regular job that will be picked by a runner. + +Some features are not implemented yet. For example, support for environments. + +[Configuration keywords](yaml/README.md) available for bridge jobs are: + +- `trigger` (to define a downstream pipeline trigger) +- `stage` +- `allow_failure` +- `only` and `except` +- `when` +- `extends` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 2ffa3d4edc7..342c2ab972a 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -19,7 +19,7 @@ If all the jobs in a stage: - Fail, the next stage is not (usually) executed and the pipeline ends early. NOTE: **Note:** -If you have a [mirrored repository that GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository that GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -220,7 +220,7 @@ For information on adding pipeline badges to projects, see [Pipeline badges](../ Pipelines for different projects can be combined and visualized together. -For more information, see [Multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html). +For more information, see [Multi-project pipelines](multi_project_pipelines.md). ## Working with pipelines @@ -266,6 +266,9 @@ Clicking on an individual job will show you its job trace, and allow you to: - Retry the job. - Erase the job trace. +NOTE: **Note:** +To prevent jobs from being bypassed or run out of order, canceled jobs can only be retried when the whole pipeline they belong to is retried. + ### Seeing the failure reason for jobs > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782) in GitLab 10.7. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 65886400c64..015f1c0dc0f 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -127,7 +127,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is pending. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 0e72bdddee7..ad80b5d8818 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -23,6 +23,63 @@ attackers can impersonate the user that exposed their trigger token publicly in their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables) to protect trigger tokens. +### CI job token + +You can use the `CI_JOB_TOKEN` [variable][predef] (used to authenticate +with the [GitLab Container Registry][registry]) in the following cases. + +#### When used with multi-project pipelines **[PREMIUM]** + +> **Note**: +The use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced][ee-2017] +in [GitLab Premium][ee] 9.3. + +This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, +and it creates a dependent pipeline relation visible on the +[pipeline graph](../multi_project_pipelines.md#overview). For example: + +```yaml +build_docs: + stage: deploy + script: + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline + only: + - tags +``` + +Pipelines triggered that way also expose a special variable: +`CI_PIPELINE_SOURCE=pipeline`. + +Read more about the [pipelines trigger API][trigapi]. + +#### When a pipeline depends on the artifacts of another pipeline **[PREMIUM]** + +> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] + in [GitLab Premium][ee] 9.5. + +With the introduction of dependencies between different projects, one of +them may need to access artifacts created by a previous one. This process +must be granted for authorized accesses, and it can be done using the +`CI_JOB_TOKEN` variable that identifies a specific job. For example: + +```yaml +build_submodule: + image: debian + stage: test + script: + - apt update && apt install -y unzip + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN" + - unzip artifacts.zip + only: + - tags +``` + +This allows you to use that for multi-project pipelines and download artifacts +from any project to which you have access as this follows the same principles +with the [permission model][permissions]. + +Read more about the [jobs API](../../api/jobs.md#download-the-artifacts-archive). + ## Adding a new trigger You can add a new trigger by going to your project's @@ -225,7 +282,10 @@ removed with one of the future versions of GitLab. You are advised to [take ownership](#taking-ownership-of-a-trigger) of any legacy triggers. [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 +[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 [ee]: https://about.gitlab.com/pricing/ [variables]: ../variables/README.md [predef]: ../variables/README.md#predefined-environment-variables [registry]: ../../user/project/container_registry.md +[permissions]: ../../user/permissions.md#job-permissions +[trigapi]: ../../api/pipeline_triggers.md diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 6313ffc584d..1ba22070abe 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -52,6 +52,33 @@ or directly in the `.gitlab-ci.yml` file and reuse them as you wish. That can be very powerful as it can be used for scripting without the need to specify the value itself. +#### Variable types + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46806) in GitLab 11.11. + +There are two types of variables supported by GitLab: + +- `env_var`: the runner will create environment variable named same as the variable key and set its value to the variable value. +- `file`: the runner will write the variable value to a temporary file and set the path to this file as the value of an environment variable named same as the variable key. + +#### Masked variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/13784) in GitLab 11.10 + +By default, variables will be created as masked variables. +This means that the value of the variable will be hidden in job logs, +though it must match certain requirements to do so: + +- The value must be in a single line. +- The value must not have escape characters. +- The value must not use variables. +- The value must not have any whitespace. +- The value must be at least 8 characters long. + +If the value does not meet the requirements above, then the CI variable will fail to save. +In order to save, either alter the value to meet the masking requirements +or disable **Masked** for the variable. + ## Getting started To get started with environment variables in the scope of GitLab @@ -104,7 +131,10 @@ let's say you want to output `HELLO WORLD` for a `TEST` variable. You can either set the variable directly in the `.gitlab-ci.yml` file or through the UI. -#### Via [`.gitlab-ci.yml`](../yaml/README.md#variables) +#### Via `.gitlab-ci.yml` + +To create a new custom `env_var` variable via [`.gitlab-ci.yml`](../yaml/README.md#variables), define their variable/value pair under +`variables`: ```yaml variables: @@ -116,11 +146,13 @@ For a deeper look into them, see [`.gitlab-ci.yml` defined variables](#gitlab-ci #### Via the UI From the UI, navigate to your project's **Settings > CI/CD** and -expand **Environment variables**. Create a new variable by naming -it in the field **Input variable key**, and define its value in the +expand **Variables**. Create a new variable by choosing its **type**, naming +it in the field **Input variable key**, and defining its value in the **Input variable value** field: - + + +You'll also see the option to mask and/or protect your variables. Once you've set the variables, call them from the `.gitlab-ci.yml` file: @@ -129,30 +161,14 @@ test_variable: stage: test script: - echo $CI_JOB_STAGE # calls a predefined variable - - echo $TEST # calls a custom variable + - echo $TEST # calls a custom variable of type `env_var` + - echo $GREETING # calls a custom variable of type `file` that contains the path to the temp file + - cat $GREETING # the temp file itself contains the variable value ``` The output will be: - - -### Masked Variables - -By default, variables will be created as masked variables. -This means that the value of the variable will be hidden in job logs, -though it must match certain requirements to do so: - -- The value must be in a single line. -- The value must contain only letters, numbers, or underscores. -- The value must not have escape characters, such as `\"` -- The value must not use variables. -- The value must not have any whitespace. -- The value must be at least 8 characters long. - -The above rules are validated using the regex `/\A\w{8,}\z/`. If the value -does not meet the requirements above, then the CI variable will fail to save. -In order to save, either alter the value to meet the masking requirements or -disable `Masked` for the variable. + ### Syntax of environment variables in job scripts @@ -299,7 +315,7 @@ use for storing things like passwords, SSH keys, and credentials. Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. -1. Inputing variable keys and values in the **Environment variables** section. +1. Inputing variable types, keys, and values in the **Variables** section. Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. Once you set them, they will be available for all subsequent pipelines. @@ -350,6 +366,25 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. +### Limiting environment scopes of environment variables **[PREMIUM]** + +> [Introduced][ee-2112] in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. + +You can limit the environment scope of a variable by +[defining which environments][envs] it can be available for. + +Wildcards can be used, and the default environment scope is `*` which means +any jobs will have this variable, not matter if an environment is defined or +not. + +For example, if the environment scope is `production`, then only the jobs +having the environment `production` defined would have this specific variable. +Wildcards (`*`) can be used along with the environment name, therefore if the +environment scope is `review/*` then any jobs with environment names starting +with `review/` would have that particular variable. + +To learn more about about scoping environments, see [Scoping environments with specs](../environments.md#scoping-environments-with-specs-premium). + ### Deployment environment variables > Introduced in GitLab 8.15. @@ -390,8 +425,7 @@ For instance, suppose you added a [custom variable `$TEST`](#creating-a-custom-environment-variable) as exemplified above and you want to override it in a manual pipeline. Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. -Choose the branch you want to run the pipeline for, then add a new variable -pair through the UI: +Choose the branch you want to run the pipeline for, then add a new variable through the UI:  @@ -662,13 +696,15 @@ MIIFQzCCBCugAwIBAgIRAL/ElDjuf15xwja1ZnCocWAwDQYJKoZIhvcNAQELBQAw' ... ``` +[ee-2112]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2112 [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI variables" -[eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" [envs]: ../environments.md [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md [shellexecutors]: https://docs.gitlab.com/runner/executors/ [triggered]: ../triggers/README.md +[trigger-job-token]: ../triggers/README.md#ci-job-token [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token [registry]: ../../user/project/container_registry.md [dependent-repositories]: ../../user/project/new_ci_build_permissions_model.md#dependent-repositories +[get-job-artifacts]: ../../api/jobs.html#get-job-artifacts diff --git a/doc/ci/variables/img/custom_variable_output.png b/doc/ci/variables/img/custom_variable_output.png Binary files differdeleted file mode 100644 index 50f3bceff9a..00000000000 --- a/doc/ci/variables/img/custom_variable_output.png +++ /dev/null diff --git a/doc/ci/variables/img/custom_variables_output.png b/doc/ci/variables/img/custom_variables_output.png Binary files differnew file mode 100644 index 00000000000..29f5c63b3d9 --- /dev/null +++ b/doc/ci/variables/img/custom_variables_output.png diff --git a/doc/ci/variables/img/new_custom_variable_example.png b/doc/ci/variables/img/new_custom_variable_example.png Binary files differdeleted file mode 100644 index d169c5f1806..00000000000 --- a/doc/ci/variables/img/new_custom_variable_example.png +++ /dev/null diff --git a/doc/ci/variables/img/new_custom_variables_example.png b/doc/ci/variables/img/new_custom_variables_example.png Binary files differnew file mode 100644 index 00000000000..4b78e0ff587 --- /dev/null +++ b/doc/ci/variables/img/new_custom_variables_example.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differindex 3bcd354e096..3c8c59720cf 100644 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 091ddcb0bae..0470cf52654 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -112,3 +112,22 @@ They are: - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - In the `only` and `except` [variables expressions](README.md#environment-variables-expressions). + +## Variables with an environment scope + +Variables defined with an environment scope are supported. Given that +there is a variable `$STAGING_SECRET` defined in a scope of +`review/staging/*`, the following job that is using dynamic environments +is going to be created, based on the matching variable expression: + +```yaml +my-job: + stage: staging + environment: + name: review/$CI_JOB_STAGE/deploy + script: + - 'deploy staging' + only: + variables: + - $STAGING_SECRET == 'something' +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 03383d11c14..dca2d953286 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -19,7 +19,7 @@ We have complete examples of configuring pipelines: - To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab-ce`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml). NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -56,7 +56,7 @@ independently from each other. Each instance of GitLab CI has an embedded debug tool called Lint, which validates the content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your -project namespace. For example, `http://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. +project namespace. For example, `https://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. ### Unavailable names for jobs @@ -101,7 +101,7 @@ The following table lists available parameters for jobs: | [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | | [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, and `environment:action`. | | [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, and `artifacts:reports:performance`. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, `artifacts:reports:performance` and `artifacts:reports:metrics`. | | [`dependencies`](#dependencies) | Other jobs that a job depends on so that you can pass artifacts between them. | | [`coverage`](#coverage) | Code coverage settings for a given job. | | [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | @@ -1457,7 +1457,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html) as artifacts. The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will @@ -1468,7 +1468,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html) +The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html) as artifacts. The collected Container Scanning report will be uploaded to GitLab as an artifact and will @@ -1479,7 +1479,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html) +The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dast/index.html) as artifacts. The collected DAST report will be uploaded to GitLab as an artifact and will @@ -1507,6 +1507,14 @@ as artifacts. The collected Performance report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. +##### `artifacts:reports:metrics` **[PREMIUM]** + +The `metrics` report collects [Metrics](../../ci/metrics_reports.md) +as artifacts. + +The collected Metrics report will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests. + ### `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1702,7 +1710,7 @@ test: from `trigger` definition is started by GitLab, a downstream pipeline gets created. -Learn more about [multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html#creating-cross-project-pipelines-from-gitlab-ci-yml). +Learn more about [multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml). #### Simple `trigger` syntax @@ -2070,8 +2078,8 @@ of the `group/my-project`: ```yaml include: - - local: : /templates/docker-build.yml - - local: : /templates/docker-testing.yml + - local: /templates/docker-build.yml + - local: /templates/docker-testing.yml ``` Our `/templates/docker-build.yml` present in `group/my-project` adds a `docker-build` job: |
