diff options
Diffstat (limited to 'doc/development/pipelines.md')
| -rw-r--r-- | doc/development/pipelines.md | 61 |
1 files changed, 44 insertions, 17 deletions
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index a8ef5701d50..3243c7ec753 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Pipelines for the GitLab project @@ -412,7 +412,7 @@ Merge Request. This prevents `rspec fail-fast` duration from exceeding the avera This number can be overridden by setting a CI variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. -NOTE: **Note:** +NOTE: This experiment is only enabled when the CI variable `RSPEC_FAIL_FAST_ENABLED=true` is set. #### Determining related test files in a Merge Request @@ -435,7 +435,7 @@ We are using a custom mapping between source file to test files, maintained in t We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html): -| PostgreSQL version | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) | +| PostgreSQL version | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | [13.7 (December 2020)](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5722) | 14.0 (May 2021?) | | ------ | --------------- | ---------------- | ---------------- | ------------------ | ------------ | -------------------- | ---------------- | | PG11 | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | `nightly` | - | | PG12 | - | - | `nightly` | `2-hour`/`nightly` | `2-hour`/`nightly` | MRs/`2-hour`/`nightly` | `2-hour`/`nightly` | @@ -457,7 +457,7 @@ Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more ### As-if-FOSS jobs -The `* as-if-foss` jobs allows to run GitLab's test suite "as-if-FOSS", meaning as if the jobs would run in the context +The `* as-if-foss` jobs allows the GitLab test suite "as-if-FOSS", meaning as if the jobs would run in the context of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: - `gitlab-org/security/gitlab` merge requests. @@ -467,7 +467,7 @@ of the `gitlab-org/gitlab-foss` project. These jobs are only created in the foll The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable set and get their EE-specific folders removed before the tests start running. -The intent is to ensure that a change won't introduce a failure once the `gitlab-org/gitlab` project will be synced to +The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to the `gitlab-org/gitlab-foss` project. ## Performance @@ -485,22 +485,24 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. All jobs must only pull caches by default. 1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. -1. We currently have 6 different caches defined in +1. We currently have several different caches defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: - `.rails-cache`. - `.static-analysis-cache`. + - `.coverage-cache` - `.qa-cache` - `.yarn-cache`. - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). 1. Only 6 specific jobs, running in 2-hourly scheduled pipelines, are pushing (i.e. updating) to the caches: - `update-rails-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-coverage-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). -1. These jobs will run in merge requests whose title include `UPDATE CACHE`. +1. These jobs run in merge requests whose title include `UPDATE CACHE`. ### Pre-clone step @@ -519,7 +521,7 @@ variable: ```shell echo "Downloading archived master..." -wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master.tar.gz +wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz if [ ! -f /tmp/gitlab.tar.gz ]; then echo "Repository cache not available, cloning a new directory..." @@ -544,8 +546,7 @@ on a scheduled pipeline, it does the following: 1. Saves the data as a `.tar.gz`. 1. Uploads it into the Google Cloud Storage bucket. -When a CI job runs with this configuration, you'll see something like -this: +When a CI job runs with this configuration, the output looks something like this: ```shell $ eval "$CI_PRE_CLONE_SCRIPT" @@ -566,7 +567,7 @@ GitLab Team Member, find credentials in the [GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). Note that this bucket should be located in the same continent as the -runner, or [network egress charges will apply](https://cloud.google.com/storage/pricing). +runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). ## CI configuration internals @@ -628,17 +629,21 @@ that are scoped to a single [configuration keyword](../ci/yaml/README.md#job-key | Job definitions | Description | |------------------|-------------| -| `.default-tags` | Ensures a job has the `gitlab-org` tag to ensure it's using our dedicated runners. | | `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). | | `.rails-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails tasks. | | `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | +| `.coverage-cache` | Allows a job to use a default `cache` definition suitable for coverage tasks. | +| `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | | `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | | `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | | `.use-pg11` | Allows a job to use the `postgres:11.6` and `redis:4.0-alpine` services. | -| `.use-pg11-ee` | Same as `.use-pg11` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | +| `.use-pg11-ee` | Same as `.use-pg11` but also use the `docker.elastic.co/elasticsearch/elasticsearch:7.9.2` services. | +| `.use-pg12` | Allows a job to use the `postgres:12` and `redis:4.0-alpine` services. | +| `.use-pg12-ee` | Same as `.use-pg12` but also use the `docker.elastic.co/elasticsearch/elasticsearch:7.9.2` services. | | `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable. | +| `.use-docker-in-docker` | Allows a job to use Docker in Docker. | ### `rules`, `if:` conditions and `changes:` patterns @@ -655,33 +660,55 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/README.md#anch #### `if:` conditions +<!-- vale gitlab.Substitutions = NO --> + | `if:` conditions | Description | Notes | |------------------|-------------|-------| -| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success\|manual`), or **not** create a job for forks (by using `when: never`). | +| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success|manual`), or **not** create a job for forks (by using `when: never`). | | `if-not-ee` | Matches if the project isn't EE (i.e. project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success|manual`), or **not** create a job if the project is EE (by using `when: never`). | | `if-not-foss` | Matches if the project isn't FOSS (i.e. project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success|manual`), or **not** create a job if the project is FOSS (by using `when: never`). | -| `if-default-refs` | Matches if the pipeline is for `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs won't be created for branches with this default configuration. | +| `if-default-refs` | Matches if the pipeline is for `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs aren't created for branches with this default configuration. | | `if-master-refs` | Matches if the current branch is `master`. | | +| `if-master-push` | Matches if the current branch is `master` and pipeline source is `push`. | | +| `if-master-schedule-2-hourly` | Matches if the current branch is `master` and pipeline runs on a 2-hourly schedule. | | +| `if-master-schedule-2-nightly` | Matches if the current branch is `master` and pipeline runs on a nightly schedule. | | +| `if-auto-deploy-branches` | Matches if the current branch is an auto-deploy one. | | | `if-master-or-tag` | Matches if the pipeline is for the `master` branch or for a tag. | | | `if-merge-request` | Matches if the pipeline is for a merge request. | | +| `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR title includes "RUN AS-IF-FOSS". | | +| `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR title includes "UPDATE CACHE". | | +| `if-merge-request-title-run-all-rspec` | Matches if the pipeline is for a merge request and the MR title includes "RUN ALL RSPEC". | | +| `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | +| `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | | `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | | `if-dot-com-gitlab-org-schedule` | Limits jobs creation to scheduled pipelines for the `gitlab-org` group on GitLab.com. | | | `if-dot-com-gitlab-org-master` | Limits jobs creation to the `master` branch for the `gitlab-org` group on GitLab.com. | | | `if-dot-com-gitlab-org-merge-request` | Limits jobs creation to merge requests for the `gitlab-org` group on GitLab.com. | | | `if-dot-com-gitlab-org-and-security-tag` | Limits job creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | +| `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | | `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | +| `if-rspec-fail-fast-disabled` | Limits jobs to pipelines with `$RSPEC_FAIL_FAST_ENABLED` variable not set to `"true"`. | | +| `if-rspec-fail-fast-skipped` | Matches if the pipeline is for a merge request and the MR title includes "SKIP RSPEC FAIL-FAST". | | +| `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | + +<!-- vale gitlab.Substitutions = YES --> #### `changes:` patterns | `changes:` patterns | Description | |------------------------------|--------------------------------------------------------------------------| -| `ci-patterns` | Only create job for CI config-related changes. | -| `yaml-patterns` | Only create job for YAML-related changes. | +| `ci-patterns` | Only create job for CI configuration-related changes. | +| `ci-build-images-patterns` | Only create job for CI configuration-related changes related to the `build-images` stage. | +| `ci-review-patterns` | Only create job for CI configuration-related changes related to the `review` stage. | +| `ci-qa-patterns` | Only create job for CI configuration-related changes related to the `qa` stage. | +| `yaml-lint-patterns` | Only create job for YAML-related changes. | | `docs-patterns` | Only create job for docs-related changes. | | `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes. | | `frontend-patterns` | Only create job for frontend-related changes. | +| `backend-patterns` | Only create job for backend-related changes. | +| `db-patterns` | Only create job for DB-related changes. | | `backstage-patterns` | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs). | | `code-patterns` | Only create job for code-related changes. | | `qa-patterns` | Only create job for QA-related changes. | |