diff options
Diffstat (limited to 'doc/development/pipelines.md')
| -rw-r--r-- | doc/development/pipelines.md | 535 |
1 files changed, 432 insertions, 103 deletions
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 9ba6dfc110a..39ca846c1cc 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -19,6 +19,9 @@ The current stages are: <https://gitlab.com/gitlab-org/gitlab-foss>. - `prepare`: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages. +- `build-images`: This stage includes jobs that prepare docker images + that are needed by jobs in subsequent stages or downstream pipelines. +- `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests. - `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs. - `post-test`: This stage includes jobs that build reports or gather data from the `test` stage's jobs (e.g. coverage, Knapsack metadata etc.). @@ -30,7 +33,6 @@ The current stages are: that is deployed in the previous stage. - `post-qa`: This stage includes jobs that build reports or gather data from the `qa` stage's jobs (e.g. Review App performance report). -- `notification`: This stage includes jobs that sends notifications about pipeline status. - `pages`: This stage includes a job that deploys the various reports as GitLab Pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>, <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>, @@ -68,12 +70,9 @@ that are scoped to a single [configuration parameter](../ci/yaml/README.md#confi | `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). | | `.default-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails and frontend tasks. | -| `.use-pg9` | Allows a job to use the `postgres:9.6.17` and `redis:alpine` services. | -| `.use-pg10` | Allows a job to use the `postgres:10.12` and `redis:alpine` services. | | `.use-pg11` | Allows a job to use the `postgres:11.6` and `redis:alpine` services. | -| `.use-pg9-ee` | Same as `.use-pg9` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | -| `.use-pg10-ee` | Same as `.use-pg10` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | | `.use-pg11-ee` | Same as `.use-pg11` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | +| `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable. | ## `workflow:rules` @@ -129,11 +128,12 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/README.md#anch | `changes:` patterns | Description | |------------------------------|--------------------------------------------------------------------------| +| `ci-patterns` | Only create job for CI config-related changes. | | `yaml-patterns` | Only create job for YAML-related changes. | | `docs-patterns` | Only create job for docs-related changes. | -| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes. | +| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes. | | `frontend-patterns` | Only create job for frontend-related changes. | -| `backstage-patterns` | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs). | +| `backstage-patterns` | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs). | | `code-patterns` | Only create job for code-related changes. | | `qa-patterns` | Only create job for QA-related changes. | | `code-backstage-patterns` | Combination of `code-patterns` and `backstage-patterns`. | @@ -151,112 +151,360 @@ request, be sure to start the `dont-interrupt-me` job before pushing. ## PostgreSQL versions testing +### Current versions testing + +| Where? | PostgreSQL version | +| ------ | ------ | +| MRs | 11 | +| `master` (non-scheduled pipelines) | 11 | +| 2-hourly scheduled pipelines | 11 | + +### Long-term plan + We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html): -| | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) | +| PostgreSQL version | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) | | ------ | ------------------ | --------------- | ---------------- | ---------------- | ------------------ | ------------ | -------------------- | ---------------- | -| PG9.6 | nightly | - | - | - | - | - | - | - | -| PG10 | `master` | - | - | - | - | - | - | - | -| PG11 | MRs/`master` | MRs/`master` | MRs/`master` | MRs/`master` | MRs/`master` | MRs/`master` | nightly | - | -| PG12 | - | - | - | - | `master` | `master` | MRs/`master` | `master` | -| PG13 | - | - | - | - | - | - | - | MRs/`master` | +| PG9.6 | MRs/`master`/`2-hour`/`nightly` | - | - | - | - | - | - | - | +| PG10 | `nightly` | - | - | - | - | - | - | - | +| PG11 | `master`/`2-hour` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | `nightly` | - | +| PG12 | - | - | - | - | `master`/`2-hour` | `master`/`2-hour` | MRs/`master`/`2-hour`/`nightly` | `master`/`2-hour` | +| PG13 | - | - | - | - | - | - | - | MRs/`master`/`2-hour`/`nightly` | + +## Pipeline types + +Since we use the [`rules:`](../ci/yaml/README.md#rules) and [`needs:`](../ci/yaml/README.md#needs) keywords extensively, +we have four main pipeline types which are described below. Note that an MR that includes multiple types of changes would +have a pipelines that include jobs from multiple types (e.g. a combination of docs-only and code-only pipelines). + +### Docs-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/135236627> + +```mermaid +graph LR + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-50["docs lint (6.75 minutes)"]; + click 1-50 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" + end +``` + +### Code-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/136295694> + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-2["build-qa-image (3.4 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-3["compile-assets pull-cache (9.06 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" + 1-5["gitlab:assets:compile pull-cache (22 minutes)"]; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-6["setup-test-env (8.22 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-7["review-stop-failed-deployment"]; + 1-8["dependency_scanning"]; + 1-9["qa:internal, qa:internal-as-if-foss"]; + 1-11["qa:selectors, qa:selectors-as-if-foss"]; + 1-14["retrieve-tests-metadata (1.5 minutes)"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["code_quality"]; + 1-16["brakeman-sast"]; + 1-17["eslint-sast"]; + 1-18["kubesec-sast"]; + 1-19["nodejs-scan-sast"]; + 1-20["secrets-sast"]; + + class 1-3 criticalPath; + class 1-6 criticalPath; + end + + 2_1-1["graphql-reference-verify (5 minutes)"]; + click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" + 2_1-2["memory-static (4.75 minutes)"]; + click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0" + 2_1-3["run-dev-fixtures (5 minutes)"]; + click 2_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356729&udv=0" + 2_1-4["run-dev-fixtures-ee (5 minutes)"]; + click 2_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356731&udv=0" + subgraph "Needs `setup-test-env`"; + 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; + end + + 2_2-1["static-analysis (17 minutes)"]; + click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" + 2_2-2["frontend-fixtures (17.2 minutes)"]; + class 2_2-2 criticalPath; + click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" + 2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"]; + click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0" + 2_2-4["memory-on-boot (7.19 minutes)"]; + click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0" + 2_2-5["webpack-dev-server (6.1 minutes)"]; + click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" + subgraph "Needs `setup-test-env` & `compile-assets`"; + 2_2-1 & 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; + 2_2-3 --> 1-6 & 1-4; + end + + 2_3-1["build-assets-image (2.5 minutes)"]; + subgraph "Needs `gitlab:assets:compile`"; + 2_3-1 --> 1-5 + end + + 2_4-1["package-and-qa (manual)"]; + subgraph "Needs `build-qa-image`"; + 2_4-1 --> 1-2; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + end + + 2_5-1["rspec & db jobs (12-22 minutes)"]; + subgraph "Needs `compile-assets`, `setup-test-env`, & `retrieve-tests-metadata`"; + 2_5-1 --> 1-3 & 1-6 & 1-14; + class 2_5-1 criticalPath; + click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" + end + + 3_1-1["jest (15 minutes)"]; + class 3_1-1 criticalPath; + click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" + 3_1-2["karma (8 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" + 3_1-3["jest-as-if-foss (19.7 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0" + 3_1-4["karma-as-if-foss (7.5 minutes)"]; + click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0" + subgraph "Needs `frontend-fixtures`"; + 3_1-1 & 3_1-2 --> 2_2-2; + 3_1-3 & 3_1-4 --> 2_2-3; + end + + 3_2-1["rspec:coverage (6.5 minutes)"]; + subgraph "Depends on `rspec` jobs"; + 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; + class 3_2-1 criticalPath; + click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" + end + + 4_1-1["coverage-frontend (3.6 minutes)"]; + subgraph "Needs `jest`"; + 4_1-1 --> 3_1-1; + class 4_1-1 criticalPath; + click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0" + end +``` + +### Frontend-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134661039> + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-2["build-qa-image (3.4 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-3["compile-assets pull-cache (9.06 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" + 1-5["gitlab:assets:compile pull-cache (22 minutes)"]; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-6["setup-test-env (8.22 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-7["review-stop-failed-deployment"]; + 1-8["dependency_scanning"]; + 1-9["qa:internal, qa:internal-as-if-foss"]; + 1-11["qa:selectors, qa:selectors-as-if-foss"]; + 1-14["retrieve-tests-metadata (1.5 minutes)"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["code_quality"]; + 1-16["brakeman-sast"]; + 1-17["eslint-sast"]; + 1-18["kubesec-sast"]; + 1-19["nodejs-scan-sast"]; + 1-20["secrets-sast"]; + + class 1-3 criticalPath; + class 1-5 criticalPath; + class 1-6 criticalPath; + end + + 2_1-1["graphql-reference-verify (5 minutes)"]; + click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" + 2_1-2["memory-static (4.75 minutes)"]; + click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0" + 2_1-3["run-dev-fixtures (5 minutes)"]; + click 2_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356729&udv=0" + 2_1-4["run-dev-fixtures-ee (5 minutes)"]; + click 2_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356731&udv=0" + subgraph "Needs `setup-test-env`"; + 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; + end + + 2_2-1["static-analysis (17 minutes)"]; + click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" + 2_2-2["frontend-fixtures (17.2 minutes)"]; + class 2_2-2 criticalPath; + click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" + 2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"]; + click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0" + 2_2-4["memory-on-boot (7.19 minutes)"]; + click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0" + 2_2-5["webpack-dev-server (6.1 minutes)"]; + click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" + subgraph "Needs `setup-test-env` & `compile-assets`"; + 2_2-1 & 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; + 2_2-3 --> 1-6 & 1-4; + end + + 2_3-1["build-assets-image (2.5 minutes)"]; + class 2_3-1 criticalPath; + subgraph "Needs `gitlab:assets:compile`"; + 2_3-1 --> 1-5 + end -## Directed acyclic graph + 2_4-1["package-and-qa (manual)"]; + subgraph "Needs `build-qa-image` & `build-assets-image`"; + 2_4-1 --> 1-2 & 2_3-1; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + end + + 2_5-1["rspec & db jobs (12-22 minutes)"]; + subgraph "Needs `compile-assets`, `setup-test-env, & `retrieve-tests-metadata`"; + 2_5-1 --> 1-3 & 1-6 & 1-14; + class 2_5-1 criticalPath; + click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" + end + + 2_6-1["review-build-cng (27.3 minutes)"]; + subgraph "Needs `build-assets-image`"; + 2_6-1 --> 2_3-1; + class 2_6-1 criticalPath; + click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914314&udv=0" + end + + 3_1-1["jest (15 minutes)"]; + class 3_1-1 criticalPath; + click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" + 3_1-2["karma (8 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" + 3_1-3["jest-as-if-foss (19.7 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0" + 3_1-4["karma-as-if-foss (7.5 minutes)"]; + click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0" + subgraph "Needs `frontend-fixtures`"; + 3_1-1 & 3_1-3 --> 2_2-2; + 3_1-2 & 3_1-4 --> 2_2-3; + end + + 3_2-1["rspec:coverage (6.5 minutes)"]; + subgraph "Depends on `rspec` jobs"; + 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; + class 3_2-1 criticalPath; + click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" + end -We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to -execute jobs out of order for the following jobs: + 4_1-1["coverage-frontend (3.6 minutes)"]; + subgraph "Needs `jest`"; + 4_1-1 --> 3_1-1; + class 4_1-1 criticalPath; + click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0" + end + + 3_3-1["review-deploy (6 minutes)"]; + subgraph "Played by `review-build-cng`"; + 3_3-1 --> 2_6-1; + class 3_3-1 criticalPath; + click 3_3-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6721130&udv=0" + end + + 4_2-1["review-qa-smoke (8 minutes)"]; + click 4_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6729805&udv=0" + 4_2-2["review-performance (4 minutes)"]; + click 4_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356817&udv=0" + 4_2-3["dast (18 minutes)"]; + click 4_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356819&udv=0" + class 4_2-3 criticalPath; + subgraph "Played by `review-deploy`"; + 4_2-1 & 4_2-2 & 4_2-3 -.->|"(don't use needs because of limitations)"| 3_3-1; + end +``` + +### QA-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134645109> ```mermaid graph RL; - A[setup-test-env]; - B["gitlab:assets:compile pull-push-cache<br/>(canonical master only)"]; - C["gitlab:assets:compile pull-cache<br/>(canonical default refs only)"]; - D["cache gems<br/>(master and tags only)"]; - E[review-build-cng]; - F[build-qa-image]; - G[review-deploy]; - I["karma, jest"]; - I2["karma-as-if-foss, jest-as-if-foss<br/>(EE default refs only)"]; - J["compile-assets pull-push-cache<br/>(master only)"]; - J2["compile-assets pull-push-cache as-if-foss<br/>(EE master only)"]; - K[compile-assets pull-cache]; - K2["compile-assets pull-cache as-if-foss<br/>(EE default refs only)"]; - U[frontend-fixtures]; - U2["frontend-fixtures-as-if-foss<br/>(EE default refs only)"]; - V["webpack-dev-server, static-analysis"]; - M[coverage]; - O[coverage-frontend]; - N["pages (master only)"]; - Q[package-and-qa]; - S["RSpec<br/>(e.g. rspec unit pg10)"] - T[retrieve-tests-metadata]; - QA["qa:internal, qa:selectors"]; - QA2["qa:internal-as-if-foss, qa:selectors-as-if-foss<br/>(EE default refs only)"]; - X["docs lint, code_quality, sast, dependency_scanning, danger-review"]; - -subgraph "`prepare` stage" - A - B - C - F - K - K2 - J - J2 - T - end - -subgraph "`fixture` stage" - U -.-> |needs and depends on| A; - U -.-> |needs and depends on| K; - U2 -.-> |needs and depends on| A; - U2 -.-> |needs and depends on| K2; - end - -subgraph "`test` stage" - D -.-> |needs| A; - I -.-> |needs and depends on| U; - I2 -.-> |needs and depends on| U2; - L -.-> |needs and depends on| A; - S -.-> |needs and depends on| A; - S -.-> |needs and depends on| K; - S -.-> |needs and depends on| T; - L["db:*, gitlab:setup, graphql-docs-verify, downtime_check"] -.-> |needs| A; - V -.-> |needs and depends on| K; - X -.-> |needs| T; - QA -.-> |needs| T; - QA2 -.-> |needs| T; - end - -subgraph "`post-test` stage" - M --> |happens after| S - O --> |needs `jest`| I - end - -subgraph "`review-prepare` stage" - E -.-> |needs| C; - end - -subgraph "`review` stage" - G -.-> |needs| E - end - -subgraph "`qa` stage" - Q -.-> |needs| C; - Q -.-> |needs| F; - QA1["review-qa-smoke, review-qa-all, review-performance, dast"] -.-> |needs| G; - end - -subgraph "`post-qa` stage" - PQA1["parallel-spec-reports"] -.-> |depends on `review-qa-all`| QA1; + classDef criticalPath fill:#f66; + + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-2["build-qa-image (3.4 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-3["compile-assets pull-cache (9.06 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" + 1-5["gitlab:assets:compile pull-cache (22 minutes)"]; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-6["setup-test-env (8.22 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-7["review-stop-failed-deployment"]; + 1-8["dependency_scanning"]; + 1-9["qa:internal, qa:internal-as-if-foss"]; + 1-11["qa:selectors, qa:selectors-as-if-foss"]; + 1-14["retrieve-tests-metadata (1.5 minutes)"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["code_quality"]; + 1-16["brakeman-sast"]; + 1-17["eslint-sast"]; + 1-18["kubesec-sast"]; + 1-19["nodejs-scan-sast"]; + 1-20["secrets-sast"]; + + class 1-5 criticalPath; + end + + 2_1-1["graphql-reference-verify (5 minutes)"]; + click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" + subgraph "Needs `setup-test-env`"; + 2_1-1 --> 1-6; + end + + 2_2-1["static-analysis (17 minutes)"]; + click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" + subgraph "Needs `setup-test-env` & `compile-assets`"; + 2_2-1 --> 1-6 & 1-3; + end + + 2_3-1["build-assets-image (2.5 minutes)"]; + subgraph "Needs `gitlab:assets:compile`"; + 2_3-1 --> 1-5 + class 2_3-1 criticalPath; end -subgraph "`pages` stage" - N -.-> |depends on| C; - N -.-> |depends on karma| I; - N -.-> |depends on| M; - N --> |happens after| PQA1 - end + 2_4-1["package-and-qa (108 minutes)"]; + subgraph "Needs `build-qa-image` & `build-assets-image`"; + 2_4-1 --> 1-2 & 2_3-1; + class 2_4-1 criticalPath; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + end ``` ## Test jobs @@ -268,6 +516,87 @@ for more information. Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. +## As-if-FOSS jobs + +The `* as-if-foss` jobs allows to run GitLab's test suite "as-if-FOSS", meaning as if the jobs would run in the context +of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: + +- `master` commits (pushes and scheduled pipelines). +- `gitlab-org/security/gitlab` merge requests. +- Merge requests which include `RUN AS-IF-FOSS` in their title. +- Merge requests that changes the CI config. + +The `* as-if-foss` jobs have the `FOSS_ONLY='1'` variable set and gets their EE-specific +folders removed before the tests start running. + +The intent is to ensure that a change won't introduce a failure once the `gitlab-org/gitlab` project will be synced to +the `gitlab-org/gitlab-foss` project. + +## Pre-clone step + +The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/issues/39134) +to seed the project with a recent archive of the repository. This is done for +several reasons: + +- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. +- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. + +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../user/gitlab_com/index.md#pre-clone-script). + +The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD +variable: + +```shell +echo "Downloading archived master..." +wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master.tar.gz + +if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit +fi + +rm -rf $CI_PROJECT_DIR +echo "Extracting tarball into $CI_PROJECT_DIR..." +mkdir -p $CI_PROJECT_DIR +cd $CI_PROJECT_DIR +tar xzf /tmp/gitlab.tar.gz +rm -f /tmp/gitlab.tar.gz +chmod a+w $CI_PROJECT_DIR +``` + +The first step of the script downloads `gitlab-master.tar.gz` from +Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) +that is responsible for keeping that archive up-to-date. Every two hours +on a scheduled pipeline, it does the following: + +1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Saves the data as a `.tar.gz`. +1. Uploads it into the Google Cloud Storage bucket. + +When a CI job runs with this configuration, you'll see something like +this: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/group/project... +Fetching changes... +Reinitialized existing Git repository in /builds/group/project/.git/ +``` + +Note that the `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. These +credentials are stored in the 1Password GitLab.com Production vault. + +Note that this bucket should be located in the same continent as the +runner, or [network egress charges will apply](https://cloud.google.com/storage/pricing). + --- [Return to Development documentation](README.md) |