Add latest changes from gitlab-org/gitlab@master

1 files changed, 216 insertions, 0 deletions

diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md
new file mode 100644
index 00000000000..448fb0f9f5a
--- /dev/null
+++ b/doc/development/pipelines.md
@@ -0,0 +1,216 @@
+# Pipelines for the GitLab project
+
+Pipelines for `gitlab-org/gitlab` and `gitlab-org/gitlab-foss` (as well as the
+`dev` instance's mirrors) are configured in the usual
+[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml)
+which itself includes files under
+[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/ci)
+for easier maintenance.
+
+We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding)
+GitLab [CI/CD features and best-practices](../ci/yaml/README.md)
+as much as possible.
+
+## Stages
+
+The current stages are:
+
+- `prepare`: This stage includes jobs that prepare artifacts that are needed by
+  jobs in subsequent stages.
+- `quick-test`: This stage includes test jobs that should run first and fail the
+  pipeline early (currently used to run Geo tests when the branch name starts
+  with `geo-`, `geo/`, or ends with `-geo`).
+- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
+- `review-prepare`: This stage includes a job that build the CNG images that are
+  later used by the (Helm) Review App deployment (see
+  [Review Apps](testing_guide/review_apps.md) for details).
+- `review`: This stage includes jobs that deploy the GitLab and Docs Review Apps.
+- `qa`: This stage includes jobs that perform QA tasks against the Review App
+  that is deployed in the previous stage.
+- `post-test`: This stage includes jobs that build reports or gather data from
+  the previous stages' jobs (e.g. coverage, Knapsack metadata etc.).
+- `pages`: This stage includes a job that deploys the various reports as
+  GitLab pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>,
+  <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>,
+  <https://gitlab-org.gitlab.io/gitlab/webpack-report/>).
+
+## Default image
+
+The default image is currently
+`dev.gitlab.org:5005/gitlab/gitlab-build-images:ruby-2.6.3-golang-1.11-git-2.22-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33`.
+It includes Ruby 2.6.3, Go 1.11, Git 2.22, Chrome 73, Node 12, Yarn 1.16,
+PostgreSQL 9.6, and Graphics Magick 1.3.33.
+
+The images used in our pipelines are configured in the
+[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images)
+project, which is push-mirrored to <https://dev.gitlab.org/gitlab/gitlab-build-images>
+for redundancy.
+
+The current version of the build images can be found in the
+["Used by GitLab CE/EE section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml).
+
+## Default variables
+
+In addition to the [predefined variables](../ci/variables/predefined_variables.md),
+each pipeline includes the following [variables](../ci/variables/README.md):
+
+- `RAILS_ENV: "test"`
+- `NODE_ENV: "test"`
+- `SIMPLECOV: "true"`
+- `GIT_DEPTH: "20"`
+- `GIT_SUBMODULE_STRATEGY: "none"`
+- `GET_SOURCES_ATTEMPTS: "3"`
+- `KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master.json`
+- `EE_KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/${CI_PROJECT_NAME}/rspec_report-master-ee.json`
+- `FLAKY_RSPEC_SUITE_REPORT_PATH: rspec_flaky/report-suite.json`
+- `BUILD_ASSETS_IMAGE: "false"`
+- `ES_JAVA_OPTS: "-Xms256m -Xmx256m"`
+- `ELASTIC_URL: "http://elastic:changeme@docker.elastic.co-elasticsearch-elasticsearch:9200"`
+
+## Common job definitions
+
+Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends)
+that are scoped to a single
+[configuration parameter](../ci/yaml/README.md#configuration-parameters).
+
+These common definitions are:
+
+- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using
+  our dedicated runners.
+- `.default-retry`: Allows a job to retry upon `unknown_failure`, `api_failure`,
+  `runner_system_failure`.
+- `.default-before_script`: Allows a job to use a default `before_script` definition
+  suitable for Ruby/Rails tasks that may need a database running (e.g. tests).
+- `.default-cache`: Allows a job to use a default `cache` definition suitable for
+  Ruby/Rails and frontend tasks.
+- `.default-only`: Restricts the cases where a job is created. This currently
+  includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches),
+  `/^\d+-\d+-auto-deploy-\d+$/` (security branches), `merge_requests`, `tags`.
+  Note that jobs won't be created for branches with this default configuration.
+- `.only-review`: Only creates a job for the `gitlab-org` namespace and if
+  Kubernetes integration is available. Also, prevents a job from being created
+  for `master` and auto-deploy branches.
+- `.only-review-schedules`: Same as `.only-review` but also restrict a job to
+  only run for [schedules](../user/project/pipelines/schedules.md).
+- `.use-pg`: Allows a job to use the `postgres:9.6.14` and `redis:alpine` services.
+- `.use-pg-10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services.
+- `.only-ee`: Only creates a job for the `gitlab` project.
+
+## Changes detection
+
+If a job extends from `.default-only` (and most of the jobs should), it can restrict
+the cases where it should be created
+[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges)
+from a commit or MR by extending from the following CI definitions:
+
+- `.only-code-changes`: Allows a job to only be created upon code-related changes.
+- `.only-qa-changes`: Allows a job to only be created upon QA-related changes.
+- `.only-docs-changes`: Allows a job to only be created upon docs-related changes.
+- `.only-code-qa-changes`: Allows a job to only be created upon code-related or QA-related changes.
+
+**See <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml>
+for the list of exact patterns.**
+
+## Directed acyclic graph
+
+We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to
+execute jobs out of order for the following jobs:
+
+```mermaid
+graph RL;
+  A[setup-test-env];
+  B["gitlab:assets:compile<br/>(master only)"];
+  C[gitlab:assets:compile pull-cache];
+  D["cache gems<br/>(master and tags only)"];
+  E[review-build-cng];
+  F[build-qa-image];
+  G[review-deploy];
+  G2["schedule:review-deploy<br/>(master only)"];
+  H[karma];
+  I[jest];
+  J["compile-assets<br/>(master only)"];
+  K[compile-assets pull-cache];
+  L[webpack-dev-server];
+  M[coverage];
+  N[pages];
+  O[static-analysis];
+  P["package-and-qa-manual:master<br/>(master schedule only)"];
+  Q[package-and-qa];
+  R[package-and-qa-manual];
+
+subgraph "`prepare` stage"
+    A
+    F
+    J
+    K
+    end
+
+subgraph "`test` stage"
+    B --> |needs| A;
+    C --> |needs| A;
+    D --> |needs| A;
+    H -.-> |depends on| A;
+    H -.-> |depends on| J;
+    H -.-> |depends on| K;
+    I -.-> |depends on| A;
+    I -.-> |depends on| J;
+    I -.-> |depends on| K;
+    L -.-> |depends on| A;
+    L -.-> |depends on| J;
+    L -.-> |depends on| K;
+    downtime_check --> |needs and depends on| A;
+    db:* --> |needs| A;
+    gitlab:setup --> |needs| A;
+    O -.-> |depends on| A;
+    O -.-> |depends on| B;
+    O -.-> |depends on| C;
+    downtime_check --> |needs and depends on| A;
+    end
+
+subgraph "`review-prepare` stage"
+    E --> |needs| C;
+    X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| B;
+    end
+
+subgraph "`review` stage"
+    G --> |needs| E;
+    G2 --> |needs| E;
+    end
+
+subgraph "`qa` stage"
+    Q --> |needs| C;
+    Q --> |needs| F;
+    R --> |needs| C;
+    R --> |needs| F;
+    P --> |needs| B;
+    P --> |needs| F;
+    review-qa-smoke -.-> |depends on| G;
+    review-qa-all -.-> |depends on| G;
+    review-qa-performance -.-> |depends on| G;
+    X2["schedule:review-performance<br/>(master only)"] -.-> |depends on| G2;
+    dast -.-> |depends on| G;
+    end
+
+subgraph "`post-test` stage"
+    M
+    end
+
+subgraph "`pages` stage"
+    N -.-> |depends on| B;
+    N -.-> |depends on| H;
+    N -.-> |depends on| M;
+    end
+```
+
+## Test jobs
+
+Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md)
+for more information.
+
+## Review app jobs
+
+Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information.
+
+---
+
+[Return to Development documentation](README.md)