diff options
Diffstat (limited to 'doc/development/cicd/templates.md')
| -rw-r--r-- | doc/development/cicd/templates.md | 247 |
1 files changed, 213 insertions, 34 deletions
diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index ed0d217c247..fd1456a1676 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, concepts, howto --- @@ -9,27 +9,214 @@ type: index, concepts, howto This document explains how to develop [GitLab CI/CD templates](../../ci/examples/README.md). -## Place the template file in a relevant directory +## Requirements for CI/CD templates + +Before submitting a merge request with a new or updated CI/CD template, you must: + +- Place the template in the correct [directory](#template-directories). +- Follow the [CI/CD template authoring guidelines](#template-authoring-guidelines). +- Name the template following the `*.gitlab-ci.yml` format. +- Use valid [`.gitlab-ci.yml` syntax](../../ci/yaml/README.md). Verify it's valid + with the [CI/CD lint tool](../../ci/lint.md). +- Include [a changelog](../changelog.md) if the merge request introduces a user-facing change. +- Follow the [template review process](#contribute-cicd-template-merge-requests). +- (Optional but highly recommended) Test the template in an example GitLab project + that reviewers can access. Reviewers might not be able to create the data or configuration + that the template requires, so an example project helps the reviewers ensure the + template is correct. The example project pipeline should succeed before submitting + the merge request for review. + +## Template directories + +All template files are saved in `lib/gitlab/ci/templates`. Save general templates +in this directory, but certain template types have a specific directory reserved for +them. The ability to [select a template in new file UI](#make-sure-the-new-template-can-be-selected-in-ui) +is determined by the directory it is in: + +| Sub-directory | Selectable in UI | Template type | +|----------------|------------------|---------------| +| `/*` (root) | Yes | General templates. | +| `/AWS/*` | No | Templates related to Cloud Deployment (AWS). | +| `/Jobs/*` | No | Templates related to Auto DevOps. | +| `/Pages/*` | Yes | Sample templates for using Static site generators with GitLab Pages. | +| `/Security/*` | Yes | Templates related to Security scanners. | +| `/Terraform/*` | No | Templates related to infrastructure as Code (Terraform). | +| `/Verify/*` | Yes | Templates related to Testing features. | +| `/Workflows/*` | No | Sample templates for using the `workflow:` keyword. | + +## Template authoring guidelines + +Use the following guidelines to ensure your template submission follows standards: + +### Template types + +Templates have two different types that impact the way the template should be written +and used. The style in a template should match one of these two types: + +A **pipeline template** provides an end-to-end CI/CD workflow that matches a project's +structure, language, and so on. It usually should be used by itself in projects that +don't have any other `.gitlab-ci.yml` files. + +When authoring pipeline templates: + +- Place any [global keywords](../../ci/yaml/README.md#global-keywords) like `image` + or `before_script` in a [`default`](../../ci/yaml/README.md#custom-default-keyword-values) + section at the top of the template. +- Note clearly in the [code comments](#explain-the-template-with-comments) if the + template is designed to be used with the `includes` keyword in an existing + `.gitlab-ci.yml` file or not. + +A **job template** provides specific jobs that can be added to an existing CI/CD +workflow to accomplish specific tasks. It usually should be used by adding it to +an existing `.gitlab-ci.yml` file by using the [`includes`](../../ci/yaml/README.md#global-keywords) +keyword. You can also copy and paste the contents into an existing `.gitlab-ci.yml` file. + +Configure job templates so that users can add them to their current pipeline with very +few or no modifications. It must be configured to reduce the risk of conflicting with +other pipeline configuration. + +When authoring job templates: + +- Do not use [global](../../ci/yaml/README.md#global-keywords) or [`default`](../../ci/yaml/README.md#custom-default-keyword-values) + keywords. When a root `.gitlab-ci.yml` includes a template, global or default keywords + might be overridden and cause unexpected behavior. If a job template requires a + specific stage, explain in the code comments that users must manually add the stage + to the main `.gitlab-ci.yml` configuration. +- Note clearly in [code comments](#explain-the-template-with-comments) that the template + is designed to be used with the `includes` keyword or copied into an existing configuration. +- Consider [versioning](#versioning) the template with latest and stable versions + to avoid [backward compatibility](#backward-compatibility) problems. + Maintenance of this type of template is more complex, because changes to templates + imported with `includes` can break pipelines for all projects using the template. + +Additional points to keep in mind when authoring templates: + +| Template design points | Pipeline templates | Job templates | +|------------------------------------------------------|--------------------|---------------| +| Can use global keywords, including `stages`. | Yes | No | +| Can define jobs. | Yes | Yes | +| Can be selected in the new file UI | Yes | Yes | +| Can include other job templates with `include` | Yes | No | +| Can include other pipeline templates with `include`. | No | No | + +### Syntax guidelines + +To make templates easier to follow, templates should all use clear syntax styles, +with a consistent format. + +#### Do not hardcode the default branch + +Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) +instead of a hardcoded `main` branch, and never use `master`: -All template files reside in the `lib/gitlab/ci/templates` directory, and are categorized by the following sub-directories: +```yaml +job1: + rules: + if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + script: + echo "example job 1" + +job2: + only: + variables: + - $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + script: + echo "example job 2" + +``` + +#### Use `rules` instead of `only` or `except` + +Avoid using [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic) if possible. +Only and except is not being developed any more, and [`rules`](../../ci/yaml/README.md#rules) +is now the preferred syntax: + +```yaml +job2: + script: + - echo + rules: + - if: $CI_COMMIT_BRANCH +``` + +#### Break up long commands + +If a command is very long, or has many command line flags, like `-o` or `--option`: + +- Split these up into a multi-line command to make it easier to see every part of the command. +- Use the long name for the flags, when available. + +For example, with a long command with short CLI flags like +`docker run --e SOURCE_CODE="$PWD" -v "$PWD":/code -v /var/run/docker.sock:/var/run/docker.sock "$CODE_QUALITY_IMAGE" /code`: + +```yaml +job1: + script: + - docker run + --env SOURCE_CODE="$PWD" + --volume "$PWD":/code + --volume /var/run/docker.sock:/var/run/docker.sock + "$CODE_QUALITY_IMAGE" /code +``` -| Sub-directory | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | -|----------------|----------------------------------------------------|-----------------------------------------------------------------------| -| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | -| `/Jobs/*` | Auto DevOps related jobs | No | -| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | -| `/Security/*` | Security related jobs | Yes | -| `/Terraform/*` | Infrastructure as Code related templates | No | -| `/Verify/*` | Verify/testing related jobs | Yes | -| `/Workflows/*` | Common uses of the `workflow:` keyword | No | -| `/*` (root) | General templates | Yes | +You can also use the `|` and `>` YAML operators to [split up multi-line commands](../../ci/yaml/script.md#split-long-commands). -## Criteria +### Explain the template with comments -The file must follow the [`.gitlab-ci.yml` syntax](../../ci/yaml/README.md). -Verify it's valid by pasting it into the CI lint tool at `https://gitlab.com/gitlab-org/gitlab/-/ci/lint`. +You can access template contents from the new file menu, and this might be the only +place users see information about the template. It's important to clearly document +the behavior of the template directly in the template itself. -Also, all templates must be named with the `*.gitlab-ci.yml` suffix. +The following guidelines cover the basic comments expected in all template submissions. +Add additional comments as needed if you think the comments can help users or +[template reviewers](#contribute-cicd-template-merge-requests). + +#### Explain requirements and expectations + +Give the details on how to use the template in `#` comments at the top of the file. +This includes: + +- Repository/project requirements. +- Expected behavior. +- Any places that need to be edited by users before using the template. +- If the template should be used by copy pasting it into a configuration file, or + by using it with the `include` keyword in an existing pipeline. +- If any variables need to be saved in the project's CI/CD settings. + +```yaml +# Use this template to publish an application that uses the ABC server. +# You can copy and paste this template into a new `.gitlab-ci.yml` file. +# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword. +# +# Requirements: +# - An ABC project with content saved in /content and tests in /test +# - A CI/CD variable named ABC-PASSWORD saved in the project CI/CD settings. The value +# should be the password used to deploy to your ABC server. +# - An ABC server configured to listen on port 12345. +# +# You must change the URL on line 123 to point to your ABC server and port. +# +# For more information, see https://gitlab.com/example/abcserver/README.md + +job1: + ... +``` + +#### Explain how variables affect template behavior + +If the template uses variables, explain them in `#` comments where they are first +defined. You can skip the comment when the variable is trivially clear: + +```yaml +variables: # Good to have a comment here, for example: + TEST_CODE_PATH: <path/to/code> # Update this variable with the relative path to your Ruby specs + +job1: + variables: + ERROR_MESSAGE: "The $TEST_CODE_PATH path is invalid" # (No need for a comment here, it's already clear) + script: + - echo ${ERROR_MESSAGE} +``` ### Backward compatibility @@ -65,18 +252,6 @@ users have to fix their `.gitlab-ci.yml` that could annoy their workflow. Please read [versioning](#versioning) section for introducing breaking change safely. -### Best practices - -- Avoid using [global keywords](../../ci/yaml/README.md#global-keywords), - such as `image`, `stages` and `variables` at top-level. - When a root `.gitlab-ci.yml` [includes](../../ci/yaml/README.md#include) - multiple templates, these global keywords could be overridden by the - others and cause an unexpected behavior. -- Include [a changelog](../changelog.md) if your merge request introduces a user-facing change. -- Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) - instead of a hardcoded `main` branch, and never use `master`. -- Use [`rules`](../../ci/yaml/README.md#rules) instead of [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic), if possible. - ## Versioning Versioning allows you to introduce a new template without modifying the existing @@ -135,7 +310,7 @@ include: ### Further reading There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about -introducing versioning concepts in GitLab CI Templates. You can check that issue to +introducing versioning concepts in GitLab CI/CD templates. You can check that issue to follow the progress. ## Testing @@ -157,7 +332,7 @@ This is useful information for reviewers to make sure the template is safe to be ### Make sure the new template can be selected in UI -Templates located under some directories are also [selectable in the **New file** UI](#place-the-template-file-in-a-relevant-directory). +Templates located under some directories are also [selectable in the **New file** UI](#template-directories). When you add a template into one of those directories, make sure that it correctly appears in the dropdown:  @@ -187,6 +362,10 @@ A template could contain malicious code. For example, a template that contains t might accidentally expose secret project CI/CD variables in a job log. If you're unsure if it's secure or not, you need to ask security experts for cross-validation. -## Contribute CI/CD Template Merge Requests +## Contribute CI/CD template merge requests -After your CI/CD Template MR is created and labeled with `ci::templates`, DangerBot suggests one reviewer and one maintainer that can review your code. When your merge request is ready for review, please `@mention` the reviewer and ask them to review your CI/CD Template changes. See details in the merge request that added [a DangerBot task for CI/CD Template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). +After your CI/CD template MR is created and labeled with `ci::templates`, DangerBot +suggests one reviewer and one maintainer that can review your code. When your merge +request is ready for review, please `@mention` the reviewer and ask them to review +your CI/CD template changes. See details in the merge request that added +[a DangerBot task for CI/CD template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). |