diff options
Diffstat (limited to 'doc/ci')
| -rw-r--r-- | doc/ci/README.md | 4 | ||||
| -rw-r--r-- | doc/ci/api/README.md | 21 | ||||
| -rw-r--r-- | doc/ci/api/builds.md | 138 | ||||
| -rw-r--r-- | doc/ci/api/runners.md | 45 | ||||
| -rw-r--r-- | doc/ci/docker/using_docker_build.md | 224 | ||||
| -rw-r--r-- | doc/ci/docker/using_docker_images.md | 2 | ||||
| -rw-r--r-- | doc/ci/environments.md | 58 | ||||
| -rw-r--r-- | doc/ci/examples/README.md | 3 | ||||
| -rw-r--r-- | doc/ci/examples/php.md | 4 | ||||
| -rw-r--r-- | doc/ci/examples/test-scala-application.md | 47 | ||||
| -rw-r--r-- | doc/ci/pipelines.md | 38 | ||||
| -rw-r--r-- | doc/ci/quick_start/README.md | 64 | ||||
| -rw-r--r-- | doc/ci/quick_start/img/pipelines_status.png | bin | 0 -> 89387 bytes | |||
| -rw-r--r-- | doc/ci/runners/README.md | 14 | ||||
| -rw-r--r-- | doc/ci/variables/README.md | 2 | ||||
| -rw-r--r-- | doc/ci/yaml/README.md | 202 |
16 files changed, 557 insertions, 309 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md index 4abc45bf9bb..3dd4e2bc230 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -5,6 +5,8 @@ - [Get started with GitLab CI](quick_start/README.md) - [CI examples for various languages](examples/README.md) - [Learn how to enable or disable GitLab CI](enable_or_disable_ci.md) +- [Pipelines and builds](pipelines.md) +- [Environments and deployments](environments.md) - [Learn how `.gitlab-ci.yml` works](yaml/README.md) - [Configure a Runner, the application that runs your builds](runners/README.md) - [Use Docker images with GitLab Runner](docker/using_docker_images.md) @@ -14,5 +16,5 @@ - [Trigger builds through the API](triggers/README.md) - [Build artifacts](build_artifacts/README.md) - [User permissions](permissions/README.md) -- [API](api/README.md) +- [API](../../api/ci/README.md) - [CI services (linked docker containers)](services/README.md) diff --git a/doc/ci/api/README.md b/doc/ci/api/README.md index aea808007fc..4ca8d92d7cc 100644 --- a/doc/ci/api/README.md +++ b/doc/ci/api/README.md @@ -1,22 +1,3 @@ # GitLab CI API -## Purpose - -Main purpose of GitLab CI API is to provide necessary data and context for -GitLab CI Runners. - -For consumer API take a look at this [documentation](../../api/README.md) where -you will find all relevant information. - -## API Prefix - -Current CI API prefix is `/ci/api/v1`. - -You need to prepend this prefix to all examples in this documentation, like: - - GET /ci/api/v1/builds/:id/artifacts - -## Resources - -- [Builds](builds.md) -- [Runners](runners.md) +This document was moved to a [new location](../../api/ci/README.md). diff --git a/doc/ci/api/builds.md b/doc/ci/api/builds.md index 79761a893da..f5bd3181c02 100644 --- a/doc/ci/api/builds.md +++ b/doc/ci/api/builds.md @@ -1,139 +1,3 @@ # Builds API -API used by runners to receive and update builds. - -_**Note:** This API is intended to be used only by Runners as their own -communication channel. For the consumer API see the -[Builds API](../../api/builds.md)._ - -## Authentication - -This API uses two types of authentication: - -1. Unique runner's token - - Token assigned to runner after it has been registered. - -2. Using build authorization token - - This is project's CI token that can be found in Continuous Integration - project settings. - - Build authorization token can be passed as a parameter or a value of - `BUILD-TOKEN` header. This method are interchangeable. - -## Builds - -### Runs oldest pending build by runner - -``` -POST /ci/api/v1/builds/register -``` - -| Attribute | Type | Required | Description | -|-----------|---------|----------|---------------------| -| `token` | string | yes | Unique runner token | - - -``` -curl -X POST "https://gitlab.example.com/ci/api/v1/builds/register" -F "token=t0k3n" -``` - -### Update details of an existing build - -``` -PUT /ci/api/v1/builds/:id -``` - -| Attribute | Type | Required | Description | -|-----------|---------|----------|----------------------| -| `id` | integer | yes | The ID of a project | -| `token` | string | yes | Unique runner token | -| `state` | string | no | The state of a build | -| `trace` | string | no | The trace of a build | - -``` -curl -X PUT "https://gitlab.example.com/ci/api/v1/builds/1234" -F "token=t0k3n" -F "state=running" -F "trace=Running git clone...\n" -``` - -### Incremental build trace update - -Using this method you need to send trace content as a request body. You also need to provide the `Content-Range` header -with a range of sent trace part. Note that you need to send parts in the proper order, so the begining of the part -must start just after the end of the previous part. If you provide the wrong part, then GitLab CI API will return `416 -Range Not Satisfiable` response with a header `Range: 0-X`, where `X` is the current trace length. - -For example, if you receive `Range: 0-11` in the response, then your next part must contain a `Content-Range: 11-...` -header and a trace part covered by this range. - -For a valid update API will return `202` response with: -* `Build-Status: {status}` header containing current status of the build, -* `Range: 0-{length}` header with the current trace length. - -``` -PATCH /ci/api/v1/builds/:id/trace.txt -``` - -Parameters: - -| Attribute | Type | Required | Description | -|-----------|---------|----------|----------------------| -| `id` | integer | yes | The ID of a build | - -Headers: - -| Attribute | Type | Required | Description | -|-----------------|---------|----------|-----------------------------------| -| `BUILD-TOKEN` | string | yes | The build authorization token | -| `Content-Range` | string | yes | Bytes range of trace that is sent | - -``` -curl -X PATCH "https://gitlab.example.com/ci/api/v1/builds/1234/trace.txt" -H "BUILD-TOKEN=build_t0k3n" -H "Content-Range=0-21" -d "Running git clone...\n" -``` - - -### Upload artifacts to build - -``` -POST /ci/api/v1/builds/:id/artifacts -``` - -| Attribute | Type | Required | Description | -|-----------|---------|----------|-------------------------------| -| `id` | integer | yes | The ID of a build | -| `token` | string | yes | The build authorization token | -| `file` | mixed | yes | Artifacts file | - -``` -curl -X POST "https://gitlab.example.com/ci/api/v1/builds/1234/artifacts" -F "token=build_t0k3n" -F "file=@/path/to/file" -``` - -### Download the artifacts file from build - -``` -GET /ci/api/v1/builds/:id/artifacts -``` - -| Attribute | Type | Required | Description | -|-----------|---------|----------|-------------------------------| -| `id` | integer | yes | The ID of a build | -| `token` | string | yes | The build authorization token | - -``` -curl "https://gitlab.example.com/ci/api/v1/builds/1234/artifacts" -F "token=build_t0k3n" -``` - -### Remove the artifacts file from build - -``` -DELETE /ci/api/v1/builds/:id/artifacts -``` - -| Attribute | Type | Required | Description | -|-----------|---------|----------|-------------------------------| -| ` id` | integer | yes | The ID of a build | -| `token` | string | yes | The build authorization token | - -``` -curl -X DELETE "https://gitlab.example.com/ci/api/v1/builds/1234/artifacts" -F "token=build_t0k3n" -``` +This document was moved to a [new location](../../api/ci/builds.md). diff --git a/doc/ci/api/runners.md b/doc/ci/api/runners.md index 2f01da4bd76..b14ea99db76 100644 --- a/doc/ci/api/runners.md +++ b/doc/ci/api/runners.md @@ -1,46 +1,3 @@ # Runners API -API used by runners to register and delete themselves. - -_**Note:** This API is intended to be used only by Runners as their own -communication channel. For the consumer API see the -[new Runners API](../../api/runners.md)._ - -## Authentication - -This API uses two types of authentication: - -1. Unique runner's token - - Token assigned to runner after it has been registered. - -2. Using runners' registration token - - This is a token that can be found in project's settings. - It can be also found in Admin area » Runners settings. - - There are two types of tokens you can pass - shared runner registration - token or project specific registration token. - -## Runners - -### Register a new runner - -Used to make GitLab CI aware of available runners. - - POST /ci/api/v1/runners/register - -Parameters: - - * `token` (required) - Registration token - - -### Delete a runner - -Used to remove runner. - - DELETE /ci/api/v1/runners/delete - -Parameters: - - * `token` (required) - Unique runner token +This document was moved to a [new location](../../api/ci/runners.md). diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index ca52a483a59..7f83f846454 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -4,14 +4,14 @@ GitLab CI allows you to use Docker Engine to build and test docker-based project **This also allows to you to use `docker-compose` and other docker-enabled tools.** -This is one of new trends in Continuous Integration/Deployment to: +One of the new trends in Continuous Integration/Deployment is to: -1. create application image, -1. run test against created image, -1. push image to remote registry, -1. deploy server from pushed image +1. create an application image, +1. run tests against the created image, +1. push image to a remote registry, and +1. deploy to a server from the pushed image. -It's also useful in case when your application already has the `Dockerfile` that can be used to create and test image: +It's also useful when your application already has the `Dockerfile` that can be used to create and test an image: ```bash $ docker build -t my-image dockerfiles/ $ docker run my-docker-image /script/to/run/tests @@ -19,24 +19,25 @@ $ docker tag my-image my-registry:5000/my-image $ docker push my-registry:5000/my-image ``` -However, this requires special configuration of GitLab Runner to enable `docker` support during build. -**This requires running GitLab Runner in privileged mode which can be harmful when untrusted code is run.** +This requires special configuration of GitLab Runner to enable `docker` support during builds. -There are two methods to enable the use of `docker build` and `docker run` during build. +## Runner Configuration -## 1. Use shell executor +There are three methods to enable the use of `docker build` and `docker run` during builds; each with their own tradeoffs. + +### Use shell executor The simplest approach is to install GitLab Runner in `shell` execution mode. -GitLab Runner then executes build scripts as `gitlab-runner` user. +GitLab Runner then executes build scripts as the `gitlab-runner` user. 1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). 1. During GitLab Runner installation select `shell` as method of executing build scripts or use command: ```bash - $ sudo gitlab-runner register -n \ + $ sudo gitlab-ci-multi-runner register -n \ --url https://gitlab.com/ci \ - --token RUNNER_TOKEN \ + --registration-token REGISTRATION_TOKEN \ --executor shell --description "My Runner" ``` @@ -70,16 +71,18 @@ GitLab Runner then executes build scripts as `gitlab-runner` user. 5. You can now use `docker` command and install `docker-compose` if needed. -6. However, by adding `gitlab-runner` to `docker` group you are effectively granting `gitlab-runner` full root permissions. -For more information please checkout [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). +> **Note:** +* By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. +For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). -## 2. Use docker-in-docker executor +### Use docker-in-docker executor -The second approach is to use the special Docker image with all tools installed +The second approach is to use the special docker-in-docker (dind) +[Docker image](https://hub.docker.com/_/docker/) with all tools installed (`docker` and `docker-compose`) and run the build script in context of that image in privileged mode. -In order to do that follow the steps: +In order to do that, follow the steps: 1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). @@ -87,9 +90,9 @@ In order to do that follow the steps: mode: ```bash - sudo gitlab-runner register -n \ + sudo gitlab-ci-multi-runner register -n \ --url https://gitlab.com/ci \ - --token RUNNER_TOKEN \ + --registration-token REGISTRATION_TOKEN \ --executor docker \ --description "My Docker Runner" \ --docker-image "docker:latest" \ @@ -119,11 +122,7 @@ In order to do that follow the steps: Insecure = false ``` - If you want to use the Shared Runners available on your GitLab CE/EE - installation in order to build Docker images, then make sure that your - Shared Runners configuration has the `privileged` mode set to `true`. - -1. You can now use `docker` from build script: +1. You can now use `docker` in the build script (note the inclusion of the `docker:dind` service): ```yaml image: docker:latest @@ -141,14 +140,177 @@ In order to do that follow the steps: - docker run my-docker-image /script/to/run/tests ``` -1. However, by enabling `--docker-privileged` you are effectively disabling all - the security mechanisms of containers and exposing your host to privilege - escalation which can lead to container breakout. - - For more information, check out the official Docker documentation on - [Runtime privilege and Linux capabilities][docker-cap]. +Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges: +* By enabling `--docker-privileged`, you are effectively disabling all of +the security mechanisms of containers and exposing your host to privilege +escalation which can lead to container breakout. For more information, check out the official Docker documentation on +[Runtime privilege and Linux capabilities][docker-cap]. +* Using docker-in-docker, each build is in a clean environment without the past +history. Concurrent builds work fine because every build gets it's own instance of docker engine so they won't conflict with each other. But this also means builds can be slower because there's no caching of layers. +* By default, `docker:dind` uses `--storage-driver vfs` which is the slowest form +offered. An example project using this approach can be found here: https://gitlab.com/gitlab-examples/docker. +### Use Docker socket binding + +The third approach is to bind-mount `/var/run/docker.sock` into the container so that docker is available in the context of that image. + +In order to do that, follow the steps: + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. Register GitLab Runner from the command line to use `docker` and share `/var/run/docker.sock`: + + ```bash + sudo gitlab-ci-multi-runner register -n \ + --url https://gitlab.com/ci \ + --registration-token REGISTRATION_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "docker:latest" \ + --docker-volumes /var/run/docker.sock:/var/run/docker.sock + ``` + + The above command will register a new Runner to use the special + `docker:latest` image which is provided by Docker. **Notice that it's using + the Docker daemon of the Runner itself, and any containers spawned by docker commands will be siblings of the Runner rather than children of the runner.** This may have complications and limitations that are unsuitable for your workflow. + + The above command will create a `config.toml` entry similar to this: + + ``` + [[runners]] + url = "https://gitlab.com/ci" + token = REGISTRATION_TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:latest" + privileged = false + disable_cache = false + volumes = ["/var/run/docker.sock", "/cache"] + [runners.cache] + Insecure = false + ``` + +1. You can now use `docker` in the build script (note that you don't need to include the `docker:dind` service as when using the Docker in Docker executor): + + ```yaml + image: docker:latest + + before_script: + - docker info + + build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +While the above method avoids using Docker in privileged mode, you should be aware of the following implications: +* By sharing the docker daemon, you are effectively disabling all +the security mechanisms of containers and exposing your host to privilege +escalation which can lead to container breakout. For example, if a project +ran `docker rm -f $(docker ps -a -q)` it would remove the GitLab Runner +containers. +* Concurrent builds may not work; if your tests +create containers with specific names, they may conflict with each other. +* Sharing files and directories from the source repo into containers may not +work as expected since volume mounting is done in the context of the host +machine, not the build container. +e.g. `docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests` + +## Using the GitLab Container Registry + +> **Note:** +This feature requires GitLab 8.8 and GitLab Runner 1.2. + +Once you've built a Docker image, you can push it up to the built-in [GitLab Container Registry](../../container_registry/README.md). For example, if you're using +docker-in-docker on your runners, this is how your `.gitlab-ci.yml` could look: + + +```yaml + build: + image: docker:latest + services: + - docker:dind + stage: build + script: + - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.example.com + - docker build -t registry.example.com/group/project:latest . + - docker push registry.example.com/group/project:latest +``` + +You have to use the credentials of the special `gitlab-ci-token` user with its +password stored in `$CI_BUILD_TOKEN` in order to push to the Registry connected +to your project. This allows you to automate building and deployment of your +Docker images. + +Here's a more elaborate example that splits up the tasks into 4 pipeline stages, +including two tests that run in parallel. The build is stored in the container +registry and used by subsequent stages, downloading the image +when needed. Changes to `master` also get tagged as `latest` and deployed using +an application-specific deploy script: + +```yaml +image: docker:latest +services: +- docker:dind + +stages: +- build +- test +- release +- deploy + +variables: + CONTAINER_TEST_IMAGE: registry.example.com/my-group/my-project:$CI_BUILD_REF_NAME + CONTAINER_RELEASE_IMAGE: registry.example.com/my-group/my-project:latest + +before_script: + - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.example.com + +build: + stage: build + script: + - docker build --pull -t $CONTAINER_TEST_IMAGE . + - docker push $CONTAINER_TEST_IMAGE + +test1: + stage: test + script: + - docker pull $CONTAINER_TEST_IMAGE + - docker run $CONTAINER_TEST_IMAGE /script/to/run/tests + +test2: + stage: test + script: + - docker pull $CONTAINER_TEST_IMAGE + - docker run $CONTAINER_TEST_IMAGE /script/to/run/another/test + +release-image: + stage: release + script: + - docker pull $CONTAINER_TEST_IMAGE + - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE + - docker push $CONTAINER_RELEASE_IMAGE + only: + - master + +deploy: + stage: deploy + script: + - ./deploy.sh + only: + - master +``` + +Some things you should be aware of when using the Container Registry: +* You must log in to the container registry before running commands. Putting this in `before_script` will run it before each build job. +* Using `docker build --pull` makes sure that Docker fetches any changes to base images before building just in case your cache is stale. It takes slightly longer, but means you don’t get stuck without security patches to base images. +* Doing an explicit `docker pull` before each `docker run` makes sure to fetch the latest image that was just built. This is especially important if you are using multiple runners that cache images locally. Using the git SHA in your image tag makes this less necessary since each build will be unique and you shouldn't ever have a stale image, but it's still possible if you re-build a given commit after a dependency has changed. +* You don't want to build directly to `latest` in case there are multiple builds happening simultaneously. + [docker-in-docker]: https://blog.docker.com/2013/09/docker-can-now-run-within-docker/ [docker-cap]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 56ac2195c49..a849905ac6b 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -23,7 +23,7 @@ To use GitLab Runner with docker you need to register a new runner to use the `docker` executor: ```bash -gitlab-runner register \ +gitlab-ci-multi-runner register \ --url "https://gitlab.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby-2.1" \ diff --git a/doc/ci/environments.md b/doc/ci/environments.md new file mode 100644 index 00000000000..d85b8a34ced --- /dev/null +++ b/doc/ci/environments.md @@ -0,0 +1,58 @@ +# Introduction to environments and deployments + +>**Note:** +Introduced in GitLab 8.9. + +## Environments + +Environments are places where code gets deployed, such as staging or production. +CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. +Defining environments in a project's `.gitlab-ci.yml` lets developers track +[deployments] to these environments. + +## Deployments + +Deployments are created when [jobs] deploy versions of code to [environments]. + +## Defining environments + +You can create and delete environments manually in the web interface, but we +recommend that you define your environments in `.gitlab-ci.yml` first, which +will automatically create environments for you after the first deploy. + +The `environment` is just a hint for GitLab that this job actually deploys to +this environment. Each time the job succeeds, a deployment is recorded, +remembering the git SHA and environment. + +Add something like this to your `.gitlab-ci.yml`: +``` +production: + stage: deploy + script: dpl... + environment: production +``` + +See full [documentation](yaml/README.md#environment). + +## Seeing environment status + +You can find the environment list under **Pipelines > Environments** for your +project. You'll see the git SHA and date of the last deployment to each +environment defined. + +>**Note:** +Only deploys that happen after your `.gitlab-ci.yml` is properly configured will +show up in the environments and deployments lists. + +## Seeing deployment history + +Clicking on an environment will show the history of deployments. + +>**Note:** +Only deploys that happen after your `.gitlab-ci.yml` is properly configured will +show up in the environments and deployments lists. + +[Pipelines]: pipelines.md +[jobs]: yaml/README.md#jobs +[environments]: #environments +[deployments]: #deployments diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 61294be599d..27bc21c2922 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -4,7 +4,8 @@ - [Test and deploy a Ruby application to Heroku](test-and-deploy-ruby-application-to-heroku.md) - [Test and deploy a Python application to Heroku](test-and-deploy-python-application-to-heroku.md) - [Test a Clojure application](test-clojure-application.md) -- [Using `dpl` as deployment tool](../deployment/README.md) +- [Test a Scala application](test-scala-application.md) +- [Using `dpl` as deployment tool](deployment/README.md) - Help your favorite programming language and GitLab by sending a merge request with a guide for that language. diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 26953014502..17e1c64bb8a 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -263,10 +263,10 @@ terminal execute: ```bash # Check using docker executor -gitlab-runner exec docker test:app +gitlab-ci-multi-runner exec docker test:app # Check using shell executor -gitlab-runner exec shell test:app +gitlab-ci-multi-runner exec shell test:app ``` ## Example project diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md new file mode 100644 index 00000000000..7412fdbbc78 --- /dev/null +++ b/doc/ci/examples/test-scala-application.md @@ -0,0 +1,47 @@ +## Test a Scala application + +This example demonstrates the integration of Gitlab CI with Scala +applications using SBT. Checkout the example +[project](https://gitlab.com/gitlab-examples/scala-sbt) and +[build status](https://gitlab.com/gitlab-examples/scala-sbt/builds). + +### Add `.gitlab-ci.yml` file to project + +The following `.gitlab-ci.yml` should be added in the root of your +repository to trigger CI: + +``` yaml +image: java:8 + +before_script: + - apt-get update -y + - apt-get install apt-transport-https -y + # Install SBT + - echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list + - apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823 + - apt-get update -y + - apt-get install sbt -y + - sbt sbt-version + +test: + script: + - sbt clean coverage test coverageReport +``` + +The `before_script` installs [SBT](http://www.scala-sbt.org/) and +displays the version that is being used. The `test` stage executes SBT +to compile and test the project. +[scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT +plugin to measure test coverage. + +You can use other versions of Scala and SBT by defining them in +`build.sbt`. + +### Display test coverage in build + +Add the `Coverage was \[\d+.\d+\%\]` regular expression in the +**Settings > Edit Project > Test coverage parsing** project setting to +retrieve the test coverage rate from the build trace and have it +displayed with your builds. + +**Builds** must be enabled for this option to appear. diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md new file mode 100644 index 00000000000..48a9f994759 --- /dev/null +++ b/doc/ci/pipelines.md @@ -0,0 +1,38 @@ +# Introduction to pipelines and builds + +>**Note:** +Introduced in GitLab 8.8. + +## Pipelines + +A pipeline is a group of [builds] that get executed in [stages] (batches). All +of the builds in a stage are executed in parallel (if there are enough +concurrent [runners]), and if they all succeed, the pipeline moves on to the +next stage. If one of the builds fails, the next stage is not (usually) +executed. + +## Builds + +Builds are individual runs of [jobs]. Not to be confused with a `build` job or +`build` stage. + +## Defining pipelines + +Pipelines are defined in `.gitlab-ci.yml` by specifying [jobs] that run in +[stages]. + +See full [documentation](yaml/README.md#jobs). + +## Seeing pipeline status + +You can find the current and historical pipeline runs under **Pipelines** for your +project. + +## Seeing build status + +Clicking on a pipeline will show the builds that were run for that pipeline. + +[builds]: #builds +[jobs]: yaml/README.md#jobs +[stages]: yaml/README.md#stages +[runners]: runners/README.md diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 386b8e29fcf..7fa1a478f34 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -4,41 +4,41 @@ is fully integrated into GitLab itself and is [enabled] by default on all projects. -The TL;DR version of how GitLab CI works is the following. - ---- - GitLab offers a [continuous integration][ci] service. If you [add a `.gitlab-ci.yml` file][yaml] to the root directory of your repository, and configure your GitLab project to use a [Runner], then each merge request or -push triggers a build. +push triggers your CI [pipeline]. -The `.gitlab-ci.yml` file tells the GitLab runner what to do. By default it -runs three [stages]: `build`, `test`, and `deploy`. +The `.gitlab-ci.yml` file tells the GitLab runner what to do. By default it runs +a pipeline with three [stages]: `build`, `test`, and `deploy`. You don't need to +use all three stages; stages with no jobs are simply ignored. If everything runs OK (no non-zero return values), you'll get a nice green checkmark associated with the pushed commit or merge request. This makes it -easy to see whether a merge request will cause any of the tests to fail before +easy to see whether a merge request caused any of the tests to fail before you even look at the code. -Most projects only use GitLab's CI service to run the test suite so that +Most projects use GitLab's CI service to run the test suite so that developers get immediate feedback if they broke something. +There's a growing trend to use continuous delivery and continuous deployment to +automatically deploy tested code to staging and production environments. + So in brief, the steps needed to have a working CI can be summed up to: 1. Add `.gitlab-ci.yml` to the root directory of your repository 1. Configure a Runner -From there on, on every push to your Git repository, the build will be -automagically started by the Runner and will appear under the project's -`/builds` page. +From there on, on every push to your Git repository, the Runner will +automagically start the pipeline and the pipeline will appear under the +project's `/pipelines` page. --- This guide assumes that you: - have a working GitLab instance of version 8.0 or higher or are using - [GitLab.com](https://gitlab.com/users/sign_in) + [GitLab.com](https://gitlab.com) - have a project in GitLab that you would like to use CI for Let's break it down to pieces and work on solving the GitLab CI puzzle. @@ -57,15 +57,14 @@ On any push to your repository, GitLab will look for the `.gitlab-ci.yml` file and start builds on _Runners_ according to the contents of the file, for that commit. -Because `.gitlab-ci.yml` is in the repository, it is version controlled, -old versions still build successfully, forks can easily make use of CI, -branches can have separate builds and you have a single source of truth for CI. -You can read more about the reasons why we are using `.gitlab-ci.yml` -[in our blog about it][blog-ci]. +Because `.gitlab-ci.yml` is in the repository and is version controlled, old +versions still build successfully, forks can easily make use of CI, branches can +have different pipelines and jobs, and you have a single source of truth for CI. +You can read more about the reasons why we are using `.gitlab-ci.yml` [in our +blog about it][blog-ci]. **Note:** `.gitlab-ci.yml` is a [YAML](https://en.wikipedia.org/wiki/YAML) file -so you have to pay extra attention to the indentation. Always use spaces, not -tabs. +so you have to pay extra attention to indentation. Always use spaces, not tabs. ### Creating a simple `.gitlab-ci.yml` file @@ -108,7 +107,7 @@ If you want to check whether your `.gitlab-ci.yml` file is valid, there is a Lint tool under the page `/ci/lint` of your GitLab instance. You can also find the link under **Settings > CI settings** in your project. -For more information and a complete `.gitlab-ci.yml` syntax, please check +For more information and a complete `.gitlab-ci.yml` syntax, please read [the documentation on .gitlab-ci.yml](../yaml/README.md). ### Push `.gitlab-ci.yml` to GitLab @@ -122,7 +121,8 @@ git commit -m "Add .gitlab-ci.yml" git push origin master ``` -Now if you go to the **Builds** page you will see that the builds are pending. +Now if you go to the **Pipelines** page you will see that the pipeline is +pending. You can also go to the **Commits** page and notice the little clock icon next to the commit SHA. @@ -138,15 +138,14 @@ Notice that there are two jobs pending which are named after what we wrote in `.gitlab-ci.yml`. The red triangle indicates that there is no Runner configured yet for these builds. -The next step is to configure a Runner so that it picks the pending jobs. +The next step is to configure a Runner so that it picks the pending builds. ## Configuring a Runner -In GitLab, Runners run the builds that you define in `.gitlab-ci.yml`. -A Runner can be a virtual machine, a VPS, a bare-metal machine, a docker -container or even a cluster of containers. GitLab and the Runners communicate -through an API, so the only needed requirement is that the machine on which the -Runner is configured to have Internet access. +In GitLab, Runners run the builds that you define in `.gitlab-ci.yml`. A Runner +can be a virtual machine, a VPS, a bare-metal machine, a docker container or +even a cluster of containers. GitLab and the Runners communicate through an API, +so the only requirement is that the Runner's machine has Internet access. A Runner can be specific to a certain project or serve multiple projects in GitLab. If it serves all projects it's called a _Shared Runner_. @@ -188,12 +187,16 @@ To enable **Shared Runners** you have to go to your project's [Read more on Shared Runners](../runners/README.md). -## Seeing the status of your build +## Seeing the status of your pipeline and builds After configuring the Runner successfully, you should see the status of your last commit change from _pending_ to either _running_, _success_ or _failed_. -You can view all builds, by going to the **Builds** page in your project. +You can view all pipelines by going to the **Pipelines** page in your project. + + + +Or you can view all builds, by going to the **Pipelines > Builds** page.  @@ -238,3 +241,4 @@ CI with various languages. [runner]: ../runners/README.md [enabled]: ../enable_or_disable_ci.md [stages]: ../yaml/README.md#stages +[pipeline]: ../pipelines.md diff --git a/doc/ci/quick_start/img/pipelines_status.png b/doc/ci/quick_start/img/pipelines_status.png Binary files differnew file mode 100644 index 00000000000..6bc97bb739c --- /dev/null +++ b/doc/ci/quick_start/img/pipelines_status.png diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index b42d7a62ebc..ddebd987650 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -63,10 +63,10 @@ instance. Now simply register the runner as any runner: ``` -sudo gitlab-runner register +sudo gitlab-ci-multi-runner register ``` -Shared runners are enabled by default as of GitLab 8.2, but can be disabled with the +Shared runners are enabled by default as of GitLab 8.2, but can be disabled with the `DISABLE SHARED RUNNERS` button. Previous versions of GitLab defaulted shared runners to disabled. @@ -93,9 +93,15 @@ setup a specific runner for this project. To register the runner, run the command below and follow instructions: ``` -sudo gitlab-runner register +sudo gitlab-ci-multi-runner register ``` +### Lock a specific runner from being enabled for other projects + +You can configure a runner to assign it exclusively to a project. When a +runner is locked this way, it can no longer be enabled for other projects. +This setting is available on each runner in *Project Settings* > *Runners*. + ### Making an existing Shared Runner Specific If you are an admin on your GitLab instance, @@ -128,7 +134,7 @@ the appropriate dependencies to run Rails test suites. ### Prevent runner with tags from picking jobs without tags You can configure a runner to prevent it from picking jobs with tags when -the runnner does not have tags assigned. This setting is available on each +the runner does not have tags assigned. This setting is available on each runner in *Project Settings* > *Runners*. ### Be careful with sensitive information diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 70fb81492d6..137b080a8f7 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -34,6 +34,7 @@ The `API_TOKEN` will take the Secure Variable value: `SECURE`. | **CI_BUILD_ID** | all | The unique id of the current build that GitLab CI uses internally | | **CI_BUILD_REPO** | all | The URL to clone the Git repository | | **CI_BUILD_TRIGGERED** | 0.5 | The flag to indicate that build was [triggered] | +| **CI_BUILD_TOKEN** | 1.2 | Token used for authenticating with the GitLab Container Registry | | **CI_PROJECT_ID** | all | The unique id of the current project that GitLab CI uses internally | | **CI_PROJECT_DIR** | all | The full path where the repository is cloned and where the build is ran | @@ -50,6 +51,7 @@ export CI_BUILD_TAG="1.0.0" export CI_BUILD_NAME="spec:other" export CI_BUILD_STAGE="test" export CI_BUILD_TRIGGERED="true" +export CI_BUILD_TOKEN="abcde-1234ABCD5678ef" export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" export CI_PROJECT_ID="34" export CI_SERVER="yes" diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 0707555e393..1892acda29b 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -13,30 +13,34 @@ If you want a quick introduction to GitLab CI, follow our **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - [.gitlab-ci.yml](#gitlab-ci-yml) - - [image and services](#image-and-services) - - [before_script](#before_script) - - [after_script](#after_script) - - [stages](#stages) - - [types](#types) - - [variables](#variables) - - [cache](#cache) - - [cache:key](#cache-key) + - [image and services](#image-and-services) + - [before_script](#before_script) + - [after_script](#after_script) + - [stages](#stages) + - [types](#types) + - [variables](#variables) + - [cache](#cache) + - [cache:key](#cache-key) - [Jobs](#jobs) - - [script](#script) - - [stage](#stage) - - [job variables](#job-variables) - - [only and except](#only-and-except) - - [tags](#tags) - - [when](#when) - - [artifacts](#artifacts) - - [artifacts:name](#artifacts-name) - - [artifacts:when](#artifacts-when) - - [dependencies](#dependencies) - - [before_script and after_script](#before_script-and-after_script) + - [script](#script) + - [stage](#stage) + - [only and except](#only-and-except) + - [job variables](#job-variables) + - [tags](#tags) + - [when](#when) + - [environment](#environment) + - [artifacts](#artifacts) + - [artifacts:name](#artifactsname) + - [artifacts:when](#artifactswhen) + - [artifacts:expire_in](#artifactsexpire_in) + - [dependencies](#dependencies) + - [before_script and after_script](#before_script-and-after_script) +- [Git Strategy](#git-strategy) +- [Shallow cloning](#shallow-cloning) - [Hidden jobs](#hidden-jobs) - [Special YAML features](#special-yaml-features) - - [Anchors](#anchors) -- [Validate the .gitlab-ci.yml](#validate-the-gitlab-ci-yml) + - [Anchors](#anchors) +- [Validate the .gitlab-ci.yml](#validate-the-gitlab-ciyml) - [Skipping builds](#skipping-builds) - [Examples](#examples) @@ -52,7 +56,7 @@ of your repository and contains definitions of how your project should be built. The YAML file defines a set of jobs with constraints stating when they should be run. The jobs are defined as top-level elements with a name and always have -to contain the `script` clause: +to contain at least the `script` clause: ```yaml job1: @@ -163,9 +167,9 @@ stages: There are also two edge cases worth mentioning: -1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, +1. If no `stages` are defined in `.gitlab-ci.yml`, then by default the `build`, `test` and `deploy` are allowed to be used as job's stage by default. -2. If a job doesn't specify `stage`, the job is assigned the `test` stage. +2. If a job doesn't specify a `stage`, the job is assigned the `test` stage. ### types @@ -176,9 +180,9 @@ Alias for [stages](#stages). >**Note:** Introduced in GitLab Runner v0.5.0. -GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build -environment. The variables are stored in the git repository and are meant to -store non-sensitive project configuration, for example: +GitLab CI allows you to add variables to `.gitlab-ci.yml` that are set in the +build environment. The variables are stored in the git repository and are meant +to store non-sensitive project configuration, for example: ```yaml variables: @@ -251,8 +255,8 @@ rspec: - binaries/ ``` -The cache is provided on best effort basis, so don't expect that cache will be -always present. For implementation details please check GitLab Runner. +The cache is provided on a best-effort basis, so don't expect that the cache +will be always present. For implementation details, please check GitLab Runner. #### cache:key @@ -353,6 +357,7 @@ job_name: | cache | no | Define list of files that should be cached between subsequent runs | | before_script | no | Override a set of commands that are executed before build | | after_script | no | Override a set of commands that are executed after build | +| environment | no | Defines a name of environment to which deployment is done by this build | ### script @@ -476,10 +481,10 @@ failure. `when` can be set to one of the following values: 1. `on_success` - execute build only when all builds from prior stages - succeeded. This is the default. + succeed. This is the default. 1. `on_failure` - execute build only when at least one build from prior stages - failed. -1. `always` - execute build despite the status of builds from prior stages. + fails. +1. `always` - execute build regardless of the status of builds from prior stages. For example: @@ -524,6 +529,36 @@ The above script will: 1. Execute `cleanup_build_job` only when `build_job` fails 2. Always execute `cleanup_job` as the last step in pipeline. +### environment + +>**Note:** +Introduced in GitLab 8.9. + +`environment` is used to define that a job deploys to a specific environment. +This allows easy tracking of all deployments to your environments straight from +GitLab. + +If `environment` is specified and no environment under that name exists, a new +one will be created automatically. + +The `environment` name must contain only letters, digits, '-' and '_'. Common +names are `qa`, `staging`, and `production`, but you can use whatever name works +with your workflow. + +--- + +**Example configurations** + +``` +deploy to production: + stage: deploy + script: git push production HEAD:master + environment: production +``` + +The `deploy to production` job will be marked as doing deployment to +`production` environment. + ### artifacts >**Notes:** @@ -531,10 +566,10 @@ The above script will: > - Introduced in GitLab Runner v0.7.0 for non-Windows platforms. > - Windows support was added in GitLab Runner v.1.0.0. > - Currently not all executors are supported. -> - Build artifacts are only collected for successful builds. +> - Build artifacts are only collected for successful builds by default. -`artifacts` is used to specify list of files and directories which should be -attached to build after success. To pass artifacts between different builds, +`artifacts` is used to specify a list of files and directories which should be +attached to the build after success. To pass artifacts between different builds, see [dependencies](#dependencies). Below are some examples. @@ -662,9 +697,9 @@ failure. `artifacts:when` can be set to one of the following values: -1. `on_success` - upload artifacts only when build succeeds. This is the default -1. `on_failure` - upload artifacts only when build fails -1. `always` - upload artifacts despite the build status +1. `on_success` - upload artifacts only when the build succeeds. This is the default. +1. `on_failure` - upload artifacts only when the build fails. +1. `always` - upload artifacts regardless of the build status. --- @@ -678,6 +713,42 @@ job: when: on_failure ``` +#### artifacts:expire_in + +>**Note:** +Introduced in GitLab 8.9 and GitLab Runner v1.3.0. + +`artifacts:expire_in` is used to delete uploaded artifacts after the specified +time. By default, artifacts are stored on GitLab forever. `expire_in` allows you +to specify how long artifacts should live before they expire, counting from the +time they are uploaded and stored on GitLab. + +You can use the **Keep** button on the build page to override expiration and +keep artifacts forever. + +After expiry, artifacts are actually deleted hourly by default (via a cron job), +but they are not accessible after expiry. + +The value of `expire_in` is an elapsed time. Examples of parseable values: +- '3 mins 4 sec' +- '2 hrs 20 min' +- '2h20min' +- '6 mos 1 day' +- '47 yrs 6 mos and 4d' +- '3 weeks and 2 days' + +--- + +**Example configurations** + +To expire artifacts 1 week after being uploaded: + +```yaml +job: + artifacts: + expire_in: 1 week +``` + ### dependencies >**Note:** @@ -752,6 +823,61 @@ job: - execute this after my script ``` +## Git Strategy + +>**Note:** +Introduced in GitLab 8.9 as an experimental feature. May change in future +releases or be removed completely. + +You can set the `GIT_STRATEGY` used for getting recent application code. `clone` +is slower, but makes sure you have a clean directory before every build. `fetch` +is faster. `GIT_STRATEGY` can be specified in the global `variables` section or +in the `variables` section for individual jobs. If it's not specified, then the +default from project settings will be used. + +``` +variables: + GIT_STRATEGY: clone +``` + +or + +``` +variables: + GIT_STRATEGY: fetch +``` + +## Shallow cloning + +>**Note:** +Introduced in GitLab 8.9 as an experimental feature. May change in future +releases or be removed completely. + +You can specify the depth of fetching and cloning using `GIT_DEPTH`. This allows +shallow cloning of the repository which can significantly speed up cloning for +repositories with a large number of commits or old, large binaries. The value is +passed to `git fetch` and `git clone`. + +>**Note:** +If you use a depth of 1 and have a queue of builds or retry +builds, jobs may fail. + +Since Git fetching and cloning is based on a ref, such as a branch name, runners +can't clone a specific commit SHA. If there are multiple builds in the queue, or +you are retrying an old build, the commit to be tested needs to be within the +git history that is cloned. Setting too small a value for `GIT_DEPTH` can make +it impossible to run these old commits. You will see `unresolved reference` in +build logs. You should then reconsider changing `GIT_DEPTH` to a higher value. + +Builds that rely on `git describe` may not work correctly when `GIT_DEPTH` is +set since only part of the git history is present. + +To fetch or clone only the last 3 commits: +``` +variables: + GIT_DEPTH: "3" +``` + ## Hidden jobs >**Note:** |
