diff options
Diffstat (limited to 'doc/ci')
| -rw-r--r-- | doc/ci/examples/README.md | 16 | ||||
| -rw-r--r-- | doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png | bin | 0 -> 22046 bytes | |||
| -rw-r--r-- | doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png | bin | 0 -> 27620 bytes | |||
| -rw-r--r-- | doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png | bin | 0 -> 18789 bytes | |||
| -rw-r--r-- | doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md | 526 | ||||
| -rw-r--r-- | doc/ci/pipelines.md | 3 | ||||
| -rw-r--r-- | doc/ci/runners/README.md | 3 | ||||
| -rw-r--r-- | doc/ci/variables/README.md | 5 | ||||
| -rw-r--r-- | doc/ci/yaml/README.md | 966 |
9 files changed, 1002 insertions, 517 deletions
diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index ffebe1618d3..f69729f602d 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -32,34 +32,38 @@ There's also a collection of repositories with [example projects](https://gitlab - **Debian**: [Continuous Deployment with GitLab: how to build and deploy a Debian Package with GitLab CI](https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/) - **Maven**: [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md) +### Game development + +- [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md) + ### Miscellaneous - [Using `dpl` as deployment tool](deployment/README.md) - [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) -### Code quality analysis +## Code quality analysis [Analyze code quality with the Code Climate CLI](code_climate.md). -### Static Application Security Testing (SAST) +## Static Application Security Testing (SAST) - **(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html) - [Scan your Docker images for vulnerabilities](sast_docker.md) -### Dynamic Application Security Testing (DAST) +## Dynamic Application Security Testing (DAST) Scan your app for vulnerabilities with GitLab [Dynamic Application Security Testing (DAST)](dast.md). -### Browser Performance Testing with Sitespeed.io +## Browser Performance Testing with Sitespeed.io Analyze your [browser performance with Sitespeed.io](browser_performance.md). -### GitLab CI/CD for Review Apps +## GitLab CI/CD for Review Apps - [Example project](https://gitlab.com/gitlab-examples/review-apps-nginx/) that shows how to use GitLab CI/CD for [Review Apps](../review_apps/index.html). - [Dockerizing GitLab Review Apps](https://about.gitlab.com/2017/07/11/dockerizing-review-apps/) -### GitLab CI/CD for GitLab Pages +## GitLab CI/CD for GitLab Pages See the documentation on [GitLab Pages](../../user/project/pages/index.md) for a complete overview. diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png Binary files differnew file mode 100644 index 00000000000..76e0295722b --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png Binary files differnew file mode 100644 index 00000000000..050a97d2726 --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png Binary files differnew file mode 100644 index 00000000000..4ab5d5f401a --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md new file mode 100644 index 00000000000..bfc8558a580 --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -0,0 +1,526 @@ +--- +author: Ryan Hall +author_gitlab: blitzgren +level: intermediary +article_type: tutorial +date: 2018-03-07 +--- + +# DevOps and Game Dev with GitLab CI/CD + +With advances in WebGL and WebSockets, browsers are extremely viable as game development +platforms without the use of plugins like Adobe Flash. Furthermore, by using GitLab and [AWS](https://aws.amazon.com/), +single game developers, as well as game dev teams, can easily host browser-based games online. + +In this tutorial, we'll focus on DevOps, as well as testing and hosting games with Continuous +Integration/Deployment methods. We assume you are familiar with GitLab, javascript, +and the basics of game development. + +## The game + +Our [demo game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) consists of a simple spaceship traveling in space that shoots by clicking the mouse in a given direction. + +Creating a strong CI/CD pipeline at the beginning of developing another game, [Dark Nova](http://darknova.io/about), +was essential for the fast pace the team worked at. This tutorial will build upon my +[previous introductory article](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) and go through the following steps: + +1. Using code from the previous article to start with a barebones [Phaser](https://phaser.io) game built by a gulp file +1. Adding and running unit tests +1. Creating a `Weapon` class that can be triggered to spawn a `Bullet` in a given direction +1. Adding a `Player` class that uses this weapon and moves around the screen +1. Adding the sprites we will use for the `Player` and `Weapon` +1. Testing and deploying with Continuous Integration and Continuous Deployment methods + +By the end, we'll have the core of a [playable game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) +that's tested and deployed on every push to the `master` branch of the [codebase](https://gitlab.com/blitzgren/gitlab-game-demo). +This will also provide +boilerplate code for starting a browser-based game with the following components: + +- Written in [Typescript](https://www.typescriptlang.org/) and [PhaserJs](https://phaser.io) +- Building, running, and testing with [Gulp](http://gulpjs.com/) +- Unit tests with [Chai](http://chaijs.com/) and [Mocha](https://mochajs.org/) +- CI/CD with GitLab +- Hosting the codebase on GitLab.com +- Hosting the game on AWS +- Deploying to AWS + +## Requirements and setup + +Please refer to my previous article [DevOps and Game Dev](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) to learn the foundational +development tools, running a Hello World-like game, and building this game using GitLab +CI/CD from every new push to master. The `master` branch for this game's [repository](https://gitlab.com/blitzgren/gitlab-game-demo) +contains a completed version with all configurations. If you would like to follow along +with this article, you can clone and work from the `devops-article` branch: + +```sh +git clone git@gitlab.com:blitzgren/gitlab-game-demo.git +git checkout devops-article +``` + +Next, we'll create a small subset of tests that exemplify most of the states I expect +this `Weapon` class to go through. To get started, create a folder called `lib/tests` +and add the following code to a new file `weaponTests.ts`: + +```ts +import { expect } from 'chai'; +import { Weapon, BulletFactory } from '../lib/weapon'; + +describe('Weapon', () => { + var subject: Weapon; + var shotsFired: number = 0; + // Mocked bullet factory + var bulletFactory: BulletFactory = <BulletFactory>{ + generate: function(px, py, vx, vy, rot) { + shotsFired++; + } + }; + var parent: any = { x: 0, y: 0 }; + + beforeEach(() => { + shotsFired = 0; + subject = new Weapon(bulletFactory, parent, 0.25, 1); + }); + + it('should shoot if not in cooldown', () => { + subject.trigger(true); + subject.update(0.1); + expect(shotsFired).to.equal(1); + }); + + it('should not shoot during cooldown', () => { + subject.trigger(true); + subject.update(0.1); + subject.update(0.1); + expect(shotsFired).to.equal(1); + }); + + it('should shoot after cooldown ends', () => { + subject.trigger(true); + subject.update(0.1); + subject.update(0.3); // longer than timeout + expect(shotsFired).to.equal(2); + }); + + it('should not shoot if not triggered', () => { + subject.update(0.1); + subject.update(0.1); + expect(shotsFired).to.equal(0); + }); +}); +``` + +To build and run these tests using gulp, let's also add the following gulp functions +to the existing `gulpfile.js` file: + +```ts +gulp.task('build-test', function () { + return gulp.src('src/tests/**/*.ts', { read: false }) + .pipe(tap(function (file) { + // replace file contents with browserify's bundle stream + file.contents = browserify(file.path, { debug: true }) + .plugin(tsify, { project: "./tsconfig.test.json" }) + .bundle(); + })) + .pipe(buffer()) + .pipe(sourcemaps.init({loadMaps: true}) ) + .pipe(gulp.dest('built/tests')); +}); + +gulp.task('run-test', function() { + gulp.src(['./built/tests/**/*.ts']).pipe(mocha()); +}); +``` + +We will start implementing the first part of our game and get these `Weapon` tests to pass. +The `Weapon` class will expose a method to trigger the generation of a bullet at a given +direction and speed. Later we will implement a `Player` class that ties together the user input +to trigger the weapon. In the `src/lib` folder create a `weapon.ts` file. We'll add two classes +to it: `Weapon` and `BulletFactory` which will encapsulate Phaser's **sprite** and +**group** objects, and the logic specific to our game. + +```ts +export class Weapon { + private isTriggered: boolean = false; + private currentTimer: number = 0; + + constructor(private bulletFactory: BulletFactory, private parent: Phaser.Sprite, private cooldown: number, private bulletSpeed: number) { + } + + public trigger(on: boolean): void { + this.isTriggered = on; + } + + public update(delta: number): void { + this.currentTimer -= delta; + + if (this.isTriggered && this.currentTimer <= 0) { + this.shoot(); + } + } + + private shoot(): void { + // Reset timer + this.currentTimer = this.cooldown; + + // Get velocity direction from player rotation + var parentRotation = this.parent.rotation + Math.PI / 2; + var velx = Math.cos(parentRotation); + var vely = Math.sin(parentRotation); + + // Apply a small forward offset so bullet shoots from head of ship instead of the middle + var posx = this.parent.x - velx * 10 + var posy = this.parent.y - vely * 10; + + this.bulletFactory.generate(posx, posy, -velx * this.bulletSpeed, -vely * this.bulletSpeed, this.parent.rotation); + } +} + +export class BulletFactory { + + constructor(private bullets: Phaser.Group, private poolSize: number) { + // Set all the defaults for this BulletFactory's bullet object + this.bullets.enableBody = true; + this.bullets.physicsBodyType = Phaser.Physics.ARCADE; + this.bullets.createMultiple(30, 'bullet'); + this.bullets.setAll('anchor.x', 0.5); + this.bullets.setAll('anchor.y', 0.5); + this.bullets.setAll('outOfBoundsKill', true); + this.bullets.setAll('checkWorldBounds', true); + } + + public generate(posx: number, posy: number, velx: number, vely: number, rot: number): Phaser.Sprite { + // Pull a bullet from Phaser's Group pool + var bullet = this.bullets.getFirstExists(false); + + // Set the few unique properties about this bullet: rotation, position, and velocity + if (bullet) { + bullet.reset(posx, posy); + bullet.rotation = rot; + bullet.body.velocity.x = velx; + bullet.body.velocity.y = vely; + } + + return bullet; + } +} +``` + +Lastly, we'll redo our entry point, `game.ts`, to tie together both `Player` and `Weapon` objects +as well as add them to the update loop. Here is what the updated `game.ts` file looks like: + +```ts +import { Player } from "./player"; +import { Weapon, BulletFactory } from "./weapon"; + +window.onload = function() { + var game = new Phaser.Game(800, 600, Phaser.AUTO, 'gameCanvas', { preload: preload, create: create, update: update }); + var player: Player; + var weapon: Weapon; + + // Import all assets prior to loading the game + function preload () { + game.load.image('player', 'assets/player.png'); + game.load.image('bullet', 'assets/bullet.png'); + } + + // Create all entities in the game, after Phaser loads + function create () { + // Create and position the player + var playerSprite = game.add.sprite(400, 550, 'player'); + playerSprite.anchor.setTo(0.5); + player = new Player(game.input, playerSprite, 150); + + var bulletFactory = new BulletFactory(game.add.group(), 30); + weapon = new Weapon(bulletFactory, player.sprite, 0.25, 1000); + + player.loadWeapon(weapon); + } + + // This function is called once every tick, default is 60fps + function update() { + var deltaSeconds = game.time.elapsedMS / 1000; // convert to seconds + player.update(deltaSeconds); + weapon.update(deltaSeconds); + } +} +``` + +Run `gulp serve` and you can run around and shoot. Wonderful! Let's update our CI +pipeline to include running the tests along with the existing build job. + +## Continuous Integration + +To ensure our changes don't break the build and all tests still pass, we utilize +Continuous Integration (CI) to run these checks automatically for every push. +Read through this article to understand [Continuous Integration, Continuous Delivery, and Continuous Deployment](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/), +and how these methods are leveraged by GitLab. +From the [last tutorial](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) we already have a `gitlab-ci.yml` file set up for building our app from +every push. We need to set up a new CI job for testing, which GitLab CI/CD will run after the build job using our generated artifacts from gulp. + +Please read through the [documentation on CI/CD configuration](../../../ci/yaml/README.md) file to explore its contents and adjust it to your needs. + +### Build your game with GitLab CI/CD + +We need to update our build job to ensure tests get run as well. Add `gulp build-test` +to the end of the `script` array for the existing `build` job. Once these commands run, +we know we will need to access everything in the `built` folder, given by GitLab CI/CD's `artifacts`. +We'll also cache `node_modules` to avoid having to do a full re-pull of those dependencies: +just pack them up in the cache. Here is the full `build` job: + +```yml +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules + artifacts: + paths: + - built +``` + +### Test your game with GitLab CI/CD + +For testing locally, we simply run `gulp run-tests`, which requires gulp to be installed +globally like in the `build` job. We pull `node_modules` from the cache, so the `npm i` +command won't have to do much. In preparation for deployment, we know we will still need +the `built` folder in the artifacts, which will be brought over as default behavior from +the previous job. Lastly, by convention, we let GitLab CI/CD know this needs to be run after +the `build` job by giving it a `test` [stage](../../../ci/yaml/README.md#stages). +Following the YAML structure, the `test` job should look like this: + +```yml +test: + stage: test + script: + - npm i gulp -g + - npm i + - gulp run-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ +``` + +We have added unit tests for a `Weapon` class that shoots on a specified interval. +The `Player` class implements `Weapon` along with the ability to move around and shoot. Also, +we've added test artifacts and a test stage to our GitLab CI/CD pipeline using `.gitlab-ci.yml`, +allowing us to run our tests by every push. +Our entire `.gitlab-ci.yml` file should now look like this: + +```yml +image: node:6 + +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ + +test: + stage: test + script: + - npm i gulp -g + - npm i + - gulp run-test + cache: + policy: pull + paths: + - node_modules/ + artifacts: + paths: + - built/ +``` + +### Run your CI/CD pipeline + +That's it! Add all your new files, commit, and push. For a reference of what our repo should +look like at this point, please refer to the [final commit related to this article on my sample repository](https://gitlab.com/blitzgren/gitlab-game-demo/commit/8b36ef0ecebcf569aeb251be4ee13743337fcfe2). +By applying both build and test stages, GitLab will run them sequentially at every push to +our repository. If all goes well you'll end up with a green check mark on each job for the pipeline: + + + +You can confirm that the tests passed by clicking on the `test` job to enter the full build logs. +Scroll to the bottom and observe, in all its passing glory: + +```sh +$ gulp run-test +[18:37:24] Using gulpfile /builds/blitzgren/gitlab-game-demo/gulpfile.js +[18:37:24] Starting 'run-test'... +[18:37:24] Finished 'run-test' after 21 ms + + + Weapon + ✓ should shoot if not in cooldown + ✓ should not shoot during cooldown + ✓ should shoot after cooldown ends + ✓ should not shoot if not triggered + + + 4 passing (18ms) + +Uploading artifacts... +built/: found 17 matching files +Uploading artifacts to coordinator... ok id=17095874 responseStatus=201 Created token=aaaaaaaa Job succeeded +``` + +## Continuous Deployment + +We have our codebase built and tested on every push. To complete the full pipeline with Continuous Deployment, +let's set up [free web hosting with AWS S3](https://aws.amazon.com/s/dm/optimization/server-side-test/free-tier/free_np/) and a job through which our build artifacts get +deployed. GitLab also has a free static site hosting service we could use, [GitLab Pages](https://about.gitlab.com/features/pages/), +however Dark Nova specifically uses other AWS tools that necessitates using `AWS S3`. +Read through this article that describes [deploying to both S3 and GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) +and further delves into the principles of GitLab CI/CD than discussed in this article. + +### Set up S3 Bucket + +1. Log into your AWS account and go to [S3](https://console.aws.amazon.com/s3/home) +1. Click the **Create Bucket** link at the top +1. Enter a name of your choosing and click next +1. Keep the default **Properties** and click next +1. Click the **Manage group permissions** and allow **Read** for the **Everyone** group, click next +1. Create the bucket, and select it in your S3 bucket list +1. On the right side, click **Properties** and enable the **Static website hosting** category +1. Update the radio button to the **Use this bucket to host a website** selection. Fill in `index.html` and `error.html` respectively + +### Set up AWS Secrets + +We need to be able to deploy to AWS with our AWS account credentials, but we certainly +don't want to put secrets into source code. Luckily GitLab provides a solution for this +with [Secret Variables](../../../ci/variables/README.md). This can get complicated +due to [IAM](https://aws.amazon.com/iam/) management. As a best practice, you shouldn't +use root security credentials. Proper IAM credential management is beyond the scope of this +article, but AWS will remind you that using root credentials is unadvised and against their +best practices, as they should. Feel free to follow best practices and use a custom IAM user's +credentials, which will be the same two credentials (Key ID and Secret). It's a good idea to +fully understand [IAM Best Practices in AWS](http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). We need to add these credentials to GitLab: + +1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) +1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later +  +1. Go to your GitLab project, click **Settings > CI/CD** on the left sidebar +1. Expand the **Secret Variables** section +  +1. Add a key named `AWS_KEY_ID` and copy the key id from Step 2 into the **Value** textbox +1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox + +### Deploy your game with GitLab CI/CD + +To deploy our build artifacts, we need to install the [AWS CLI](https://aws.amazon.com/cli/) on +the Shared Runner. The Shared Runner also needs to be able to authenticate with your AWS +account to deploy the artifacts. By convention, AWS CLI will look for `AWS_ACCESS_KEY_ID` +and `AWS_SECRET_ACCESS_KEY`. GitLab's CI gives us a way to pass the secret variables we +set up in the prior section using the `variables` portion of the `deploy` job. At the end, +we add directives to ensure deployment `only` happens on pushes to `master`. This way, every +single branch still runs through CI, and only merging (or committing directly) to master will +trigger the `deploy` job of our pipeline. Put these together to get the following: + +```yml +deploy: + stage: deploy + variables: + AWS_ACCESS_KEY_ID: "$AWS_KEY_ID" + AWS_SECRET_ACCESS_KEY: "$AWS_KEY_SECRET" + script: + - apt-get update + - apt-get install -y python3-dev python3-pip + - easy_install3 -U pip + - pip3 install --upgrade awscli + - aws s3 sync ./built s3://gitlab-game-demo --region "us-east-1" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers --cache-control "no-cache, no-store, must-revalidate" --delete + only: + - master +``` + +Be sure to update the region and S3 URL in that last script command to fit your setup. +Our final configuration file `.gitlab-ci.yml` looks like: + +```yml +image: node:6 + +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ + +test: + stage: test + script: + - npm i gulp -g + - gulp run-test + cache: + policy: pull + paths: + - node_modules/ + artifacts: + paths: + - built/ + +deploy: + stage: deploy + variables: + AWS_ACCESS_KEY_ID: "$AWS_KEY_ID" + AWS_SECRET_ACCESS_KEY: "$AWS_KEY_SECRET" + script: + - apt-get update + - apt-get install -y python3-dev python3-pip + - easy_install3 -U pip + - pip3 install --upgrade awscli + - aws s3 sync ./built s3://gitlab-game-demo --region "us-east-1" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers --cache-control "no-cache, no-store, must-revalidate" --delete + only: + - master +``` + +## Conclusion + +Within the [demo repository](https://gitlab.com/blitzgren/gitlab-game-demo) you can also find a handful of boilerplate code to get +[Typescript](https://www.typescriptlang.org/), [Mocha](https://mochajs.org/), [Gulp](http://gulpjs.com/) and [Phaser](https://phaser.io) all playing +together nicely with GitLab CI/CD, which is the result of lessons learned while making [Dark Nova](http://darknova.io/). +Using a combination of free and open source software, we have a full CI/CD pipeline, a game foundation, +and unit tests, all running and deployed at every push to master - with shockingly little code. +Errors can be easily debugged through GitLab's build logs, and within minutes of a successful commit, +you can see the changes live on your game. + +Setting up Continous Integration and Continuous Deployment from the start with Dark Nova enables +rapid but stable development. We can easily test changes in a separate [environment](../../../ci/environments.md#introduction-to-environments-and-deployments), +or multiple environments if needed. Balancing and updating a multiplayer game can be ongoing +and tedious, but having faith in a stable deployment with GitLab CI/CD allows +a lot of breathing room in quickly getting changes to players. + +## Further settings + +Here are some ideas to further investigate that can speed up or improve your pipeline: + +- [Yarn](https://yarnpkg.com) instead of npm +- Setup a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ci-yml) image that can preload dependencies and tools (like AWS CLI) +- Forward a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to your game's S3 static website +- Combine jobs if you find it unnecessary for a small project +- Avoid the queues and set up your own [custom GitLab CI/CD runner](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index ac4a9b0ed27..856d7f264e4 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -121,8 +121,9 @@ The basic requirements is that there are two numbers separated with one of the following (you can even use them interchangeably): - a space -- a backslash (`/`) +- a forward slash (`/`) - a colon (`:`) +- a dot (`.`) >**Note:** More specifically, [it uses][regexp] this regular expression: `\d+[\s:\/\\]+\d+\s*`. diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 03aa6ff8e7c..f879ed62010 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -163,8 +163,7 @@ in your `.gitlab-ci.yml`. Behind the scenes, this works by increasing a counter in the database, and the value of that counter is used to create the key for the cache. After a push, a -new key is generated and the old cache is not valid anymore. Eventually, the -Runner's garbage collector will remove it form the filesystem. +new key is generated and the old cache is not valid anymore. ## How shared Runners pick jobs diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index f30a85b114e..bd4aeb006bd 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -12,7 +12,7 @@ this order: 1. [Trigger variables][triggers] or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables) (take precedence over all) 1. Project-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) 1. Group-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) -1. YAML-defined [job-level variables](../yaml/README.md#job-variables) +1. YAML-defined [job-level variables](../yaml/README.md#variables) 1. YAML-defined [global variables](../yaml/README.md#variables) 1. [Deployment variables](#deployment-variables) 1. [Predefined variables](#predefined-variables-environment-variables) (are the @@ -56,6 +56,9 @@ future GitLab releases.** | **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | | **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | | **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | +| **CI_RUNNER_VERSION** | all | 10.6 | GitLab Runner version that is executing the current job | +| **CI_RUNNER_REVISION** | all | 10.6 | GitLab Runner revision that is executing the current job | +| **CI_RUNNER_EXECUTABLE_ARCH** | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | | **CI_PIPELINE_ID** | 8.10 | 0.5 | The unique id of the current pipeline that GitLab CI uses internally | | **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | | **CI_PIPELINE_SOURCE** | 10.0 | all | The source for this pipeline, one of: push, web, trigger, schedule, api, external. Pipelines created before 9.5 will have unknown as source | diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 80ab63468f2..ae200f9b6e2 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -3,18 +3,19 @@ This document describes the usage of `.gitlab-ci.yml`, the file that is used by GitLab Runner to manage your project's jobs. -If you want a quick introduction to GitLab CI, follow our -[quick start guide](../quick_start/README.md). - -## .gitlab-ci.yml - From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root of your repository and contains definitions of how your project should be built. +If you want a quick introduction to GitLab CI, follow our +[quick start guide](../quick_start/README.md). + +## Jobs + The YAML file defines a set of jobs with constraints stating when they should -be run. The jobs are defined as top-level elements with a name and always have -to contain at least the `script` clause: +be run. You can specify an unlimited number of jobs which are defined as +top-level elements with an arbitrary name and always have to contain at least +the `script` clause. ```yaml job1: @@ -24,9 +25,8 @@ job2: script: "execute-script-for-job2" ``` -The above example is the simplest possible CI configuration with two separate +The above example is the simplest possible CI/CD configuration with two separate jobs, where each of the jobs executes a different command. - Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. @@ -34,78 +34,115 @@ Jobs are picked up by [Runners](../runners/README.md) and executed within the environment of the Runner. What is important, is that each job is run independently from each other. -The YAML syntax allows for using more complex job specifications than in the -above example: +Each job must have a unique name, but there are a few **reserved `keywords` that +cannot be used as job names**: -```yaml -image: ruby:2.1 -services: - - postgres +- `image` +- `services` +- `stages` +- `types` +- `before_script` +- `after_script` +- `variables` +- `cache` -before_script: - - bundle install +A job is defined by a list of parameters that define the job behavior. -after_script: - - rm secrets +| Keyword | Required | Description | +|---------------|----------|-------------| +| script | yes | Defines a shell script which is executed by Runner | +| image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | +| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | +| stage | no | Defines a job stage (default: `test`) | +| type | no | Alias for `stage` | +| variables | no | Define job variables on a job level | +| only | no | Defines a list of git refs for which job is created | +| except | no | Defines a list of git refs for which job is not created | +| tags | no | Defines a list of tags which are used to select Runner | +| allow_failure | no | Allow job to fail. Failed job doesn't contribute to commit status | +| when | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | +| dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them| +| artifacts | no | Define list of [job artifacts](#artifacts) | +| cache | no | Define list of files that should be cached between subsequent runs | +| before_script | no | Override a set of commands that are executed before job | +| after_script | no | Override a set of commands that are executed after job | +| environment | no | Defines a name of environment to which deployment is done by this job | +| coverage | no | Define code coverage settings for a given job | +| retry | no | Define how many times a job can be auto-retried in case of a failure | -stages: - - build - - test - - deploy +### `pages` -job1: - stage: build +`pages` is a special job that is used to upload static content to GitLab that +can be used to serve your website. It has a special syntax, so the two +requirements below must be met: + +1. Any static content must be placed under a `public/` directory +1. `artifacts` with a path to the `public/` directory must be defined + +The example below simply moves all files from the root of the project to the +`public/` directory. The `.public` workaround is so `cp` doesn't also copy +`public/` to itself in an infinite loop: + +``` +pages: + stage: deploy script: - - execute-script-for-job1 + - mkdir .public + - cp -r * .public + - mv .public public + artifacts: + paths: + - public only: - - master - tags: - - docker + - master ``` -There are a few reserved `keywords` that **cannot** be used as job names: - -| Keyword | Required | Description | -|---------------|----------|-------------| -| image | no | Use docker image, covered in [Use Docker](../docker/README.md) | -| services | no | Use docker services, covered in [Use Docker](../docker/README.md) | -| stages | no | Define build stages | -| types | no | Alias for `stages` (deprecated) | -| before_script | no | Define commands that run before each job's script | -| after_script | no | Define commands that run after each job's script | -| variables | no | Define build variables | -| cache | no | Define list of files that should be cached between subsequent runs | +Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). -### image and services +## `image` and `services` This allows to specify a custom Docker image and a list of services that can be used for time of the job. The configuration of this feature is covered in [a separate document](../docker/README.md). -### before_script - -`before_script` is used to define the command that should be run before all -jobs, including deploy jobs, but after the restoration of artifacts. This can -be an array or a multi-line string. - -### after_script +## `before_script` and `after_script` > Introduced in GitLab 8.7 and requires Gitlab Runner v1.2 +`before_script` is used to define the command that should be run before all +jobs, including deploy jobs, but after the restoration of [artifacts](#artifacts). +This can be an array or a multi-line string. + `after_script` is used to define the command that will be run after for all jobs, including failed ones. This has to be an array or a multi-line string. -> **Note:** The `before_script` and the main `script` are concatenated and run in a single context/container. The `after_script` is run separately, so depending on the executor, changes done outside of the working tree might not be visible, e.g. software installed in the `before_script`. -### stages +It's possible to overwrite the globally defined `before_script` and `after_script` +if you set it per-job: -`stages` is used to define stages that can be used by jobs. -The specification of `stages` allows for having flexible multi stage pipelines. +```yaml +before_script: +- global before script + +job: + before_script: + - execute this instead of global before script + script: + - my command + after_script: + - execute this after my script +``` + +## `stages` +`stages` is used to define stages that can be used by jobs and is defined +globally. + +The specification of `stages` allows for having flexible multi stage pipelines. The ordering of elements in `stages` defines the ordering of jobs' execution: 1. Jobs of the same stage are run in parallel. @@ -134,280 +171,45 @@ There are also two edge cases worth mentioning: `test` and `deploy` are allowed to be used as job's stage by default. 2. If a job doesn't specify a `stage`, the job is assigned the `test` stage. -### types - -> Deprecated, and could be removed in one of the future releases. Use [stages](#stages) instead. - -Alias for [stages](#stages). - -### variables - -> Introduced in GitLab Runner v0.5.0. - -GitLab CI allows you to add variables to `.gitlab-ci.yml` that are set in the -job environment. The variables are stored in the Git repository and are meant -to store non-sensitive project configuration, for example: - -```yaml -variables: - DATABASE_URL: "postgres://postgres@postgres/my_database" -``` - ->**Note:** -Integers (as well as strings) are legal both for variable's name and value. -Floats are not legal and cannot be used. - -These variables can be later used in all executed commands and scripts. -The YAML-defined variables are also set to all created service containers, -thus allowing to fine tune them. Variables can be also defined on a -[job level](#job-variables). - -Except for the user defined variables, there are also the ones set up by the -Runner itself. One example would be `CI_COMMIT_REF_NAME` which has the value of -the branch or tag name for which project is built. Apart from the variables -you can set in `.gitlab-ci.yml`, there are also the so called secret variables -which can be set in GitLab's UI. - -[Learn more about variables.][variables] - -### cache - -> -**Notes:** -- Introduced in GitLab Runner v0.7.0. -- Prior to GitLab 9.2, caches were restored after artifacts. -- From GitLab 9.2, caches are restored before artifacts. - -`cache` is used to specify a list of files and directories which should be -cached between jobs. You can only use paths that are within the project -workspace. - -**By default caching is enabled and shared between pipelines and jobs, -starting from GitLab 9.0** - -If `cache` is defined outside the scope of jobs, it means it is set -globally and all jobs will use that definition. - -Cache all files in `binaries` and `.config`: - -```yaml -rspec: - script: test - cache: - paths: - - binaries/ - - .config -``` - -Cache all Git untracked files: - -```yaml -rspec: - script: test - cache: - untracked: true -``` - -Cache all Git untracked files and files in `binaries`: - -```yaml -rspec: - script: test - cache: - untracked: true - paths: - - binaries/ -``` - -Locally defined cache overrides globally defined options. The following `rspec` -job will cache only `binaries/`: - -```yaml -cache: - paths: - - my/files - -rspec: - script: test - cache: - key: rspec - paths: - - binaries/ -``` - -Note that since cache is shared between jobs, if you're using different -paths for different jobs, you should also set a different **cache:key** -otherwise cache content can be overwritten. - -The cache is provided on a best-effort basis, so don't expect that the cache -will be always present. For implementation details, please check GitLab Runner. - -#### cache:key - -> Introduced in GitLab Runner v1.0.0. - -The `key` directive allows you to define the affinity of caching -between jobs, allowing to have a single cache for all jobs, -cache per-job, cache per-branch or any other way you deem proper. - -This allows you to fine tune caching, allowing you to cache data between -different jobs or even different branches. - -The `cache:key` variable can use any of the [predefined variables](../variables/README.md). - -The default key is **default** across the project, therefore everything is -shared between each pipelines and jobs by default, starting from GitLab 9.0. - ->**Note:** The `cache:key` variable cannot contain the `/` character, or the equivalent URI encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. - ---- - -**Example configurations** - -To enable per-job caching: - -```yaml -cache: - key: "$CI_JOB_NAME" - untracked: true -``` - -To enable per-branch caching: - -```yaml -cache: - key: "$CI_COMMIT_REF_SLUG" - untracked: true -``` - -To enable per-job and per-branch caching: - -```yaml -cache: - key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" - untracked: true -``` - -To enable per-branch and per-stage caching: - -```yaml -cache: - key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" - untracked: true -``` - -If you use **Windows Batch** to run your shell scripts you need to replace -`$` with `%`: - -```yaml -cache: - key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%" - untracked: true -``` +## `stage` -If you use **Windows PowerShell** to run your shell scripts you need to replace -`$` with `$env:`: - -```yaml -cache: - key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG" - untracked: true -``` - -### cache:policy - -> Introduced in GitLab 9.4. - -The default behaviour of a caching job is to download the files at the start of -execution, and to re-upload them at the end. This allows any changes made by the -job to be persisted for future runs, and is known as the `pull-push` cache -policy. - -If you know the job doesn't alter the cached files, you can skip the upload step -by setting `policy: pull` in the job specification. Typically, this would be -twinned with an ordinary cache job at an earlier stage to ensure the cache -is updated from time to time: +`stage` is defined per-job and relies on [`stages`](#stages) which is defined +globally. It allows to group jobs into different stages, and jobs of the same +`stage` are executed in `parallel`. For example: ```yaml stages: - - setup + - build - test + - deploy -prepare: - stage: setup - cache: - key: gems - paths: - - vendor/bundle - script: - - bundle install --deployment - -rspec: - stage: test - cache: - key: gems - paths: - - vendor/bundle - policy: pull - script: - - bundle exec rspec ... -``` - -This helps to speed up job execution and reduce load on the cache server, -especially when you have a large number of cache-using jobs executing in -parallel. - -Additionally, if you have a job that unconditionally recreates the cache without -reference to its previous contents, you can use `policy: push` in that job to -skip the download step. - -## Jobs +job 1: + stage: build + script: make build dependencies -`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job -must have a unique name, which is not one of the keywords mentioned above. -A job is defined by a list of parameters that define the job behavior. +job 2: + stage: build + script: make build artifacts -```yaml -job_name: - script: - - rake spec - - coverage +job 3: stage: test - only: - - master - except: - - develop - tags: - - ruby - - postgres - allow_failure: true + script: make test + +job 4: + stage: deploy + script: make deploy ``` -| Keyword | Required | Description | -|---------------|----------|-------------| -| script | yes | Defines a shell script which is executed by Runner | -| image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| stage | no | Defines a job stage (default: `test`) | -| type | no | Alias for `stage` | -| variables | no | Define job variables on a job level | -| only | no | Defines a list of git refs for which job is created | -| except | no | Defines a list of git refs for which job is not created | -| tags | no | Defines a list of tags which are used to select Runner | -| allow_failure | no | Allow job to fail. Failed job doesn't contribute to commit status | -| when | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | -| dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them| -| artifacts | no | Define list of [job artifacts](../../user/project/pipelines/job_artifacts.md) | -| cache | no | Define list of files that should be cached between subsequent runs | -| before_script | no | Override a set of commands that are executed before job | -| after_script | no | Override a set of commands that are executed after job | -| environment | no | Defines a name of environment to which deployment is done by this job | -| coverage | no | Define code coverage settings for a given job | -| retry | no | Define how many times a job can be auto-retried in case of a failure | +## `types` -### script +CAUTION: **Deprecated:** +`types` is deprecated, and could be removed in one of the future releases. +Use [stages](#stages) instead. -`script` is a shell script which is executed by the Runner. For example: +## `script` + +`script` is the only required keyword that a job needs. It's a shell script +which is executed by the Runner. For example: ```yaml job: @@ -429,13 +231,7 @@ that the YAML parser knows to interpret the whole thing as a string rather than a "key: value" pair. Be careful when using special characters: `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. -### stage - -`stage` allows to group jobs into different stages. Jobs of the same `stage` -are executed in `parallel`. For more info about the use of `stage` please check -[stages](#stages). - -### only and except (simplified) +## `only` and `except` (simplified) `only` and `except` are two parameters that set a job policy to limit when jobs are created: @@ -505,12 +301,13 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except master. -### only and except (complex) +## `only` and `except` (complex) > Introduced in GitLab 10.0 -> This an _alpha_ feature, and it it subject to change at any time without - prior notice! +CAUTION: **Warning:** +This an _alpha_ feature, and it it subject to change at any time without +prior notice! Since GitLab 10.0 it is possible to define a more elaborate only/except job policy configuration. @@ -535,24 +332,7 @@ job: kubernetes: active ``` -### Job variables - -It is possible to define job variables using a `variables` keyword on a job -level. It works basically the same way as its [global-level equivalent](#variables), -but allows you to define job-specific variables. - -When the `variables` keyword is used on a job level, it overrides the global YAML -job variables and predefined ones. To turn off global defined variables -in your job, define an empty hash: - -```yaml -job_name: - variables: {} -``` - -Job variables priority is defined in the [variables documentation][variables]. - -### tags +## `tags` `tags` is used to select specific Runners from the list of all Runners that are allowed to run this project. @@ -573,7 +353,7 @@ job: The specification above, will make sure that `job` is built by a Runner that has both `ruby` AND `postgres` tags defined. -### allow_failure +## `allow_failure` `allow_failure` is used when you want to allow a job to fail without impacting the rest of the CI suite. Failed jobs don't contribute to the commit status. @@ -606,7 +386,7 @@ job3: - deploy_to_staging ``` -### when +## `when` `when` is used to implement jobs that are run in case of failure or despite the failure. @@ -619,7 +399,7 @@ failure. fails. 1. `always` - execute job regardless of the status of jobs from prior stages. 1. `manual` - execute job manually (added in GitLab 8.10). Read about - [manual actions](#manual-actions) below. + [manual actions](#when-manual) below. For example: @@ -667,42 +447,41 @@ The above script will: success or failure. 3. Allow you to manually execute `deploy_job` from GitLab's UI. -#### Manual actions - -> Introduced in GitLab 8.10. -> Blocking manual actions were introduced in GitLab 9.0 -> Protected actions were introduced in GitLab 9.2 +### `when:manual` -Manual actions are a special type of job that are not executed automatically; -they need to be explicitly started by a user. Manual actions can be started -from pipeline, build, environment, and deployment views. +> **Notes:** +- Introduced in GitLab 8.10. +- Blocking manual actions were introduced in GitLab 9.0. +- Protected actions were introduced in GitLab 9.2. -An example usage of manual actions is deployment to production. +Manual actions are a special type of job that are not executed automatically, +they need to be explicitly started by a user. An example usage of manual actions +would be a deployment to a production environment. Manual actions can be started +from the pipeline, job, environment, and deployment views. Read more at the +[environments documentation][env-manual]. -Read more at the [environments documentation][env-manual]. - -Manual actions can be either optional or blocking. Blocking manual action will -block execution of the pipeline at stage this action is defined in. It is +Manual actions can be either optional or blocking. Blocking manual actions will +block the execution of the pipeline at the stage this action is defined in. It's possible to resume execution of the pipeline when someone executes a blocking -manual actions by clicking a _play_ button. +manual action by clicking a _play_ button. -When pipeline is blocked it will not be merged if Merge When Pipeline Succeeds +When a pipeline is blocked, it will not be merged if Merge When Pipeline Succeeds is set. Blocked pipelines also do have a special status, called _manual_. - Manual actions are non-blocking by default. If you want to make manual action blocking, it is necessary to add `allow_failure: false` to the job's definition in `.gitlab-ci.yml`. -Optional manual actions have `allow_failure: true` set by default. +Optional manual actions have `allow_failure: true` set by default and their +Statuses do not contribute to the overall pipeline status. So, if a manual +action fails, the pipeline will eventually succeed. -**Statuses of optional actions do not contribute to overall pipeline status.** +Manual actions are considered to be write actions, so permissions for +[protected branches](../../user/project/protected_branches.md) are used when +user wants to trigger an action. In other words, in order to trigger a manual +action assigned to a branch that the pipeline is running for, user needs to +have ability to merge to this branch. -**Manual actions are considered to be write actions, so permissions for -protected branches are used when user wants to trigger an action. In other -words, in order to trigger a manual action assigned to a branch that the -pipeline is running for, user needs to have ability to merge to this branch.** - -### environment +## `environment` > **Notes:** @@ -727,7 +506,7 @@ deploy to production: In the above example, the `deploy to production` job will be marked as doing a deployment to the `production` environment. -#### environment:name +### `environment:name` > **Notes:** @@ -766,7 +545,7 @@ deploy to production: name: production ``` -#### environment:url +### `environment:url` > **Notes:** @@ -793,7 +572,7 @@ deploy to production: url: https://prod.example.com ``` -#### environment:on_stop +### `environment:on_stop` > **Notes:** @@ -808,7 +587,7 @@ the environment. Read the `environment:action` section for an example. -#### environment:action +### `environment:action` > [Introduced][ce-6669] in GitLab 8.13. @@ -849,7 +628,7 @@ The `stop_review_app` job is **required** to have the following keywords defined - `stage` should be the same as the `review_app` in order for the environment to stop automatically when the branch is deleted -#### dynamic environments +### Dynamic environments > **Notes:** @@ -885,13 +664,204 @@ The common use case is to create dynamic environments for branches and use them as Review Apps. You can see a simple example using Review Apps at <https://gitlab.com/gitlab-examples/review-apps-nginx/>. -### artifacts +## `cache` + +> +**Notes:** +- Introduced in GitLab Runner v0.7.0. +- `cache` can be set globally and per-job. +- From GitLab 9.0, caching is enabled and shared between pipelines and jobs + by default. +- From GitLab 9.2, caches are restored before [artifacts](#artifacts). + +`cache` is used to specify a list of files and directories which should be +cached between jobs. You can only use paths that are within the project +workspace. + +If `cache` is defined outside the scope of jobs, it means it is set +globally and all jobs will use that definition. + +Cache all files in `binaries` and `.config`: + +```yaml +rspec: + script: test + cache: + paths: + - binaries/ + - .config +``` + +Cache all Git untracked files: + +```yaml +rspec: + script: test + cache: + untracked: true +``` + +Cache all Git untracked files and files in `binaries`: + +```yaml +rspec: + script: test + cache: + untracked: true + paths: + - binaries/ +``` + +Locally defined cache overrides globally defined options. The following `rspec` +job will cache only `binaries/`: + +```yaml +cache: + paths: + - my/files + +rspec: + script: test + cache: + key: rspec + paths: + - binaries/ +``` + +Note that since cache is shared between jobs, if you're using different +paths for different jobs, you should also set a different **cache:key** +otherwise cache content can be overwritten. + +NOTE: **Note:** +The cache is provided on a best-effort basis, so don't expect that the cache +will be always present. + +### `cache:key` + +> Introduced in GitLab Runner v1.0.0. + +The `key` directive allows you to define the affinity of caching +between jobs, allowing to have a single cache for all jobs, +cache per-job, cache per-branch or any other way that fits your needs. + +This way, you can fine tune caching, allowing you to cache data between +different jobs or even different branches. + +The `cache:key` variable can use any of the +[predefined variables](../variables/README.md), and the default key, if not set, +is set as `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and +per-branch". It is the default across the project, therefore everything is +shared between pipelines and jobs running on the same branch by default. + +NOTE: **Note:** +The `cache:key` variable cannot contain the `/` character, or the equivalent +URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. + +**Example configurations** + +To enable per-job caching: + +```yaml +cache: + key: "$CI_JOB_NAME" + untracked: true +``` + +To enable per-branch caching: + +```yaml +cache: + key: "$CI_COMMIT_REF_SLUG" + untracked: true +``` + +To enable per-job and per-branch caching: + +```yaml +cache: + key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" + untracked: true +``` + +To enable per-branch and per-stage caching: + +```yaml +cache: + key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" + untracked: true +``` + +If you use **Windows Batch** to run your shell scripts you need to replace +`$` with `%`: + +```yaml +cache: + key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%" + untracked: true +``` + +If you use **Windows PowerShell** to run your shell scripts you need to replace +`$` with `$env:`: + +```yaml +cache: + key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG" + untracked: true +``` + +### `cache:policy` + +> Introduced in GitLab 9.4. + +The default behaviour of a caching job is to download the files at the start of +execution, and to re-upload them at the end. This allows any changes made by the +job to be persisted for future runs, and is known as the `pull-push` cache +policy. + +If you know the job doesn't alter the cached files, you can skip the upload step +by setting `policy: pull` in the job specification. Typically, this would be +twinned with an ordinary cache job at an earlier stage to ensure the cache +is updated from time to time: + +```yaml +stages: + - setup + - test + +prepare: + stage: setup + cache: + key: gems + paths: + - vendor/bundle + script: + - bundle install --deployment + +rspec: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - bundle exec rspec ... +``` + +This helps to speed up job execution and reduce load on the cache server, +especially when you have a large number of cache-using jobs executing in +parallel. + +Additionally, if you have a job that unconditionally recreates the cache without +reference to its previous contents, you can use `policy: push` in that job to +skip the download step. + +## `artifacts` > **Notes:** - Introduced in GitLab Runner v0.7.0 for non-Windows platforms. - Windows support was added in GitLab Runner v.1.0.0. -- Prior to GitLab 9.2, caches were restored after artifacts. - From GitLab 9.2, caches are restored before artifacts. - Currently not all executors are supported. - Job artifacts are only collected for successful jobs by default. @@ -960,7 +930,9 @@ release-job: The artifacts will be sent to GitLab after the job finishes successfully and will be available for download in the GitLab UI. -#### artifacts:name +[Read more about artifacts.](../../user/project/pipelines/job_artifacts.md) + +### `artifacts:name` > Introduced in GitLab 8.6 and GitLab Runner v1.1.0. @@ -970,10 +942,6 @@ useful when you'd like to download the archive from GitLab. The `artifacts:name` variable can make use of any of the [predefined variables](../variables/README.md). The default name is `artifacts`, which becomes `artifacts.zip` when downloaded. ---- - -**Example configurations** - To create an archive with a name of the current job: ```yaml @@ -1033,7 +1001,7 @@ job: untracked: true ``` -#### artifacts:when +### `artifacts:when` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. @@ -1046,11 +1014,7 @@ failure. 1. `on_failure` - upload artifacts only when the job fails. 1. `always` - upload artifacts regardless of the job status. ---- - -**Example configurations** - -To upload artifacts only when job fails. +To upload artifacts only when job fails: ```yaml job: @@ -1058,22 +1022,23 @@ job: when: on_failure ``` -#### artifacts:expire_in +### `artifacts:expire_in` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. -`artifacts:expire_in` is used to delete uploaded artifacts after the specified -time. By default, artifacts are stored on GitLab forever. `expire_in` allows you -to specify how long artifacts should live before they expire, counting from the -time they are uploaded and stored on GitLab. +`expire_in` allows you to specify how long artifacts should live before they +expire and therefore deleted, counting from the time they are uploaded and +stored on GitLab. If the expiry time is not defined, it defaults to the +[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) +(30 days by default, forever on GitLab.com). You can use the **Keep** button on the job page to override expiration and keep artifacts forever. -After expiry, artifacts are actually deleted hourly by default (via a cron job), -but they are not accessible after expiry. +After their expiry, artifacts are deleted hourly by default (via a cron job), +and are not accessible anymore. -The value of `expire_in` is an elapsed time. Examples of parseable values: +The value of `expire_in` is an elapsed time. Examples of parsable values: - '3 mins 4 sec' - '2 hrs 20 min' @@ -1082,10 +1047,6 @@ The value of `expire_in` is an elapsed time. Examples of parseable values: - '47 yrs 6 mos and 4d' - '3 weeks and 2 days' ---- - -**Example configurations** - To expire artifacts 1 week after being uploaded: ```yaml @@ -1094,7 +1055,7 @@ job: expire_in: 1 week ``` -### dependencies +## `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1153,7 +1114,7 @@ deploy: script: make deploy ``` -#### When a dependent job will fail +### When a dependent job will fail > Introduced in GitLab 10.3. @@ -1167,27 +1128,9 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. -### before_script and after_script - -It's possible to overwrite the globally defined `before_script` and `after_script`: +## `coverage` -```yaml -before_script: -- global before script - -job: - before_script: - - execute this instead of global before script - script: - - my command - after_script: - - execute this after my script -``` - -### coverage - -**Notes:** -- [Introduced][ce-7447] in GitLab 8.17. +> [Introduced][ce-7447] in GitLab 8.17. `coverage` allows you to configure how code coverage will be extracted from the job output. @@ -1205,10 +1148,9 @@ job1: coverage: '/Code coverage: \d+\.\d+/' ``` -### retry +## `retry` -**Notes:** -- [Introduced][ce-3442] in GitLab 9.5. +> [Introduced][ce-3442] in GitLab 9.5. `retry` allows you to configure how many times a job is going to be retried in case of a failure. @@ -1228,16 +1170,57 @@ test: retry: 2 ``` -## Git Strategy +## `variables` + +> Introduced in GitLab Runner v0.5.0. + +NOTE: **Note:** +Integers (as well as strings) are legal both for variable's name and value. +Floats are not legal and cannot be used. + +GitLab CI/CD allows you to define variables inside `.gitlab-ci.yml` that are +then passed in the job environment. They can be set globally and per-job. +When the `variables` keyword is used on a job level, it overrides the global +YAML variables and predefined ones. + +They are stored in the Git repository and are meant to store non-sensitive +project configuration, for example: + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. +The YAML-defined variables are also set to all created service containers, +thus allowing to fine tune them. + +To turn off global defined variables in a specific job, define an empty hash: + +```yaml +job_name: + variables: {} +``` + +Except for the user defined variables, there are also the ones [set up by the +Runner itself](../variables/README.md#predefined-variables-environment-variables). +One example would be `CI_COMMIT_REF_NAME` which has the value of +the branch or tag name for which project is built. Apart from the variables +you can set in `.gitlab-ci.yml`, there are also the so called +[secret variables](../variables/README.md#secret-variables) +which can be set in GitLab's UI. + +[Learn more about variables and their priority.][variables] + +### Git strategy > Introduced in GitLab 8.9 as an experimental feature. May change or be removed completely in future releases. `GIT_STRATEGY=none` requires GitLab Runner v1.7+. You can set the `GIT_STRATEGY` used for getting recent application code, either -in the global [`variables`](#variables) section or the [`variables`](#job-variables) -section for individual jobs. If left unspecified, the default from project -settings will be used. +globally or per-job in the [`variables`](#variables) section. If left +unspecified, the default from project settings will be used. There are three possible values: `clone`, `fetch`, and `none`. @@ -1269,44 +1252,13 @@ variables: GIT_STRATEGY: none ``` -## Git Checkout - -> Introduced in GitLab Runner 9.3 - -The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either -`clone` or `fetch` to specify whether a `git checkout` should be run. If not -specified, it defaults to true. Like `GIT_STRATEGY`, it can be set in either the -global [`variables`](#variables) section or the [`variables`](#job-variables) -section for individual jobs. - -If set to `false`, the Runner will: - -- when doing `fetch` - update the repository and leave working copy on - the current revision, -- when doing `clone` - clone the repository and leave working copy on the - default branch. - -Having this setting set to `true` will mean that for both `clone` and `fetch` -strategies the Runner will checkout the working copy to a revision related -to the CI pipeline: - -```yaml -variables: - GIT_STRATEGY: clone - GIT_CHECKOUT: "false" -script: - - git checkout master - - git merge $CI_BUILD_REF_NAME -``` - -## Git Submodule Strategy +### Git submodule strategy > Requires GitLab Runner v1.10+. The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git -submodules are included when fetching the code before a build. Like -`GIT_STRATEGY`, it can be set in either the global [`variables`](#variables) -section or the [`variables`](#job-variables) section for individual jobs. +submodules are included when fetching the code before a build. You can set them +globally or per-job in the [`variables`](#variables) section. There are three possible values: `none`, `normal`, and `recursive`: @@ -1336,8 +1288,36 @@ Note that for this feature to work correctly, the submodules must be configured - a relative path to another repository on the same GitLab server. See the [Git submodules](../git_submodules.md) documentation. +### Git checkout + +> Introduced in GitLab Runner 9.3 + +The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either +`clone` or `fetch` to specify whether a `git checkout` should be run. If not +specified, it defaults to true. You can set them globally or per-job in the +[`variables`](#variables) section. + +If set to `false`, the Runner will: + +- when doing `fetch` - update the repository and leave working copy on + the current revision, +- when doing `clone` - clone the repository and leave working copy on the + default branch. + +Having this setting set to `true` will mean that for both `clone` and `fetch` +strategies the Runner will checkout the working copy to a revision related +to the CI pipeline: -## Job stages attempts +```yaml +variables: + GIT_STRATEGY: clone + GIT_CHECKOUT: "false" +script: + - git checkout master + - git merge $CI_BUILD_REF_NAME +``` + +### Job stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. @@ -1359,10 +1339,9 @@ variables: GET_SOURCES_ATTEMPTS: 3 ``` -You can set them in the global [`variables`](#variables) section or the -[`variables`](#job-variables) section for individual jobs. +You can set them globally or per-job in the [`variables`](#variables) section. -## Shallow cloning +### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. May change in future releases or be removed completely. @@ -1393,7 +1372,17 @@ variables: GIT_DEPTH: "3" ``` -## Hidden keys (jobs) +You can set it globally or per-job in the [`variables`](#variables) section. + +## Special YAML features + +It's possible to use special YAML features like anchors (`&`), aliases (`*`) +and map merging (`<<`), which will allow you to greatly reduce the complexity +of `.gitlab-ci.yml`. + +Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). + +### Hidden keys (jobs) > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1419,14 +1408,6 @@ Use this feature to ignore jobs, or use the [special YAML features](#special-yaml-features) and transform the hidden keys into templates. -## Special YAML features - -It's possible to use special YAML features like anchors (`&`), aliases (`*`) -and map merging (`<<`), which will allow you to greatly reduce the complexity -of `.gitlab-ci.yml`. - -Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). - ### Anchors > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1556,34 +1537,10 @@ with an API call. [Read more in the triggers documentation.](../triggers/README.md) -### pages - -`pages` is a special job that is used to upload static content to GitLab that -can be used to serve your website. It has a special syntax, so the two -requirements below must be met: - -1. Any static content must be placed under a `public/` directory -1. `artifacts` with a path to the `public/` directory must be defined - -The example below simply moves all files from the root of the project to the -`public/` directory. The `.public` workaround is so `cp` doesn't also copy -`public/` to itself in an infinite loop: - -``` -pages: - stage: deploy - script: - - mkdir .public - - cp -r * .public - - mv .public public - artifacts: - paths: - - public - only: - - master -``` +## Skipping jobs -Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). +If your commit message contains `[ci skip]` or `[skip ci]`, using any +capitalization, the commit will be created but the pipeline will be skipped. ## Validate the .gitlab-ci.yml @@ -1595,11 +1552,6 @@ You can find the link under `/ci/lint` of your gitlab instance. If you get validation error when using specific values (e.g., `true` or `false`), try to quote them, or change them to a different form (e.g., `/bin/true`). -## Skipping jobs - -If your commit message contains `[ci skip]` or `[skip ci]`, using any -capitalization, the commit will be created but the jobs will be skipped. - ## Examples Visit the [examples README][examples] to see a list of examples using GitLab |
