diff options
Diffstat (limited to 'doc/ci')
53 files changed, 3195 insertions, 0 deletions
diff --git a/doc/ci/README.md b/doc/ci/README.md new file mode 100644 index 00000000000..e3534c6991f --- /dev/null +++ b/doc/ci/README.md @@ -0,0 +1,27 @@ +## GitLab CI Documentation + +### User documentation + ++ [Quick Start](quick_start/README.md) ++ [Configuring project (.gitlab-ci.yml)](yaml/README.md) ++ [Configuring runner](runners/README.md) ++ [Configuring deployment](deployment/README.md) ++ [Using Docker Images](docker/using_docker_images.md) ++ [Using Docker Build](docker/using_docker_build.md) ++ [Using Variables](variables/README.md) + +### Examples + ++ [Test and deploy Ruby applications to Heroku](examples/test-and-deploy-ruby-application-to-heroku.md) ++ [Test and deploy Python applications to Heroku](examples/test-and-deploy-python-application-to-heroku.md) ++ [Test Clojure applications](examples/test-clojure-application.md) ++ Help your favorite programming language and GitLab by sending a merge request with a guide for that language. + +### Administrator documentation + ++ [Install](install/README.md) ++ [Update](update/README.md) ++ [User permissions](permissions/README.md) ++ [Backup/Restore](raketasks/backup_restore.md) ++ [Migrating to packaged CI](migration_to_omnibus/README.md) ++ [API](api/README.md) diff --git a/doc/ci/api/README.md b/doc/ci/api/README.md new file mode 100644 index 00000000000..95fe2f837a5 --- /dev/null +++ b/doc/ci/api/README.md @@ -0,0 +1,87 @@ +# GitLab CI API + +## Resources + +- [Projects](projects.md) +- [Runners](runners.md) +- [Commits](commits.md) +- [Builds](builds.md) +- [Forks](forks.md) + + +## Authentication + +GitLab CI API uses different types of authentication depends on what API you use. +Each API document has section with information about authentication you need to use. + +GitLab CI API has 4 authentication methods: + +* GitLab user token & GitLab url +* GitLab CI project token +* GitLab CI runners registration token +* GitLab CI runner token + + +### Authentication #1: GitLab user token & GitLab url + +Authentication is done by +sending the `private-token` of a valid user and the `url` of an +authorized Gitlab instance via a query string along with the API +request: + + GET http://ci.example.com/api/v1/projects?private_token=QVy1PB7sTxfy4pqfZM1U&url=http://demo.gitlab.com/ + +If preferred, you may instead send the `private-token` as a header in +your request: + + curl --header "PRIVATE-TOKEN: QVy1PB7sTxfy4pqfZM1U" "http://ci.example.com/api/v1/projects?url=http://demo.gitlab.com/" + + +### Authentication #2: GitLab CI project token + +Each project in GitLab CI has it own token. +It can be used to get project commits and builds information. +You can use project token only for certain project. + +### Authentication #3: GitLab CI runners registration token + +This token is not persisted and is generated on each application start. +It can be used only for registering new runners in system. You can find it on +GitLab CI Runners web page https://gitlab-ci.example.com/admin/runners + +### Authentication #4: GitLab CI runner token + +Every GitLab CI runner has it own token that allow it to receive and update +GitLab CI builds. This token exists of internal purposes and should be used only +by runners + +## JSON + +All API requests are serialized using JSON. You don't need to specify +`.json` at the end of API URL. + +## Status codes + +The API is designed to return different status codes according to context and action. In this way if a request results in an error the caller is able to get insight into what went wrong, e.g. status code `400 Bad Request` is returned if a required attribute is missing from the request. The following list gives an overview of how the API functions generally behave. + +API request types: + +- `GET` requests access one or more resources and return the result as JSON +- `POST` requests return `201 Created` if the resource is successfully created and return the newly created resource as JSON +- `GET`, `PUT` and `DELETE` return `200 OK` if the resource is accessed, modified or deleted successfully, the (modified) result is returned as JSON +- `DELETE` requests are designed to be idempotent, meaning a request a resource still returns `200 OK` even it was deleted before or is not available. The reasoning behind it is the user is not really interested if the resource existed before or not. + +The following list shows the possible return codes for API requests. + +Return values: + +- `200 OK` - The `GET`, `PUT` or `DELETE` request was successful, the resource(s) itself is returned as JSON +- `201 Created` - The `POST` request was successful and the resource is returned as JSON +- `400 Bad Request` - A required attribute of the API request is missing, e.g. the title of an issue is not given +- `401 Unauthorized` - The user is not authenticated, a valid user token is necessary, see above +- `403 Forbidden` - The request is not allowed, e.g. the user is not allowed to delete a project +- `404 Not Found` - A resource could not be accessed, e.g. an ID for a resource could not be found +- `405 Method Not Allowed` - The request is not supported +- `409 Conflict` - A conflicting resource already exists, e.g. creating a project with a name that already exists +- `422 Unprocessable` - The entity could not be processed +- `500 Server Error` - While handling the request something went wrong on the server side diff --git a/doc/ci/api/builds.md b/doc/ci/api/builds.md new file mode 100644 index 00000000000..54749bc6fa1 --- /dev/null +++ b/doc/ci/api/builds.md @@ -0,0 +1,41 @@ +# Builds API + +This API used by runners to receive and update builds. + +__Authentication is done by runner token__ + +## Builds + +### Runs oldest pending build by runner + + POST /builds/register + +Parameters: + + * `token` (required) - The unique token of runner + +Returns: + +```json +{ + "id" : 79, + "commands" : "", + "path" : "", + "ref" : "", + "sha" : "", + "project_id" : 6, + "repo_url" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "before_sha" : "" +} +``` + + +### Update details of an existing build + + PUT /builds/:id + +Parameters: + + * `id` (required) - The ID of a project + * `state` (optional) - The state of a build + * `trace` (optional) - The trace of a build diff --git a/doc/ci/api/commits.md b/doc/ci/api/commits.md new file mode 100644 index 00000000000..0015a62a38f --- /dev/null +++ b/doc/ci/api/commits.md @@ -0,0 +1,101 @@ +# Commits API + +__Authentication is done by GitLab CI project token__ + +## Commits + +### Retrieve all commits per project + +Get list of commits per project + + GET /commits + +Parameters: + + * `project_id` (required) - The ID of a project + * `project_token` (requires) - Project token + * `page` (optional) + * `per_page` (optional) - items per request (default is 20) + +Returns: + +```json +[{ + "id": 3, + "ref": "master", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "project_id": 2, + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "created_at": "2014-11-05T09:46:35.247Z", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "duration": 5.062692165374756, + "git_commit_message": "wow\n", + "git_author_name": "Administrator", + "git_author_email": "admin@example.com", + "builds": [{ + "id": 7, + "project_id": 2, + "ref": "master", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "created_at": "2014-11-05T09:46:35.259Z", + "updated_at": "2014-11-05T09:46:44.255Z", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "started_at": "2014-11-05T09:46:39.192Z", + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "runner_id": 1, + "coverage": null, + "commit_id": 3 + }] +}] +``` + +### Create commit + +Inform GitLab CI about new commit you want it to build. + +__If commit already exists in GitLab CI it will not be created__ + + + POST /commits + +Parameters: + + * `project_id` (required) - The ID of a project + * `project_token` (requires) - Project token + * `data` (required) - Push data. For example see comment in `lib/api/commits.rb` + +Returns: + +```json +{ + "id": 3, + "ref": "master", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "project_id": 2, + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "created_at": "2014-11-05T09:46:35.247Z", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "duration": 5.062692165374756, + "git_commit_message": "wow\n", + "git_author_name": "Administrator", + "git_author_email": "admin@example.com", + "builds": [{ + "id": 7, + "project_id": 2, + "ref": "master", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "created_at": "2014-11-05T09:46:35.259Z", + "updated_at": "2014-11-05T09:46:44.255Z", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "started_at": "2014-11-05T09:46:39.192Z", + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "runner_id": 1, + "coverage": null, + "commit_id": 3 + }] +} +``` diff --git a/doc/ci/api/forks.md b/doc/ci/api/forks.md new file mode 100644 index 00000000000..1a5ea8041d8 --- /dev/null +++ b/doc/ci/api/forks.md @@ -0,0 +1,23 @@ +# Forks API + +This API is intended to aid in the setup and configuration of +forked projects on Gitlab CI. + +__Authentication is done by GitLab user token & GitLab project token__ + +## Forks + +### Create fork for project + + + +``` +POST /forks +``` + +Parameters: + + project_id (required) - The ID of a project + project_token (requires) - Project token + private_token(required) - User private token + data (required) - GitLab project data (name_with_namespace, web_url, default_branch, ssh_url_to_repo) diff --git a/doc/ci/api/projects.md b/doc/ci/api/projects.md new file mode 100644 index 00000000000..c24d48f829f --- /dev/null +++ b/doc/ci/api/projects.md @@ -0,0 +1,154 @@ +# Projects API + +This API is intended to aid in the setup and configuration of +projects on Gitlab CI. + +__Authentication is done by GitLab user token & GitLab url__ + +## Projects + +### List Authorized Projects + +Lists all projects that the authenticated user has access to. + +``` +GET /projects +``` + +Returns: + +```json + [ + { + "id" : 271, + "name" : "gitlabhq", + "timeout" : 1800, + "token" : "iPWx6WM4lhHNedGfBpPJNP", + "default_ref" : "master", + "gitlab_url" : "http://demo.gitlabhq.com/gitlab/gitlab-shell", + "path" : "gitlab/gitlab-shell", + "always_build" : false, + "polling_interval" : null, + "public" : false, + "ssh_url_to_repo" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "gitlab_id" : 3 + }, + { + "id" : 272, + "name" : "gitlab-ci", + "timeout" : 1800, + "token" : "iPWx6WM4lhHNedGfBpPJNP", + "default_ref" : "master", + "gitlab_url" : "http://demo.gitlabhq.com/gitlab/gitlab-shell", + "path" : "gitlab/gitlab-shell", + "always_build" : false, + "polling_interval" : null, + "public" : false, + "ssh_url_to_repo" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "gitlab_id" : 4 + } +] +``` + +### List Owned Projects + +Lists all projects that the authenticated user owns. + +``` +GET /projects/owned +``` + +Returns: + +```json +[ + { + "id" : 272, + "name" : "gitlab-ci", + "timeout" : 1800, + "token" : "iPWx6WM4lhHNedGfBpPJNP", + "default_ref" : "master", + "gitlab_url" : "http://demo.gitlabhq.com/gitlab/gitlab-shell", + "path" : "gitlab/gitlab-shell", + "always_build" : false, + "polling_interval" : null, + "public" : false, + "ssh_url_to_repo" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "gitlab_id" : 4 + } +] +``` + +### Single Project + +Returns information about a single project for which the user is +authorized. + + GET /projects/:id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + +### Create Project + +Creates a Gitlab CI project using Gitlab project details. + + POST /projects + +Parameters: + + * `name` (required) - The name of the project + * `gitlab_id` (required) - The ID of the project on the Gitlab instance + * `path` (required) - The gitlab project path + * `ssh_url_to_repo` (required) - The gitlab SSH url to the repo + * `default_ref` (optional) - The branch to run on (default to `master`) + +### Update Project + +Updates a Gitlab CI project using Gitlab project details that the +authenticated user has access to. + + PUT /projects/:id + +Parameters: + + * `name` - The name of the project + * `gitlab_id` - The ID of the project on the Gitlab instance + * `path` - The gitlab project path + * `ssh_url_to_repo` - The gitlab SSH url to the repo + * `default_ref` - The branch to run on (default to `master`) + +### Remove Project + +Removes a Gitlab CI project that the authenticated user has access to. + + DELETE /projects/:id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + +### Link Project to Runner + +Links a runner to a project so that it can make builds (only via +authorized user). + + POST /projects/:id/runners/:runner_id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + * `runner_id` (required) - The ID of the Gitlab CI runner + +### Remove Project from Runner + +Removes a runner from a project so that it can not make builds (only +via authorized user). + + DELETE /projects/:id/runners/:runner_id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + * `runner_id` (required) - The ID of the Gitlab CI runner
\ No newline at end of file diff --git a/doc/ci/api/runners.md b/doc/ci/api/runners.md new file mode 100644 index 00000000000..68b5851617a --- /dev/null +++ b/doc/ci/api/runners.md @@ -0,0 +1,77 @@ +# Runners API + +## Runners + +### Retrieve all runners + +__Authentication is done by GitLab user token & GitLab url__ + +Used to get information about all runners registered on the Gitlab CI +instance. + + GET /runners + +Returns: + +```json +[ + { + "id" : 85, + "token" : "12b68e90394084703135" + }, + { + "id" : 86, + "token" : "76bf894e969364709864" + }, +] +``` + +### Register a new runner + + +__Authentication is done with a Shared runner registration token or a project Specific runner registration token__ + +Used to make Gitlab CI aware of available runners. + + POST /runners/register + +Parameters: + + * `token` (required) - The registration token. It is 2 types of token you can pass here. + +1. Shared runner registration token +2. Project specific registration token + +Returns: + +```json +{ + "id" : 85, + "token" : "12b68e90394084703135" +} +``` + +### Delete a runner + + +__Authentication is done by runner token__ + +Used to removing runners. + + DELETE /runners/delete + +Parameters: + + * `token` (required) - The runner token. + +Returns: + +```json +{ + "id" : 1, + "token" : "d14963981a428f70121777e50643d1", + "created_at" : "2015-02-26T11:39:39.232Z", + "updated_at" : "2015-02-26T11:39:39.232Z", + "description" : "awesome runner" +} +```
\ No newline at end of file diff --git a/doc/ci/deployment/README.md b/doc/ci/deployment/README.md new file mode 100644 index 00000000000..ffd841ca9e7 --- /dev/null +++ b/doc/ci/deployment/README.md @@ -0,0 +1,98 @@ +## Using Dpl as deployment tool +Dpl (dee-pee-ell) is a deploy tool made for continuous deployment that's developed and used by Travis CI, but can also be used with GitLab CI. + +**We recommend to use Dpl, if you're deploying to any of these of these services: https://github.com/travis-ci/dpl#supported-providers**. + +### Requirements +To use Dpl you need at least Ruby 1.8.7 with ability to install gems. + +### Basic usage +The Dpl can be installed on any machine with: +``` +gem install dpl +``` + +This allows you to test all commands from your shell, rather than having to test it on a CI server. + +If you don't have Ruby installed you can do it on Debian-compatible Linux with: +``` +apt-get update +apt-get install ruby-dev +``` + +The Dpl provides support for vast number of services, including: Heroku, Cloud Foundry, AWS/S3, and more. +To use it simply define provider and any additional parameters required by the provider. + +For example if you want to use it to deploy your application to heroku, you need to specify `heroku` as provider, specify `api-key` and `app`. +There's more and all possible parameters can be found here: https://github.com/travis-ci/dpl#heroku + +``` +staging: + type: deploy + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api-key=$HEROKU_STAGING_API_KEY +``` + +In the above example we use Dpl to deploy `my-app-staging` to Heroku server with api-key stored in `HEROKU_STAGING_API_KEY` secure variable. + +To use different provider take a look at long list of [Supported Providers](https://github.com/travis-ci/dpl#supported-providers). + +### Using Dpl with Docker +When you use GitLab Runner you most likely configured it to use your server's shell commands. +This means that all commands are run in context of local user (ie. gitlab_runner or gitlab_ci_multi_runner). +It also means that most probably in your Docker container you don't have the Ruby runtime installed. +You will have to install it: +``` +staging: + type: deploy + - apt-get update -yq + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master +``` + +The first line `apt-get update -yq` updates the list of available packages, where second `apt-get install -y ruby-dev` install `Ruby` runtime on system. +The above example is valid for all Debian-compatible systems. + +### Usage in staging and production +It's pretty common in developer workflow to have staging (development) and production environment. +If we consider above example: we would like to deploy `master` branch to `staging` and `all tags` to `production` environment. +The final `.gitlab-ci.yml` for that setup would look like this: + +``` +staging: + type: deploy + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + type: deploy + - gem install dpl + - dpl --provider=heroku --app=my-app-production --api-key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +We created two deploy jobs that are executed on different events: +1. `staging` is executed for all commits that were pushed to `master` branch, +2. `production` is executed for all pushed tags. + +We also use two secure variables: +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +2. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +### Storing API keys +In GitLab CI 7.12 a new feature was introduced: Secure Variables. +Secure Variables can added by going to `Project > Variables > Add Variable`. +**This feature requires `gitlab-runner` with version equal or greater than 0.4.0.** +The variables that are defined in the project settings are send along with the build script to the runner. +The secure variables are stored out of the repository. Never store secrets in your projects' .gitlab-ci.yml. +It is also important that secret's value is hidden in the build log. + +You access added variable by prefixing it's name with `$` (on non-Windows runners) or `%` (for Windows Batch runners): +1. `$SECRET_VARIABLE` - use it for non-Windows runners +2. `%SECRET_VARIABLE%` - use it for Windows Batch runners diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md new file mode 100644 index 00000000000..84eaf29efd1 --- /dev/null +++ b/doc/ci/docker/README.md @@ -0,0 +1,4 @@ +# Docker integration + ++ [Using Docker Images](using_docker_images.md) ++ [Using Docker Build](using_docker_build.md)
\ No newline at end of file diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md new file mode 100644 index 00000000000..702a6c6b587 --- /dev/null +++ b/doc/ci/docker/using_docker_build.md @@ -0,0 +1,112 @@ +# Using Docker Build + +GitLab CI can allows you to use Docker Engine to build and test docker-based projects. + +**This also allows to you to use `docker-compose` and other docker-enabled tools.** + +This is one of new trends in Continuous Integration/Deployment to: + +1. create application image, +1. run test against created image, +1. push image to remote registry, +1. deploy server from pushed image + +It's also useful in case when your application already has the `Dockerfile` that can be used to create and test image: +```bash +$ docker build -t my-image dockerfiles/ +$ docker run my-docker-image /script/to/run/tests +$ docker tag my-image my-registry:5000/my-image +$ docker push my-registry:5000/my-image +``` + +However, this requires special configuration of GitLab Runner to enable `docker` support during build. +**This requires running GitLab Runner in privileged mode which can be harmful when untrusted code is run.** + +There are two methods to enable the use of `docker build` and `docker run` during build. + +## 1. Use shell executor + +The simplest approach is to install GitLab Runner in `shell` execution mode. +GitLab Runner then executes build scripts as `gitlab-runner` user. + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. During GitLab Runner installation select `shell` as method of executing build scripts or use command: + + ```bash + $ sudo gitlab-runner register -n \ + --url http://ci.gitlab.com \ + --token RUNNER_TOKEN \ + --executor shell + --description "My Runner" + ``` + +2. Install Docker on server. + + For more information how to install Docker on different systems checkout the [Supported installations](https://docs.docker.com/installation/). + +3. Add `gitlab-runner` user to `docker` group: + + ```bash + $ sudo usermod -aG docker gitlab-runner + ``` + +4. Verify that `gitlab-runner` has access to Docker: + + ```bash + $ sudo -u gitlab-runner -H docker info + ``` + + You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: + ```yaml + before_script: + - docker info + + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +5. You can now use `docker` command and install `docker-compose` if needed. + +6. However, by adding `gitlab-runner` to `docker` group you are effectively granting `gitlab-runner` full root permissions. +For more information please checkout [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). + +## 2. Use docker-in-docker executor + +Second approach is to use special Docker image with all tools installed (`docker` and `docker-compose`) and run build script in context of that image in privileged mode. +In order to do that follow the steps: + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. Register GitLab Runner from command line to use `docker` and `privileged` mode: + + ```bash + $ sudo gitlab-runner register -n \ + --url http://ci.gitlab.com \ + --token RUNNER_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "gitlab/dind:latest" \ + --docker-privileged + ``` + + The above command will register new Runner to use special [gitlab/dind](https://registry.hub.docker.com/u/gitlab/dind/) image which is provided by GitLab Inc. + The image at the start runs Docker daemon in [docker-in-docker](https://blog.docker.com/2013/09/docker-can-now-run-within-docker/) mode. + +1. You can now use `docker` from build script: + + ```yaml + before_script: + - docker info + + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +1. However, by enabling `--docker-privileged` you are effectively disables all security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. +For more information you could be interested in checking out [Runtime privilege](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration). + diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md new file mode 100644 index 00000000000..ef449cd45bc --- /dev/null +++ b/doc/ci/docker/using_docker_images.md @@ -0,0 +1,203 @@ +# Using Docker Images +GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. + +Docker is an open-source project that allows to use predefined images to run applications +in independent "containers" that are run within a single Linux instance. +[Docker Hub](https://registry.hub.docker.com/) have rich database of built images that can be used to build applications. + +Docker when used with GitLab CI runs each build in separate and isolated container using predefined image and always from scratch. +It makes it easier to have simple and reproducible build environment that can also be run on your workstation. +This allows you to test all commands from your shell, rather than having to test them on a CI server. + +### Register Docker runner +To use GitLab Runner with Docker you need to register new runner to use `docker` executor: + +```bash +gitlab-ci-multi-runner register \ + --url "https://ci.gitlab.com/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "docker-ruby-2.1" \ + --executor "docker" \ + --docker-image ruby:2.1 \ + --docker-postgres latest \ + --docker-mysql latest +``` + +**The registered runner will use `ruby:2.1` image and will run two services (`postgres:latest` and `mysql:latest`) that will be accessible for time of the build.** + +### What is image? +The image is the name of any repository that is present in local Docker Engine or any repository that can be found at [Docker Hub](https://registry.hub.docker.com/). +For more information about the image and Docker Hub please read the [Docker Fundamentals](https://docs.docker.com/introduction/understanding-docker/). + +### What is service? +Service is just another image that is run for time of your build and is linked to your build. This allows you to access the service image during build time. +The service image can run any application, but most common use case is to run some database container, ie.: `mysql`. +It's easier and faster to use existing image, run it as additional container than install `mysql` every time project is built. + +#### How is service linked to the build? +There's good document that describes how Docker linking works: [Linking containers together](https://docs.docker.com/userguide/dockerlinks/). +To summarize: if you add `mysql` as service to your application, the image will be used to create container that is linked to build container. +The service container for MySQL will be accessible under hostname `mysql`. +So, **to access your database service you have to connect to host: `mysql` instead of socket or `localhost`**. + +### How to use other images as services? +You are not limited to have only database services. +You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). +Look for `[runners.docker]` section: +``` +[runners.docker] + image = "ruby:2.1" + services = ["mysql:latest", "postgres:latest"] +``` + +For example you need `wordpress` instance to test some API integration with `Wordpress`. +You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). +This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: +``` +[runners.docker] + image = "ruby:2.1" + services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] +``` + +Next time when you run your application the `tutum/wordpress` will be started +and you will have access to it from your build container under hostname: `tutum_wordpress`. + +Alias hostname for the service is made from the image name: +1. Everything after `:` is stripped, +2. '/' is replaced to `_`. + +### Configuring services +Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. + +GitLab Runner 0.5.0 and up passes all YAML-defined variables to created service containers. + +1. To configure database name for [postgres](https://registry.hub.docker.com/u/library/postgres/) service, +you need to set POSTGRES_DB. + + ```yaml + services: + - postgres + + variables: + POSTGRES_DB: gitlab + ``` + +1. To use [mysql](https://registry.hub.docker.com/u/library/mysql/) service with empty password for time of build, +you need to set MYSQL_ALLOW_EMPTY_PASSWORD. + + ```yaml + services: + - mysql + + variables: + MYSQL_ALLOW_EMPTY_PASSWORD: yes + ``` + +For other possible configuration variables check the +https://registry.hub.docker.com/u/library/mysql/ or https://registry.hub.docker.com/u/library/postgres/ +or README page for any other Docker image. + +**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** + +### Overwrite image and services +It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. +If you add to your YAML the `image` and the `services` these parameters +be used instead of the ones that were specified during runner's registration. +``` +image: ruby:2.2 +services: + - postgres:9.3 +before_install: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +It's possible to define image and service per-job: +``` +before_install: + - bundle install + +test:2.1: + image: ruby:2.1 + services: + - postgres:9.3 + script: + - bundle exec rake spec + +test:2.2: + image: ruby:2.2 + services: + - postgres:9.4 + script: + - bundle exec rake spec +``` + +#### How to enable overwriting? +To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). +You can do that by hand modifying runner configuration: `config.toml`. +Please go to section where is `[runners.docker]` definition for your runner. +Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: +``` +[runners.docker] + image = "ruby:2.1" + allowed_images = ["ruby:*", "python:*"] + allowed_services = ["mysql:*", "redis:*"] +``` +This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. +You will be able to pick only `ruby` and `python` images. +The same rule can be applied to limit services. + +If you are courageous enough, you can make it fully open and accept everything: +``` +[runners.docker] + image = "ruby:2.1" + allowed_images = ["*", "*/*"] + allowed_services = ["*", "*/*"] +``` + +**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** + +### How Docker integration works +1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. +1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). +1. Create build container and link any service container to build container. +1. Start build container and send build script to the container. +1. Run build script. +1. Checkout code in: `/builds/group-name/project-name/`. +1. Run any step defined in `.gitlab-ci.yml`. +1. Check exit status of build script. +1. Remove build container and all created service containers. + +### How to debug a build locally +1. Create a file with build script: +```bash +$ cat <<EOF > build_script +git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner +cd /builds/gitlab-org/gitlab-ci-multi-runner +make <- or any other build step +EOF +``` + +1. Create service containers: +``` +$ docker run -d -n service-mysql mysql:latest +$ docker run -d -n service-postgres postgres:latest +``` +This will create two service containers (MySQL and PostgreSQL). + +1. Create a build container and execute script in its context: +``` +$ cat build_script | docker run -n build -i -l mysql:service-mysql -l postgres:service-postgres ruby:2.1 /bin/bash +``` +This will create build container that has two service containers linked. +The build_script is piped using STDIN to bash interpreter which executes the build script in container. + +1. At the end remove all containers: +``` +docker rm -f -v build service-mysql service-postgres +``` +This will forcefully (the `-f` switch) remove build container and service containers +and all volumes (the `-v` switch) that were created with the container creation. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md new file mode 100644 index 00000000000..e0b9fa0e25d --- /dev/null +++ b/doc/ci/examples/README.md @@ -0,0 +1,5 @@ +# Build script examples + ++ [Test and deploy Ruby Application to Heroku](test-and-deploy-ruby-application-to-heroku.md) ++ [Test and deploy Python Application to Heroku](test-and-deploy-python-application-to-heroku.md) ++ [Test Clojure applications](examples/test-clojure-application.md) diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md new file mode 100644 index 00000000000..859adf5f465 --- /dev/null +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -0,0 +1,72 @@ +## Test and Deploy a python application +This example will guide you how to run tests in your Python application and deploy it automatically as Heroku application. + +You can checkout the example [source](https://gitlab.com/ayufan/python-getting-started) and check [CI status](https://ci.gitlab.com/projects/4080). + +### Configure project +This is what the `.gitlab-ci.yml` file looks like for this project: +```yaml +test: + script: + # this configures django application to use attached postgres database that is run on `postgres` host + - export DATABASE_URL=postgres://postgres:@postgres:5432/python-test-app + - apt-get update -qy + - apt-get install -y python-dev python-pip + - pip install -r requirements.txt + - python manage.py test + +staging: + type: deploy + script: + - apt-get update -qy + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-python-test-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + type: deploy + script: + - apt-get update -qy + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-python-test-prod --api-key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +This project has three jobs: +1. `test` - used to test rails application, +2. `staging` - used to automatically deploy staging environment every push to `master` branch +3. `production` - used to automatically deploy production environmnet for every created tag + +### Store API keys +You'll need to create two variables in `Project > Variables`: +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +2. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/account). + +### Create Heroku application +For each of your environments, you'll need to create a new Heroku application. +You can do this through the [Dashboard](https://dashboard.heroku.com/). + +### Create runner +First install [Docker Engine](https://docs.docker.com/installation/). +To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner). +You can use public runners available on `ci.gitlab.com`, but you can register your own: +``` +gitlab-ci-multi-runner register \ + --non-interactive \ + --url "https://ci.gitlab.com/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "python-3.2" \ + --executor "docker" \ + --docker-image python:3.2 \ + --docker-postgres latest +``` + +With the command above, you create a runner that uses [python:3.2](https://registry.hub.docker.com/u/library/python/) image and uses [postgres](https://registry.hub.docker.com/u/library/postgres/) database. + +To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password. diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md new file mode 100644 index 00000000000..a1265ae8833 --- /dev/null +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -0,0 +1,67 @@ +## Test and Deploy a ruby application +This example will guide you how to run tests in your Ruby application and deploy it automatiacally as Heroku application. + +You can checkout the example [source](https://gitlab.com/ayufan/ruby-getting-started) and check [CI status](https://ci.gitlab.com/projects/4050). + +### Configure project +This is what the `.gitlab-ci.yml` file looks like for this project: +```yaml +test: + script: + - apt-get update -qy + - apt-get install -y nodejs + - bundle install --path /cache + - bundle exec rake db:create RAILS_ENV=test + - bundle exec rake test + +staging: + type: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-ruby-test-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + type: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-ruby-test-prod --api-key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +This project has three jobs: +1. `test` - used to test rails application, +2. `staging` - used to automatically deploy staging environment every push to `master` branch +3. `production` - used to automatically deploy production environmnet for every created tag + +### Store API keys +You'll need to create two variables in `Project > Variables`: +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +2. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/account). + +### Create Heroku application +For each of your environments, you'll need to create a new Heroku application. +You can do this through the [Dashboard](https://dashboard.heroku.com/). + +### Create runner +First install [Docker Engine](https://docs.docker.com/installation/). +To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner). +You can use public runners available on `ci.gitlab.com`, but you can register your own: +``` +gitlab-ci-multi-runner register \ + --non-interactive \ + --url "https://ci.gitlab.com/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "ruby-2.1" \ + --executor "docker" \ + --docker-image ruby:2.1 \ + --docker-postgres latest +``` + +With the command above, you create a runner that uses [ruby:2.1](https://registry.hub.docker.com/u/library/ruby/) image and uses [postgres](https://registry.hub.docker.com/u/library/postgres/) database. + +To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password.
\ No newline at end of file diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md new file mode 100644 index 00000000000..6c6faf8f928 --- /dev/null +++ b/doc/ci/examples/test-clojure-application.md @@ -0,0 +1,35 @@ +## Test Clojure applications + +This example will guide you how to run tests in your Clojure application. + +You can checkout the example [source](https://gitlab.com/dzaporozhets/clojure-web-application) and check [CI status](https://ci.gitlab.com/projects/6306). + +### Configure project + +This is what the `.gitlab-ci.yml` file looks like for this project: + +```yaml +variables: + POSTGRES_DB: sample-test + DATABASE_URL: "postgresql://postgres@postgres:5432/sample-test" + +before_script: + - apt-get update -y + - apt-get install default-jre postgresql-client -y + - wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein + - chmod a+x lein + - export LEIN_ROOT=1 + - PATH=$PATH:. + - lein deps + - lein migratus migrate + +test: + script: + - lein test +``` + +In before script we install JRE and [Leiningen](http://leiningen.org/). +Sample project uses [migratus](https://github.com/yogthos/migratus) library to manage database migrations. +So we added database migration as last step of `before_script` section + +You can use public runners available on `ci.gitlab.com` for testing your application with such configuration. diff --git a/doc/ci/install/README.md b/doc/ci/install/README.md new file mode 100644 index 00000000000..8cbc858458c --- /dev/null +++ b/doc/ci/install/README.md @@ -0,0 +1,276 @@ +# Select Version to Install +Make sure you view this installation guide from the branch (version) of GitLab CI you would like to install. In most cases +this should be the highest numbered stable branch (example shown below). + + + +If this is unclear check the [GitLab Blog](http://blog.gitlab.org/) for installation guide links by version. + +## GitLab CI 7.12 requires GitLab 7.12 or newer + +other [requirements](requirements.md) + +# Setup: + +## 1. Packages / Dependencies + +`sudo` is not installed on Debian by default. Make sure your system is +up-to-date and install it. + + sudo apt-get update + sudo apt-get upgrade + +**Note:** +During this installation some files will need to be edited manually. If +you are familiar with vim set it as default editor with the commands +below. If you are not familiar with vim please skip this and keep using +the default editor. + + # Install vim + sudo apt-get install vim + sudo update-alternatives --set editor /usr/bin/vim.basic + +Install the required packages: + + sudo apt-get install wget curl gcc checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libreadline6-dev libc6-dev libssl-dev libmysql++-dev make build-essential zlib1g-dev openssh-server git-core libyaml-dev postfix libpq-dev libicu-dev openssl nodejs + sudo apt-get install redis-server + +# 2. Ruby + +Download Ruby and compile it: + + mkdir /tmp/ruby && cd /tmp/ruby + curl --progress http://cache.ruby-lang.org/pub/ruby/ruby-2.1.6.tar.bz2 | tar xj + cd ruby-2.1.6/ + ./configure --disable-install-rdoc + make + sudo make install + +Install the Bundler Gem: + + sudo gem install bundler --no-ri --no-rdoc + + +## 3. GitLab CI user: + + sudo adduser --disabled-login --gecos 'GitLab CI' gitlab_ci + + +## 4. Prepare the database + +We recommend PostgreSQL but you can also use MySQL + +### MySQL + + # Install the database packages + sudo apt-get install mysql-server mysql-client libmysqlclient-dev + + # Login to MySQL + $ mysql -u root -p + + # Create the GitLab CI database + mysql> CREATE DATABASE IF NOT EXISTS `gitlab_ci_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; + + # Create the MySQL User change $password to a real password + mysql> CREATE USER 'gitlab_ci'@'localhost' IDENTIFIED BY '$password'; + + # Grant proper permissions to the MySQL User + mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `gitlab_ci_production`.* TO 'gitlab_ci'@'localhost'; + + # Logout MYSQL + mysql> exit; + +### PostgreSQL + + # Install the database packages + sudo apt-get install -y postgresql-9.1 libpq-dev + + # Login to PostgreSQL + sudo -u postgres psql -d template1 + + # Create a user for GitLab CI. We do not specify a password because we are using peer authentication. + template1=# CREATE USER gitlab_ci; + + # Create the GitLab CI production database & grant all privileges on database + template1=# CREATE DATABASE gitlab_ci_production OWNER gitlab_ci; + + # Quit the database session + template1=# \q + + # Try connecting to the new database with the new user + sudo -u gitlab_ci -H psql -d gitlab_ci_production + +## 5. Get code + + cd /home/gitlab_ci/ + + sudo -u gitlab_ci -H git clone https://gitlab.com/gitlab-org/gitlab-ci.git + + cd gitlab-ci + + sudo -u gitlab_ci -H git checkout 7-12-stable + +## 6. Setup application + + # Edit application settings + # Production + sudo -u gitlab_ci -H cp config/application.yml.example config/application.yml + sudo -u gitlab_ci -H editor config/application.yml + # Development + #sudo -u gitlab_ci -H cp config/application.yml.example.development config/application.yml + + # Copy the example secrets file + sudo -u gitlab_ci -H cp config/secrets.yml.example config/secrets.yml + sudo -u gitlab_ci -H chmod 0600 config/secrets.yml + + # Edit web server settings + sudo -u gitlab_ci -H cp config/unicorn.rb.example config/unicorn.rb + sudo -u gitlab_ci -H editor config/unicorn.rb + + # Create socket and pid directories + sudo -u gitlab_ci -H mkdir -p tmp/sockets/ + sudo chmod -R u+rwX tmp/sockets/ + sudo -u gitlab_ci -H mkdir -p tmp/pids/ + sudo chmod -R u+rwX tmp/pids/ + + # Change the permissions of the directory where build traces are stored + sudo chmod -R u+rwX builds/ + +### Install gems + + # For MySQL (note, the option says "without ... postgres") + sudo -u gitlab_ci -H bundle install --without development test postgres --deployment + + # Or for PostgreSQL (note, the option says "without ... mysql") + sudo -u gitlab_ci -H bundle install --without development test mysql --deployment + +### Setup db + + # mysql + sudo -u gitlab_ci -H cp config/database.yml.mysql config/database.yml + + # postgres + sudo -u gitlab_ci -H cp config/database.yml.postgresql config/database.yml + + # Edit user/password (not necessary with default Postgres setup) + sudo -u gitlab_ci -H editor config/database.yml + + # Setup tables + sudo -u gitlab_ci -H bundle exec rake setup RAILS_ENV=production + + # Setup schedules + sudo -u gitlab_ci -H bundle exec whenever -w RAILS_ENV=production + +### Secure secrets.yml + +The `secrets.yml` file stores encryption keys for sessions and secure variables. +Backup `secrets.yml` someplace safe, but don't store it in the same place as your database backups. +Otherwise your secrets are exposed if one of your backups is compromised. + +## 8. Install Init Script + +Copy the init script (will be /etc/init.d/gitlab_ci): + + sudo cp /home/gitlab_ci/gitlab-ci/lib/support/init.d/gitlab_ci /etc/init.d/gitlab_ci + +Make GitLab CI start on boot: + + sudo update-rc.d gitlab_ci defaults 21 + + +Start your GitLab CI instance: + + sudo service gitlab_ci start + # or + sudo /etc/init.d/gitlab_ci start + + +# 8. Nginx + + +## Installation + + sudo apt-get install nginx + +## Site Configuration + +Download an example site config: + + sudo cp /home/gitlab_ci/gitlab-ci/lib/support/nginx/gitlab_ci /etc/nginx/sites-available/gitlab_ci + sudo ln -s /etc/nginx/sites-available/gitlab_ci /etc/nginx/sites-enabled/gitlab_ci + +Make sure to edit the config file to match your setup: + + # Change **YOUR_SERVER_IP** and **YOUR_SERVER_FQDN** + # to the IP address and fully-qualified domain name + # of your host serving GitLab CI + sudo editor /etc/nginx/sites-enabled/gitlab_ci + +## Check your configuration + + sudo nginx -t + +## Start nginx + + sudo /etc/init.d/nginx start + +# 9. GitLab OAuth2 application + + +Go to the admin area of GitLab, to the `Application` section. Create an application for the GitLab CI +For callback URL use: `http://ci.example.com/user_sessions/callback` if you use http, or `https://ci.example.com/user_sessions/callback` if you use https. + +When `app_id` and `app_secret` are generated add them to the GitLab CI config: + +``` +production: + gitlab_server: + url: 'http://gitlab.example.com' + app_id: XXXXXX + app_secret: XXXXXX + +``` + + +# 10. Runners + + +Now you need Runners to process your builds. +Checkout the [Gitlab Runner section](https://about.gitlab.com/gitlab-ci/#gitlab-runner) to install it + +# Done! + + +Visit YOUR_SERVER for your first GitLab CI login. +You will be asked to authorize with your GitLab credentials. + +**Enjoy!** + +## Advanced settings + +### SMTP email settings + +If you want to use SMTP do next: + + # Copy config file + sudo -u gitlab_ci -H cp config/initializers/smtp_settings.rb.sample config/initializers/smtp_settings.rb + + # Edit it with your settings + sudo -u gitlab_ci -H editor config/initializers/smtp_settings.rb + +Restart application + +### Custom Redis Connection + +If you'd like Resque to connect to a Redis server on a non-standard port or on +a different host, you can configure its connection string via the +`config/resque.yml` file. + + # example + production: redis://redis.example.tld:6379 + +If you want to connect the Redis server via socket, then use the "unix:" URL scheme +and the path to the Redis socket file in the `config/resque.yml` file. + + # example + production: unix:/path/to/redis/socket diff --git a/doc/ci/install/requirements.md b/doc/ci/install/requirements.md new file mode 100644 index 00000000000..6c12607c3f8 --- /dev/null +++ b/doc/ci/install/requirements.md @@ -0,0 +1,61 @@ +# Requirements + +## Operating Systems + +### Supported Unix distributions + +- Ubuntu +- Debian +- CentOS +- Red Hat Enterprise Linux (please use the CentOS packages and instructions) +- Scientific Linux (please use the CentOS packages and instructions) +- Oracle Linux (please use the CentOS packages and instructions) + +For the installations options please see [the installation page on the GitLab website](https://about.gitlab.com/installation/). + +### Unsupported Unix distributions + +- OS X +- Arch Linux +- Fedora +- Gentoo +- FreeBSD + +### Non-Unix operating systems such as Windows + +GitLab CI is developed for Unix operating systems. +GitLab CI does **not** run on Windows and we have no plans of supporting it in the near future. +Please consider using a virtual machine to run GitLab CI. + +## Ruby versions + +GitLab requires Ruby (MRI) 2.0 or 2.1 +You will have to use the standard MRI implementation of Ruby. +We love [JRuby](http://jruby.org/) and [Rubinius](http://rubini.us/) but GitLab CI needs several Gems that have native extensions. + + +### Memory + +You need at least 1GB of addressable memory (RAM + swap) to install and use GitLab CI! + +## Unicorn Workers + +It's possible to increase the amount of unicorn workers and this will usually help for to reduce the response time of the applications and increase the ability to handle parallel requests. + +For most instances we recommend using: CPU cores + 1 = unicorn workers. +So for a machine with 2 cores, 3 unicorn workers is ideal. + +For all machines that have 1GB and up we recommend a minimum of three unicorn workers. +If you have a 512MB machine with a magnetic (non-SSD) swap drive we recommend to configure only one Unicorn worker to prevent excessive swapping. +With one Unicorn worker only git over ssh access will work because the git over HTTP access requires two running workers (one worker to receive the user request and one worker for the authorization check). +If you have a 512MB machine with a SSD drive you can use two Unicorn workers, this will allow HTTP access although it will be slow due to swapping. + +To change the Unicorn workers when you have the Omnibus package please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). + +## Supported web browsers + +- Chrome (Latest stable version) +- Firefox (Latest released version and [latest ESR version](https://www.mozilla.org/en-US/firefox/organizations/)) +- Safari 7+ (known problem: required fields in html5 do not work) +- Opera (Latest released version) +- IE 10+ diff --git a/doc/ci/migration_to_omnibus/README.md b/doc/ci/migration_to_omnibus/README.md new file mode 100644 index 00000000000..ae46f59a7bb --- /dev/null +++ b/doc/ci/migration_to_omnibus/README.md @@ -0,0 +1,29 @@ +## Migrating to packaged CI + +Since version 5.1 GitLab CI is shipping as part of the GitLab omnibus package. This guide describes how to migrate GitLab CI from a source installation to an Omnibus package. + +### 1. Update GitLab + +Update GitLab CI manually to the version that you will install using the omnibus package (at least 7.11). Follow the update [manual for installation from sourse](update/README.md) + +### 2. Backup + +``` +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production +``` + +This command will create a backup file in the tmp folder +(`/home/gitlab_ci/gitlab_ci/tmp/backups/*_gitlab_ci_backup.tar.gz`). You can read more in the [GitLab CI backup/restore documentation](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/raketasks/backup_restore.md) + +### 2. Install a packaged GitLab CI + +This process is described in the [instruction for enabling GitLab CI](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/gitlab-ci/README.md) + +### 4. Restore backup + +Put backup file to directory `/var/opt/gitlab/backups`. +Run the restore command: + +``` +sudo gitlab-ci-rake backup:restore +``` diff --git a/doc/ci/permissions/README.md b/doc/ci/permissions/README.md new file mode 100644 index 00000000000..d77061c14cd --- /dev/null +++ b/doc/ci/permissions/README.md @@ -0,0 +1,24 @@ +# Users Permissions + +GitLab CI relies on user's role on the GitLab. There are three permissions levels on GitLab CI: admin, master, developer, other. + +Admin user can perform any actions on GitLab CI in scope of instance and project. Also user with admin permission can use admin interface. + + + + +| Action | Guest, Reporter | Developer | Master | Admin | +|---------------------------------------|-----------------|-------------|----------|--------| +| See commits and builds | ✓ | ✓ | ✓ | ✓ | +| Retry or cancel build | | ✓ | ✓ | ✓ | +| Remove project | | | ✓ | ✓ | +| Create project | | | ✓ | ✓ | +| Change project configuration | | | ✓ | ✓ | +| Add specific runners | | | ✓ | ✓ | +| Add shared runners | | | | ✓ | +| See events in the system | | | | ✓ | +| Admin interface | | | | ✓ | + + + + diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md new file mode 100644 index 00000000000..3b9156f7409 --- /dev/null +++ b/doc/ci/quick_start/README.md @@ -0,0 +1,119 @@ +# Quick Start + +To start building projects with GitLab CI a few steps needs to be done. + +## 1. Install GitLab and CI + +First you need to have a working GitLab and GitLab CI instance. + +You can omit this step if you use [GitLab.com](http://GitLab.com/). + +## 2. Create repository on GitLab + +Once you login on your GitLab add a new repository where you will store your source code. +Push your application to that repository. + +## 3. Add project to CI + +The next part is to login to GitLab CI. +Point your browser to the URL you have set GitLab CI or use [ci.gitlab.com](http://ci.gitlab.com/) that is linked to [GitLab.com](http://GitLab.com/). + +On the first screen you will see a list of GitLab's projects that you have access to: + + + +Click **Add Project to CI**. +This will create project in CI and authorize GitLab CI to fetch sources from GitLab. + +> GitLab CI creates unique token that is used to configure GitLab CI service in GitLab. +> This token allows to access GitLab's repository and configures GitLab to trigger GitLab CI webhook on **Push events** and **Tag push events**. +> You can see that token by going to Project's Settings > Services > GitLab CI. +> You will see there token, the same token is assigned in GitLab CI settings of project. + +## 4. Create project's configuration - .gitlab-ci.yml + +The next: You have to define how your project will be built. +GitLab CI uses [YAML](https://en.wikipedia.org/wiki/YAML) file to store build configuration. +You need to create `.gitlab-ci.yml` in root directory of your repository: + +```yaml +before_script: + - bundle install + +rspec: + script: + - bundle exec rspec + +rubocop: + script: + - bundle exec rubocop +``` + +This is the simplest possible build configuration that will work for most Ruby applications: +1. Define two jobs `rspec` and `rubocop` with two different commands to be executed. +1. Before every job execute commands defined by `before_script`. + +The `.gitlab-ci.yml` defines set of jobs with constrains how and when they should be run. +The jobs are defined as top-level elements with name and always have to contain the `script`. +Jobs are used to create builds, which are then picked by [runners](../runners/README.md) and executed within environment of the runner. +What is important that each job is run independently from each other. + +For more information and complete `.gitlab-ci.yml` syntax, please check the [Configuring project (.gitlab-ci.yml)](../yaml/README.md). + +## 5. Add file and push .gitlab-ci.yml to repository + +Once you created `.gitlab-ci.yml` you should add it to git repository and push it to GitLab. + +```bash +git add .gitlab-ci.yml +git commit +git push origin master +``` + +If you refresh the project's page on GitLab CI you will notice a one new commit: + + + +However the commit has status **pending** which means that commit was not yet picked by runner. + +## 6. Configure runner + +In GitLab CI, Runners run your builds. +A runner is a machine (can be virtual, bare-metal or VPS) that picks up builds through the coordinator API of GitLab CI. + +A runner can be specific to a certain project or serve any project in GitLab CI. +A runner that serves all projects is called a shared runner. +More information about different runner types can be found in [Configuring runner](../runners/README.md). + +To check if you have runners assigned to your project go to **Runners**. You will find there information how to setup project specific runner: + +1. Install GitLab Runner software. Checkout the [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner) section to install it. +1. Specify following URL during runner setup: https://ci.gitlab.com/ +1. Use the following registration token during setup: TOKEN + +If you do it correctly your runner should be shown under **Runners activated for this project**: + + + +### Shared runners + +If you use [ci.gitlab.com](http://ci.gitlab.com/) you can use **Shared runners** provided by GitLab Inc. +These are special virtual machines that are run on GitLab's infrastructure that can build any project. +To enable **Shared runners** you have to go to **Runners** and click **Enable shared runners** for this project. + +## 7. Check status of commit + +If everything went OK and you go to commit, the status of the commit should change from **pending** to either **running**, **success** or **failed**. + + + +You can click **Build ID** to view build log for specific job. + +## 8. Congratulations! + +You managed to build your first project using GitLab CI. +You may need to tune your `.gitlab-ci.yml` file to implement build plan for your project. +A few examples how it can be done you can find on [Examples](../examples/README.md) page. + +GitLab CI also offers **the Lint** tool to verify validity of your `.gitlab-ci.yml` which can be useful to troubleshoot potential problems. +The Lint is available from project's settings or by adding `/lint` to GitLab CI url. diff --git a/doc/ci/quick_start/build_status.png b/doc/ci/quick_start/build_status.png Binary files differnew file mode 100644 index 00000000000..333259e6acd --- /dev/null +++ b/doc/ci/quick_start/build_status.png diff --git a/doc/ci/quick_start/commit_status.png b/doc/ci/quick_start/commit_status.png Binary files differnew file mode 100644 index 00000000000..725b79e6f91 --- /dev/null +++ b/doc/ci/quick_start/commit_status.png diff --git a/doc/ci/quick_start/new_commit.png b/doc/ci/quick_start/new_commit.png Binary files differnew file mode 100644 index 00000000000..3839e893c17 --- /dev/null +++ b/doc/ci/quick_start/new_commit.png diff --git a/doc/ci/quick_start/projects.png b/doc/ci/quick_start/projects.png Binary files differnew file mode 100644 index 00000000000..0b3430a69db --- /dev/null +++ b/doc/ci/quick_start/projects.png diff --git a/doc/ci/quick_start/runners.png b/doc/ci/quick_start/runners.png Binary files differnew file mode 100644 index 00000000000..25b4046bc00 --- /dev/null +++ b/doc/ci/quick_start/runners.png diff --git a/doc/ci/quick_start/runners_activated.png b/doc/ci/quick_start/runners_activated.png Binary files differnew file mode 100644 index 00000000000..c934bd12f41 --- /dev/null +++ b/doc/ci/quick_start/runners_activated.png diff --git a/doc/ci/raketasks/README.md b/doc/ci/raketasks/README.md new file mode 100644 index 00000000000..872be4dc966 --- /dev/null +++ b/doc/ci/raketasks/README.md @@ -0,0 +1,3 @@ +# Rake Tasks + ++ [Backup/Restore](backup_restore.md)
\ No newline at end of file diff --git a/doc/ci/raketasks/backup_restore.md b/doc/ci/raketasks/backup_restore.md new file mode 100644 index 00000000000..eed12c46247 --- /dev/null +++ b/doc/ci/raketasks/backup_restore.md @@ -0,0 +1,237 @@ +# Backup restore + +## Create a backup of the GitLab CI + +A backup creates an archive file that contains the database and builds files. +This archive will be saved in backup_path (see `config/application.yml`). +The filename will be `[TIMESTAMP]_gitlab_ci_backup.tar.gz`. This timestamp can be used to restore an specific backup. +You can only restore a backup to exactly the same version of GitLab CI that you created it on, for example 7.10.1. + +*If you are interested in the GitLab backup please follow to the [GitLab backup documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md)* + +``` +# use this command if you've installed GitLab CI with the Omnibus package +sudo gitlab-ci-rake backup:create + +# if you've installed GitLab from source +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production +``` + + +Example output: + +``` +Dumping database ... +Dumping PostgreSQL database gitlab_ci_development ... [DONE] +done +Dumping builds ... +done +Creating backup archive: 1430930060_gitlab_ci_backup.tar.gz ... done +Uploading backup archive to remote storage ... skipped +Deleting tmp directories ... done +done +Deleting old backups ... skipping +``` + +## Upload backups to remote (cloud) storage + +You can let the backup script upload the '.tar.gz' file it creates. +It uses the [Fog library](http://fog.io/) to perform the upload. +In the example below we use Amazon S3 for storage. +But Fog also lets you use [other storage providers](http://fog.io/storage/). + +For omnibus packages: + +```ruby +gitlab_ci['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' +} +gitlab_ci['backup_upload_remote_directory'] = 'my.s3.bucket' +gitlab_ci['backup_multipart_chunk_size'] = 104857600 +``` + +For installations from source: + +```yaml + backup: + # snip + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: AWS + region: eu-west-1 + aws_access_key_id: AKIAKIAKI + aws_secret_access_key: 'secret123' + # The remote 'directory' to store your backups. For S3, this would be the bucket name. + remote_directory: 'my.s3.bucket' + multipart_chunk_size: 104857600 +``` + +If you are uploading your backups to S3 you will probably want to create a new +IAM user with restricted access rights. To give the upload user access only for +uploading backups create the following IAM profile, replacing `my.s3.bucket` +with the name of your bucket: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1412062044000", + "Effect": "Allow", + "Action": [ + "s3:AbortMultipartUpload", + "s3:GetBucketAcl", + "s3:GetBucketLocation", + "s3:GetObject", + "s3:GetObjectAcl", + "s3:ListBucketMultipartUploads", + "s3:PutObject", + "s3:PutObjectAcl" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket/*" + ] + }, + { + "Sid": "Stmt1412062097000", + "Effect": "Allow", + "Action": [ + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "*" + ] + }, + { + "Sid": "Stmt1412062128000", + "Effect": "Allow", + "Action": [ + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket" + ] + } + ] +} +``` + +## Storing configuration files + +Please be informed that a backup does not store your configuration and secret files. +If you use an Omnibus package please see the [instructions in the readme to backup your configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#backup-and-restore-omnibus-gitlab-configuration). +If you have a cookbook installation there should be a copy of your configuration in Chef. +If you have an installation from source: +1. please backup `config/secrets.yml` file that contains key to encrypt variables in database, +but don't store it in the same place as your database backups. +Otherwise your secrets are exposed in case one of your backups is compromised. +1. please consider backing up your `application.yml` file, +1. any SSL keys and certificates, +1. and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +## Restore a previously created backup + +You can only restore a backup to exactly the same version of GitLab CI that you created it on, for example 7.10.1. + +### Installation from source + +``` +sudo -u gitlab_ci -H bundle exec rake backup:restore RAILS_ENV=production +``` + +Options + +``` +BACKUP=timestamp_of_backup (required if more than one backup exists) +``` + +### Omnibus package installation + +We will assume that you have installed GitLab CI from an omnibus package and run +`sudo gitlab-ctl reconfigure` at least once. + +First make sure your backup tar file is in `/var/opt/gitlab/backups`. + +```shell +sudo cp 1393513186_gitlab_ci_backup.tar.gz /var/opt/gitlab/backups/ +``` + +Next, restore the backup by running the restore command. You need to specify the +timestamp of the backup you are restoring. + +```shell +# Stop processes that are connected to the database +sudo gitlab-ctl stop ci-unicorn +sudo gitlab-ctl stop ci-sidekiq + +# This command will overwrite the contents of your GitLab CI database! +sudo gitlab-ci-rake backup:restore BACKUP=1393513186 + +# Start GitLab +sudo gitlab-ctl start +``` + +If there is a GitLab version mismatch between your backup tar file and the installed +version of GitLab, the restore command will abort with an error. Install a package for +the [required version](https://www.gitlab.com/downloads/archives/) and try again. + + + +## Configure cron to make daily backups + +### For installation from source: +``` +cd /home/git/gitlab +sudo -u gitlab_ci -H editor config/application.yml # Enable keep_time in the backup section to automatically delete old backups +sudo -u gitlab_ci crontab -e # Edit the crontab for the git user +``` + +Add the following lines at the bottom: + +``` +# Create a backup of the GitLab CI every day at 4am +0 4 * * * cd /home/gitlab_ci/gitlab_ci && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake backup:create RAILS_ENV=production CRON=1 +``` + +The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors. +This is recommended to reduce cron spam. + +### Omnibus package installation + +To schedule a cron job that backs up your GitLab CI, use the root user: + +``` +sudo su - +crontab -e +``` + +There, add the following line to schedule the backup for everyday at 2 AM: + +``` +0 2 * * * /opt/gitlab/bin/gitlab-ci-rake backup:create CRON=1 +``` + +You may also want to set a limited lifetime for backups to prevent regular +backups using all your disk space. To do this add the following lines to +`/etc/gitlab/gitlab.rb` and reconfigure: + +``` +# limit backup lifetime to 7 days - 604800 seconds +gitlab_ci['backup_keep_time'] = 604800 +``` + +NOTE: This cron job does not [backup your omnibus-gitlab configuration](#backup-and-restore-omnibus-gitlab-configuration). + +## Known issues + +If you’ve been using GitLab CI since 7.11 or before using MySQL and the official installation guide, you will probably get the following error while making a backup: `Dumping MySQL database gitlab_ci_production ... mysqldump: Got error: 1044: Access denied for user 'gitlab_ci'@'localhost' to database 'gitlab_ci_production' when using LOCK TABLES` .This can be resolved by adding a LOCK TABLES permission to the gitlab_ci MySQL user. Add this permission with: +``` +$ mysql -u root -p +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `gitlab_ci_production`.* TO 'gitlab_ci'@'localhost'; +``` + diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md new file mode 100644 index 00000000000..68dcfe23ffb --- /dev/null +++ b/doc/ci/runners/README.md @@ -0,0 +1,145 @@ +# Runners + +In GitLab CI, Runners run your [yaml](../yaml/README.md). +A runner is an isolated (virtual) machine that picks up builds +through the coordinator API of GitLab CI. + +A runner can be specific to a certain project or serve any project +in GitLab CI. A runner that serves all projects is called a shared runner. + +## Shared vs. Specific Runners + +A runner that is specific only runs for the specified project. A shared runner +can run jobs for every project that has enabled the option +`Allow shared runners`. + +**Shared runners** are useful for jobs that have similar requirements, +between multiple projects. Rather than having multiple runners idling for +many projects, you can have a single or a small number of runners that handle +multiple projects. This makes it easier to maintain and update runners. + +**Specific runners** are useful for jobs that have special requirements or for +projects with a very demand. If a job has certain requirements, you can set +up the specific runner with this in mind, while not having to do this for all +runners. For example, if you want to deploy a certain project, you can setup +a specific runner to have the right credentials for this. + +Projects with high demand of CI activity can also benefit from using specific runners. +By having dedicated runners you are guaranteed that the runner is not being held +up by another project's jobs. + +You can set up a specific runner to be used by multiple projects. The difference +with a shared runner is that you have to enable each project explicitly for +the runner to be able to run its jobs. + +Specific runners do not get shared with forked projects automatically. +A fork does copy the CI settings (jobs, allow shared, etc) of the cloned repository. + +# Creating and Registering a Runner + +There are several ways to create a runner. Only after creation, upon +registration its status as Shared or Specific is determined. + +[See the documentation for](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation) +the different methods of installing a Runner instance. + +After installing the runner, you can either register it as `Shared` or as `Specific`. +You can only register a Shared Runner if you have admin access to the GitLab instance. + +## Registering a Shared Runner + +You can only register a shared runner if you are an admin on the linked +GitLab instance. + +Grab the shared-runner token on the `admin/runners` page of your GitLab CI +instance. + + + +Now simply register the runner as any runner: + +``` +sudo gitlab-runner register +``` + +Note that you will have to enable `Allows shared runners` for each project +that you want to make use of a shared runner. This is by default `off`. + +## Registering a Specific Runner + +Registering a specific can be done in two ways: + +1. Creating a runner with the project registration token +1. Converting a shared runner into a specific runner (one-way, admin only) + +There are several ways to create a runner instance. The steps below only +concern registering the runner on GitLab CI. + +### Registering a Specific Runner with a Project Registration token + +To create a specific runner without having admin rights to the GitLab instance, +visit the project you want to make the runner work for in GitLab CI. + +Click on the runner tab and use the registration token you find there to +setup a specific runner for this project. + + + +To register the runner, run the command below and follow instructions: + +``` +sudo gitlab-runner register +``` + +### Making an existing Shared Runner Specific + +If you are an admin on your GitLab instance, +you can make any shared runner a specific runner, _but you can not +make a specific runner a shared runner_. + +To make a shared runner specific, go to the runner page (`/admin/runners`) +and find your runner. Add any projects on the left to make this runner +run exclusively for these projects, therefore making it a specific runner. + + + +## Using Shared Runners Effectively + +If you are planning to use shared runners, there are several things you +should keep in mind. + +### Use Tags + +You must setup a runner to be able to run all the different types of jobs +that it may encounter on the projects it's shared over. This would be +problematic for large amounts of projects, if it wasn't for tags. + +By tagging a Runner for the types of jobs it can handle, you can make sure +shared runners will only run the jobs they are equipped to run. + +For instance, at GitLab we have runners tagged with "rails" if they contain +the appropriate dependencies to run Rails test suites. + +### Be Careful with Sensitive Information + +If you can run a build on a runner, you can get access to any code it runs +and get the token of the runner. With shared runners, this means that anyone +that runs jobs on the runner, can access anyone else's code that runs on the runner. + +In addition, because you can get access to the runner token, it is possible +to create a clone of a runner and submit false builds, for example. + +The above is easily avoided by restricting the usage of shared runners +on large public GitLab instances and controlling access to your GitLab instance. + +### Forks + +Whenever a project is forked, it copies the settings of the jobs that relate +to it. This means that if you have shared runners setup for a project and +someone forks that project, the shared runners will also serve jobs of this +project. + +# Attack vectors in runners + +Mentioned briefly earlier, but the following things of runners can be exploited. +We're always looking for contributions that can mitigate these [Security Considerations](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md). diff --git a/doc/ci/runners/project_specific.png b/doc/ci/runners/project_specific.png Binary files differnew file mode 100644 index 00000000000..f51ea694e78 --- /dev/null +++ b/doc/ci/runners/project_specific.png diff --git a/doc/ci/runners/shared_runner.png b/doc/ci/runners/shared_runner.png Binary files differnew file mode 100644 index 00000000000..9755144eb08 --- /dev/null +++ b/doc/ci/runners/shared_runner.png diff --git a/doc/ci/runners/shared_to_specific_admin.png b/doc/ci/runners/shared_to_specific_admin.png Binary files differnew file mode 100644 index 00000000000..44a4bef22f7 --- /dev/null +++ b/doc/ci/runners/shared_to_specific_admin.png diff --git a/doc/ci/update/3.0-to-3.1.md b/doc/ci/update/3.0-to-3.1.md new file mode 100644 index 00000000000..039e781a6c8 --- /dev/null +++ b/doc/ci/update/3.0-to-3.1.md @@ -0,0 +1,30 @@ +# Update from 3.0 to 3.1 + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 3-1-stable +``` + +### 4. Install libs, migrations etc + +``` +bundle install --without development test --deployment +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/3.1-to-3.2.md b/doc/ci/update/3.1-to-3.2.md new file mode 100644 index 00000000000..dc6ab6514c6 --- /dev/null +++ b/doc/ci/update/3.1-to-3.2.md @@ -0,0 +1,30 @@ +# Update from 3.1 to 3.2 + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 3-2-stable +``` + +### 4. Install libs, migrations etc + +``` +bundle install --without development test --deployment +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/3.2-to-4.0.md b/doc/ci/update/3.2-to-4.0.md new file mode 100644 index 00000000000..50c26e49224 --- /dev/null +++ b/doc/ci/update/3.2-to-4.0.md @@ -0,0 +1,35 @@ +# Update from 3.2 to 4.0 + +## GitLab CI 4.0 requires GitLab 6.3 or higher + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 4-0-stable +``` + +### 4. Install libs, migrations etc + +``` +bundle install --without development test --deployment +bundle exec rake db:migrate RAILS_ENV=production +bundle exec whenever -w +``` + +### 5. Start web application + + sudo service gitlab_ci start + +### 6. Update your runners to version 4.0 diff --git a/doc/ci/update/4.0-to-4.1.md b/doc/ci/update/4.0-to-4.1.md new file mode 100644 index 00000000000..e749b324ee7 --- /dev/null +++ b/doc/ci/update/4.0-to-4.1.md @@ -0,0 +1,49 @@ +# Update from 4.0 to 4.1 + +## GitLab CI 4.x requires GitLab 6.3 or higher + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 4-1-stable +``` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production + +# Update cron +bundle exec whenever -w +``` + +### 5. Start web application + + sudo service gitlab_ci start + +### 6. Make sure your runners are version 4.0 + + + cd path_to_runner + cat VERSION + +To update runners follow this instructions https://github.com/gitlabhq/gitlab-ci-runner#update diff --git a/doc/ci/update/4.1-to-4.2.md b/doc/ci/update/4.1-to-4.2.md new file mode 100644 index 00000000000..81394aa216e --- /dev/null +++ b/doc/ci/update/4.1-to-4.2.md @@ -0,0 +1,47 @@ +# Update from 4.1 to 4.2 + +## GitLab CI 4.x requires GitLab 6.3 or higher + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 4-2-stable +``` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Install the new init script +As a user with sudo rights: + +``` +cd /home/gitlab_ci/gitlab-ci +sudo cp lib/support/init.d/gitlab_ci /etc/init.d/gitlab_ci +sudo chmod +x /etc/init.d/gitlab_ci +``` + +### 6. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/4.2-to-4.3.md b/doc/ci/update/4.2-to-4.3.md new file mode 100644 index 00000000000..f340cb61411 --- /dev/null +++ b/doc/ci/update/4.2-to-4.3.md @@ -0,0 +1,61 @@ +# Update from 4.2 to 4.3 + +## GitLab CI 4.x requires GitLab 6.3 or higher + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 4-3-stable +``` + +### 4. Install libs, migrations etc + +Edit web server settings + +``` +cp config/unicorn.rb.example config/unicorn.rb +editor config/unicorn.rb +``` + +Then: + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Setup unicorn +``` +cp config/unicorn.rb.example config/unicorn.rb +``` + +### 6. Install the new init script +As a user with sudo rights: + +``` +cd /home/gitlab_ci/gitlab-ci +sudo cp lib/support/init.d/gitlab_ci /etc/init.d/gitlab_ci +sudo chmod +x /etc/init.d/gitlab_ci +``` + +### 7. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/4.3-to-5.0.md b/doc/ci/update/4.3-to-5.0.md new file mode 100644 index 00000000000..23327eac608 --- /dev/null +++ b/doc/ci/update/4.3-to-5.0.md @@ -0,0 +1,42 @@ +# Update from 4.3 to 5.0 + +__GitLab CI 5.0 requires GitLab 6.3 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. get latest code + +``` +git fetch +git checkout 5-0-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start
\ No newline at end of file diff --git a/doc/ci/update/5.0-to-5.1.md b/doc/ci/update/5.0-to-5.1.md new file mode 100644 index 00000000000..9a37a87a882 --- /dev/null +++ b/doc/ci/update/5.0-to-5.1.md @@ -0,0 +1,42 @@ +# Update from 5.0 to 5.1 + +__GitLab CI 5.1 requires GitLab 6.3 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 5-1-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start
\ No newline at end of file diff --git a/doc/ci/update/5.1-to-5.2.md b/doc/ci/update/5.1-to-5.2.md new file mode 100644 index 00000000000..6dcf00276f6 --- /dev/null +++ b/doc/ci/update/5.1-to-5.2.md @@ -0,0 +1,42 @@ +# Update from 5.1 to 5.2 + +__GitLab CI 5.2 requires GitLab 7.5 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 5-2-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start
\ No newline at end of file diff --git a/doc/ci/update/5.2-to-5.3.md b/doc/ci/update/5.2-to-5.3.md new file mode 100644 index 00000000000..2b70f2146ac --- /dev/null +++ b/doc/ci/update/5.2-to-5.3.md @@ -0,0 +1,42 @@ +# Update from 5.2 to 5.3 + +__GitLab CI 5.3 requires GitLab 7.5 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 5-3-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start
\ No newline at end of file diff --git a/doc/ci/update/5.3-to-5.4.md b/doc/ci/update/5.3-to-5.4.md new file mode 100644 index 00000000000..6d1cd61389a --- /dev/null +++ b/doc/ci/update/5.3-to-5.4.md @@ -0,0 +1,60 @@ +# Update from 5.3 to 5.4 + +__GitLab CI 5.4 requires GitLab 7.5 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 5-4-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Update config +GitLab CI 5.4 and above make use of the OAuth2 protocol for authentication with GitLab. This means that after updating GitLab (CI), +you need to create an OAuth2 application in GitLab admin area, which gives you the APP_ID and APP_SECRET. + +For callback URL use: `http://ci.example.com/user_sessions/callback` if you use http, or `https://ci.example.com/user_sessions/callback` if you use https. + +You will have to add APP_ID and APP_SECRET to the GitLab CI config, as such: + +``` +production: + gitlab_server: + url: 'http://gitlab.example.com' + app_id: XXXXXX + app_secret: XXXXXX + +``` + + +### 6. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/5.4-to-7.8.md b/doc/ci/update/5.4-to-7.8.md new file mode 100644 index 00000000000..71561d047e6 --- /dev/null +++ b/doc/ci/update/5.4-to-7.8.md @@ -0,0 +1,65 @@ +# Update from 5.3 to 7.8 + +## Notice + +With this release we are bumping the GitLab CI version to 7.8 in order to be on par with the current GitLab version and +to avoid naming confusion. + +__GitLab CI 7.8 requires GitLab 7.8 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 7-8-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + +``` +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Update config +GitLab CI 5.4 and above make use of the OAuth2 protocol for authentication with GitLab. This means that after updating GitLab (CI), +you need to create an OAuth2 application in GitLab admin area, which gives you the APP_ID and APP_SECRET. + +For callback URL use: `http://ci.example.com/user_sessions/callback` if you use http, or `https://ci.example.com/user_sessions/callback` if you use https. + +You will have to add APP_ID and APP_SECRET to the GitLab CI config, as such: + +``` +production: + gitlab_server: + url: 'http://gitlab.example.com' + app_id: XXXXXX + app_secret: XXXXXX + +``` + + +### 6. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/7.10-to-7.11.md b/doc/ci/update/7.10-to-7.11.md new file mode 100644 index 00000000000..4e04509e84d --- /dev/null +++ b/doc/ci/update/7.10-to-7.11.md @@ -0,0 +1,45 @@ +# Update from 7.10 to 7.11 + +## Notice + +__GitLab CI 7.11 requires GitLab 7.11 or higher and GitLab Multi Runner 0.3.0 and higher + +### 1. Stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 7-11-stable +``` + +### 4. Install libs, migrations etc + + +``` +# Install nodejs dependency: +sudo apt-get install nodejs + +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + + +### 5. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/7.11-to-7.12.md b/doc/ci/update/7.11-to-7.12.md new file mode 100644 index 00000000000..ad45ea3a660 --- /dev/null +++ b/doc/ci/update/7.11-to-7.12.md @@ -0,0 +1,67 @@ +# Update from 7.11 to 7.12 + +## Notice + +__GitLab CI 7.12 requires GitLab 7.12 or higher and GitLab Multi Runner 0.4.0 or higher + +### 1. Stop CI server + + sudo service gitlab_ci stop + +### 2. Update ruby if needed + +If your ruby version is older than 2.0.0 please update it. + +Update packages: + + sudo apt-get update + sudo apt-get upgrade + +Download Ruby and compile it: + + mkdir /tmp/ruby && cd /tmp/ruby + curl --progress http://cache.ruby-lang.org/pub/ruby/ruby-2.1.6.tar.bz2 | tar xj + cd ruby-2.1.6/ + ./configure --disable-install-rdoc + make + sudo make install + +Install the Bundler Gem: + + sudo gem install bundler --no-ri --no-rdoc + +### 3. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 4. Get latest code + +``` +git fetch +git checkout 7-12-stable +``` + +### 5. Install libs, migrations etc + + +``` +# Install nodejs dependency: +sudo apt-get install nodejs + +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + + +### 6. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/7.12-to-7.13.md b/doc/ci/update/7.12-to-7.13.md new file mode 100644 index 00000000000..2877c297d6f --- /dev/null +++ b/doc/ci/update/7.12-to-7.13.md @@ -0,0 +1,63 @@ +# Update from 7.12 to 7.13 + +## Notice + +__GitLab CI 7.13 requires GitLab 7.12 or higher and GitLab Multi Runner 0.5.0 or higher + +### 1. Stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 7-13-stable +``` + +### 4. Make sure GitLab CI can write to the builds/ directory + +``` +sudo chmod -R u+rwX builds +``` + +### 4. Copy secrets + +The `secrets.yml` file is used to store keys to encrypt sessions and encrypt secure variables. +When you run migrations make sure to store it someplace safe. +Don't store it in the same place as your database backups, +otherwise your secrets are exposed if one of your backups is compromised. + +``` +sudo -u gitlab_ci -H cp config/secrets.yml.example config/secrets.yml +sudo -u gitlab_ci -H chmod 0600 config/secrets.yml +``` + +### 5. Install libs, migrations etc + + +``` +# Install nodejs dependency: +sudo apt-get install nodejs + +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + + +### 5. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/7.8-to-7.9.md b/doc/ci/update/7.8-to-7.9.md new file mode 100644 index 00000000000..fc1cefe9ba5 --- /dev/null +++ b/doc/ci/update/7.8-to-7.9.md @@ -0,0 +1,66 @@ +# Update from 7.8 to 7.9 + +## Notice + +__GitLab CI 7.9 requires GitLab 7.9 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 7-9-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + + +``` +# Install nodejs dependency: +sudo apt-get install nodejs + +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Update config +GitLab CI 5.4 and above make use of the OAuth2 protocol for authentication with GitLab. This means that after updating GitLab (CI), +you need to create an OAuth2 application in GitLab admin area, which gives you the APP_ID and APP_SECRET. + +For callback URL use: `http://ci.example.com/user_sessions/callback` if you use http, or `https://ci.example.com/user_sessions/callback` if you use https. + +You will have to add APP_ID and APP_SECRET to the GitLab CI config, as such: + +``` +production: + gitlab_server: + url: 'http://gitlab.example.com' + app_id: XXXXXX + app_secret: XXXXXX + +``` + + +### 6. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/7.9-to-7.10.md b/doc/ci/update/7.9-to-7.10.md new file mode 100644 index 00000000000..2b54874daf4 --- /dev/null +++ b/doc/ci/update/7.9-to-7.10.md @@ -0,0 +1,49 @@ +# Update from 7.9 to 7.10 + +## Notice + +__GitLab CI 7.10 requires GitLab 7.10 or higher and GitLab CI Runner v5__ + +### 1. stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git fetch +git checkout 7-10-stable +``` + +#### Redis config + +If you have `config/resque.yml` file - please update it with recent `config/resque.yml.example` + +### 4. Install libs, migrations etc + + +``` +# Install nodejs dependency: +sudo apt-get install nodejs + +# For MySQL users +bundle install --without postgres development test --deployment + +# For Postgres users +bundle install --without mysql development test --deployment + +# Run migrations +bundle exec rake db:migrate RAILS_ENV=production +``` + + +### 5. Start web application + + sudo service gitlab_ci start diff --git a/doc/ci/update/README.md b/doc/ci/update/README.md new file mode 100644 index 00000000000..7a615ef7978 --- /dev/null +++ b/doc/ci/update/README.md @@ -0,0 +1,2 @@ ++ [The indivual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update) ++ [Patch versions](patch_versions.md) diff --git a/doc/ci/update/patch_versions.md b/doc/ci/update/patch_versions.md new file mode 100644 index 00000000000..c13f69c03c9 --- /dev/null +++ b/doc/ci/update/patch_versions.md @@ -0,0 +1,59 @@ +# Universal update guide for patch versions. For example from 4.0.0 to 4.0.1, also see the [semantic versioning specification](http://semver.org/). + +### 1. Stop CI server + + sudo service gitlab_ci stop + +### 2. Switch to your gitlab_ci user + +``` +sudo su gitlab_ci +cd /home/gitlab_ci/gitlab-ci +``` + +### 3. Get latest code + +``` +git pull origin STABLE_BRANCH +``` + +### 4. Install libs, migrations etc + +``` +bundle install --without development test --deployment +bundle exec rake db:migrate RAILS_ENV=production +``` + +### 5. Start web application + + sudo service gitlab_ci start + + +# One line upgrade command + +You have read through the entire guide and probably already did all the steps one by one. + +Here is a one line command with all above steps for the next time you upgrade: + +``` + sudo service gitlab_ci stop && \ + cd /home/gitlab_ci/gitlab-ci && \ + sudo -u gitlab_ci -H git pull origin `git rev-parse --abbrev-ref HEAD` && \ + sudo -u gitlab_ci -H bundle install --without development test --deployment && \ + sudo -u gitlab_ci -H bundle exec rake db:migrate RAILS_ENV=production && \ + cd && \ + sudo service gitlab_ci start +``` + +Since when we start this `gitlab_ci` service, the document `db/schema.rb` is shown always as modified for git, you could even do like this, **if and only if**, you are sure you only have that modification: + +``` + sudo service gitlab_ci stop && \ + cd /home/gitlab_ci/gitlab-ci && \ + sudo -u gitlab_ci -H git checkout -f `git rev-parse --abbrev-ref HEAD` && \ + sudo -u gitlab_ci -H git pull origin `git rev-parse --abbrev-ref HEAD` && \ + sudo -u gitlab_ci -H bundle install --without development test --deployment && \ + sudo -u gitlab_ci -H bundle exec rake db:migrate RAILS_ENV=production && \ + cd && \ + sudo service gitlab_ci start +``` diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md new file mode 100644 index 00000000000..04c6bf1e3a3 --- /dev/null +++ b/doc/ci/variables/README.md @@ -0,0 +1,95 @@ +## Variables +When receiving a build from GitLab CI, the runner prepares the build environment. +It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables** + +The variables can be overwritten. They take precedence over each other in this order: +1. Secure variables +1. YAML-defined variables +1. Predefined variables + +For example, if you define: +1. API_TOKEN=SECURE as Secure Variable +1. API_TOKEN=YAML as YAML-defined variable + +The API_TOKEN will take the Secure Variable value: `SECURE`. + +### Predefined variables (Environment Variables) + +| Variable | Description | +|-------------------------|-------------| +| **CI** | Mark that build is executed in CI environment | +| **GITLAB_CI** | Mark that build is executed in GitLab CI environment | +| **CI_SERVER** | Mark that build is executed in CI environment | +| **CI_SERVER_NAME** | CI server that is used to coordinate builds | +| **CI_SERVER_VERSION** | Not yet defined | +| **CI_SERVER_REVISION** | Not yet defined | +| **CI_BUILD_REF** | The commit revision for which project is built | +| **CI_BUILD_BEFORE_SHA** | The first commit that were included in push request | +| **CI_BUILD_REF_NAME** | The branch or tag name for which project is built | +| **CI_BUILD_ID** | The unique id of the current build that GitLab CI uses internally | +| **CI_BUILD_REPO** | The URL to clone the Git repository | +| **CI_PROJECT_ID** | The unique id of the current project that GitLab CI uses internally | +| **CI_PROJECT_DIR** | The full path where the repository is cloned and where the build is ran | + +Example values: + +```bash +export CI_BUILD_BEFORE_SHA="9df57456fa9de2a6d335ca5edf9750ed812b9df0" +export CI_BUILD_ID="50" +export CI_BUILD_REF="1ecfd275763eff1d6b4844ea3168962458c9f27a" +export CI_BUILD_REF_NAME="master" +export CI_BUILD_REPO="https://gitlab.com/gitlab-org/gitlab-ce.git" +export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" +export CI_PROJECT_ID="34" +export CI_SERVER="yes" +export CI_SERVER_NAME="GitLab CI" +export CI_SERVER_REVISION="" +export CI_SERVER_VERSION="" +``` + +### YAML-defined variables +**This feature requires GitLab Runner 0.5.0 or higher** + +GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment. +The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL. + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. + +The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. + +More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md). + +### User-defined variables (Secure Variables) +**This feature requires GitLab Runner 0.4.0 or higher** + +GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. +The secure variables are stored out of the repository (the `.gitlab-ci.yml`). +These variables are securely stored in GitLab CI database and are hidden in the build log. +It's desired method to use them for storing passwords, secret keys or whatever you want. + +Secure Variables can added by going to `Project > Variables > Add Variable`. + +They will be available for all subsequent builds. + +### Use variables +The variables are set as environment variables in build environment and are accessible with normal methods that are used to access such variables. +In most cases the **bash** is used to execute build script. +To access variables (predefined and user-defined) in bash environment, prefix the variable name with `$`: +``` +job_name: + script: + - echo $CI_BUILD_ID +``` + +You can also list all environment variables with `export` command, +but be aware that this will also expose value of all **Secure Variables** in build log: +``` +job_name: + script: + - export +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md new file mode 100644 index 00000000000..4caeccacb7f --- /dev/null +++ b/doc/ci/yaml/README.md @@ -0,0 +1,204 @@ +# Configuration of your builds with .gitlab-ci.yml +From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (**.gitlab-ci.yml**) for the project configuration. +It is placed in the root of your repository and contains definitions of how your project should be built. + +The YAML file defines a set of jobs with constraints stating when they should be run. +The jobs are defined as top-level elements with a name and always have to contain the `script` clause: + +```yaml +job1: + script: "execute-script-for-job1" + +job2: + script: "execute-script-for-job2" +``` + +The above example is the simplest possible CI configuration with two separate jobs, +where each of the jobs executes a different command. +Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. + +Jobs are used to create builds, which are then picked up by [runners](../runners/README.md) and executed within the environment of the runner. +What is important, is that each job is run independently from each other. + +## .gitlab-ci.yml +The YAML syntax allows for using more complex job specifications than in the above example: + +```yaml +image: ruby:2.1 +services: + - postgres + +before_script: + - bundle_install + +stages: + - build + - test + - deploy + +job1: + stage: build + script: + - execute-script-for-job1 + only: + - master + tags: + - docker +``` + +There are a few `keywords` that can't be used as job names: + +| keyword | required | description | +|---------------|----------|-------------| +| image | optional | Use docker image, covered in [Use Docker](../docker/README.md) | +| services | optional | Use docker services, covered in [Use Docker](../docker/README.md) | +| stages | optional | Define build stages | +| types | optional | Alias for `stages` | +| before_script | optional | Define commands prepended for each job's script | +| variables | optional | Define build variables | + +### image and services +This allows to specify a custom Docker image and a list of services that can be used for time of the build. +The configuration of this feature is covered in separate document: [Use Docker](../docker/README.md). + +### before_script +`before_script` is used to define the command that should be run before all builds, including deploy builds. This can be an array or a multiline string. + +### stages +`stages` is used to define build stages that can be used by jobs. +The specification of `stages` allows for having flexible multi stage pipelines. + +The ordering of elements in `stages` defines the ordering of builds' execution: + +1. Builds of the same stage are run in parallel. +1. Builds of next stage are run after success. + +Let's consider the following example, which defines 3 stages: +``` +stages: + - build + - test + - deploy +``` + +1. First all jobs of `build` are executed in parallel. +1. If all jobs of `build` succeeds, the `test` jobs are executed in parallel. +1. If all jobs of `test` succeeds, the `deploy` jobs are executed in parallel. +1. If all jobs of `deploy` succeeds, the commit is marked as `success`. +1. If any of the previous jobs fails, the commit is marked as `failed` and no jobs of further stage are executed. + +There are also two edge cases worth mentioning: + +1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, `test` and `deploy` are allowed to be used as job's stage by default. +2. If a job doesn't specify `stage`, the job is assigned the `test` stage. + +### types +Alias for [stages](#stages). + +### variables +**This feature requires `gitlab-runner` with version equal or greater than 0.5.0.** + +GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment. +The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL. + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. + +The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. + +## Jobs +`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. +Each job has to have a unique `job_name`, which is not one of the keywords mentioned above. +A job is defined by a list of parameters that define the build behaviour. + +```yaml +job_name: + script: + - rake spec + - coverage + stage: test + only: + - master + except: + - develop + tags: + - ruby + - postgres + allow_failure: true +``` + +| keyword | required | description | +|---------------|----------|-------------| +| script | required | Defines a shell script which is executed by runner | +| stage | optional (default: test) | Defines a build stage | +| type | optional | Alias for `stage` | +| only | optional | Defines a list of git refs for which build is created | +| except | optional | Defines a list of git refs for which build is not created | +| tags | optional | Defines a list of tags which are used to select runner | +| allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status | + +### script +`script` is a shell script which is executed by runner. The shell script is prepended with `before_script`. + +```yaml +job: + script: "bundle exec rspec" +``` + +This parameter can also contain several commands using an array: +```yaml +job: + script: + - uname -a + - bundle exec rspec +``` + +### stage +`stage` allows to group build into different stages. Builds of the same `stage` are executed in `parallel`. +For more info about the use of `stage` please check the [stages](#stages). + +### only and except +This are two parameters that allow for setting a refs policy to limit when jobs are built: +1. `only` defines the names of branches and tags for which job will be built. +2. `except` defines the names of branches and tags for which the job wil **not** be built. + +There are a few rules that apply to usage of refs policy: + +1. `only` and `except` are exclusive. If both `only` and `except` are defined in job specification only `only` is taken into account. +1. `only` and `except` allow for using the regexp expressions. +1. `only` and `except` allow for using special keywords: `branches` and `tags`. +These names can be used for example to exclude all tags and all branches. + +```yaml +job: + only: + - /^issue-.*$/ # use regexp + except: + - branches # use special keyword +``` + +### tags +`tags` is used to select specific runners from the list of all runners that are allowed to run this project. + +During registration of a runner, you can specify the runner's tags, ie.: `ruby`, `postgres`, `development`. +`tags` allow you to run builds with runners that have the specified tags assigned: + +``` +job: + tags: + - ruby + - postgres +``` + +The above specification will make sure that `job` is built by a runner that have `ruby` AND `postgres` tags defined. + +## Validate the .gitlab-ci.yml +Each instance of GitLab CI has an embedded debug tool called Lint. +You can find the link to the Lint in the project's settings page or use short url `/lint`. + +## Skipping builds +There is one more way to skip all builds, if your commit message contains tag [ci skip]. In this case, commit will be created but builds will be skipped
\ No newline at end of file |
