diff options
Diffstat (limited to 'doc/ci/docker/using_docker_images.md')
| -rw-r--r-- | doc/ci/docker/using_docker_images.md | 358 |
1 files changed, 302 insertions, 56 deletions
diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 96834e15bb9..d3433594eb7 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,4 +1,4 @@ -# Using Docker Images +# Using Docker images GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use [Docker Engine](https://www.docker.com/) to test and build any application. @@ -17,14 +17,16 @@ can also run on your workstation. The added benefit is that you can test all the commands that we will explore later from your shell, rather than having to test them on a dedicated CI server. -## Register docker runner +## Register Docker Runner -To use GitLab Runner with docker you need to register a new runner to use the -`docker` executor: +To use GitLab Runner with Docker you need to [register a new Runner][register] +to use the `docker` executor. + +A one-line example can be seen below: ```bash -gitlab-ci-multi-runner register \ - --url "https://gitlab.com/" \ +sudo gitlab-runner register \ + --url "https://gitlab.example.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby-2.1" \ --executor "docker" \ @@ -33,26 +35,26 @@ gitlab-ci-multi-runner register \ --docker-mysql latest ``` -The registered runner will use the `ruby:2.1` docker image and will run two +The registered runner will use the `ruby:2.1` Docker image and will run two services, `postgres:latest` and `mysql:latest`, both of which will be accessible during the build process. ## What is an image -The `image` keyword is the name of the docker image the docker executor -will run to perform the CI tasks. +The `image` keyword is the name of the Docker image the Docker executor +will run to perform the CI tasks. -By default the executor will only pull images from [Docker Hub][hub], +By default, the executor will only pull images from [Docker Hub][hub], but this can be configured in the `gitlab-runner/config.toml` by setting -the [docker pull policy][] to allow using local images. +the [Docker pull policy][] to allow using local images. For more information about images and Docker Hub please read the [Docker Fundamentals][] documentation. ## What is a service -The `services` keyword defines just another docker image that is run during -your job and is linked to the docker image that the `image` keyword defines. +The `services` keyword defines just another Docker image that is run during +your job and is linked to the Docker image that the `image` keyword defines. This allows you to access the service image during build time. The service image can run any application, but the most common use case is to @@ -60,6 +62,11 @@ run a database container, eg. `mysql`. It's easier and faster to use an existing image and run it as an additional container than install `mysql` every time the project is built. +You are not limited to have only database services. You can add as many +services you need to `.gitlab-ci.yml` or manually modify `config.toml`. +Any image found at [Docker Hub][hub] or your private Container Registry can be +used as a service. + You can see some widely used services examples in the relevant documentation of [CI services examples](../services/README.md). @@ -73,22 +80,49 @@ then be used to create a container that is linked to the job container. The service container for MySQL will be accessible under the hostname `mysql`. So, in order to access your database service you have to connect to the host -named `mysql` instead of a socket or `localhost`. +named `mysql` instead of a socket or `localhost`. Read more in [accessing the +services](#accessing-the-services). -## Overwrite image and services +### Accessing the services -See [How to use other images as services](#how-to-use-other-images-as-services). +Let's say that you need a Wordpress instance to test some API integration with +your application. -## How to use other images as services +You can then use for example the [tutum/wordpress][] image in your +`.gitlab-ci.yml`: -You are not limited to have only database services. You can add as many -services you need to `.gitlab-ci.yml` or manually modify `config.toml`. -Any image found at [Docker Hub][hub] can be used as a service. +```yaml +services: +- tutum/wordpress:latest +``` -## Define image and services from `.gitlab-ci.yml` +If you don't [specify a service alias](#available-settings-for-services-entry), +when the job is run, `tutum/wordpress` will be started and you will have +access to it from your build container under two hostnames to choose from: + +- `tutum-wordpress` +- `tutum__wordpress` + +>**Note:** +Hostnames with underscores are not RFC valid and may cause problems in 3rd party +applications. + +The default aliases for the service's hostname are created from its image name +following these rules: + +- Everything after the colon (`:`) is stripped +- Slash (`/`) is replaced with double underscores (`__`) and the primary alias + is created +- Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is + created (requires GitLab Runner v1.1.0 or higher) + +To override the default behavior, you can +[specify a service alias](#available-settings-for-services-entry). + +## Define `image` and `services` from `.gitlab-ci.yml` You can simply define an image that will be used for all jobs and a list of -services that you want to use during build time. +services that you want to use during build time: ```yaml image: ruby:2.2 @@ -125,6 +159,203 @@ test:2.2: - bundle exec rake spec ``` +Or you can pass some [extended configuration options](#extended-docker-configuration-options) +for `image` and `services`: + +```yaml +image: + name: ruby:2.2 + entrypoint: ["/bin/bash"] + +services: +- name: my-postgres:9.4 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + +before_script: +- bundle install + +test: + script: + - bundle exec rake spec +``` + +## Extended Docker configuration options + +> **Note:** +This feature requires GitLab 9.4 and GitLab Runner 9.4 or higher. + +When configuring the `image` or `services` entries, you can use a string or a map as +options: + +- when using a string as an option, it must be the full name of the image to use + (including the Registry part if you want to download the image from a Registry + other than Docker Hub) +- when using a map as an option, then it must contain at least the `name` + option, which is the same name of the image as used for the string setting + +For example, the following two definitions are equal: + +1. Using a string as an option to `image` and `services`: + + ```yaml + image: "registry.example.com/my/image:latest" + + services: + - postgresql:9.4 + - redis:latest + ``` + +1. Using a map as an option to `image` and `services`. The use of `image:name` is + required: + + ```yaml + image: + name: "registry.example.com/my/image:latest" + + services: + - name: postgresql:9.4 + - name: redis:latest + ``` + +### Available settings for `image` + +> **Note:** +This feature requires GitLab 9.4 and GitLab Runner 9.4 or higher. + +| Setting | Required | Description | +|------------|----------|-------------| +| `name` | yes, when used with any other option | Full name of the image that should be used. It should contain the Registry part if needed. | +| `entrypoint` | no | Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`][entrypoint] directive, where each shell token is a separate string in the array. | + +### Available settings for `services` + +> **Note:** +This feature requires GitLab 9.4 and GitLab Runner 9.4 or higher. + +| Setting | Required | Description | +|------------|----------|-------------| +| `name` | yes, when used with any other option | Full name of the image that should be used. It should contain the Registry part if needed. | +| `entrypoint` | no | Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`][entrypoint] directive, where each shell token is a separate string in the array. | +| `command` | no | Command or script that should be used as the container's command. It will be translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`][cmd] directive, where each shell token is a separate string in the array. | +| `alias` | no | Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | + +### Starting multiple services from the same image + +Before the new extended Docker configuration options, the following configuration +would not work properly: + +```yaml +services: +- mysql:latest +- mysql:latest +``` + +The Runner would start two containers using the `mysql:latest` image, but both +of them would be added to the job's container with the `mysql` alias based on +the [default hostname naming](#accessing-the-services). This would end with one +of the services not being accessible. + +After the new extended Docker configuration options, the above example would +look like: + +```yaml +services: +- name: mysql:latest + alias: mysql-1 +- name: mysql:latest + alias: mysql-2 +``` + +The Runner will still start two containers using the `mysql:latest` image, +but now each of them will also be accessible with the alias configured +in `.gitlab-ci.yml` file. + +### Setting a command for the service + +Let's assume you have a `super/sql:latest` image with some SQL database +inside it and you would like to use it as a service for your job. Let's also +assume that this image doesn't start the database process while starting +the container and the user needs to manually use `/usr/bin/super-sql run` as +a command to start the database. + +Before the new extended Docker configuration options, you would need to create +your own image based on the `super/sql:latest` image, add the default command, +and then use it in job's configuration, like: + +```Dockerfile +# my-super-sql:latest image's Dockerfile + +FROM super/sql:latest +CMD ["/usr/bin/super-sql", "run"] +``` + +```yaml +# .gitlab-ci.yml + +services: +- my-super-sql:latest +``` + +After the new extended Docker configuration options, you can now simply +set a `command` in `.gitlab-ci.yml`, like: + +```yaml +# .gitlab-ci.yml + +services: +- name: super/sql:latest + command: ["/usr/bin/super-sql", "run"] +``` + +As you can see, the syntax of `command` is similar to [Dockerfile's `CMD`][cmd]. + +### Overriding the entrypoint of an image + +Let's assume you have a `super/sql:experimental` image with some SQL database +inside it and you would like to use it as a base image for your job because you +want to execute some tests with this database binary. Let's also assume that +this image is configured with `/usr/bin/super-sql run` as an entrypoint. That +means, that when starting the container without additional options, it will run +the database's process, while Runner expects that the image will have no +entrypoint or at least will start with a shell as its entrypoint. + +Previously we would need to create our own image based on the +`super/sql:experimental` image, set the entrypoint to a shell, and then use +it in job's configuration, e.g.: + +Before the new extended Docker configuration options, you would need to create +your own image based on the `super/sql:experimental` image, set the entrypoint +to a shell and then use it in job's configuration, like: + +```Dockerfile +# my-super-sql:experimental image's Dockerfile + +FROM super/sql:experimental +ENTRYPOINT ["/bin/sh"] +``` + +```yaml +# .gitlab-ci.yml + +image: my-super-sql:experimental +``` + +After the new extended Docker configuration options, you can now simply +set an `entrypoint` in `.gitlab-ci.yml`, like: + +```yaml +# .gitlab-ci.yml + +image: + name: super/sql:experimental + entrypoint: ["/bin/sh"] +``` + +As you can see the syntax of `entrypoint` is similar to +[Dockerfile's `ENTRYPOINT`][entrypoint]. + ## Define image and services in `config.toml` Look for the `[runners.docker]` section: @@ -138,49 +369,60 @@ Look for the `[runners.docker]` section: The image and services defined this way will be added to all job run by that runner. -## Define an image from a private Docker registry +## Define an image from a private Container Registry -Starting with GitLab Runner 0.6.0, you are able to define images located to -private registries that could also require authentication. +> **Notes:** +- This feature requires GitLab Runner **1.8** or higher +- For GitLab Runner versions **>= 0.6, <1.8** there was a partial + support for using private registries, which required manual configuration + of credentials on runner's host. We recommend to upgrade your Runner to + at least version **1.8** if you want to use private registries. +- If the repository is private you need to authenticate your GitLab Runner in the + registry. Learn more about how [GitLab Runner works in this case][runner-priv-reg]. -All you have to do is be explicit on the image definition in `.gitlab-ci.yml`. +As an example, let's assume that you want to use the `registry.example.com/private/image:latest` +image which is private and requires you to login into a private container registry. +To configure access for `registry.example.com`, follow these steps: -```yaml -image: my.registry.tld:5000/namespace/image:tag -``` +1. Do a `docker login` on your computer: -In the example above, GitLab Runner will look at `my.registry.tld:5000` for the -image `namespace/image:tag`. + ```bash + docker login registry.example.com --username my_username --password my_password + ``` -If the repository is private you need to authenticate your GitLab Runner in the -registry. Learn how to do that on -[GitLab Runner's documentation][runner-priv-reg]. +1. Copy the content of `~/.docker/config.json` +1. Create a [secret variable] `DOCKER_AUTH_CONFIG` with the content of the + Docker configuration file as the value: -## Accessing the services + ```json + { + "auths": { + "registry.example.com": { + "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=" + } + } + } + ``` -Let's say that you need a Wordpress instance to test some API integration with -your application. +1. Do a `docker logout` on your computer if you don't need access to the + registry from it: -You can then use for example the [tutum/wordpress][] image in your -`.gitlab-ci.yml`: - -```yaml -services: -- tutum/wordpress:latest -``` + ```bash + docker logout registry.example.com + ``` -When the job is run, `tutum/wordpress` will be started and you will have -access to it from your build container under the hostnames `tutum-wordpress` -(requires GitLab Runner v1.1.0 or newer) and `tutum__wordpress`. +1. You can now use any private image from `registry.example.com` defined in + `image` and/or `services` in your [`.gitlab-ci.yml` file][yaml-priv-reg]: -*Note: hostname with underscores is not RFC valid and may cause problems in 3rd party applications.* + ```yaml + image: my.registry.tld:5000/namespace/image:tag + ``` -The alias hostnames for the service are made from the image name following these -rules: + In the example above, GitLab Runner will look at `my.registry.tld:5000` for the + image `namespace/image:tag`. -1. Everything after `:` is stripped -2. Slash (`/`) is replaced with double underscores (`__`) - primary alias -3. Slash (`/`) is replaced with dash (`-`) - secondary alias, requires GitLab Runner v1.1.0 or newer +You can add configuration for as many registries as you want, adding more +registries to the `"auths"` hash as described above. ## Configuring services @@ -208,7 +450,7 @@ See the specific documentation for ## How Docker integration works -Below is a high level overview of the steps performed by docker during job +Below is a high level overview of the steps performed by Docker during job time. 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. @@ -225,7 +467,7 @@ time. ## How to debug a job locally *Note: The following commands are run without root privileges. You should be -able to run docker with your regular user account.* +able to run Docker with your regular user account.* First start with creating a file named `build_script`: @@ -283,4 +525,8 @@ creation. [tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/ [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ -[runner-priv-reg]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry +[runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry +[secret variable]: ../variables/README.md#secret-variables +[entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint +[cmd]: https://docs.docker.com/engine/reference/builder/#cmd +[register]: https://docs.gitlab.com/runner/register/ |