diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/README.md | 12 | ||||
| -rw-r--r-- | doc/ci/README.md | 12 | ||||
| -rw-r--r-- | doc/ci/docker/using_docker_images.md | 115 | ||||
| -rw-r--r-- | doc/ci/languages/README.md | 3 | ||||
| -rw-r--r-- | doc/ci/languages/php.md | 178 | ||||
| -rw-r--r-- | doc/ci/services/README.md | 6 | ||||
| -rw-r--r-- | doc/ci/services/docker-services.md | 5 | ||||
| -rw-r--r-- | doc/ci/services/mysql.md | 72 | ||||
| -rw-r--r-- | doc/ci/services/postgres.md | 70 | ||||
| -rw-r--r-- | doc/ci/services/redis.md | 40 | ||||
| -rw-r--r-- | doc/ci/ssh_keys/README.md | 114 |
11 files changed, 559 insertions, 68 deletions
diff --git a/doc/README.md b/doc/README.md index 58ab5dd08e0..a7025f7af10 100644 --- a/doc/README.md +++ b/doc/README.md @@ -24,9 +24,21 @@ - [Using Docker Images](ci/docker/using_docker_images.md) - [Using Docker Build](ci/docker/using_docker_build.md) - [Using Variables](ci/variables/README.md) +- [Using SSH keys](ci/ssh_keys/README.md) - [User permissions](ci/permissions/README.md) - [API](ci/api/README.md) +### CI Languages + ++ [Testing PHP](ci/languages/php.md) + +## CI Services + ++ [Using MySQL](ci/services/mysql.md) ++ [Using PostgreSQL](ci/services/postgres.md) ++ [Using Redis](ci/services/redis.md) ++ [Using Other Services](ci/docker/using_docker_images.html#how-to-use-other-images-as-services) + ### CI Examples - [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md) diff --git a/doc/ci/README.md b/doc/ci/README.md index 97325069ceb..ae921b6f988 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -9,6 +9,18 @@ + [Using Docker Images](docker/using_docker_images.md) + [Using Docker Build](docker/using_docker_build.md) + [Using Variables](variables/README.md) ++ [Using SSH keys](ssh_keys/README.md) + +### Languages + ++ [Testing PHP](languages/php.md) + +### Services + ++ [Using MySQL](services/mysql.md) ++ [Using PostgreSQL](services/postgres.md) ++ [Using Redis](services/redis.md) ++ [Using Other Services](docker/using_docker_images.html#how-to-use-other-images-as-services) ### Examples diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 1feae62b1c7..2f0ca19cd0f 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -42,7 +42,44 @@ So, **to access your database service you have to connect to host: `mysql` inste ### How to use other images as services? You are not limited to have only database services. -You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). +You can add the services to `.gitlab-ci.yml` or hand modify the `config.toml`. +You can use any image as service found at [Docker Hub](https://registry.hub.docker.com/). + +### Define image and services from `.gitlab-ci.yml` +You can simply define image or list services that you want to use for the build time. +``` +image: ruby:2.2 +services: + - postgres:9.3 +before_install: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +It's possible to define image and service per-job: +``` +before_install: + - bundle install + +test:2.1: + image: ruby:2.1 + services: + - postgres:9.3 + script: + - bundle exec rake spec + +test:2.2: + image: ruby:2.2 + services: + - postgres:9.4 + script: + - bundle exec rake spec +``` + +### Define image and services in `config.toml` Look for `[runners.docker]` section: ``` [runners.docker] @@ -50,13 +87,16 @@ Look for `[runners.docker]` section: services = ["mysql:latest", "postgres:latest"] ``` +The image and services defined these way will be added to all builds run by that runner. + +### Accessing the services For example you need `wordpress` instance to test some API integration with `Wordpress`. -You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). -This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: +You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). + ``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] +# .gitlab-ci.yml +services: +- tutum/wordpress:latest ``` Next time when you run your application the `tutum/wordpress` will be started @@ -64,7 +104,7 @@ and you will have access to it from your build container under hostname: `tutum_ Alias hostname for the service is made from the image name: 1. Everything after `:` is stripped, -2. '/' is replaced with `__`. +2. '/' is replaced to `__`. ### Configuring services Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. @@ -99,67 +139,6 @@ or README page for any other Docker image. **Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** -### Overwrite image and services -It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. -If you add to your YAML the `image` and the `services` these parameters -be used instead of the ones that were specified during runner's registration. -``` -image: ruby:2.2 -services: - - postgres:9.3 -before_install: - - bundle install - -test: - script: - - bundle exec rake spec -``` - -It's possible to define image and service per-job: -``` -before_install: - - bundle install - -test:2.1: - image: ruby:2.1 - services: - - postgres:9.3 - script: - - bundle exec rake spec - -test:2.2: - image: ruby:2.2 - services: - - postgres:9.4 - script: - - bundle exec rake spec -``` - -#### How to enable overwriting? -To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). -You can do that by hand modifying runner configuration: `config.toml`. -Please go to section where is `[runners.docker]` definition for your runner. -Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: -``` -[runners.docker] - image = "ruby:2.1" - allowed_images = ["ruby:*", "python:*"] - allowed_services = ["mysql:*", "redis:*"] -``` -This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. -You will be able to pick only `ruby` and `python` images. -The same rule can be applied to limit services. - -If you are courageous enough, you can make it fully open and accept everything: -``` -[runners.docker] - image = "ruby:2.1" - allowed_images = ["*", "*/*"] - allowed_services = ["*", "*/*"] -``` - -**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** - ### How Docker integration works 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. 1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). diff --git a/doc/ci/languages/README.md b/doc/ci/languages/README.md new file mode 100644 index 00000000000..375adf58d18 --- /dev/null +++ b/doc/ci/languages/README.md @@ -0,0 +1,3 @@ +### Languages + ++ [Testing PHP](php.md) diff --git a/doc/ci/languages/php.md b/doc/ci/languages/php.md new file mode 100644 index 00000000000..e0589182003 --- /dev/null +++ b/doc/ci/languages/php.md @@ -0,0 +1,178 @@ +## Testing PHP projects + +This guide covers basic of building PHP projects. + +Is it possible to test PHP apps on any system. +However, it will require manual configuration. +The simplest is to use Docker executor as described below. + +### PHP projects on Docker executor +It's possible to official [PHP](https://hub.docker.com/_/php/) repositories on Docker Hub. +They allow to test PHP projects against different versions of the runtime. +However, they require additional configuration. + +To build PHP project you need to create valid `.gitlab-ci.yml` describing the build environment: +1. First you need to specify PHP image as described here: http://doc.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-image. To your `.gitlab-ci.yml` add: + + image: php:5.6 + +2. The official images are great, but they are lacking a few useful tools for testing. We need to install them first in build environment. Create `ci/docker_install.sh` file with following content: + + #!/bin/bash + + # We need to install dependencies only for Docker + [[ ! -e /.dockerinit ]] && exit 0 + + set -xe + + # Install git (the php image doesn't have it) which is required by composer + apt-get update -yqq + apt-get install git -yqq + + # Install phpunit, the tool that we will use for testing + curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar + chmod +x /usr/local/bin/phpunit + + # Install mysql driver + # Here you can install any other extension that you need + docker-php-ext-install pdo_mysql + +3. From your `.gitlab-ci.yml` run the created script: + + before_script: + - bash ci/docker_install.sh > /dev/null + +4. Now you can run your tests. Usually it will be `phpunit` with arguments: + + test:app: + script: + - phpunit --configuration phpunit_myapp.xml --coverage-text + +5. Commit your files, and push them to GitLab to see if it works. With GitLab Runner 1.0 you can also test the changes locally. From your terminal execute: + + # Check using docker executor + gitlab-runner exec docker test:app + + # Check using shell executor + gitlab-runner exec shell test:app + +The final `.gitlab-ci.yml` should look similar to this: + + # Select image from https://hub.docker.com/_/php/ + image: php:5.6 + + before_script: + # Install dependencies + - ci/docker_install.sh > /dev/null + + test:app: + script: + - phpunit --configuration phpunit_myapp.xml --coverage-text + +#### Test against different PHP versions in Docker builds + +You can also test against multiple version of PHP runtime: + + before_script: + # Install dependencies + - ci/docker_install.sh > /dev/null + + # We test PHP5.6 + test:5.6: + image: php:5.6 + script: + - phpunit --configuration phpunit_myapp.xml --coverage-text + + # We test PHP7.0 + test:7.0: + image: php:7.0 + script: + - phpunit --configuration phpunit_myapp.xml --coverage-text + +#### Custom PHP configuration in Docker builds + +You can customise your PHP environment by putting your .ini file into `/usr/local/etc/php/conf.d/`: + + before_script: + - cp my_php.ini /usr/local/etc/php/conf.d/test.ini + +### Test PHP projects using Shell + +Shell executor runs your builds in terminal session of your server. Thus in order to test your projects you need to have all dependencies installed as root. + +1. Install PHP dependencies: + + sudo apt-get update -qy + sudo apt-get install phpunit php5-mysql -y + + This will install the PHP version available for your distribution. + +2. Now you can run your tests. Usually it will be `phpunit` with arguments: + + test:app: + script: + - phpunit --configuration phpunit_myapp.xml --coverage-text + +#### Test against different PHP versions in Shell builds + +The [phpenv](https://github.com/phpenv/phpenv) allows you to easily manage different PHP with they own configs. +This is specially usefull when testing PHP project with Shell executor. + +Login as `gitlab-runner` user and follow [the installation guide](https://github.com/phpenv/phpenv#installation). + +Using phpenv also allows to easily configure PHP environment with: `phpenv config-add my_config.ini`. + +#### Install custom extensions + +Since we have pretty bare installation of our PHP environment you may need some extensions that are not present on your installation. + +To install additional extensions simply execute.: + + pecl install <extension> + + It's not advised to add this to the `.gitlab-ci.yml`. + You should execute this command once, only to setup the build environment. + +### Extend your tests + +#### Using atoum + +Instead of PHPUnit, you can use any other tool to run unit tests. For example [atoum](https://github.com/atoum/atoum): + + before_script: + - wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar + + test:atoum: + script: + - php mageekguy.atoum.phar + +#### Using Composer + +Majority of the PHP projects use Composer for managing the packages. +It's very simple to execute the Composer before running your tests. +To your `.gitlab-ci.yml` add: + + # The composer stores all downloaded packages in vendor/ + # Remove it if you committed the vendor/ directory + cache: + paths: + - vendor/ + + before_script: + # Install composer dependencies + - curl -sS https://getcomposer.org/installer | php + - php composer.phar install + +### Access private packages / dependencies + +You need to configure [the SSH keys](../ssh_keys/README.md) in order to checkout the repositories. + +### Use databases or other services + +Please checkout the docs about configuring [the CI services](../services/README.md). + +### Example project + +You maybe interested in our [Example Project](https://gitlab.com/gitlab-examples/php) that runs on [GitLab.com](https://gitlab.com) using our publically available shared runners. + +Want to hack it? Simply fork it, commit and push changes. Within a few moments the changes will be picked and rebuilt by public runners. diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md new file mode 100644 index 00000000000..0550e9435a3 --- /dev/null +++ b/doc/ci/services/README.md @@ -0,0 +1,6 @@ +## GitLab CI Services + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) ++ [Using Other Services](../docker/using_docker_images.html#how-to-use-other-images-as-services) diff --git a/doc/ci/services/docker-services.md b/doc/ci/services/docker-services.md new file mode 100644 index 00000000000..df36ebaf7d4 --- /dev/null +++ b/doc/ci/services/docker-services.md @@ -0,0 +1,5 @@ +## GitLab CI Services + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md new file mode 100644 index 00000000000..3155af6b3e1 --- /dev/null +++ b/doc/ci/services/mysql.md @@ -0,0 +1,72 @@ +## Using MySQL + +It's possible to use MySQL database test your apps during builds. + +### Use MySQL with Docker executor + +If you are using our Docker integration you basically have everything already. + +1. Add this to your `.gitlab-ci.yml`: + + services: + - mysql + + variables: + # Configure mysql service (https://hub.docker.com/_/mysql/) + MYSQL_DATABASE: hello_world_test + MYSQL_ROOT_PASSWORD: mysql + +2. Configure your application to use the database: + + Host: mysql + User: root + Password: mysql + Database: hello_world_test + +3. You can also use any other available on [DockerHub](https://hub.docker.com/_/mysql/). For example: `mysql:5.5`. + +Example: https://gitlab.com/gitlab-examples/mysql/blob/master/.gitlab-ci.yml + +### Use MySQL with Shell executor + +It's possible to use MySQL on manually configured servers that are using GitLab Runner with Shell executor. + +1. First install the MySQL server: + + sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev + + # Pick a MySQL root password (can be anything), type it and press enter + # Retype the MySQL root password and press enter + +2. Create an user: + + mysql -u root -p + + # Create a user which will be used by your apps + # do not type the 'mysql>', this is part of the prompt + # change $password in the command below to a real password you pick + mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; + + # Ensure you can use the InnoDB engine which is necessary to support long indexes + # If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off" + mysql> SET storage_engine=INNODB; + + # Create the database + mysql> CREATE DATABASE IF NOT EXISTS `hello_world_test` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; + + # Grant necessary permissions on the database + mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `hello_world_test`.* TO 'runner'@'localhost'; + + # Quit the database session + mysql> \q + +3. Try to connect to database: + + sudo -u gitlab-runner -H mysql -u runner -p -D hello_world_test + +4. Configure your application to use the database: + + Host: localhost + User: runner + Password: $password + Database: hello_world_test diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md new file mode 100644 index 00000000000..e57f8c5944a --- /dev/null +++ b/doc/ci/services/postgres.md @@ -0,0 +1,70 @@ +## Using PostgreSQL + +It's possible to use PostgreSQL database test your apps during builds. + +### Use PostgreSQL with Docker executor + +If you are using our Docker integration you basically have everything already. + +1. Add this to your `.gitlab-ci.yml`: + + services: + - postgres + + variables: + # Configure postgres service (https://hub.docker.com/_/postgres/) + POSTGRES_DB: hello_world_test + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "" + +2. Configure your application to use the database: + + Host: postgres + User: postgres + Password: postgres + Database: hello_world_test + +3. You can also use any other available on [DockerHub](https://hub.docker.com/_/postgres/). For example: `postgres:9.3`. + +Example: https://gitlab.com/gitlab-examples/postgres/blob/master/.gitlab-ci.yml + +### Use PostgreSQL with Shell executor + +It's possible to use PostgreSQL on manually configured servers that are using GitLab Runner with Shell executor. + +1. First install the PostgreSQL server: + + sudo apt-get install -y postgresql postgresql-client libpq-dev + +2. Create an user: + + # Install the database packages + sudo apt-get install -y postgresql postgresql-client libpq-dev + + # Login to PostgreSQL + sudo -u postgres psql -d template1 + + # Create a user for runner + # Do not type the 'template1=#', this is part of the prompt + template1=# CREATE USER runner CREATEDB; + + # Create the database & grant all privileges on database + template1=# CREATE DATABASE hello_world_test OWNER runner; + + # Quit the database session + template1=# \q + +3. Try to connect to database: + + # Try connecting to the new database with the new user + sudo -u gitlab-runner -H psql -d hello_world_test + + # Quit the database session + hello_world_test> \q + +4. Configure your application to use the database: + + Host: localhost + User: runner + Password: + Database: hello_world_test diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md new file mode 100644 index 00000000000..523634a457e --- /dev/null +++ b/doc/ci/services/redis.md @@ -0,0 +1,40 @@ +## Using Redis + +It's possible to use Redis database test your apps during builds. + +### Use Redis with Docker executor + +If you are using our Docker integration you basically have everything already. + +1. Add this to your `.gitlab-ci.yml`: + + services: + - redis + +2. Configure your application to use the database: + + Host: redis + +3. You can also use any other available on [DockerHub](https://hub.docker.com/_/redis/). For example: `redis:2.6`. + +Example: https://gitlab.com/gitlab-examples/redis/blob/master/.gitlab-ci.yml + +### Use Redis with Shell executor + +It's possible to use Redis on manually configured servers that are using GitLab Runner with Shell executor. + +1. First install the Redis server: + + sudo apt-get install redis-server + +2. Try to connect to the server: + + # Try connecting the the Redis server + sudo -u gitlab-runner -H redis-cli + + # Quit the session + 127.0.0.1:6379> quit + +4. Configure your application to use the database: + + Host: localhost diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md new file mode 100644 index 00000000000..515194e5f5e --- /dev/null +++ b/doc/ci/ssh_keys/README.md @@ -0,0 +1,114 @@ +# Using SSH keys + +GitLab currently doesn't have built-in support for SSH keys in build environment. + +The SSH keys can be useful when: +1. You want to checkout internal submodules, +2. You want to download private packages using your package manager (ie. bundler), +3. You want to deploy your app (ex. to Heroku or own server), +4. You want to execute ssh commands from build environment on remote server, +5. You want to rsync files from your build to remote server. + +If anyone of the above holds true, then you most likely need SSH key. + +There are two possibilities to add SSH keys to build environment. + +## Inject keys in your build environment +The most widely supported is to inject SSH key into your build environment by extending your .gitlab-ci.yml. +This is the universal solution which works with any type of executor (docker, shell, etc.). + +### How it works? +1. We create a new SSH private key with [ssh-keygen](http://linux.die.net/man/1/ssh-keygen). +2. We add the private key as the Secure Variable to project. +3. We run the [ssh-agent](http://linux.die.net/man/1/ssh-agent) during build to load the private key. + +The example [.gitlab-ci.yml](https://gitlab.com/gitlab-examples/ssh-private-key/blob/master/.gitlab-ci.yml) looks like this. + +### Make it work? +1. First, go to terminal and generate a new SSH key: +```bash +$ ssh-keygen -t rsa -f my_key + +Generating public/private rsa key pair. +Enter passphrase (empty for no passphrase): +Enter same passphrase again: +Your identification has been saved in my_key. +Your public key has been saved in my_key.pub. +The key fingerprint is: +SHA256:tBJEfyJUGTMNmPCiPg4UHywHs67MxlM2iEBAlI/W+TY fingeprint +The key's randomart image is: ++---[RSA 2048]----+ +|=*. .o++*= | +|..= +o..o. | +|.+++o + + . | +|+o*=.. + + | +|o+.=. . S | +|*.o .E . | +|o*o . . | +|.o.. | +| . | ++----[SHA256]-----+ +``` + +2. Create a new **Secure Variable** in your project settings on GitLab and name it: `SSH_PRIVATE_KEY`. + +3. Copy the content of `my_key` and paste it as a **Value** of **SSH_PRIVATE_KEY**. + +4. Next you need to modify your `.gitlab-ci.yml` and at the top of the file add: +``` +before_script: +# install ssh-agent (it is required for Docker, change apt-get to yum if you use CentOS-based image) +- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + +# run ssh-agent (in build environment) +- eval $(ssh-agent -s) + +# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store +- ssh-add <(echo "$SSH_PRIVATE_KEY") + +# for Docker builds disable host key checking, by adding that you are suspectible to man-in-the-middle attack +- mkdir -p ~/.ssh +- '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config` +``` + +5. Add the public key from `my_key.pub` to services that you want to have an access from build. + +6. If your builds are run using `shell` executor, you may need to login to server and execute the `ssh <address-of-my-server>` to store the fingerprint of remote server. + +## SSH keys when using Shell executor +If use `shell`, not `docker` it can be easier to have the SSH key. + +We can generate the SSH key for the machine that holds `gitlab-runner` and use that key for all projects that are run on this machine. + +1. First, login to server that runs your builds. + +2. From terminal login as `gitlab-runner` user and generate the SSH private key: +```bash +$ ssh-keygen -t rsa +Generating public/private rsa key pair. +Enter passphrase (empty for no passphrase): +Enter same passphrase again: +Your identification has been saved in ~/.ssh/id_rsa. +Your public key has been saved in ~/.ssh/id_rsa.pub. +The key fingerprint is: +SHA256:tBJEfyJUGTMNmPCiPg4UHywHs67MxlM2iEBAlI/W+TY fingeprint +The key's randomart image is: ++---[RSA 2048]----+ +|=*. .o++*= | +|..= +o..o. | +|.+++o + + . | +|+o*=.. + + | +|o+.=. . S | +|*.o .E . | +|o*o . . | +|.o.. | +| . | ++----[SHA256]-----+ +``` + +3. Add the public key from `~/.ssh/id_rsa.pub` to services that you want to have an access from build. + +4. Try to login for the first time and accept fingerprint: +```bash +ssh <address-of-my-server +``` |