Rework CI documentation by adding informations about YAML, the use of Docker images and building them

21 files changed, 527 insertions, 630 deletions

diff --git a/doc/README.md b/doc/README.md
index 19ebad9..f6e411c 100644
--- a/doc/README.md
+++ b/doc/README.md
@@ -1,12 +1,25 @@
-## The GitLab Documentation covers the following subjects
+## GitLab CI Documentation
 
-+ [API](api/README.md)
-+ [Examples](examples/README.md)
-+ [Install](install/installation.md)
+### User documentation
+
++ [Quick Start](quick_start/README.md)
++ [Configuring project (.gitlab-ci.yml)](yaml/README.md)
++ [Configuring runner](runners/README.md)
++ [Configuring deployment](deployment/README.md)
++ [Using Docker Images](docker/using_docker_images.md)
++ [Using Docker Build](docker/using_docker_build.md)
++ [Using Variables](variables/README.md)
+
+### Examples
+
++ [Test and deploy Ruby applications to Heroku](examples/test-and-deploy-ruby-application-to-heroku.md)
++ [Test and deploy Python applications to Heroku](examples/test-and-deploy-python-application-to-heroku.md)
+
+### Administrator documentation
+
++ [Install](install/README.md)
 + [Update](update/README.md)
-+ [Runners](runners/README.md)
-+ [Configuration of your builds with .gitlab-ci.yml](builds_configuration/README.md)
-+ [Deployment](deployment/README.md)
-+ [Permissions](permissions/README.md) User permissions
-+ [Rake Tasks](raketasks/README.md) Backup and restore take tasks
++ [User permissions](permissions/README.md)
++ [Backup/Restore](raketasks/backup_restore.md)
 + [Migrating to packaged CI](migration_to_omnibus/README.md)
++ [API](api/README.md)
diff --git a/doc/builds_configuration/README.md b/doc/builds_configuration/README.md
deleted file mode 100644
index 0afac2a..0000000
--- a/doc/builds_configuration/README.md
+++ /dev/null
@@ -1,113 +0,0 @@
-## Configuration of your builds with .gitlab-ci.yml
-
-From version 7.12, GitLab CI uses a .gitlab-ci.yml file for the configuration of your builds. It is placed in the root of your repository and contains three type of objects: before_script, builds and deploy_builds. Here is an example of how it looks:
-
-```yaml
-before_script: 
-  - gem install bundler
-  - bundle install 
-  - bundle exec rake db:create
-
-rspec: 
-  script: "rake spec"
-  tags: 
-    - ruby
-    - postgres
-  only: 
-    - branches
-
-staging: 
-  script: "cap deploy stating"
-  type: deploy
-  tags: 
-    - capistrano
-    - debian
-  except:
-    - stable
-    - /^deploy-.*$/
-
-```
-
-Let's have a close look at each section.
-
-### builds
-Here you can specify parameters of your builds:
-
-```yaml
-rspec: 
-  script: "rake spec"  # (required) - shell command for runner
-  tags:                # (optional) - runner tags, only runners which have these tags will be used
-    - ruby
-    - postgres
-  only:                # (optional) - git refs (branches and tags)
-    - master
-
-```
-
-`rspec` is a key of this object and it determines the name of your build
-
-`script` is a shell script which is used by runner. It will also be prepanded with `before_script`. This parameter can also contain several commands using array:
-
-```yaml
-script:
-  - uname -a
-  - bundle exec rspec
-```
-
-You can read about `only` and `except` parameters in the [refs settings explanation](#refs-settings-explanation)
-
-### deploy_builds
-Deploy Builds will be ran when all other builds have succeeded. Define them using simple syntax:
-
-```yaml
-production: 
-  script: "cap deploy production" # (required) - shell command for runner
-  type: deploy
-  tags: 
-    - ruby
-    - postgres
-  only:
-    - master
-```
-`production` - is a name of deploy build
-
-`script` - is a shell script which will be prepended with `before_script`
-
-`type: deploy` is a parameter which indicates that it is a deploy job
-
-You can read about `only` and `except` parameters in the [refs settings explanation](#refs-settings-explanation)
-
-### before_script
-`before_script` is used to define the command that should be ran before all builds, including deploy builds. This can be an array or a multiline string
-
-### Refs settings explanation
-There are two parameters that will help you set up the refs policy for your build or deploy build on CI
-```
-only:
-  - master
-```
-`only` defines the exact name of the branch or the tag which will be ran. It also supports the regexp expressions:
-
-```
-only:
-  - /^issue-.*$/
-```
-You can also use an `except` parameter:
-```
-except:
-  - "deploy"
-```
-This parameter is used to exclude some refs. It is also supporting regexp expressions
-
-There are also special keys like `branches` or `tags`. These parameters can be used to exclude all tags or branches
-```
-except:
-  - branches
-```
-
-## Debugging your builds with .gitlab-ci.yml
-
-Each instance of GitLab CI has an embeded debug tool Lint. You can find the link to the Lint in the project's settings page or use short url `/lint`
-
-## Skipping builds
-There is one more way to skip all builds, if your commit message contains tag [ci skip]. In this case, commit will be created but builds will be skipped
diff --git a/doc/docker/README.md b/doc/docker/README.md
new file mode 100644
index 0000000..84eaf29
--- /dev/null
+++ b/doc/docker/README.md
@@ -0,0 +1,4 @@
+# Docker integration
+
++ [Using Docker Images](using_docker_images.md)
++ [Using Docker Build](using_docker_build.md)
\ No newline at end of file
diff --git a/doc/docker/using_docker_build.md b/doc/docker/using_docker_build.md
new file mode 100644
index 0000000..fc63b68
--- /dev/null
+++ b/doc/docker/using_docker_build.md
@@ -0,0 +1,108 @@
+# Using Docker Build
+GitLab CI can allows you to use Docker Engine to build and test docker-based projects.
+This is one of new trends in Continuous Integration/Deployment to:
+
+1. create application image,
+1. run test against created image,
+1. push image to remote registry, 
+1. deploy server from pushed image
+
+It's also useful in case when your application already has the `Dockerfile` that can be used to create and test image:
+```bash
+$ docker build -t my-image dockerfiles/
+$ docker run my-docker-image /script/to/run/tests
+$ docker tag my-image my-registry:5000/my-image
+$ docker push my-registry:5000/my-image
+```
+
+However, this requires special configuration of GitLab Runner to enable `docker` support during build.
+**This requires running GitLab Runner in privileged mode which can be harmful when untrusted code is run.**
+
+There are two methods to enable the use of `docker build` and `docker run` during build or even `docker-compose` if needed.
+
+## 1. Use `shell` executor
+
+The simplest approach is to install GitLab Runner in `shell` execution mode.
+GitLab Runner then executes build scripts as `gitlab-runner` user.
+
+1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation).
+
+1. During GitLab Runner installation select `shell` as method of executing build scripts or use command:
+
+```bash
+$ sudo gitlab-runner register -n \
+  --url http://ci.gitlab.com \
+  --token RUNNER_TOKEN \
+  --executor shell
+  --description "My Runner"
+```
+
+2. Install Docker on server.
+For more information how to install Docker on different systems checkout the [Supported installations](https://docs.docker.com/installation/).
+
+3. Add `gitlab-runner` user to `docker` group:
+
+```bash
+$ sudo usermod -aG docker gitlab-runner
+```
+
+4. Verify that `gitlab-runner` has access to Docker:
+
+```bash
+$ sudo -u gitlab-runner -H docker info
+```
+
+You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`:
+
+```yaml
+before_script:
+  - docker info
+
+build_image:
+  script:
+    - docker build -t my-docker-image .
+    - docker run my-docker-image /script/to/run/tests
+```
+
+5. You can now use `docker` command and install `docker-compose` if needed.
+
+6. However, by adding `gitlab-runner` to `docker` group you are effectively granting `gitlab-runner` full root permissions.
+For more information please checkout [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful).
+
+## 2. Use `docker`-in-docker executor
+
+Second approach is to use special Docker image with all tools installed (`docker` and `docker-compose`) and run build script in context of that image in privileged mode.
+In order to do that follow the steps:
+
+1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation).
+
+1. Register GitLab Runner from command line to use `docker` and `privileged` mode:
+
+```bash
+$ sudo gitlab-runner register -n \
+  --url http://ci.gitlab.com \
+  --token RUNNER_TOKEN \
+  --executor docker \
+  --description "My Docker Runner" \
+  --docker-image "gitlab/dind:latest" \
+  --docker-privileged
+```
+
+The above command will register new Runner to use special [gitlab/dind](https://registry.hub.docker.com/u/gitlab/dind/) image which is provided by GitLab Inc.
+The image at the start runs Docker daemon in [docker-in-docker](https://blog.docker.com/2013/09/docker-can-now-run-within-docker/) mode.
+
+1. You can now use `docker` from build script:
+
+```yaml
+before_script:
+  - docker info
+
+build_image:
+  script:
+    - docker build -t my-docker-image .
+    - docker run my-docker-image /script/to/run/tests
+```
+
+1. However, by enabling `--docker-privileged` you are effectively disables all security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout.
+For more information you could be interested in checking out [Runtime privilege](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration).
+
diff --git a/doc/builds_configuration/docker.md b/doc/docker/using_docker_images.md
index 1ee7d10..c88dd95 100644
--- a/doc/builds_configuration/docker.md
+++ b/doc/docker/using_docker_images.md
@@ -1,4 +1,4 @@
-# Docker integration
+# Using Docker Images
 GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. 
 
 Docker is an open-source project that allows to use predefined images to run applications 
@@ -127,7 +127,7 @@ If you are courageous enough, you can make it fully open and accept everything:
 
 **It the feature is not enabled, or image isn't allowed the error message will be put into the build log.**
 
-### How it works?
+### How Docker integration works
 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`.
 1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example).
 1. Create build container and link any service container to build container.
@@ -138,25 +138,33 @@ If you are courageous enough, you can make it fully open and accept everything:
 1. Check exit status of build script.
 1. Remove build container and all created service containers.
 
-### How to debug build locally?
-1. Create build environment locally first using following commands:
+### How to debug a build locally
+1. Create a file with build script:
+```bash
+$ cat <<EOF > build_script
+git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner
+cd /builds/gitlab-org/gitlab-ci-multi-runner
+make <- or any other build step
+EOF
+```
+
+1. Create service containers:
 ```
-$ docker run -d -n build-mysql mysql:latest
-$ docker run -d -n build-postgres postgres:latest
-$ docker run -n build -it -l mysql:build-mysql -l postgres:build-postgres ruby:2.1 /bin/bash
+$ docker run -d -n service-mysql mysql:latest
+$ docker run -d -n service-postgres postgres:latest
 ```
-This will create two service containers (MySQL and PostgreSQL) that are linked to build container created as a last one.
+This will create two service containers (MySQL and PostgreSQL).
 
-1. Then in context of docker container you can copy-paste your build script:
+1. Create a build container and execute script in its context:
 ```
-$ git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner
-$ cd /builds/gitlab-org/gitlab-ci-multi-runner
-$ make <- or any other build step
+$ cat build_script | docker run -n build -i -l mysql:service-mysql -l postgres:service-postgres ruby:2.1 /bin/bash
 ```
+This will create build container that has two service containers linked.
+The build_script is piped using STDIN to bash interpreter which executes the build script in container. 
 
 1. At the end remove all containers:
 ```
-docker rm -f -v build build-mysql build-postgres
+docker rm -f -v build service-mysql service-postgres
 ```
 This will forcefully (the `-f` switch) remove build container and service containers 
 and all volumes (the `-v` switch) that were created with the container creation.
diff --git a/doc/examples/README.md b/doc/examples/README.md
index 23e9740..78dc1e0 100644
--- a/doc/examples/README.md
+++ b/doc/examples/README.md
@@ -1,47 +1,4 @@
-# How your build script is run
-
-The runner runs the line below before it runs the commands in your build script:
-
-    cd /gitlab-ci-runner/tmp/builds && git clone git@gitlab_server_fqdn:group/project.git project-1 && cd project-1 && git checkout master
-
-# Build script example
-
-    bundle install
-    bundle exec rake db:create RAILS_ENV=test
-    bundle exec rake db:migrate RAILS_ENV=test
-    script/run_all_tests
-
-# Environmental variables
-
-The runner sets the following environmental variables:
-
-```
-CI=true
-CI_SERVER=true
-CI_SERVER_NAME=GitLab CI
-GITLAB_CI=true
-CI_SERVER_VERSION
-CI_SERVER_REVISION
-CI_BUILD_REF
-CI_BUILD_BEFORE_SHA
-CI_BUILD_REF_NAME (branch)
-CI_BUILD_ID
-CI_BUILD_REPO
-CI_PROJECT_DIR
-```
-
 # Build script examples
 
-+ [Build script for Omniauth LDAP](build-script-for-omniauth-ldap.md)
-+ [Build script GitLab CE](build_script_gitlab_ce.md)
-+ [Build script for Sencha deploy PhoneGapBuild](build_script_sencha_deploy_phonegapbuild.md)
 + [Test and deploy Ruby Application to Heroku](test-and-deploy-ruby-application-to-heroku.md)
 + [Test and deploy Python Application to Heroku](test-and-deploy-python-application-to-heroku.md)
-
-# Configuring runner examples
-
-+ [For Ruby](configure/ruby.md)
-+ We welcome contributions of examples for other environments.
-
-Please see [cookbook-gitlab-test](https://gitlab.com/gitlab-org/cookbook-gitlab-test/blob/master/README.md)
-for instructions how to prepare a server to run CI tests for GitLab.
\ No newline at end of file
diff --git a/doc/examples/build-script-for-omniauth-ldap.md b/doc/examples/build-script-for-omniauth-ldap.md
deleted file mode 100644
index 28951dd..0000000
--- a/doc/examples/build-script-for-omniauth-ldap.md
+++ /dev/null
@@ -1,10 +0,0 @@
-Build script for omniauth-ldap
-==========================
-
-This is a simple script to test [the GitLab fork of Omniauth-LDAP](https://github.com/gitlabhq/omniauth-ldap)
-
-
-```bash
-bundle
-bundle exec rspec spec
-```
diff --git a/doc/examples/build_script_gitlab_ce.md b/doc/examples/build_script_gitlab_ce.md
deleted file mode 100644
index e14d045..0000000
--- a/doc/examples/build_script_gitlab_ce.md
+++ /dev/null
@@ -1,92 +0,0 @@
-Build script to run the tests of GitLab CE
-=================================
-
-# Build script used at ci.gitlab.org to test the private GitLab B.V. repo at dev.gitlab.org
-
-This build script can run both with the docker or shell executor in [gitlab-ci-multi-runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner).
-
-If you are using a shell executor, runner must be configured to have mysql and ruby, see a [configuration example](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/examples/configure/ruby.md).
-
-```bash
-if [ -f /.dockerinit ]; then
-    wget -q http://ftp.de.debian.org/debian/pool/main/p/phantomjs/phantomjs_1.9.0-1+b1_amd64.deb
-    dpkg -i phantomjs_1.9.0-1+b1_amd64.deb
-
-    apt-get update -qq
-    apt-get install -y -qq libicu-dev libkrb5-dev cmake nodejs
-
-    cp config/database.yml.mysql config/database.yml
-    sed -i 's/username:.*/username: root/g' config/database.yml
-    sed -i 's/password:.*/password:/g' config/database.yml
-    sed -i 's/# socket:.*/host: mysql/g' config/database.yml
-
-    cp config/resque.yml.example config/resque.yml
-    sed -i 's/localhost/redis/g' config/resque.yml
-    FLAGS=(--deployment --path /cache)
-else
-    export PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin
-    cp config/database.yml.mysql config/database.yml
-    sed "s/username\:.*$/username\: runner/" -i config/database.yml
-    sed "s/password\:.*$/password\: 'password'/" -i config/database.yml
-    sed "s/gitlabhq_test/gitlabhq_test_$((RANDOM/5000))/" -i config/database.yml
-fi
-
-ruby -v
-which ruby
-gem install bundler --no-ri --no-rdoc
-
-cp config/gitlab.yml.example config/gitlab.yml
-touch log/application.log
-touch log/test.log
-
-bundle install --without postgres production --jobs $(nproc)  "${FLAGS[@]}"
-RAILS_ENV=test bundle exec rake db:create
-RAILS_ENV=test SIMPLECOV=true bundle exec rake test
-```
-
-# Build script on [GitHost.io](https://gitlab-ce.githost.io/projects/4/) to test the [GitLab.com repo](https://gitlab.com/gitlab-org/gitlab-ce)
-
-```bash
-# Install dependencies: phantomjs, redis, cmake
-if [ ! -f ~/.runner_setup ]; then
-    echo "Setting up runner"
-    sudo wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2
-    sudo tar xjf phantomjs-1.9.7-linux-x86_64.tar.bz2
-    sudo mv phantomjs-1.9.7-linux-x86_64/bin/phantomjs /usr/local/bin/phantomjs
-    sudo apt-get update
-    sudo apt-get -y -q install mysql-server redis-server build-essential cmake curl
-    touch ~/.runner_setup
-    echo "Done setting up runner"
-fi
-
-# Install ruby
-\curl -sSL https://get.rvm.io | bash -s stable --ruby
-source ~/.rvm/scripts/rvm
-
-# Prepare GitLab and run tests
-ruby -v
-gem install bundler
-bundle install
-cp config/database.yml.mysql config/database.yml
-cp config/gitlab.yml.example config/gitlab.yml
-RAILS_ENV=test bundle exec rake db:drop db:create
-RAILS_ENV=test bundle exec rake test
-```
-
-# Troubleshooting
-
-## InvalidByteSequenceError
-
-Test pass locally but on CI there is an error with encoding.
-One of the possible solutions for error: `Encoding::InvalidByteSequenceError: "\xF0" on US-ASCII` during build is setting the correct locale in the build job:
-
-```
-export LC_CTYPE=en_US.UTF-8
-
-```
-
-or
-
-```
-export LANG=en_US.UTF-8
-```
diff --git a/doc/examples/build_script_sencha_deploy_phonegapbuild.md b/doc/examples/build_script_sencha_deploy_phonegapbuild.md
deleted file mode 100644
index 30c31fe..0000000
--- a/doc/examples/build_script_sencha_deploy_phonegapbuild.md
+++ /dev/null
@@ -1,303 +0,0 @@
-script-build-sencha-deploy-phonegapbuild
-=================================
-
-Build an app with Sencha Cmd and deploy to PhoneGap Build.
-
-This is a build script originally written for Gitlab CI but should be fine for whatever CI server you're using.
-
-The script does the following:
-
-- compile the app with sencha cmd ("app build package")
-- push the "build" subdirectory to PhoneGap Build using REST APIs
-
-If you just need the first or second part you can comment out in the script the section you don't need, it should be pretty straightforward.
-
-HOWTO
------
-
-- (one-time) configure the script editing it and filling in the "### Config ###' section
-- chdir to your sencha app root
-- execute the script
-
-Dependencies
-------------
-
-- JSON.sh from https://github.com/dominictarr/JSON.sh
-- XMLLint from http://xmlsoft.org/xmllint.html (or libxml2-utils on Debian-based distros)
-
-License
--------
-
-This script is licensed under MIT Expat, i.e. you can do anything including commercial purposes, just don't blame it on me if anything goes wrong! :)
-
-Bash
---------
-
-```bash
-#!/bin/bash
-
-# 8p8@Mobimentum 2013-09-04 Build a Sencha app and publish repo to user PhoneGapBuild
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-### Config ###
-
-# PhoneGap Build basic auth (include ":" at the end with no password)
-PGB_AUTH_USER="youruser@yourdomain.com:"
-
-# PhoneGap Build auth token 
-PGB_AUTH_TOKEN=""
-
-# JSON.sh, cfr. https://github.com/dominictarr/JSON.sh
-JSON_SH="$(dirname $0)/JSON.sh"
-
-# XMLLint is in package libxml2-utils (Debian)
-XMLLINT="$(which xmllint)"
-
-# Sencha CMD
-SENCHA_CMD="/opt/sencha-cmd"
-
-### Script ###
-
-# Compile
-$SENCHA_CMD app build package
-status=$?
-
-# Add config.xml and zip content
-app=$(grep -P '^\s*app.name=' .sencha/app/sencha.cfg | awk -F'=' '{print $2}')
-cp config.xml "build/package/$app/"
-cd "build/package/$app"
-zip -r "${app}.zip" *
-
-# Check if app is already uploaded to PG:B
-# cfr. https://build.phonegap.com/docs/read_api
-app_pkg="$($XMLLINT --xpath "/*[local-name()='widget']/@id" config.xml | sed 's/\(^ id=\"\|\"$\)//g')"
-app_id=$(curl -u "$PGB_AUTH_USER" "https://build.phonegap.com/api/v1/apps?auth_token=$PGB_AUTH_TOKEN" \
-	| $JSON_SH -b | grep -E '\["apps",([0-9]+),"(package|id)"\]' \
-	| grep -A 1 "$app_pkg" | tail -n 1 | awk -F'\t' '{print $2}')
-
-# Upload app
-# cfr. https://build.phonegap.com/docs/write_api
-if [[ ! -z "$app_id" ]]
-then
-	# App already uploaded
-	curl -u "$PGB_AUTH_USER" -X PUT -F "file=@${app}.zip" \
-		"https://build.phonegap.com/api/v1/apps/$app_id?auth_token=$PGB_AUTH_TOKEN"
-else
-	# New app
-	# Upload to PG:B
-	curl -u "$PGB_AUTH_USER" -F "file=@${app}.zip" -F "data={\"title\":\"$app\",\"create_method\":\"file\"}" \
-		"https://build.phonegap.com/api/v1/apps?auth_token=$PGB_AUTH_TOKEN"
-fi
-
-# TODO: improve status detection
-exit $status
-```
-
-JSON
---------
-
-```bash
-#!/usr/bin/env bash
-
-throw () {
-  echo "$*" >&2
-  exit 1
-}
-
-BRIEF=0
-LEAFONLY=0
-PRUNE=0
-
-usage() {
-  echo
-  echo "Usage: JSON.sh [-b] [-l] [-p] [-h]"
-  echo
-  echo "-p - Prune empty. Exclude fields with empty values."
-  echo "-l - Leaf only. Only show leaf nodes, which stops data duplication."
-  echo "-b - Brief. Combines 'Leaf only' and 'Prune empty' options."
-  echo "-h - This help text."
-  echo
-}
-
-parse_options() {
-  set -- "$@"
-  local ARGN=$#
-  while [ $ARGN -ne 0 ]
-  do
-    case $1 in
-      -h) usage
-          exit 0
-      ;;
-      -b) BRIEF=1
-          LEAFONLY=1
-          PRUNE=1
-      ;;
-      -l) LEAFONLY=1
-      ;;
-      -p) PRUNE=1
-      ;;
-      ?*) echo "ERROR: Unknown option."
-          usage
-          exit 0
-      ;;
-    esac
-    shift 1
-    ARGN=$((ARGN-1))
-  done
-}
-
-awk_egrep () {
-  local pattern_string=$1
-
-  gawk '{
-    while ($0) {
-      start=match($0, pattern);
-      token=substr($0, start, RLENGTH);
-      print token;
-      $0=substr($0, start+RLENGTH);
-    }
-  }' pattern=$pattern_string
-}
-
-tokenize () {
-  local GREP
-  local ESCAPE
-  local CHAR
-
-  if echo "test string" | egrep -ao --color=never "test" &>/dev/null
-  then
-    GREP='egrep -ao --color=never'
-  else
-    GREP='egrep -ao'
-  fi
-
-  if echo "test string" | egrep -o "test" &>/dev/null
-  then
-    ESCAPE='(\\[^u[:cntrl:]]|\\u[0-9a-fA-F]{4})'
-    CHAR='[^[:cntrl:]"\\]'
-  else
-    GREP=awk_egrep
-    ESCAPE='(\\\\[^u[:cntrl:]]|\\u[0-9a-fA-F]{4})'
-    CHAR='[^[:cntrl:]"\\\\]'
-  fi
-
-  local STRING="\"$CHAR*($ESCAPE$CHAR*)*\""
-  local NUMBER='-?(0|[1-9][0-9]*)([.][0-9]*)?([eE][+-]?[0-9]*)?'
-  local KEYWORD='null|false|true'
-  local SPACE='[[:space:]]+'
-
-  $GREP "$STRING|$NUMBER|$KEYWORD|$SPACE|." | egrep -v "^$SPACE$"
-}
-
-parse_array () {
-  local index=0
-  local ary=''
-  read -r token
-  case "$token" in
-    ']') ;;
-    *)
-      while :
-      do
-        parse_value "$1" "$index"
-        index=$((index+1))
-        ary="$ary""$value" 
-        read -r token
-        case "$token" in
-          ']') break ;;
-          ',') ary="$ary," ;;
-          *) throw "EXPECTED , or ] GOT ${token:-EOF}" ;;
-        esac
-        read -r token
-      done
-      ;;
-  esac
-  [ "$BRIEF" -eq 0 ] && value=`printf '[%s]' "$ary"` || value=
-  :
-}
-
-parse_object () {
-  local key
-  local obj=''
-  read -r token
-  case "$token" in
-    '}') ;;
-    *)
-      while :
-      do
-        case "$token" in
-          '"'*'"') key=$token ;;
-          *) throw "EXPECTED string GOT ${token:-EOF}" ;;
-        esac
-        read -r token
-        case "$token" in
-          ':') ;;
-          *) throw "EXPECTED : GOT ${token:-EOF}" ;;
-        esac
-        read -r token
-        parse_value "$1" "$key"
-        obj="$obj$key:$value"        
-        read -r token
-        case "$token" in
-          '}') break ;;
-          ',') obj="$obj," ;;
-          *) throw "EXPECTED , or } GOT ${token:-EOF}" ;;
-        esac
-        read -r token
-      done
-    ;;
-  esac
-  [ "$BRIEF" -eq 0 ] && value=`printf '{%s}' "$obj"` || value=
-  :
-}
-
-parse_value () {
-  local jpath="${1:+$1,}$2" isleaf=0 isempty=0 print=0
-  case "$token" in
-    '{') parse_object "$jpath" ;;
-    '[') parse_array  "$jpath" ;;
-    # At this point, the only valid single-character tokens are digits.
-    ''|[!0-9]) throw "EXPECTED value GOT ${token:-EOF}" ;;
-    *) value=$token
-       isleaf=1
-       [ "$value" = '""' ] && isempty=1
-       ;;
-  esac
-  [ "$value" = '' ] && return
-  [ "$LEAFONLY" -eq 0 ] && [ "$PRUNE" -eq 0 ] && print=1
-  [ "$LEAFONLY" -eq 1 ] && [ "$isleaf" -eq 1 ] && [ $PRUNE -eq 0 ] && print=1
-  [ "$LEAFONLY" -eq 0 ] && [ "$PRUNE" -eq 1 ] && [ "$isempty" -eq 0 ] && print=1
-  [ "$LEAFONLY" -eq 1 ] && [ "$isleaf" -eq 1 ] && \
-    [ $PRUNE -eq 1 ] && [ $isempty -eq 0 ] && print=1
-  [ "$print" -eq 1 ] && printf "[%s]\t%s\n" "$jpath" "$value"
-  :
-}
-
-parse () {
-  read -r token
-  parse_value
-  read -r token
-  case "$token" in
-    '') ;;
-    *) throw "EXPECTED EOF GOT $token" ;;
-  esac
-}
-
-parse_options "$@"
-
-if ([ "$0" = "$BASH_SOURCE" ] || ! [ -n "$BASH_SOURCE" ]);
-then
-  tokenize | parse
-fi
-```
diff --git a/doc/examples/configure/ruby.md b/doc/examples/configure/ruby.md
deleted file mode 100644
index 3ecdaac..0000000
--- a/doc/examples/configure/ruby.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# Example configuring runner for Ruby
-
-In this example, we configure ruby and mysql for testing environment:
-
-```
-# as root
-(
-set -e
-apt-get update
-apt-get upgrade -y
-apt-get install -y curl
-cd /root
-rm -rf cookbooks cookbook-gitlab-test.git
-curl 'https://gitlab.com/gitlab-org/cookbook-gitlab-test/repository/archive.tar.gz?ref=master' | tar -xvz
-mkdir cookbooks
-mv cookbook-gitlab-test.git cookbooks/cookbook-gitlab-test
-curl -L https://www.chef.io/chef/install.sh | bash
-chef-client -z -r 'recipe[cookbook-gitlab-test::ruby], recipe[cookbook-gitlab-test::mysql]'
-)
-
-
-### Register your runner instance with a GitLab CI Coordinator
-gitlab-ci-multi-runner register
-
-```
\ No newline at end of file
diff --git a/doc/install/installation.md b/doc/install/README.md
index 8cbc858..8cbc858 100644
--- a/doc/install/installation.md
+++ b/doc/install/README.md
diff --git a/doc/quick_start/README.md b/doc/quick_start/README.md
new file mode 100644
index 0000000..3b9156f
--- /dev/null
+++ b/doc/quick_start/README.md
@@ -0,0 +1,119 @@
+# Quick Start
+
+To start building projects with GitLab CI a few steps needs to be done.
+
+## 1. Install GitLab and CI
+
+First you need to have a working GitLab and GitLab CI instance.
+
+You can omit this step if you use [GitLab.com](http://GitLab.com/).
+
+## 2. Create repository on GitLab
+
+Once you login on your GitLab add a new repository where you will store your source code.
+Push your application to that repository.
+
+## 3. Add project to CI
+
+The next part is to login to GitLab CI.
+Point your browser to the URL you have set GitLab CI or use [ci.gitlab.com](http://ci.gitlab.com/) that is linked to [GitLab.com](http://GitLab.com/).
+
+On the first screen you will see a list of GitLab's projects that you have access to:
+
+![Projects](projects.png)
+
+Click **Add Project to CI**.
+This will create project in CI and authorize GitLab CI to fetch sources from GitLab.
+
+> GitLab CI creates unique token that is used to configure GitLab CI service in GitLab.
+> This token allows to access GitLab's repository and configures GitLab to trigger GitLab CI webhook on **Push events** and **Tag push events**.
+> You can see that token by going to Project's Settings > Services > GitLab CI.
+> You will see there token, the same token is assigned in GitLab CI settings of project.
+
+## 4. Create project's configuration - .gitlab-ci.yml
+
+The next: You have to define how your project will be built.
+GitLab CI uses [YAML](https://en.wikipedia.org/wiki/YAML) file to store build configuration.
+You need to create `.gitlab-ci.yml` in root directory of your repository:
+
+```yaml
+before_script:
+  - bundle install
+
+rspec:
+  script:
+    - bundle exec rspec
+
+rubocop:
+  script:
+    - bundle exec rubocop
+```
+
+This is the simplest possible build configuration that will work for most Ruby applications:
+1. Define two jobs `rspec` and `rubocop` with two different commands to be executed.
+1. Before every job execute commands defined by `before_script`.
+
+The `.gitlab-ci.yml` defines set of jobs with constrains how and when they should be run.
+The jobs are defined as top-level elements with name and always have to contain the `script`.
+Jobs are used to create builds, which are then picked by [runners](../runners/README.md) and executed within environment of the runner.
+What is important that each job is run independently from each other. 
+
+For more information and complete `.gitlab-ci.yml` syntax, please check the [Configuring project (.gitlab-ci.yml)](../yaml/README.md).
+
+## 5. Add file and push .gitlab-ci.yml to repository
+
+Once you created `.gitlab-ci.yml` you should add it to git repository and push it to GitLab.
+
+```bash
+git add .gitlab-ci.yml
+git commit
+git push origin master
+```
+
+If you refresh the project's page on GitLab CI you will notice a one new commit:
+
+![](new_commit.png)
+
+However the commit has status **pending** which means that commit was not yet picked by runner.
+
+## 6. Configure runner
+
+In GitLab CI, Runners run your builds.
+A runner is a machine (can be virtual, bare-metal or VPS) that picks up builds through the coordinator API of GitLab CI.
+
+A runner can be specific to a certain project or serve any project in GitLab CI.
+A runner that serves all projects is called a shared runner.
+More information about different runner types can be found in [Configuring runner](../runners/README.md).
+
+To check if you have runners assigned to your project go to **Runners**. You will find there information how to setup project specific runner:
+
+1. Install GitLab Runner software. Checkout the [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner) section to install it.
+1. Specify following URL during runner setup: https://ci.gitlab.com/
+1. Use the following registration token during setup: TOKEN
+
+If you do it correctly your runner should be shown under **Runners activated for this project**:
+
+![](runners_activated.png)
+
+### Shared runners
+
+If you use [ci.gitlab.com](http://ci.gitlab.com/) you can use **Shared runners** provided by GitLab Inc.
+These are special virtual machines that are run on GitLab's infrastructure that can build any project.
+To enable **Shared runners** you have to go to **Runners** and click **Enable shared runners** for this project.
+
+## 7. Check status of commit
+
+If everything went OK and you go to commit, the status of the commit should change from **pending** to either **running**, **success** or **failed**.
+
+![](commit_status.png)
+
+You can click **Build ID** to view build log for specific job.
+
+## 8. Congratulations!
+
+You managed to build your first project using GitLab CI.
+You may need to tune your `.gitlab-ci.yml` file to implement build plan for your project.
+A few examples how it can be done you can find on [Examples](../examples/README.md) page.
+
+GitLab CI also offers **the Lint** tool to verify validity of your `.gitlab-ci.yml` which can be useful to troubleshoot potential problems.
+The Lint is available from project's settings or by adding `/lint` to GitLab CI url.
diff --git a/doc/quick_start/build_status.png b/doc/quick_start/build_status.png
new file mode 100644
index 0000000..333259e
--- /dev/null
+++ b/doc/quick_start/build_status.png
diff --git a/doc/quick_start/commit_status.png b/doc/quick_start/commit_status.png
new file mode 100644
index 0000000..725b79e
--- /dev/null
+++ b/doc/quick_start/commit_status.png
diff --git a/doc/quick_start/new_commit.png b/doc/quick_start/new_commit.png
new file mode 100644
index 0000000..3839e89
--- /dev/null
+++ b/doc/quick_start/new_commit.png
diff --git a/doc/quick_start/projects.png b/doc/quick_start/projects.png
new file mode 100644
index 0000000..0b3430a
--- /dev/null
+++ b/doc/quick_start/projects.png
diff --git a/doc/quick_start/runners.png b/doc/quick_start/runners.png
new file mode 100644
index 0000000..25b4046
--- /dev/null
+++ b/doc/quick_start/runners.png
diff --git a/doc/quick_start/runners_activated.png b/doc/quick_start/runners_activated.png
new file mode 100644
index 0000000..c934bd1
--- /dev/null
+++ b/doc/quick_start/runners_activated.png
diff --git a/doc/runners/README.md b/doc/runners/README.md
index 63a3ffe..68dcfe2 100644
--- a/doc/runners/README.md
+++ b/doc/runners/README.md
@@ -1,6 +1,6 @@
 # Runners
 
-In GitLab CI, Runners run your [jobs](jobs/README.md).
+In GitLab CI, Runners run your [yaml](../yaml/README.md).
 A runner is an isolated (virtual) machine that picks up builds
 through the coordinator API of GitLab CI.
 
@@ -40,10 +40,10 @@ A fork does copy the CI settings (jobs, allow shared, etc) of the cloned reposit
 There are several ways to create a runner. Only after creation, upon
 registration its status as Shared or Specific is determined.
 
-[See the documentation for](https://about.gitlab.com/gitlab-ci/#gitlab-runner)
-the different methods of creating a Runner instance.
+[See the documentation for](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation)
+the different methods of installing a Runner instance.
 
-After creating the runner, you can either register it as `Shared` or as `Specific`.
+After installing the runner, you can either register it as `Shared` or as `Specific`.
 You can only register a Shared Runner if you have admin access to the GitLab instance.
 
 ## Registering a Shared Runner
@@ -59,12 +59,7 @@ instance.
 Now simply register the runner as any runner:
 
 ```
-sudo /opt/gitlab-runner/bin/setup -C /home/gitlab-runner
-```
-
-Then restart the Upstart script:
-```
-sudo service gitlab-runner restart
+sudo gitlab-runner register
 ```
 
 Note that you will have to enable `Allows shared runners` for each project
@@ -93,12 +88,7 @@ setup a specific runner for this project.
 To register the runner, run the command below and follow instructions:
 
 ```
-sudo /opt/gitlab-runner/bin/setup -C /home/gitlab-runner
-```
-
-Then restart the Upstart script:
-```
-sudo service gitlab-runner restart
+sudo gitlab-runner register
 ```
 
 ###  Making an existing Shared Runner Specific
@@ -152,9 +142,4 @@ project.
 # Attack vectors in runners
 
 Mentioned briefly earlier, but the following things of runners can be exploited.
-We're always looking for contributions that can mitigate these.
-
-- anyone that can run a job on a runner can access any code it runs
-- when setting `Allow running on shared runners` anyone that can run their
-code on a shared runner can access any code
--
+We're always looking for contributions that can mitigate these [Security Considerations](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md).
diff --git a/doc/variables/README.md b/doc/variables/README.md
new file mode 100644
index 0000000..7f7ebdf
--- /dev/null
+++ b/doc/variables/README.md
@@ -0,0 +1,66 @@
+## Variables
+When receiving a build from GitLab CI, the runner prepares the build environment.
+It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables** (Secure Variables)
+
+### Predefined variables (Environment Variables)
+| Variable | Description |
+|--|--|
+| **CI** | Mark that build is executed in CI environment |
+| **GITLAB_CI** | Mark that build is executed in GitLab CI environment |
+| **CI_SERVER** | Mark that build is executed in CI environment |
+| **CI_SERVER_NAME** | CI server that is used to coordinate builds |
+| **CI_SERVER_VERSION** | Not yet defined |
+| **CI_SERVER_REVISION** | Not yet defined |
+| **CI_BUILD_REF** | The commit revision for which project is built |
+| **CI_BUILD_BEFORE_SHA** | The first commit that were included in push request |
+| **CI_BUILD_REF_NAME** | The branch or tag name for which project is built |
+| **CI_BUILD_ID** | The unique id of the current build that GitLab CI uses internally |
+| **CI_BUILD_REPO** | The URL to clone the Git repository |
+| **CI_PROJECT_ID** | The unique id of the current project that GitLab CI uses internally |
+| **CI_PROJECT_DIR** | The full path where the repository is cloned and where the build is ran |
+
+Example values:
+
+```bash
+export CI_BUILD_BEFORE_SHA="9df57456fa9de2a6d335ca5edf9750ed812b9df0"
+export CI_BUILD_ID="50"
+export CI_BUILD_REF="1ecfd275763eff1d6b4844ea3168962458c9f27a"
+export CI_BUILD_REF_NAME="master"
+export CI_BUILD_REPO="https://gitlab.com/gitlab-org/gitlab-ce.git"
+export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce"
+export CI_PROJECT_ID="34"
+export CI_SERVER="yes"
+export CI_SERVER_NAME="GitLab CI"
+export CI_SERVER_REVISION=""
+export CI_SERVER_VERSION=""
+```
+
+### User-defined variables (Secure Variables)
+**This feature requires `gitlab-runner` with version equal or greater than 0.4.0.**
+
+GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. 
+The secure variables are stored out of the repository (the `.gitlab-ci.yml`).
+These variables are securely stored in GitLab CI database and are hidden in the build log.
+It's desired method to use them for storing passwords, secret keys or whatever you want.
+
+Secure Variables can added by going to `Project > Variables > Add Variable`.
+
+They will be available for all subsequent builds.
+
+### Use variables
+The variables are set as environment variables in build environment and are accessible with normal methods that are used to access such variables.
+In most cases the **bash** is used to execute build script.
+To access variables (predefined and user-defined) in bash environment, prefix the variable name with `$`:
+```
+job_name:
+  script:
+    - echo $CI_BUILD_ID
+```
+
+You can also list all environment variables with `export` command,
+but be aware that this will also expose value of all **Secure Variables** in build log:
+```
+job_name:
+  script:
+    - export
+```
diff --git a/doc/yaml/README.md b/doc/yaml/README.md
new file mode 100644
index 0000000..6600988
--- /dev/null
+++ b/doc/yaml/README.md
@@ -0,0 +1,180 @@
+# Configuration of your builds with .gitlab-ci.yml
+From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (**.gitlab-ci.yml**) for the project configuration.
+It is placed in the root of your repository and contains definition how project should be built.
+
+The YAML defines set of jobs with constrains when they should be run.
+The jobs are defined as top-level elements with name and always have to contain the `script`: 
+```yaml
+job1:
+  script: "execute-script-for-job1"
+
+job2:
+  script: "execute-script-for-job2"
+```
+
+The above example is the simplest possible CI configuration with two separate jobs,
+where each of the job executes different script.
+
+Jobs are used to create builds, which are then picked by [runners](../runners/README.md) and executed within environment of the runner.
+What is important that each job is run independently from each other. 
+
+## .gitlab-ci.yml
+The YAML syntax allows to use more complex jobs specification then above example:
+```yaml
+image: ruby:2.1
+services:
+  - postgres
+
+before_script:
+  - bundle_install
+
+types:
+  - build
+  - test
+  - deploy
+
+job1:
+  type: build
+  script:
+    - execute-script-for-job1
+  only:
+    - master
+  tags:
+    - docker
+```
+
+There are a few `keywords` that can't be used as job names:
+
+| keyword       | required | description |
+|---------------|----------|-------------|
+| image         | optional | Use docker image, covered in [Use Docker](../docker/README.md) |
+| services      | optional | Use docker services, covered in [Use Docker](../docker/README.md) |
+| types         | optional | Define build types |
+| before_script | optional | Define commands prepended for each job's script |
+
+### image and services
+This allows to specify custom Docker image and list of services that can be used for time of the build.
+The configuration of this feature is covered in separate document: [Use Docker](../docker/README.md).
+
+### before_script
+`before_script` is used to define the command that should be ran before all builds, including deploy builds. This can be an array or a multiline string
+
+### types
+`types` is used to define build types that can be used by jobs.
+The specification of `types` allow to have flexible multi stage pipeline. 
+
+The ordering of elements in `types` defines the ordering of builds execution:
+
+1. Builds of the same type are run in parallel. 
+1. Builds of next type are run after success.
+
+Let's consider following example that defines 3 types:
+```
+types:
+  - build
+  - test
+  - deploy
+```
+
+1. First all jobs of `build` are executed in parallel.
+1. If all jobs of `build` succeeds, the `test` jobs are executed in parallel.
+1. If all jobs of `test` succeeds, the `deploy` jobs are executed in parallel.
+1. If all jobs of `deploy` succeeds, the commit is marked as `success`.
+1. If any of the previous jobs fails the commit is marked as `failed` and no jobs of further type is executed.
+
+There are also two edge cases worth mentioning:
+
+1. If no `types` is defined in `.gitlab-ci.yml` by default: build, test and deploy is allowed to be used as job's type.
+2. If job doesn't specify `type`, the job is assigned to `test`.
+
+## Jobs
+`.gitlab-ci.yml` allows to specify unlimited number of jobs.
+Each job has to have unique `job_name`, that is not the one of the keywords. 
+Job is defined by a list of parameters that define the build behaviour.
+
+```yaml
+job_name:
+  script:
+    - rake spec
+    - coverage
+  type: test
+  only:
+    - master
+  except:
+    - develop
+  tags:
+    - ruby
+    - postgres
+  allow_failure: true
+```
+
+| keyword       | required | description |
+|---------------|----------|-------------|
+| script        | required | Defines a shell script which is executed by runner |
+| type          | optional (default: test) | Defines a build type |
+| only          | optional | Defines a list of git refs for which build is created |
+| except        | optional | Defines a list of git refs for which build is not created |
+| tags          | optional | Defines a list of tags which are used to select runner |
+| allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status |
+
+### script
+`script` is a shell script which is executed by runner. The shell script is prepended with `before_script`.
+
+```yaml
+job:
+  script: "bundle exec rspec"
+```
+
+This parameter can also contain several commands using array:
+```yaml
+job:
+  script:
+    - uname -a
+    - bundle exec rspec
+```
+
+### type
+`type` allows to group build into different stages. Builds of the same `type` are executed in `parallel`.
+For more info about the use of `type` please check the [types](#types).
+
+### only and except
+This are two parameters that allows to set refs policy to limit when jobs are built:
+1. `only` defines the names of branches and tags for which job will be build.
+2. `except` defines the names of branches and tags that will be excluded from building specific job.
+
+There are a few rules that apply to usage of refs policy:
+
+1. `only` and `except` are exclusive. If both `only` and `except` are defined in job specification only `only` is taken into account.
+1. `only` and `except` allows to use the regexp expressions.
+1. `only` and `except` allows to use special keywords: `branches` and `tags`.
+These names can be used for example to exclude all tags and all branches.
+
+```yaml
+job:
+  only:
+    - /^issue-.*$/ # use regexp
+  except:
+    - branches # use special keyword
+```
+
+### tags
+`tags` is used to select specific runner from the list of all runners that are allowed to run this project.
+
+During registration of runner you can specify runner's tags, ie.: `ruby`, `postgres`, `development`.
+`tags` allows you to run builds by runners that have the specified tags assigned:
+
+```
+job:
+  tags:
+    - ruby
+    - postgres
+```
+
+The above specification will make sure that `job` is built by runner that have `ruby` AND `postgres` tags defined.
+
+## Validate the .gitlab-ci.yml
+Each instance of GitLab CI has an embedded debug tool Lint.
+You can find the link to the Lint in the project's settings page or use short url `/lint`.
+
+## Skipping builds
+There is one more way to skip all builds, if your commit message contains tag [ci skip]. In this case, commit will be created but builds will be skipped