diff options
| author | Micaël Bergeron <mbergeron@gitlab.com> | 2018-03-22 09:06:10 -0400 |
|---|---|---|
| committer | Micaël Bergeron <mbergeron@gitlab.com> | 2018-03-22 09:06:10 -0400 |
| commit | 9c6663ea079128bb730ec2a168b43961cd9462ec (patch) | |
| tree | 4f62c2e745c7f3e8571ee8c023abcce316c68275 /doc | |
| parent | 6801a93e5e7447199b091e44f33c96d22a1a1960 (diff) | |
| parent | c01697539c3da4e72b1812662ff35d1f709d1dcc (diff) | |
| download | gitlab-ce-9c6663ea079128bb730ec2a168b43961cd9462ec.tar.gz | |
Merge remote-tracking branch 'origin/master' into 40781-os-to-ce
Diffstat (limited to 'doc')
60 files changed, 2532 insertions, 1040 deletions
diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 6c5a466ced5..27a3710632d 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -187,6 +187,7 @@ for a real-world example of this exploit. ```sh sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart ``` 1. Verify that everything is configured correctly: diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex b3c6bc474e3..48212f6276a 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png Binary files differnew file mode 100644 index 00000000000..52176df9ecd --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index b9464945cea..dc4f685d843 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -11,12 +11,18 @@ It allows you to see (from left to right): - the timing of the page (backend, frontend) - time taken and number of DB queries, click through for details of these queries  +- time taken and number of [Gitaly] calls, click through for details of these calls + +- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). + - time taken and number of calls to Redis - time taken and number of background jobs created by Sidekiq -- profile of the code used to generate the page, line by line for either _all_, _app & lib_ , or _views_. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). - - time taken and number of Ruby GC calls +On the far right is a request selector that allows you to view the same metrics +(excluding the page timing and line profiler) for any requests made while the +page was open. Only the first two requests per unique URL are captured. + ## Enable the Performance Bar via the Admin panel GitLab Performance Bar is disabled by default. To enable it for a given group, @@ -39,3 +45,5 @@ You can toggle the Bar using the same shortcut.  --- + +[Gitaly]: ../../gitaly/index.md diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index d73d9422d2c..7d34d35e7d1 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -84,12 +84,14 @@ checks using those checksums can be run. These checks also detect missing files. Currently, integrity checks are supported for the following types of file: -* LFS objects -* User uploads +* CI artifacts (Available from version 10.7.0) +* LFS objects (Available from version 10.6.0) +* User uploads (Available from version 10.6.0) **Omnibus Installation** ``` +sudo gitlab-rake gitlab:artifacts:check sudo gitlab-rake gitlab:lfs:check sudo gitlab-rake gitlab:uploads:check ``` @@ -97,6 +99,7 @@ sudo gitlab-rake gitlab:uploads:check **Source Installation** ```bash +sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production ``` @@ -112,6 +115,7 @@ Variable | Type | Description `VERBOSE` | boolean | Causes failures to be listed individually, rather than being summarized. ```bash +sudo gitlab-rake gitlab:artifacts:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:lfs:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:uploads:check BATCH=100 ID_FROM=50 ID_TO=250 ``` diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index ecf92c379fd..2b4252a7b1d 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -240,3 +240,31 @@ sudo gitlab-rake gitlab:tcp_check[example.com,80] cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:tcp_check[example.com,80] RAILS_ENV=production ``` + +## Clear exclusive lease (DANGER) + +GitLab uses a shared lock mechanism: `ExclusiveLease` to prevent simultaneous operations +in a shared resource. An example is running periodic garbage collection on repositories. + +In very specific situations, a operation locked by an Exclusive Lease can fail without +releasing the lock. If you can't wait for it to expire, you can run this task to manually +clear it. + +To clear all exclusive leases: + +DANGER: **DANGER**: +Don't run it while GitLab or Sidekiq is running + +```bash +sudo gitlab-rake gitlab:exclusive_lease:clear +``` + +To specify a lease `type` or lease `type + id`, specify a scope: + +```bash +# to clear all leases for repository garbage collection: +sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:*] + +# to clear a lease for repository garbage collection in a specific project: (id=4) +sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:4] +``` diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index 677765368a8..de5207fc5e4 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -15,9 +15,10 @@ POST /projects/:id/export | Attribute | Type | Required | Description | | --------- | -------------- | -------- | ---------------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `description` | string | no | Overrides the project description | ```console -curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/export +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --form "description=Foo Bar" https://gitlab.example.com/api/v4/projects/1/export ``` ```json diff --git a/doc/api/search.md b/doc/api/search.md index d441b556186..107ddaffa6a 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -289,7 +289,7 @@ Search within the specified group. If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. ``` -GET /groups/:id/-/search +GET /groups/:id/search ``` | Attribute | Type | Required | Description | @@ -305,7 +305,7 @@ The response depends on the requested scope. ### Scope: projects ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/-/search?scope=projects&search=flight +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=projects&search=flight ``` Example response: @@ -336,7 +336,7 @@ Example response: ### Scope: issues ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/-/search?scope=issues&search=file +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=issues&search=file ``` Example response: @@ -401,7 +401,7 @@ Example response: ### Scope: merge_requests ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/-/search?scope=merge_requests&search=file +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=merge_requests&search=file ``` Example response: @@ -478,7 +478,7 @@ Example response: ### Scope: milestones ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/-/search?scope=milestones&search=release +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=milestones&search=release ``` Example response: @@ -507,7 +507,7 @@ Search within the specified project. If a user is not a member of a project and the project is private, a `GET` request on that project will result to a `404` status code. ``` -GET /projects/:id/-/search +GET /projects/:id/search ``` | Attribute | Type | Required | Description | @@ -524,7 +524,7 @@ The response depends on the requested scope. ### Scope: issues ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/12/-/search?scope=issues&search=file +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/12/search?scope=issues&search=file ``` Example response: @@ -589,7 +589,7 @@ Example response: ### Scope: merge_requests ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/-/search?scope=merge_requests&search=file +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=merge_requests&search=file ``` Example response: @@ -666,7 +666,7 @@ Example response: ### Scope: milestones ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/12/-/search?scope=milestones&search=release +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/12/search?scope=milestones&search=release ``` Example response: @@ -691,7 +691,7 @@ Example response: ### Scope: notes ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/-/search?scope=notes&search=maxime +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime ``` Example response: @@ -723,7 +723,7 @@ Example response: ### Scope: wiki_blobs ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/-/search?scope=wiki_blobs&search=bye +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=wiki_blobs&search=bye ``` Example response: @@ -746,7 +746,7 @@ Example response: ### Scope: commits ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/-/search?scope=commits&search=bye +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=commits&search=bye ``` Example response: @@ -777,7 +777,7 @@ Example response: ### Scope: blobs ```bash -curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/-/search?scope=blobs&search=installation +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=installation ``` Example response: diff --git a/doc/ci/README.md b/doc/ci/README.md index 532ae52a184..6aa0e5885db 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -65,7 +65,8 @@ learn how to leverage its potential even more. environments and use them for different purposes like testing, building and deploying - [Job artifacts](../user/project/pipelines/job_artifacts.md) -- [Git submodules](git_submodules.md): How to run your CI jobs when Git +- [Caching dependencies](caching/index.md) +- [Git submodules](git_submodules.md) - How to run your CI jobs when Git submodules are involved - [Use SSH keys in your build environment](ssh_keys/README.md) - [Trigger pipelines through the GitLab API](triggers/README.md) diff --git a/doc/ci/caching/img/clear_runners_cache.png b/doc/ci/caching/img/clear_runners_cache.png Binary files differnew file mode 100644 index 00000000000..e5db4a47b3e --- /dev/null +++ b/doc/ci/caching/img/clear_runners_cache.png diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md new file mode 100644 index 00000000000..c159198d16b --- /dev/null +++ b/doc/ci/caching/index.md @@ -0,0 +1,518 @@ +# Cache dependencies in GitLab CI/CD + +GitLab CI/CD provides a caching mechanism that can be used to save time +when your jobs are running. + +Caching is about speeding the time a job is executed by reusing the same +content of a previous job. It can be particularly useful when your are +developing software that depends on other libraries which are fetched via the +internet during build time. + +If caching is enabled, it's shared between pipelines and jobs by default, +starting from GitLab 9.0. + +Make sure you read the [`cache` reference](../yaml/README.md#cache) to learn +how it is defined in `.gitlab-ci.yml`. + +## Good caching practices + +We have the cache from the perspective of the developers (who consume a cache +within the job) and the cache from the perspective of the Runner. Depending on +which type of Runner you are using, cache can act differently. + +From the perspective of the developer, to ensure maximum availability of the +cache, when declaring `cache` in your jobs, use one or a mix of the following: + +- [Tag your Runners](../runners/README.md#using-tags) and use the tag on jobs + that share their cache. +- [Use sticky Runners](../runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) + that will be only available to a particular project. +- [Use a `key`](../yaml/README.md#cache-key) that fits your workflow (e.g., + different caches on each branch). For that, you can take advantage of the + [CI/CD predefined variables](../variables/README.md#predefined-variables-environment-variables). + +TIP: **Tip:** +Using the same Runner for your pipeline, is the most simple and efficient way to +cache files in one stage or pipeline, and pass this cache to subsequent stages +or pipelines in a guaranteed manner. + +From the perspective of the Runner, in order for cache to work effectively, one +of the following must be true: + +- Use a single Runner for all your jobs +- Use multiple Runners (in autoscale mode or not) that use + [distributed caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching), + where the cache is stored in S3 buckets (like shared Runners on GitLab.com) +- Use multiple Runners (not in autoscale mode) of the same architecture that + share a common network-mounted directory (using NFS or something similar) + where the cache will be stored + +TIP: **Tip:** +Read about the [availability of the cache](#availability-of-the-cache) +to learn more about the internals and get a better idea how cache works. + +### Sharing caches across the same branch + +Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each +branch always use the same cache: + +```yaml +cache: + key: ${CI_COMMIT_REF_SLUG} +``` + +While this feels like it might be safe from accidentally overwriting the cache, +it means merge requests get slow first pipelines, which might be a bad +developer experience. The next time a new commit is pushed to the branch, the +cache will be re-used. + +To enable per-job and per-branch caching: + +```yaml +cache: + key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" +``` + +To enable per-branch and per-stage caching: + +```yaml +cache: + key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" +``` + +### Sharing caches across different branches + +If the files you are caching need to be shared across all branches and all jobs, +you can use the same key for all of them: + +```yaml +cache: + key: one-key-to-rull-them-all +``` + +To share the same cache between branches, but separate them by job: + +```yaml +cache: + key: ${CI_JOB_NAME} +``` + +### Disabling cache on specific jobs + +If you have defined the cache globally, it means that each job will use the +same definition. You can override this behavior per-job, and if you want to +disable it completely, use an empty hash: + +```yaml +job: + cache: {} +``` + +For more fine tuning, read also about the +[`cache: policy`](../yaml/README.md#cache-policy). + +## Common use cases + +The most common use case of cache is to preserve contents between subsequent +runs of jobs for things like dependencies and commonly used libraries +(Nodejs packages, PHP packages, rubygems, python libraries, etc.), +so they don't have to be re-fetched from the public internet. + +NOTE: **Note:** +For more examples, check the [GitLab CI Yml](https://gitlab.com/gitlab-org/gitlab-ci-yml) +project. + +### Caching Nodejs dependencies + +Assuming your project is using [npm](https://www.npmjs.com/) or +[Yarn](https://yarnpkg.com/en/) to install the Nodejs dependencies, the +following example defines `cache` globally so that all jobs inherit it. +Nodejs modules are installed in `node_modules/` and are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Nodejs.gitlab-ci.yml +# +image: node:latest + +# Cache modules in between jobs +cache: + key: ${CI_COMMIT_REF_SLUG} + paths: + - node_modules/ + +before_script: + - npm install + +test_async: + script: + - node ./specs/start.js ./specs/async.spec.js +``` + +### Caching PHP dependencies + +Assuming your project is using [Composer](https://getcomposer.org/) to install +the PHP dependencies, the following example defines `cache` globally so that +all jobs inherit it. PHP libraries modules are installed in `vendor/` and +are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/PHP.gitlab-ci.yml +# +image: php:7.2 + +# Cache libraries in between jobs +cache: + key: ${CI_COMMIT_REF_SLUG} + paths: + - vendor/ + +before_script: +# Install and run Composer +- curl --show-error --silent https://getcomposer.org/installer | php +- php composer.phar install + +test: + script: + - vendor/bin/phpunit --configuration phpunit.xml --coverage-text --colors=never +``` + +### Caching Python dependencies + +Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install +the python dependencies, the following example defines `cache` globally so that +all jobs inherit it. Python libraries are installed in a virtualenv under `venv/`, +pip's cache is defined under `.cache/pip/` and both are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Python.gitlab-ci.yml +# +image: python:latest + +# Change pip's cache directory to be inside the project directory since we can +# only cache local items. +variables: + PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache" + +# Pip's cache doesn't store the python packages +# https://pip.pypa.io/en/stable/reference/pip_install/#caching +# +# If you want to also cache the installed packages, you have to install +# them in a virtualenv and cache it as well. +cache: + paths: + - .cache/ + - venv/ + +before_script: + - python -V # Print out python version for debugging + - pip install virtualenv + - virtualenv venv + - source venv/bin/activate + +test: + script: + - python setup.py test + - pip install flake8 + - flake8 . +``` + +### Caching Ruby dependencies + +Assuming your project is using [Bundler](https://bundler.io) to install the +gem dependencies, the following example defines `cache` globally so that all +jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Ruby.gitlab-ci.yml +# +image: ruby:2.5 + +# Cache gems in between builds +cache: + key: ${CI_COMMIT_REF_SLUG} + paths: + - vendor/ruby + +before_script: + - ruby -v # Print out ruby version for debugging + - gem install bundler --no-ri --no-rdoc # Bundler is not installed with the image + - bundle install -j $(nproc) --path vendor # Install dependencies into ./vendor/ruby + +rspec: + script: + - rspec spec +``` + +## Availability of the cache + +Caching is an optimization, but isn't guaranteed to always work, so you need to +be prepared to regenerate any cached files in each job that needs them. + +Assuming you have properly [defined `cache` in `.gitlab-ci.yml`](../yaml/README.md#cache) +according to your workflow, the availability of the cache ultimately depends on +how the Runner has been configured (the executor type and whether different +Runners are used for passing the cache between jobs). + +### Where the caches are stored + +Since the Runner is the one responsible for storing the cache, it's essential +to know **where** it's stored. All the cache paths defined under a job in +`.gitlab-ci.yml` are archived in a single `cache.zip` file and stored in the +Runner's configured cache location. By default, they are stored locally in the +machine where the Runner is installed and depends on the type of the executor. + +| GitLab Runner executor | Default path of the cache | +| ---------------------- | ------------------------- | +| [Shell](https://docs.gitlab.com/runner/executors/shell.html) | Locally, stored under the `gitlab-runner` user's home directory: `/home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache.zip`. | +| [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | +| [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale Runners) | Behaves the same as the Docker executor. | + +### How archiving and extracting works + +In the most simple scenario, consider that you use only one machine where the +Runner is installed, and all jobs of your project run on the same host. + +Let's see the following example of two jobs that belong to two consecutive +stages: + +```yaml +stages: +- build +- test + +before_script: +- echo "Hello" + +job A: + stage: build + script: + - mkdir vendor/ + - echo "build" > vendor/hello.txt + cache: + key: build-cache + paths: + - vendor/ + after_script: + - echo "World" + +job B: + stage: test + script: + - cat vendor/hello.txt + cache: + key: build-cache +``` + +Here's what happens behind the scenes: + +1. Pipeline starts +1. `job A` runs +1. `before_script` is executed +1. `script` is executed +1. `after_script` is executed +1. `cache` runs and the `vendor/` directory is zipped into `cache.zip`. + This file is then saved in the directory based on the + [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. +1. `job B` runs +1. The cache is extracted (if found) +1. `before_script` is executed +1. `script` is executed +1. Pipeline finishes + +By using a single Runner on a single machine, you'll not have the issue where +`job B` might execute on a Runner different from `job A`, thus guaranteeing the +cache between stages. That will only work if the build goes from stage `build` +to `test` in the same Runner/machine, otherwise, you [might not have the cache +available](#cache-mismatch). + +During the caching process, there's also a couple of things to consider: + +- If some other job, with another cache configuration had saved its + cache in the same zip file, it is overwritten. If the S3 based shared cache is + used, the file is additionally uploaded to S3 to an object based on the cache + key. So, two jobs with different paths, but the same cache key, will overwrite + their cache. +- When extracting the cache from `cache.zip`, everything in the zip file is + extracted in the job's working directory (usually the repository which is + pulled down), and the Runner doesn't mind if the archive of `job A` overwrites + things in the archive of `job B`. + +The reason why it works this way is because the cache created for one Runner +often will not be valid when used by a different one which can run on a +**different architecture** (e.g., when the cache includes binary files). And +since the different steps might be executed by Runners running on different +machines, it is a safe default. + +### Cache mismatch + +In the following table, you can see some reasons where you might hit a cache +mismatch and a few ideas how to fix it. + +| Reason of a cache mismatch | How to fix it | +| -------------------------- | ------------- | +| You use multiple standalone Runners (not in autoscale mode) attached to one project without a shared cache | Use only one Runner for your project or use multiple Runners with distributed cache enabled | +| You use Runners in autoscale mode without a distributed cache enabled | Configure the autoscale Runner to use a distributed cache | +| The machine the Runner is installed on is low on disk space or, if you've set up distributed cache, the S3 bucket where the cache is stored doesn't have enough space | Make sure you clear some space to allow new caches to be stored. Currently, there's no automatic way to do this. | +| You use the same `key` for jobs where they cache different paths. | Use different cache keys to that the cache archive is stored to a different location and doesn't overwrite wrong caches. | + +Let's explore some examples. + +--- + +Let's assume you have only one Runner assigned to your project, so the cache +will be stored in the Runner's machine by default. If two jobs, A and B, +have the same cache key, but they cache different paths, cache B would overwrite +cache A, even if their `paths` don't match: + +We want `job A` and `job B` to re-use their +cache when the pipeline is run for a second time. + +```yaml +stages: +- build +- test + +job A: + stage: build + script: make build + cache: + key: same-key + paths: + - public/ + +job B: + stage: test + script: make test + cache: + key: same-key + paths: + - vendor/ +``` + +1. `job A` runs +1. `public/` is cached as cache.zip +1. `job B` runs +1. The previous cache, if any, is unzipped +1. `vendor/` is cached as cache.zip and overwrites the previous one +1. The next time `job A` runs it will use the cache of `job B` which is different + and thus will be ineffective + +To fix that, use different `keys` for each job. + +--- + +In another case, let's assume you have more than one Runners assigned to your +project, but the distributed cache is not enabled. We want the second time the +pipeline is run, `job A` and `job B` to re-use their cache (which in this case +will be different): + +```yaml +stages: +- build +- test + +job A: + stage: build + script: build + cache: + key: keyA + paths: + - vendor/ + +job B: + stage: test + script: test + cache: + key: keyB + paths: + - vendor/ +``` + +In that case, even if the `key` is different (no fear of overwriting), you +might experience the cached files to "get cleaned" before each stage if the +jobs run on different Runners in the subsequent pipelines. + +## Clearing the cache + +GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution +of your jobs by reusing existing data. This however, can sometimes lead to an +inconsistent behavior. + +To start with a fresh copy of the cache, there are two ways to do that. + +### Clearing the cache by changing `cache:key` + +All you have to do is set a new `cache: key` in your `.gitlab-ci.yml`. In the +next run of the pipeline, the cache will be stored in a different location. + +### Clearing the cache manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41249) in GitLab 10.4. + +If you want to avoid editing `.gitlab-ci.yml`, you can easily clear the cache +via GitLab's UI: + +1. Navigate to your project's **CI/CD > Pipelines** page +1. Click on the **Clear Runner caches** button to clean up the cache + +  + +1. On the next push, your CI/CD job will use a new cache + +Behind the scenes, this works by increasing a counter in the database, and the +value of that counter is used to create the key for the cache by appending an +integer to it: `-1`, `-2`, etc. After a push, a new key is generated and the +old cache is not valid anymore. + +## Cache vs artifacts + +NOTE: **Note:** +Be careful if you use cache and artifacts to store the same path in your jobs +as **caches are restored before artifacts** and the content would be overwritten. + +Don't mix the caching with passing artifacts between stages. Caching is not +designed to pass artifacts between stages. Cache is for runtime dependencies +needed to compile the project: + +- `cache` - **Use for temporary storage for project dependencies.** Not useful + for keeping intermediate build results, like `jar` or `apk` files. + Cache was designed to be used to speed up invocations of subsequent runs of a + given job, by keeping things like dependencies (e.g., npm packages, Go vendor + packages, etc.) so they don't have to be re-fetched from the public internet. + While the cache can be abused to pass intermediate build results between stages, + there may be cases where artifacts are a better fit. +- `artifacts` - **Use for stage results that will be passed between stages.** + Artifacts were designed to upload some compiled/generated bits of the build, + and they can be fetched by any number of concurrent Runners. They are + guaranteed to be available and are there to pass data between jobs. They are + also exposed to be downloaded from the UI. + +It's sometimes confusing because the name artifact sounds like something that +is only useful outside of the job, like for downloading a final image. But +artifacts are also available in between stages within a pipeline. So if you +build your application by downloading all the required modules, you might want +to declare them as artifacts so that each subsequent stage can depend on them +being there. There are some optimizations like declaring an +[expiry time](../yaml/README.md#artifacts-expire_in) so you don't keep artifacts +around too long, and using [dependencies](../yaml/README.md#dependencies) to +control exactly where artifacts are passed around. + +So, to sum up: +- Caches are disabled if not defined globally or per job (using `cache:`) +- Caches are available for all jobs in your `.gitlab-ci.yml` if enabled globally +- Caches can be used by subsequent pipelines of that very same job (a script in + a stage) in which the cache was created (if not defined globally). +- Caches are stored where the Runner is installed **and** uploaded to S3 if + [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) +- Caches defined per job are only used either a) for the next pipeline of that job, + or b) if that same cache is also defined in a subsequent job of the same pipeline +- Artifacts are disabled if not defined per job (using `artifacts:`) +- Artifacts can only be enabled per job, not globally +- Artifacts are created during a pipeline and can be used by the subsequent + jobs of that currently active pipeline +- Artifacts are always uploaded to GitLab (known as coordinator) +- Artifacts can have an expiration value for controlling disk usage (30 days by default) diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index fb5bfe26bb0..bc5d3840368 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -58,7 +58,7 @@ your job and is linked to the Docker image that the `image` keyword defines. This allows you to access the service image during build time. The service image can run any application, but the most common use case is to -run a database container, eg. `mysql`. It's easier and faster to use an +run a database container, e.g., `mysql`. It's easier and faster to use an existing image and run it as an additional container than install `mysql` every time the project is built. @@ -83,6 +83,67 @@ So, in order to access your database service you have to connect to the host named `mysql` instead of a socket or `localhost`. Read more in [accessing the services](#accessing-the-services). +### How the health check of services works + +Services are designed to provide additional functionality which is **network accessible**. +It may be a database like MySQL, or Redis, and even `docker:dind` which +allows you to use Docker in Docker. It can be practically anything that is +required for the CI/CD job to proceed and is accessed by network. + +To make sure this works, the Runner: + +1. checks which ports are exposed from the container by default +1. starts a special container that waits for these ports to be accessible + +When the second stage of the check fails, either because there is no opened port in the +service, or the service was not started properly before the timeout and the port is not +responding, it prints the warning: `*** WARNING: Service XYZ probably didn't start properly`. + +In most cases it will affect the job, but there may be situations when the job +will still succeed even if that warning was printed. For example: + +- The service was started a little after the warning was raised, and the job is + not using the linked service from the very beginning. In that case, when the + job needed to access the service, it may have been already there waiting for + connections. +- The service container is not providing any networking service, but it's doing + something with the job's directory (all services have the job directory mounted + as a volume under `/builds`). In that case, the service will do its job, and + since the job is not trying to connect to it, it won't fail. + +### What services are not for + +As it was mentioned before, this feature is designed to provide **network accessible** +services. A database is the simplest example of such a service. + +NOTE: **Note:** +The services feature is not designed to, and will not add any software from the +defined `services` image(s) to the job's container. + +For example, if you have the following `services` defined in your job, the `php`, +`node` or `go` commands will **not** be available for your script, and thus +the job will fail: + +```yaml +job: + services: + - php:7 + - node:latest + - golang:1.10 + image: alpine:3.7 + script: + - php -v + - node -v + - go version +``` + +If you need to have `php`, `node` and `go` available for your script, you should +either: + +- choose an existing Docker image that contains all required tools, or +- create your own Docker image, which will have all the required tools included + and use that in your job + ### Accessing the services Let's say that you need a Wordpress instance to test some API integration with diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index ffebe1618d3..c1e258aedca 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -32,34 +32,41 @@ There's also a collection of repositories with [example projects](https://gitlab - **Debian**: [Continuous Deployment with GitLab: how to build and deploy a Debian Package with GitLab CI](https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/) - **Maven**: [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md) +### Game development + +- [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md) + ### Miscellaneous - [Using `dpl` as deployment tool](deployment/README.md) - [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) -### Code quality analysis +## Code quality analysis [Analyze code quality with the Code Climate CLI](code_climate.md). -### Static Application Security Testing (SAST) +## Static Application Security Testing (SAST) + +**(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html) + +## Container Scanning -- **(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html) -- [Scan your Docker images for vulnerabilities](sast_docker.md) +[Scan your Docker images for vulnerabilities](container_scanning.md) -### Dynamic Application Security Testing (DAST) +## Dynamic Application Security Testing (DAST) Scan your app for vulnerabilities with GitLab [Dynamic Application Security Testing (DAST)](dast.md). -### Browser Performance Testing with Sitespeed.io +## Browser Performance Testing with Sitespeed.io Analyze your [browser performance with Sitespeed.io](browser_performance.md). -### GitLab CI/CD for Review Apps +## GitLab CI/CD for Review Apps - [Example project](https://gitlab.com/gitlab-examples/review-apps-nginx/) that shows how to use GitLab CI/CD for [Review Apps](../review_apps/index.html). - [Dockerizing GitLab Review Apps](https://about.gitlab.com/2017/07/11/dockerizing-review-apps/) -### GitLab CI/CD for GitLab Pages +## GitLab CI/CD for GitLab Pages See the documentation on [GitLab Pages](../../user/project/pages/index.md) for a complete overview. diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md index d7df53494ed..ec5e5afb8c6 100644 --- a/doc/ci/examples/code_climate.md +++ b/doc/ci/examples/code_climate.md @@ -16,18 +16,26 @@ codequality: - docker:dind script: - docker pull codeclimate/codeclimate - - docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 init - - docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 analyze -f json > codeclimate.json || true + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run + --env SOURCE_CODE="$PWD" \ + --volume "$PWD":/code \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code artifacts: paths: [codeclimate.json] ``` -This will create a `codequality` job in your CI pipeline and will allow you to -download and analyze the report artifact in JSON format. +The above example will create a `codequality` job in your CI/CD pipeline which +will scan your source code for code quality issues. The report will be saved +as an artifact that you can later download and analyze. -For [GitLab Starter][ee] users, this information can be automatically -extracted and shown right in the merge request widget. [Learn more on code quality -diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). +TIP: **Tip:** +Starting with [GitLab Starter][ee] 9.3, this information will +be automatically extracted and shown right in the merge request widget. To do +so, the CI/CD job must be named `codequality` and the artifact path must be +`codeclimate.json`. +[Learn more on code quality diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). [cli]: https://github.com/codeclimate/codeclimate [dind]: ../docker/using_docker_build.md#use-docker-in-docker-executor diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md new file mode 100644 index 00000000000..3437b63748a --- /dev/null +++ b/doc/ci/examples/container_scanning.md @@ -0,0 +1,55 @@ +# Container Scanning with GitLab CI/CD + +You can check your Docker images (or more precisely the containers) for known +vulnerabilities by using [Clair](https://github.com/coreos/clair) and +[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools +for Vulnerability Static Analysis for containers. + +All you need is a GitLab Runner with the Docker executor (the shared Runners on +GitLab.com will work fine). You can then add a new job to `.gitlab-ci.yml`, +called `sast:container`: + +```yaml +sast:container: + image: docker:latest + variables: + DOCKER_DRIVER: overlay2 + ## Define two new variables based on GitLab's CI/CD predefined variables + ## https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG + CI_APPLICATION_TAG: $CI_COMMIT_SHA + allow_failure: true + services: + - docker:dind + script: + - docker run -d --name db arminc/clair-db:latest + - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 + - apk add -U wget ca-certificates + - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} + - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 + - mv clair-scanner_linux_amd64 clair-scanner + - chmod +x clair-scanner + - touch clair-whitelist.yml + - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-sast-container-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true + artifacts: + paths: [gl-sast-container-report.json] +``` + +The above example will create a `sast:container` job in your CI/CD pipeline, pull +the image from the [Container Registry](../../user/project/container_registry.md) +(whose name is defined from the two `CI_APPLICATION_` variables) and scan it +for possible vulnerabilities. The report will be saved as an artifact that you +can later download and analyze. + +If you want to whitelist some specific vulnerabilities, you can do so by defining +them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), +in our case its named `clair-whitelist.yml`. + +TIP: **Tip:** +Starting with [GitLab Ultimate][ee] 10.4, this information will +be automatically extracted and shown right in the merge request widget. To do +so, the CI/CD job must be named `sast:container` and the artifact path must be +`gl-sast-container-report.json`. +[Learn more on container scanning results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). + +[ee]: https://about.gitlab.com/products/ diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png Binary files differnew file mode 100644 index 00000000000..76e0295722b --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png Binary files differnew file mode 100644 index 00000000000..050a97d2726 --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png Binary files differnew file mode 100644 index 00000000000..4ab5d5f401a --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md new file mode 100644 index 00000000000..bfc8558a580 --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -0,0 +1,526 @@ +--- +author: Ryan Hall +author_gitlab: blitzgren +level: intermediary +article_type: tutorial +date: 2018-03-07 +--- + +# DevOps and Game Dev with GitLab CI/CD + +With advances in WebGL and WebSockets, browsers are extremely viable as game development +platforms without the use of plugins like Adobe Flash. Furthermore, by using GitLab and [AWS](https://aws.amazon.com/), +single game developers, as well as game dev teams, can easily host browser-based games online. + +In this tutorial, we'll focus on DevOps, as well as testing and hosting games with Continuous +Integration/Deployment methods. We assume you are familiar with GitLab, javascript, +and the basics of game development. + +## The game + +Our [demo game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) consists of a simple spaceship traveling in space that shoots by clicking the mouse in a given direction. + +Creating a strong CI/CD pipeline at the beginning of developing another game, [Dark Nova](http://darknova.io/about), +was essential for the fast pace the team worked at. This tutorial will build upon my +[previous introductory article](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) and go through the following steps: + +1. Using code from the previous article to start with a barebones [Phaser](https://phaser.io) game built by a gulp file +1. Adding and running unit tests +1. Creating a `Weapon` class that can be triggered to spawn a `Bullet` in a given direction +1. Adding a `Player` class that uses this weapon and moves around the screen +1. Adding the sprites we will use for the `Player` and `Weapon` +1. Testing and deploying with Continuous Integration and Continuous Deployment methods + +By the end, we'll have the core of a [playable game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) +that's tested and deployed on every push to the `master` branch of the [codebase](https://gitlab.com/blitzgren/gitlab-game-demo). +This will also provide +boilerplate code for starting a browser-based game with the following components: + +- Written in [Typescript](https://www.typescriptlang.org/) and [PhaserJs](https://phaser.io) +- Building, running, and testing with [Gulp](http://gulpjs.com/) +- Unit tests with [Chai](http://chaijs.com/) and [Mocha](https://mochajs.org/) +- CI/CD with GitLab +- Hosting the codebase on GitLab.com +- Hosting the game on AWS +- Deploying to AWS + +## Requirements and setup + +Please refer to my previous article [DevOps and Game Dev](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) to learn the foundational +development tools, running a Hello World-like game, and building this game using GitLab +CI/CD from every new push to master. The `master` branch for this game's [repository](https://gitlab.com/blitzgren/gitlab-game-demo) +contains a completed version with all configurations. If you would like to follow along +with this article, you can clone and work from the `devops-article` branch: + +```sh +git clone git@gitlab.com:blitzgren/gitlab-game-demo.git +git checkout devops-article +``` + +Next, we'll create a small subset of tests that exemplify most of the states I expect +this `Weapon` class to go through. To get started, create a folder called `lib/tests` +and add the following code to a new file `weaponTests.ts`: + +```ts +import { expect } from 'chai'; +import { Weapon, BulletFactory } from '../lib/weapon'; + +describe('Weapon', () => { + var subject: Weapon; + var shotsFired: number = 0; + // Mocked bullet factory + var bulletFactory: BulletFactory = <BulletFactory>{ + generate: function(px, py, vx, vy, rot) { + shotsFired++; + } + }; + var parent: any = { x: 0, y: 0 }; + + beforeEach(() => { + shotsFired = 0; + subject = new Weapon(bulletFactory, parent, 0.25, 1); + }); + + it('should shoot if not in cooldown', () => { + subject.trigger(true); + subject.update(0.1); + expect(shotsFired).to.equal(1); + }); + + it('should not shoot during cooldown', () => { + subject.trigger(true); + subject.update(0.1); + subject.update(0.1); + expect(shotsFired).to.equal(1); + }); + + it('should shoot after cooldown ends', () => { + subject.trigger(true); + subject.update(0.1); + subject.update(0.3); // longer than timeout + expect(shotsFired).to.equal(2); + }); + + it('should not shoot if not triggered', () => { + subject.update(0.1); + subject.update(0.1); + expect(shotsFired).to.equal(0); + }); +}); +``` + +To build and run these tests using gulp, let's also add the following gulp functions +to the existing `gulpfile.js` file: + +```ts +gulp.task('build-test', function () { + return gulp.src('src/tests/**/*.ts', { read: false }) + .pipe(tap(function (file) { + // replace file contents with browserify's bundle stream + file.contents = browserify(file.path, { debug: true }) + .plugin(tsify, { project: "./tsconfig.test.json" }) + .bundle(); + })) + .pipe(buffer()) + .pipe(sourcemaps.init({loadMaps: true}) ) + .pipe(gulp.dest('built/tests')); +}); + +gulp.task('run-test', function() { + gulp.src(['./built/tests/**/*.ts']).pipe(mocha()); +}); +``` + +We will start implementing the first part of our game and get these `Weapon` tests to pass. +The `Weapon` class will expose a method to trigger the generation of a bullet at a given +direction and speed. Later we will implement a `Player` class that ties together the user input +to trigger the weapon. In the `src/lib` folder create a `weapon.ts` file. We'll add two classes +to it: `Weapon` and `BulletFactory` which will encapsulate Phaser's **sprite** and +**group** objects, and the logic specific to our game. + +```ts +export class Weapon { + private isTriggered: boolean = false; + private currentTimer: number = 0; + + constructor(private bulletFactory: BulletFactory, private parent: Phaser.Sprite, private cooldown: number, private bulletSpeed: number) { + } + + public trigger(on: boolean): void { + this.isTriggered = on; + } + + public update(delta: number): void { + this.currentTimer -= delta; + + if (this.isTriggered && this.currentTimer <= 0) { + this.shoot(); + } + } + + private shoot(): void { + // Reset timer + this.currentTimer = this.cooldown; + + // Get velocity direction from player rotation + var parentRotation = this.parent.rotation + Math.PI / 2; + var velx = Math.cos(parentRotation); + var vely = Math.sin(parentRotation); + + // Apply a small forward offset so bullet shoots from head of ship instead of the middle + var posx = this.parent.x - velx * 10 + var posy = this.parent.y - vely * 10; + + this.bulletFactory.generate(posx, posy, -velx * this.bulletSpeed, -vely * this.bulletSpeed, this.parent.rotation); + } +} + +export class BulletFactory { + + constructor(private bullets: Phaser.Group, private poolSize: number) { + // Set all the defaults for this BulletFactory's bullet object + this.bullets.enableBody = true; + this.bullets.physicsBodyType = Phaser.Physics.ARCADE; + this.bullets.createMultiple(30, 'bullet'); + this.bullets.setAll('anchor.x', 0.5); + this.bullets.setAll('anchor.y', 0.5); + this.bullets.setAll('outOfBoundsKill', true); + this.bullets.setAll('checkWorldBounds', true); + } + + public generate(posx: number, posy: number, velx: number, vely: number, rot: number): Phaser.Sprite { + // Pull a bullet from Phaser's Group pool + var bullet = this.bullets.getFirstExists(false); + + // Set the few unique properties about this bullet: rotation, position, and velocity + if (bullet) { + bullet.reset(posx, posy); + bullet.rotation = rot; + bullet.body.velocity.x = velx; + bullet.body.velocity.y = vely; + } + + return bullet; + } +} +``` + +Lastly, we'll redo our entry point, `game.ts`, to tie together both `Player` and `Weapon` objects +as well as add them to the update loop. Here is what the updated `game.ts` file looks like: + +```ts +import { Player } from "./player"; +import { Weapon, BulletFactory } from "./weapon"; + +window.onload = function() { + var game = new Phaser.Game(800, 600, Phaser.AUTO, 'gameCanvas', { preload: preload, create: create, update: update }); + var player: Player; + var weapon: Weapon; + + // Import all assets prior to loading the game + function preload () { + game.load.image('player', 'assets/player.png'); + game.load.image('bullet', 'assets/bullet.png'); + } + + // Create all entities in the game, after Phaser loads + function create () { + // Create and position the player + var playerSprite = game.add.sprite(400, 550, 'player'); + playerSprite.anchor.setTo(0.5); + player = new Player(game.input, playerSprite, 150); + + var bulletFactory = new BulletFactory(game.add.group(), 30); + weapon = new Weapon(bulletFactory, player.sprite, 0.25, 1000); + + player.loadWeapon(weapon); + } + + // This function is called once every tick, default is 60fps + function update() { + var deltaSeconds = game.time.elapsedMS / 1000; // convert to seconds + player.update(deltaSeconds); + weapon.update(deltaSeconds); + } +} +``` + +Run `gulp serve` and you can run around and shoot. Wonderful! Let's update our CI +pipeline to include running the tests along with the existing build job. + +## Continuous Integration + +To ensure our changes don't break the build and all tests still pass, we utilize +Continuous Integration (CI) to run these checks automatically for every push. +Read through this article to understand [Continuous Integration, Continuous Delivery, and Continuous Deployment](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/), +and how these methods are leveraged by GitLab. +From the [last tutorial](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) we already have a `gitlab-ci.yml` file set up for building our app from +every push. We need to set up a new CI job for testing, which GitLab CI/CD will run after the build job using our generated artifacts from gulp. + +Please read through the [documentation on CI/CD configuration](../../../ci/yaml/README.md) file to explore its contents and adjust it to your needs. + +### Build your game with GitLab CI/CD + +We need to update our build job to ensure tests get run as well. Add `gulp build-test` +to the end of the `script` array for the existing `build` job. Once these commands run, +we know we will need to access everything in the `built` folder, given by GitLab CI/CD's `artifacts`. +We'll also cache `node_modules` to avoid having to do a full re-pull of those dependencies: +just pack them up in the cache. Here is the full `build` job: + +```yml +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules + artifacts: + paths: + - built +``` + +### Test your game with GitLab CI/CD + +For testing locally, we simply run `gulp run-tests`, which requires gulp to be installed +globally like in the `build` job. We pull `node_modules` from the cache, so the `npm i` +command won't have to do much. In preparation for deployment, we know we will still need +the `built` folder in the artifacts, which will be brought over as default behavior from +the previous job. Lastly, by convention, we let GitLab CI/CD know this needs to be run after +the `build` job by giving it a `test` [stage](../../../ci/yaml/README.md#stages). +Following the YAML structure, the `test` job should look like this: + +```yml +test: + stage: test + script: + - npm i gulp -g + - npm i + - gulp run-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ +``` + +We have added unit tests for a `Weapon` class that shoots on a specified interval. +The `Player` class implements `Weapon` along with the ability to move around and shoot. Also, +we've added test artifacts and a test stage to our GitLab CI/CD pipeline using `.gitlab-ci.yml`, +allowing us to run our tests by every push. +Our entire `.gitlab-ci.yml` file should now look like this: + +```yml +image: node:6 + +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ + +test: + stage: test + script: + - npm i gulp -g + - npm i + - gulp run-test + cache: + policy: pull + paths: + - node_modules/ + artifacts: + paths: + - built/ +``` + +### Run your CI/CD pipeline + +That's it! Add all your new files, commit, and push. For a reference of what our repo should +look like at this point, please refer to the [final commit related to this article on my sample repository](https://gitlab.com/blitzgren/gitlab-game-demo/commit/8b36ef0ecebcf569aeb251be4ee13743337fcfe2). +By applying both build and test stages, GitLab will run them sequentially at every push to +our repository. If all goes well you'll end up with a green check mark on each job for the pipeline: + + + +You can confirm that the tests passed by clicking on the `test` job to enter the full build logs. +Scroll to the bottom and observe, in all its passing glory: + +```sh +$ gulp run-test +[18:37:24] Using gulpfile /builds/blitzgren/gitlab-game-demo/gulpfile.js +[18:37:24] Starting 'run-test'... +[18:37:24] Finished 'run-test' after 21 ms + + + Weapon + ✓ should shoot if not in cooldown + ✓ should not shoot during cooldown + ✓ should shoot after cooldown ends + ✓ should not shoot if not triggered + + + 4 passing (18ms) + +Uploading artifacts... +built/: found 17 matching files +Uploading artifacts to coordinator... ok id=17095874 responseStatus=201 Created token=aaaaaaaa Job succeeded +``` + +## Continuous Deployment + +We have our codebase built and tested on every push. To complete the full pipeline with Continuous Deployment, +let's set up [free web hosting with AWS S3](https://aws.amazon.com/s/dm/optimization/server-side-test/free-tier/free_np/) and a job through which our build artifacts get +deployed. GitLab also has a free static site hosting service we could use, [GitLab Pages](https://about.gitlab.com/features/pages/), +however Dark Nova specifically uses other AWS tools that necessitates using `AWS S3`. +Read through this article that describes [deploying to both S3 and GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) +and further delves into the principles of GitLab CI/CD than discussed in this article. + +### Set up S3 Bucket + +1. Log into your AWS account and go to [S3](https://console.aws.amazon.com/s3/home) +1. Click the **Create Bucket** link at the top +1. Enter a name of your choosing and click next +1. Keep the default **Properties** and click next +1. Click the **Manage group permissions** and allow **Read** for the **Everyone** group, click next +1. Create the bucket, and select it in your S3 bucket list +1. On the right side, click **Properties** and enable the **Static website hosting** category +1. Update the radio button to the **Use this bucket to host a website** selection. Fill in `index.html` and `error.html` respectively + +### Set up AWS Secrets + +We need to be able to deploy to AWS with our AWS account credentials, but we certainly +don't want to put secrets into source code. Luckily GitLab provides a solution for this +with [Secret Variables](../../../ci/variables/README.md). This can get complicated +due to [IAM](https://aws.amazon.com/iam/) management. As a best practice, you shouldn't +use root security credentials. Proper IAM credential management is beyond the scope of this +article, but AWS will remind you that using root credentials is unadvised and against their +best practices, as they should. Feel free to follow best practices and use a custom IAM user's +credentials, which will be the same two credentials (Key ID and Secret). It's a good idea to +fully understand [IAM Best Practices in AWS](http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). We need to add these credentials to GitLab: + +1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) +1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later +  +1. Go to your GitLab project, click **Settings > CI/CD** on the left sidebar +1. Expand the **Secret Variables** section +  +1. Add a key named `AWS_KEY_ID` and copy the key id from Step 2 into the **Value** textbox +1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox + +### Deploy your game with GitLab CI/CD + +To deploy our build artifacts, we need to install the [AWS CLI](https://aws.amazon.com/cli/) on +the Shared Runner. The Shared Runner also needs to be able to authenticate with your AWS +account to deploy the artifacts. By convention, AWS CLI will look for `AWS_ACCESS_KEY_ID` +and `AWS_SECRET_ACCESS_KEY`. GitLab's CI gives us a way to pass the secret variables we +set up in the prior section using the `variables` portion of the `deploy` job. At the end, +we add directives to ensure deployment `only` happens on pushes to `master`. This way, every +single branch still runs through CI, and only merging (or committing directly) to master will +trigger the `deploy` job of our pipeline. Put these together to get the following: + +```yml +deploy: + stage: deploy + variables: + AWS_ACCESS_KEY_ID: "$AWS_KEY_ID" + AWS_SECRET_ACCESS_KEY: "$AWS_KEY_SECRET" + script: + - apt-get update + - apt-get install -y python3-dev python3-pip + - easy_install3 -U pip + - pip3 install --upgrade awscli + - aws s3 sync ./built s3://gitlab-game-demo --region "us-east-1" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers --cache-control "no-cache, no-store, must-revalidate" --delete + only: + - master +``` + +Be sure to update the region and S3 URL in that last script command to fit your setup. +Our final configuration file `.gitlab-ci.yml` looks like: + +```yml +image: node:6 + +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ + +test: + stage: test + script: + - npm i gulp -g + - gulp run-test + cache: + policy: pull + paths: + - node_modules/ + artifacts: + paths: + - built/ + +deploy: + stage: deploy + variables: + AWS_ACCESS_KEY_ID: "$AWS_KEY_ID" + AWS_SECRET_ACCESS_KEY: "$AWS_KEY_SECRET" + script: + - apt-get update + - apt-get install -y python3-dev python3-pip + - easy_install3 -U pip + - pip3 install --upgrade awscli + - aws s3 sync ./built s3://gitlab-game-demo --region "us-east-1" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers --cache-control "no-cache, no-store, must-revalidate" --delete + only: + - master +``` + +## Conclusion + +Within the [demo repository](https://gitlab.com/blitzgren/gitlab-game-demo) you can also find a handful of boilerplate code to get +[Typescript](https://www.typescriptlang.org/), [Mocha](https://mochajs.org/), [Gulp](http://gulpjs.com/) and [Phaser](https://phaser.io) all playing +together nicely with GitLab CI/CD, which is the result of lessons learned while making [Dark Nova](http://darknova.io/). +Using a combination of free and open source software, we have a full CI/CD pipeline, a game foundation, +and unit tests, all running and deployed at every push to master - with shockingly little code. +Errors can be easily debugged through GitLab's build logs, and within minutes of a successful commit, +you can see the changes live on your game. + +Setting up Continous Integration and Continuous Deployment from the start with Dark Nova enables +rapid but stable development. We can easily test changes in a separate [environment](../../../ci/environments.md#introduction-to-environments-and-deployments), +or multiple environments if needed. Balancing and updating a multiplayer game can be ongoing +and tedious, but having faith in a stable deployment with GitLab CI/CD allows +a lot of breathing room in quickly getting changes to players. + +## Further settings + +Here are some ideas to further investigate that can speed up or improve your pipeline: + +- [Yarn](https://yarnpkg.com) instead of npm +- Setup a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ci-yml) image that can preload dependencies and tools (like AWS CLI) +- Forward a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to your game's S3 static website +- Combine jobs if you find it unnecessary for a small project +- Avoid the queues and set up your own [custom GitLab CI/CD runner](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md index 57a9c4bcfc1..9f4a63e296d 100644 --- a/doc/ci/examples/sast_docker.md +++ b/doc/ci/examples/sast_docker.md @@ -1,55 +1 @@ -# Static Application Security Testing for Docker containers with GitLab CI/CD - -You can check your Docker images (or more precisely the containers) for known -vulnerabilities by using [Clair](https://github.com/coreos/clair) and -[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools -for Vulnerability Static Analysis for containers. - -All you need is a GitLab Runner with the Docker executor (the shared Runners on -GitLab.com will work fine). You can then add a new job to `.gitlab-ci.yml`, -called `sast:container`: - -```yaml -sast:container: - image: docker:latest - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-sast-container-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - paths: [gl-sast-container-report.json] -``` - -The above example will create a `sast:container` job in your CI/CD pipeline, pull -the image from the [Container Registry](../../user/project/container_registry.md) -(whose name is defined from the two `CI_APPLICATION_` variables) and scan it -for possible vulnerabilities. The report will be saved as an artifact that you -can later download and analyze. - -If you want to whitelist some specific vulnerabilities, you can do so by defining -them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), -in our case its named `clair-whitelist.yml`. - -TIP: **Tip:** -Starting with [GitLab Ultimate][ee] 10.4, this information will -be automatically extracted and shown right in the merge request widget. To do -so, the CI/CD job must be named `sast:container` and the artifact path must be -`gl-sast-container-report.json`. -[Learn more on application security testing results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/sast_docker.html). - -[ee]: https://about.gitlab.com/products/ +This document was moved to [another location](./container_scanning.md).
\ No newline at end of file diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index f879ed62010..7a7b50b294d 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -146,24 +146,7 @@ To protect/unprotect Runners: ## Manually clearing the Runners cache -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41249) in GitLab 10.4. - -GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution -of your jobs by reusing existing data. This however, can sometimes lead to an -inconsistent behavior. - -To start with a fresh copy of the cache, you can easily do it via GitLab's UI: - -1. Navigate to your project's **CI/CD > Pipelines** page. -1. Click on the **Clear Runner caches** to clean up the cache. -1. On the next push, your CI/CD job will use a new cache. - -That way, you don't have to change the [cache key](../yaml/README.md#cache-key) -in your `.gitlab-ci.yml`. - -Behind the scenes, this works by increasing a counter in the database, and the -value of that counter is used to create the key for the cache. After a push, a -new key is generated and the old cache is not valid anymore. +Read [clearing the cache](../caching/index.md#clearing-the-cache). ## How shared Runners pick jobs @@ -227,15 +210,16 @@ that it may encounter on the projects it's shared over. This would be problematic for large amounts of projects, if it wasn't for tags. By tagging a Runner for the types of jobs it can handle, you can make sure -shared Runners will only run the jobs they are equipped to run. +shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). For instance, at GitLab we have Runners tagged with "rails" if they contain the appropriate dependencies to run Rails test suites. ### Preventing Runners with tags from picking jobs without tags -You can configure a Runner to prevent it from picking jobs with tags when -the Runner does not have tags assigned. This setting can be enabled the first +You can configure a Runner to prevent it from picking +[jobs with tags](../yaml/README.md#tags) when the Runner does not have tags +assigned. This setting can be enabled the first time you [register a Runner][register] and can be changed afterwards under each Runner's settings. @@ -280,3 +264,36 @@ We're always looking for contributions that can mitigate these [register]: http://docs.gitlab.com/runner/register/ [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md + +## Determining the IP address of a Runner + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17286) in GitLab 10.6. + +It may be useful to know the IP address of a Runner so you can troubleshoot +issues with that Runner. GitLab stores and displays the IP address by viewing +the source of the HTTP requests it makes to GitLab when polling for jobs. The +IP address is always kept up to date so if the Runner IP changes it will be +automatically updated in GitLab. + +The IP address for shared Runners and specific Runners can be found in +different places. + +### Shared Runners + +To view the IP address of a shared Runner you must have admin access to +the GitLab instance. To determine this: + +1. Visit **Admin area ➔ Overview ➔ Runners** +1. Look for the Runner in the table and you should see a column for "IP Address" + + + +### Specific Runners + +You can find the IP address of a Runner for a specific project by: + +1. Visit your project's **Settings ➔ CI/CD** +1. Find the Runner and click on it's ID which links you to the details page +1. On the details page you should see a row for "IP Address" + + diff --git a/doc/ci/runners/img/shared_runner_ip_address.png b/doc/ci/runners/img/shared_runner_ip_address.png Binary files differnew file mode 100644 index 00000000000..3b1542d59d3 --- /dev/null +++ b/doc/ci/runners/img/shared_runner_ip_address.png diff --git a/doc/ci/runners/img/specific_runner_ip_address.png b/doc/ci/runners/img/specific_runner_ip_address.png Binary files differnew file mode 100644 index 00000000000..3b4c3e9f2eb --- /dev/null +++ b/doc/ci/runners/img/specific_runner_ip_address.png diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 23ce6a5f210..bd4aeb006bd 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -12,7 +12,7 @@ this order: 1. [Trigger variables][triggers] or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables) (take precedence over all) 1. Project-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) 1. Group-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) -1. YAML-defined [job-level variables](../yaml/README.md#job-variables) +1. YAML-defined [job-level variables](../yaml/README.md#variables) 1. YAML-defined [global variables](../yaml/README.md#variables) 1. [Deployment variables](#deployment-variables) 1. [Predefined variables](#predefined-variables-environment-variables) (are the diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 80ab63468f2..7184f3367be 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -3,18 +3,19 @@ This document describes the usage of `.gitlab-ci.yml`, the file that is used by GitLab Runner to manage your project's jobs. -If you want a quick introduction to GitLab CI, follow our -[quick start guide](../quick_start/README.md). - -## .gitlab-ci.yml - From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root of your repository and contains definitions of how your project should be built. +If you want a quick introduction to GitLab CI, follow our +[quick start guide](../quick_start/README.md). + +## Jobs + The YAML file defines a set of jobs with constraints stating when they should -be run. The jobs are defined as top-level elements with a name and always have -to contain at least the `script` clause: +be run. You can specify an unlimited number of jobs which are defined as +top-level elements with an arbitrary name and always have to contain at least +the `script` clause. ```yaml job1: @@ -24,9 +25,8 @@ job2: script: "execute-script-for-job2" ``` -The above example is the simplest possible CI configuration with two separate +The above example is the simplest possible CI/CD configuration with two separate jobs, where each of the jobs executes a different command. - Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. @@ -34,78 +34,115 @@ Jobs are picked up by [Runners](../runners/README.md) and executed within the environment of the Runner. What is important, is that each job is run independently from each other. -The YAML syntax allows for using more complex job specifications than in the -above example: +Each job must have a unique name, but there are a few **reserved `keywords` that +cannot be used as job names**: -```yaml -image: ruby:2.1 -services: - - postgres +- `image` +- `services` +- `stages` +- `types` +- `before_script` +- `after_script` +- `variables` +- `cache` -before_script: - - bundle install +A job is defined by a list of parameters that define the job behavior. -after_script: - - rm secrets +| Keyword | Required | Description | +|---------------|----------|-------------| +| script | yes | Defines a shell script which is executed by Runner | +| image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | +| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | +| stage | no | Defines a job stage (default: `test`) | +| type | no | Alias for `stage` | +| variables | no | Define job variables on a job level | +| only | no | Defines a list of git refs for which job is created | +| except | no | Defines a list of git refs for which job is not created | +| tags | no | Defines a list of tags which are used to select Runner | +| allow_failure | no | Allow job to fail. Failed job doesn't contribute to commit status | +| when | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | +| dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them| +| artifacts | no | Define list of [job artifacts](#artifacts) | +| cache | no | Define list of files that should be cached between subsequent runs | +| before_script | no | Override a set of commands that are executed before job | +| after_script | no | Override a set of commands that are executed after job | +| environment | no | Defines a name of environment to which deployment is done by this job | +| coverage | no | Define code coverage settings for a given job | +| retry | no | Define how many times a job can be auto-retried in case of a failure | -stages: - - build - - test - - deploy +### `pages` -job1: - stage: build +`pages` is a special job that is used to upload static content to GitLab that +can be used to serve your website. It has a special syntax, so the two +requirements below must be met: + +1. Any static content must be placed under a `public/` directory +1. `artifacts` with a path to the `public/` directory must be defined + +The example below simply moves all files from the root of the project to the +`public/` directory. The `.public` workaround is so `cp` doesn't also copy +`public/` to itself in an infinite loop: + +``` +pages: + stage: deploy script: - - execute-script-for-job1 + - mkdir .public + - cp -r * .public + - mv .public public + artifacts: + paths: + - public only: - - master - tags: - - docker + - master ``` -There are a few reserved `keywords` that **cannot** be used as job names: - -| Keyword | Required | Description | -|---------------|----------|-------------| -| image | no | Use docker image, covered in [Use Docker](../docker/README.md) | -| services | no | Use docker services, covered in [Use Docker](../docker/README.md) | -| stages | no | Define build stages | -| types | no | Alias for `stages` (deprecated) | -| before_script | no | Define commands that run before each job's script | -| after_script | no | Define commands that run after each job's script | -| variables | no | Define build variables | -| cache | no | Define list of files that should be cached between subsequent runs | +Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). -### image and services +## `image` and `services` This allows to specify a custom Docker image and a list of services that can be used for time of the job. The configuration of this feature is covered in [a separate document](../docker/README.md). -### before_script - -`before_script` is used to define the command that should be run before all -jobs, including deploy jobs, but after the restoration of artifacts. This can -be an array or a multi-line string. - -### after_script +## `before_script` and `after_script` > Introduced in GitLab 8.7 and requires Gitlab Runner v1.2 +`before_script` is used to define the command that should be run before all +jobs, including deploy jobs, but after the restoration of [artifacts](#artifacts). +This can be an array or a multi-line string. + `after_script` is used to define the command that will be run after for all jobs, including failed ones. This has to be an array or a multi-line string. -> **Note:** The `before_script` and the main `script` are concatenated and run in a single context/container. The `after_script` is run separately, so depending on the executor, changes done outside of the working tree might not be visible, e.g. software installed in the `before_script`. -### stages +It's possible to overwrite the globally defined `before_script` and `after_script` +if you set it per-job: -`stages` is used to define stages that can be used by jobs. -The specification of `stages` allows for having flexible multi stage pipelines. +```yaml +before_script: +- global before script + +job: + before_script: + - execute this instead of global before script + script: + - my command + after_script: + - execute this after my script +``` + +## `stages` + +`stages` is used to define stages that can be used by jobs and is defined +globally. +The specification of `stages` allows for having flexible multi stage pipelines. The ordering of elements in `stages` defines the ordering of jobs' execution: 1. Jobs of the same stage are run in parallel. @@ -134,280 +171,45 @@ There are also two edge cases worth mentioning: `test` and `deploy` are allowed to be used as job's stage by default. 2. If a job doesn't specify a `stage`, the job is assigned the `test` stage. -### types - -> Deprecated, and could be removed in one of the future releases. Use [stages](#stages) instead. +## `stage` -Alias for [stages](#stages). - -### variables - -> Introduced in GitLab Runner v0.5.0. - -GitLab CI allows you to add variables to `.gitlab-ci.yml` that are set in the -job environment. The variables are stored in the Git repository and are meant -to store non-sensitive project configuration, for example: - -```yaml -variables: - DATABASE_URL: "postgres://postgres@postgres/my_database" -``` - ->**Note:** -Integers (as well as strings) are legal both for variable's name and value. -Floats are not legal and cannot be used. - -These variables can be later used in all executed commands and scripts. -The YAML-defined variables are also set to all created service containers, -thus allowing to fine tune them. Variables can be also defined on a -[job level](#job-variables). - -Except for the user defined variables, there are also the ones set up by the -Runner itself. One example would be `CI_COMMIT_REF_NAME` which has the value of -the branch or tag name for which project is built. Apart from the variables -you can set in `.gitlab-ci.yml`, there are also the so called secret variables -which can be set in GitLab's UI. - -[Learn more about variables.][variables] - -### cache - -> -**Notes:** -- Introduced in GitLab Runner v0.7.0. -- Prior to GitLab 9.2, caches were restored after artifacts. -- From GitLab 9.2, caches are restored before artifacts. - -`cache` is used to specify a list of files and directories which should be -cached between jobs. You can only use paths that are within the project -workspace. - -**By default caching is enabled and shared between pipelines and jobs, -starting from GitLab 9.0** - -If `cache` is defined outside the scope of jobs, it means it is set -globally and all jobs will use that definition. - -Cache all files in `binaries` and `.config`: - -```yaml -rspec: - script: test - cache: - paths: - - binaries/ - - .config -``` - -Cache all Git untracked files: - -```yaml -rspec: - script: test - cache: - untracked: true -``` - -Cache all Git untracked files and files in `binaries`: - -```yaml -rspec: - script: test - cache: - untracked: true - paths: - - binaries/ -``` - -Locally defined cache overrides globally defined options. The following `rspec` -job will cache only `binaries/`: - -```yaml -cache: - paths: - - my/files - -rspec: - script: test - cache: - key: rspec - paths: - - binaries/ -``` - -Note that since cache is shared between jobs, if you're using different -paths for different jobs, you should also set a different **cache:key** -otherwise cache content can be overwritten. - -The cache is provided on a best-effort basis, so don't expect that the cache -will be always present. For implementation details, please check GitLab Runner. - -#### cache:key - -> Introduced in GitLab Runner v1.0.0. - -The `key` directive allows you to define the affinity of caching -between jobs, allowing to have a single cache for all jobs, -cache per-job, cache per-branch or any other way you deem proper. - -This allows you to fine tune caching, allowing you to cache data between -different jobs or even different branches. - -The `cache:key` variable can use any of the [predefined variables](../variables/README.md). - -The default key is **default** across the project, therefore everything is -shared between each pipelines and jobs by default, starting from GitLab 9.0. - ->**Note:** The `cache:key` variable cannot contain the `/` character, or the equivalent URI encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. - ---- - -**Example configurations** - -To enable per-job caching: - -```yaml -cache: - key: "$CI_JOB_NAME" - untracked: true -``` - -To enable per-branch caching: - -```yaml -cache: - key: "$CI_COMMIT_REF_SLUG" - untracked: true -``` - -To enable per-job and per-branch caching: - -```yaml -cache: - key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" - untracked: true -``` - -To enable per-branch and per-stage caching: - -```yaml -cache: - key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" - untracked: true -``` - -If you use **Windows Batch** to run your shell scripts you need to replace -`$` with `%`: - -```yaml -cache: - key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%" - untracked: true -``` - -If you use **Windows PowerShell** to run your shell scripts you need to replace -`$` with `$env:`: - -```yaml -cache: - key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG" - untracked: true -``` - -### cache:policy - -> Introduced in GitLab 9.4. - -The default behaviour of a caching job is to download the files at the start of -execution, and to re-upload them at the end. This allows any changes made by the -job to be persisted for future runs, and is known as the `pull-push` cache -policy. - -If you know the job doesn't alter the cached files, you can skip the upload step -by setting `policy: pull` in the job specification. Typically, this would be -twinned with an ordinary cache job at an earlier stage to ensure the cache -is updated from time to time: +`stage` is defined per-job and relies on [`stages`](#stages) which is defined +globally. It allows to group jobs into different stages, and jobs of the same +`stage` are executed in `parallel`. For example: ```yaml stages: - - setup + - build - test + - deploy -prepare: - stage: setup - cache: - key: gems - paths: - - vendor/bundle - script: - - bundle install --deployment - -rspec: - stage: test - cache: - key: gems - paths: - - vendor/bundle - policy: pull - script: - - bundle exec rspec ... -``` - -This helps to speed up job execution and reduce load on the cache server, -especially when you have a large number of cache-using jobs executing in -parallel. - -Additionally, if you have a job that unconditionally recreates the cache without -reference to its previous contents, you can use `policy: push` in that job to -skip the download step. - -## Jobs +job 1: + stage: build + script: make build dependencies -`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job -must have a unique name, which is not one of the keywords mentioned above. -A job is defined by a list of parameters that define the job behavior. +job 2: + stage: build + script: make build artifacts -```yaml -job_name: - script: - - rake spec - - coverage +job 3: stage: test - only: - - master - except: - - develop - tags: - - ruby - - postgres - allow_failure: true + script: make test + +job 4: + stage: deploy + script: make deploy ``` -| Keyword | Required | Description | -|---------------|----------|-------------| -| script | yes | Defines a shell script which is executed by Runner | -| image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| stage | no | Defines a job stage (default: `test`) | -| type | no | Alias for `stage` | -| variables | no | Define job variables on a job level | -| only | no | Defines a list of git refs for which job is created | -| except | no | Defines a list of git refs for which job is not created | -| tags | no | Defines a list of tags which are used to select Runner | -| allow_failure | no | Allow job to fail. Failed job doesn't contribute to commit status | -| when | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | -| dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them| -| artifacts | no | Define list of [job artifacts](../../user/project/pipelines/job_artifacts.md) | -| cache | no | Define list of files that should be cached between subsequent runs | -| before_script | no | Override a set of commands that are executed before job | -| after_script | no | Override a set of commands that are executed after job | -| environment | no | Defines a name of environment to which deployment is done by this job | -| coverage | no | Define code coverage settings for a given job | -| retry | no | Define how many times a job can be auto-retried in case of a failure | +## `types` + +CAUTION: **Deprecated:** +`types` is deprecated, and could be removed in one of the future releases. +Use [stages](#stages) instead. -### script +## `script` -`script` is a shell script which is executed by the Runner. For example: +`script` is the only required keyword that a job needs. It's a shell script +which is executed by the Runner. For example: ```yaml job: @@ -429,13 +231,7 @@ that the YAML parser knows to interpret the whole thing as a string rather than a "key: value" pair. Be careful when using special characters: `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. -### stage - -`stage` allows to group jobs into different stages. Jobs of the same `stage` -are executed in `parallel`. For more info about the use of `stage` please check -[stages](#stages). - -### only and except (simplified) +## `only` and `except` (simplified) `only` and `except` are two parameters that set a job policy to limit when jobs are created: @@ -505,12 +301,13 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except master. -### only and except (complex) +## `only` and `except` (complex) > Introduced in GitLab 10.0 -> This an _alpha_ feature, and it it subject to change at any time without - prior notice! +CAUTION: **Warning:** +This an _alpha_ feature, and it it subject to change at any time without +prior notice! Since GitLab 10.0 it is possible to define a more elaborate only/except job policy configuration. @@ -535,24 +332,7 @@ job: kubernetes: active ``` -### Job variables - -It is possible to define job variables using a `variables` keyword on a job -level. It works basically the same way as its [global-level equivalent](#variables), -but allows you to define job-specific variables. - -When the `variables` keyword is used on a job level, it overrides the global YAML -job variables and predefined ones. To turn off global defined variables -in your job, define an empty hash: - -```yaml -job_name: - variables: {} -``` - -Job variables priority is defined in the [variables documentation][variables]. - -### tags +## `tags` `tags` is used to select specific Runners from the list of all Runners that are allowed to run this project. @@ -573,7 +353,7 @@ job: The specification above, will make sure that `job` is built by a Runner that has both `ruby` AND `postgres` tags defined. -### allow_failure +## `allow_failure` `allow_failure` is used when you want to allow a job to fail without impacting the rest of the CI suite. Failed jobs don't contribute to the commit status. @@ -606,7 +386,7 @@ job3: - deploy_to_staging ``` -### when +## `when` `when` is used to implement jobs that are run in case of failure or despite the failure. @@ -619,7 +399,7 @@ failure. fails. 1. `always` - execute job regardless of the status of jobs from prior stages. 1. `manual` - execute job manually (added in GitLab 8.10). Read about - [manual actions](#manual-actions) below. + [manual actions](#when-manual) below. For example: @@ -667,42 +447,41 @@ The above script will: success or failure. 3. Allow you to manually execute `deploy_job` from GitLab's UI. -#### Manual actions - -> Introduced in GitLab 8.10. -> Blocking manual actions were introduced in GitLab 9.0 -> Protected actions were introduced in GitLab 9.2 +### `when:manual` -Manual actions are a special type of job that are not executed automatically; -they need to be explicitly started by a user. Manual actions can be started -from pipeline, build, environment, and deployment views. +> **Notes:** +- Introduced in GitLab 8.10. +- Blocking manual actions were introduced in GitLab 9.0. +- Protected actions were introduced in GitLab 9.2. -An example usage of manual actions is deployment to production. +Manual actions are a special type of job that are not executed automatically, +they need to be explicitly started by a user. An example usage of manual actions +would be a deployment to a production environment. Manual actions can be started +from the pipeline, job, environment, and deployment views. Read more at the +[environments documentation][env-manual]. -Read more at the [environments documentation][env-manual]. - -Manual actions can be either optional or blocking. Blocking manual action will -block execution of the pipeline at stage this action is defined in. It is +Manual actions can be either optional or blocking. Blocking manual actions will +block the execution of the pipeline at the stage this action is defined in. It's possible to resume execution of the pipeline when someone executes a blocking -manual actions by clicking a _play_ button. +manual action by clicking a _play_ button. -When pipeline is blocked it will not be merged if Merge When Pipeline Succeeds +When a pipeline is blocked, it will not be merged if Merge When Pipeline Succeeds is set. Blocked pipelines also do have a special status, called _manual_. - Manual actions are non-blocking by default. If you want to make manual action blocking, it is necessary to add `allow_failure: false` to the job's definition in `.gitlab-ci.yml`. -Optional manual actions have `allow_failure: true` set by default. +Optional manual actions have `allow_failure: true` set by default and their +Statuses do not contribute to the overall pipeline status. So, if a manual +action fails, the pipeline will eventually succeed. -**Statuses of optional actions do not contribute to overall pipeline status.** +Manual actions are considered to be write actions, so permissions for +[protected branches](../../user/project/protected_branches.md) are used when +user wants to trigger an action. In other words, in order to trigger a manual +action assigned to a branch that the pipeline is running for, user needs to +have ability to merge to this branch. -**Manual actions are considered to be write actions, so permissions for -protected branches are used when user wants to trigger an action. In other -words, in order to trigger a manual action assigned to a branch that the -pipeline is running for, user needs to have ability to merge to this branch.** - -### environment +## `environment` > **Notes:** @@ -727,7 +506,7 @@ deploy to production: In the above example, the `deploy to production` job will be marked as doing a deployment to the `production` environment. -#### environment:name +### `environment:name` > **Notes:** @@ -766,7 +545,7 @@ deploy to production: name: production ``` -#### environment:url +### `environment:url` > **Notes:** @@ -793,7 +572,7 @@ deploy to production: url: https://prod.example.com ``` -#### environment:on_stop +### `environment:on_stop` > **Notes:** @@ -808,7 +587,7 @@ the environment. Read the `environment:action` section for an example. -#### environment:action +### `environment:action` > [Introduced][ce-6669] in GitLab 8.13. @@ -849,7 +628,7 @@ The `stop_review_app` job is **required** to have the following keywords defined - `stage` should be the same as the `review_app` in order for the environment to stop automatically when the branch is deleted -#### dynamic environments +### Dynamic environments > **Notes:** @@ -885,13 +664,186 @@ The common use case is to create dynamic environments for branches and use them as Review Apps. You can see a simple example using Review Apps at <https://gitlab.com/gitlab-examples/review-apps-nginx/>. -### artifacts +## `cache` + +> +**Notes:** +- Introduced in GitLab Runner v0.7.0. +- `cache` can be set globally and per-job. +- From GitLab 9.0, caching is enabled and shared between pipelines and jobs + by default. +- From GitLab 9.2, caches are restored before [artifacts](#artifacts). + +TIP: **Learn more:** +Read how caching works and find out some good practices in the +[caching dependencies documentation](../caching/index.md). + +`cache` is used to specify a list of files and directories which should be +cached between jobs. You can only use paths that are within the project +workspace. + +If `cache` is defined outside the scope of jobs, it means it is set +globally and all jobs will use that definition. + +### `cache:paths` + +Use the `paths` directive to choose which files or directories will be cached. +Wildcards can be used as well. + +Cache all files in `binaries` that end in `.apk` and the `.config` file: + +```yaml +rspec: + script: test + cache: + paths: + - binaries/*.apk + - .config +``` + +Locally defined cache overrides globally defined options. The following `rspec` +job will cache only `binaries/`: + +```yaml +cache: + paths: + - my/files + +rspec: + script: test + cache: + paths: + - binaries/ +``` + +### `cache:key` + +> Introduced in GitLab Runner v1.0.0. + +Since the cache is shared between jobs, if you're using different +paths for different jobs, you should also set a different `cache:key` +otherwise cache content can be overwritten. + +The `key` directive allows you to define the affinity of caching between jobs, +allowing to have a single cache for all jobs, cache per-job, cache per-branch +or any other way that fits your workflow. This way, you can fine tune caching, +allowing you to cache data between different jobs or even different branches. + +The `cache:key` variable can use any of the +[predefined variables](../variables/README.md), and the default key, if not set, +is `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and +per-branch". It is the default across the project, therefore everything is +shared between pipelines and jobs running on the same branch by default. + +NOTE: **Note:** +The `cache:key` variable cannot contain the `/` character, or the equivalent +URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. + +For example, to enable per-branch caching: + +```yaml +cache: + key: "$CI_COMMIT_REF_SLUG" + paths: + - binaries/ +``` + +If you use **Windows Batch** to run your shell scripts you need to replace +`$` with `%`: + +```yaml +cache: + key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%" + paths: + - binaries/ +``` + +If you use **Windows PowerShell** to run your shell scripts you need to replace +`$` with `$env:`: + +```yaml +cache: + key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG" + paths: + - binaries/ +``` + +### `cache:untracked` + +Set `untracked: true` to cache all files that are untracked in your Git +repository: + +```yaml +rspec: + script: test + cache: + untracked: true +``` + +Cache all Git untracked files and files in `binaries`: + +```yaml +rspec: + script: test + cache: + untracked: true + paths: + - binaries/ +``` + +### `cache:policy` + +> Introduced in GitLab 9.4. + +The default behaviour of a caching job is to download the files at the start of +execution, and to re-upload them at the end. This allows any changes made by the +job to be persisted for future runs, and is known as the `pull-push` cache +policy. + +If you know the job doesn't alter the cached files, you can skip the upload step +by setting `policy: pull` in the job specification. Typically, this would be +twinned with an ordinary cache job at an earlier stage to ensure the cache +is updated from time to time: + +```yaml +stages: + - setup + - test + +prepare: + stage: setup + cache: + key: gems + paths: + - vendor/bundle + script: + - bundle install --deployment + +rspec: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - bundle exec rspec ... +``` + +This helps to speed up job execution and reduce load on the cache server, +especially when you have a large number of cache-using jobs executing in +parallel. + +Additionally, if you have a job that unconditionally recreates the cache without +reference to its previous contents, you can use `policy: push` in that job to +skip the download step. + +## `artifacts` > **Notes:** - Introduced in GitLab Runner v0.7.0 for non-Windows platforms. - Windows support was added in GitLab Runner v.1.0.0. -- Prior to GitLab 9.2, caches were restored after artifacts. - From GitLab 9.2, caches are restored before artifacts. - Currently not all executors are supported. - Job artifacts are only collected for successful jobs by default. @@ -960,7 +912,9 @@ release-job: The artifacts will be sent to GitLab after the job finishes successfully and will be available for download in the GitLab UI. -#### artifacts:name +[Read more about artifacts.](../../user/project/pipelines/job_artifacts.md) + +### `artifacts:name` > Introduced in GitLab 8.6 and GitLab Runner v1.1.0. @@ -970,10 +924,6 @@ useful when you'd like to download the archive from GitLab. The `artifacts:name` variable can make use of any of the [predefined variables](../variables/README.md). The default name is `artifacts`, which becomes `artifacts.zip` when downloaded. ---- - -**Example configurations** - To create an archive with a name of the current job: ```yaml @@ -998,7 +948,7 @@ tag including only the files that are untracked by Git: ```yaml job: artifacts: - name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}" + name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" untracked: true ``` @@ -1007,7 +957,7 @@ To create an archive with a name of the current [stage](#stages) and branch name ```yaml job: artifacts: - name: "${CI_JOB_STAGE}_${CI_COMMIT_REF_NAME}" + name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" untracked: true ``` @@ -1019,7 +969,7 @@ If you use **Windows Batch** to run your shell scripts you need to replace ```yaml job: artifacts: - name: "%CI_JOB_STAGE%_%CI_COMMIT_REF_NAME%" + name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" untracked: true ``` @@ -1029,11 +979,11 @@ If you use **Windows PowerShell** to run your shell scripts you need to replace ```yaml job: artifacts: - name: "$env:CI_JOB_STAGE_$env:CI_COMMIT_REF_NAME" + name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" untracked: true ``` -#### artifacts:when +### `artifacts:when` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. @@ -1046,11 +996,7 @@ failure. 1. `on_failure` - upload artifacts only when the job fails. 1. `always` - upload artifacts regardless of the job status. ---- - -**Example configurations** - -To upload artifacts only when job fails. +To upload artifacts only when job fails: ```yaml job: @@ -1058,22 +1004,23 @@ job: when: on_failure ``` -#### artifacts:expire_in +### `artifacts:expire_in` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. -`artifacts:expire_in` is used to delete uploaded artifacts after the specified -time. By default, artifacts are stored on GitLab forever. `expire_in` allows you -to specify how long artifacts should live before they expire, counting from the -time they are uploaded and stored on GitLab. +`expire_in` allows you to specify how long artifacts should live before they +expire and therefore deleted, counting from the time they are uploaded and +stored on GitLab. If the expiry time is not defined, it defaults to the +[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) +(30 days by default, forever on GitLab.com). You can use the **Keep** button on the job page to override expiration and keep artifacts forever. -After expiry, artifacts are actually deleted hourly by default (via a cron job), -but they are not accessible after expiry. +After their expiry, artifacts are deleted hourly by default (via a cron job), +and are not accessible anymore. -The value of `expire_in` is an elapsed time. Examples of parseable values: +The value of `expire_in` is an elapsed time. Examples of parsable values: - '3 mins 4 sec' - '2 hrs 20 min' @@ -1082,10 +1029,6 @@ The value of `expire_in` is an elapsed time. Examples of parseable values: - '47 yrs 6 mos and 4d' - '3 weeks and 2 days' ---- - -**Example configurations** - To expire artifacts 1 week after being uploaded: ```yaml @@ -1094,7 +1037,7 @@ job: expire_in: 1 week ``` -### dependencies +## `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1153,7 +1096,7 @@ deploy: script: make deploy ``` -#### When a dependent job will fail +### When a dependent job will fail > Introduced in GitLab 10.3. @@ -1167,27 +1110,9 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. -### before_script and after_script - -It's possible to overwrite the globally defined `before_script` and `after_script`: +## `coverage` -```yaml -before_script: -- global before script - -job: - before_script: - - execute this instead of global before script - script: - - my command - after_script: - - execute this after my script -``` - -### coverage - -**Notes:** -- [Introduced][ce-7447] in GitLab 8.17. +> [Introduced][ce-7447] in GitLab 8.17. `coverage` allows you to configure how code coverage will be extracted from the job output. @@ -1205,10 +1130,9 @@ job1: coverage: '/Code coverage: \d+\.\d+/' ``` -### retry +## `retry` -**Notes:** -- [Introduced][ce-3442] in GitLab 9.5. +> [Introduced][ce-12909] in GitLab 9.5. `retry` allows you to configure how many times a job is going to be retried in case of a failure. @@ -1228,16 +1152,57 @@ test: retry: 2 ``` -## Git Strategy +## `variables` + +> Introduced in GitLab Runner v0.5.0. + +NOTE: **Note:** +Integers (as well as strings) are legal both for variable's name and value. +Floats are not legal and cannot be used. + +GitLab CI/CD allows you to define variables inside `.gitlab-ci.yml` that are +then passed in the job environment. They can be set globally and per-job. +When the `variables` keyword is used on a job level, it overrides the global +YAML variables and predefined ones. + +They are stored in the Git repository and are meant to store non-sensitive +project configuration, for example: + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. +The YAML-defined variables are also set to all created service containers, +thus allowing to fine tune them. + +To turn off global defined variables in a specific job, define an empty hash: + +```yaml +job_name: + variables: {} +``` + +Except for the user defined variables, there are also the ones [set up by the +Runner itself](../variables/README.md#predefined-variables-environment-variables). +One example would be `CI_COMMIT_REF_NAME` which has the value of +the branch or tag name for which project is built. Apart from the variables +you can set in `.gitlab-ci.yml`, there are also the so called +[secret variables](../variables/README.md#secret-variables) +which can be set in GitLab's UI. + +[Learn more about variables and their priority.][variables] + +### Git strategy > Introduced in GitLab 8.9 as an experimental feature. May change or be removed completely in future releases. `GIT_STRATEGY=none` requires GitLab Runner v1.7+. You can set the `GIT_STRATEGY` used for getting recent application code, either -in the global [`variables`](#variables) section or the [`variables`](#job-variables) -section for individual jobs. If left unspecified, the default from project -settings will be used. +globally or per-job in the [`variables`](#variables) section. If left +unspecified, the default from project settings will be used. There are three possible values: `clone`, `fetch`, and `none`. @@ -1269,44 +1234,13 @@ variables: GIT_STRATEGY: none ``` -## Git Checkout - -> Introduced in GitLab Runner 9.3 - -The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either -`clone` or `fetch` to specify whether a `git checkout` should be run. If not -specified, it defaults to true. Like `GIT_STRATEGY`, it can be set in either the -global [`variables`](#variables) section or the [`variables`](#job-variables) -section for individual jobs. - -If set to `false`, the Runner will: - -- when doing `fetch` - update the repository and leave working copy on - the current revision, -- when doing `clone` - clone the repository and leave working copy on the - default branch. - -Having this setting set to `true` will mean that for both `clone` and `fetch` -strategies the Runner will checkout the working copy to a revision related -to the CI pipeline: - -```yaml -variables: - GIT_STRATEGY: clone - GIT_CHECKOUT: "false" -script: - - git checkout master - - git merge $CI_BUILD_REF_NAME -``` - -## Git Submodule Strategy +### Git submodule strategy > Requires GitLab Runner v1.10+. The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git -submodules are included when fetching the code before a build. Like -`GIT_STRATEGY`, it can be set in either the global [`variables`](#variables) -section or the [`variables`](#job-variables) section for individual jobs. +submodules are included when fetching the code before a build. You can set them +globally or per-job in the [`variables`](#variables) section. There are three possible values: `none`, `normal`, and `recursive`: @@ -1336,8 +1270,36 @@ Note that for this feature to work correctly, the submodules must be configured - a relative path to another repository on the same GitLab server. See the [Git submodules](../git_submodules.md) documentation. +### Git checkout + +> Introduced in GitLab Runner 9.3 + +The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either +`clone` or `fetch` to specify whether a `git checkout` should be run. If not +specified, it defaults to true. You can set them globally or per-job in the +[`variables`](#variables) section. + +If set to `false`, the Runner will: + +- when doing `fetch` - update the repository and leave working copy on + the current revision, +- when doing `clone` - clone the repository and leave working copy on the + default branch. + +Having this setting set to `true` will mean that for both `clone` and `fetch` +strategies the Runner will checkout the working copy to a revision related +to the CI pipeline: + +```yaml +variables: + GIT_STRATEGY: clone + GIT_CHECKOUT: "false" +script: + - git checkout master + - git merge $CI_BUILD_REF_NAME +``` -## Job stages attempts +### Job stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. @@ -1359,10 +1321,9 @@ variables: GET_SOURCES_ATTEMPTS: 3 ``` -You can set them in the global [`variables`](#variables) section or the -[`variables`](#job-variables) section for individual jobs. +You can set them globally or per-job in the [`variables`](#variables) section. -## Shallow cloning +### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. May change in future releases or be removed completely. @@ -1393,7 +1354,17 @@ variables: GIT_DEPTH: "3" ``` -## Hidden keys (jobs) +You can set it globally or per-job in the [`variables`](#variables) section. + +## Special YAML features + +It's possible to use special YAML features like anchors (`&`), aliases (`*`) +and map merging (`<<`), which will allow you to greatly reduce the complexity +of `.gitlab-ci.yml`. + +Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). + +### Hidden keys (jobs) > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1419,14 +1390,6 @@ Use this feature to ignore jobs, or use the [special YAML features](#special-yaml-features) and transform the hidden keys into templates. -## Special YAML features - -It's possible to use special YAML features like anchors (`&`), aliases (`*`) -and map merging (`<<`), which will allow you to greatly reduce the complexity -of `.gitlab-ci.yml`. - -Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). - ### Anchors > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1556,34 +1519,10 @@ with an API call. [Read more in the triggers documentation.](../triggers/README.md) -### pages - -`pages` is a special job that is used to upload static content to GitLab that -can be used to serve your website. It has a special syntax, so the two -requirements below must be met: - -1. Any static content must be placed under a `public/` directory -1. `artifacts` with a path to the `public/` directory must be defined - -The example below simply moves all files from the root of the project to the -`public/` directory. The `.public` workaround is so `cp` doesn't also copy -`public/` to itself in an infinite loop: - -``` -pages: - stage: deploy - script: - - mkdir .public - - cp -r * .public - - mv .public public - artifacts: - paths: - - public - only: - - master -``` +## Skipping jobs -Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). +If your commit message contains `[ci skip]` or `[skip ci]`, using any +capitalization, the commit will be created but the pipeline will be skipped. ## Validate the .gitlab-ci.yml @@ -1595,11 +1534,6 @@ You can find the link under `/ci/lint` of your gitlab instance. If you get validation error when using specific values (e.g., `true` or `false`), try to quote them, or change them to a different form (e.g., `/bin/true`). -## Skipping jobs - -If your commit message contains `[ci skip]` or `[skip ci]`, using any -capitalization, the commit will be created but the jobs will be skipped. - ## Examples Visit the [examples README][examples] to see a list of examples using GitLab @@ -1613,5 +1547,5 @@ CI with various languages. [variables]: ../variables/README.md [ce-7983]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7983 [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 -[ce-3442]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3442 +[ce-12909]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12909 [schedules]: ../../user/project/pipelines/schedules.md diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md index 6fe5f647d6c..41e3412c7ff 100644 --- a/doc/development/doc_styleguide.md +++ b/doc/development/doc_styleguide.md @@ -1,89 +1,28 @@ -# Documentation styleguide +# Documentation style guidelines -This styleguide recommends best practices to improve documentation and to keep -it organized and easy to find. +The documentation style guide defines the markup structure used in +GitLab documentation. Check the +[documentation guidelines](writing_documentation.md) for general development instructions. -See also [writing documentation](writing_documentation.md). - -## Location and naming of documents - ->**Note:** -These guidelines derive from the discussion taken place in issue [#3349][ce-3349]. - -The documentation hierarchy can be vastly improved by providing a better layout -and organization of directories. - -Having a structured document layout, we will be able to have meaningful URLs -like `docs.gitlab.com/user/project/merge_requests.html`. With this pattern, -you can immediately tell that you are navigating a user related documentation -and is about the project and its merge requests. - -Do not create summaries of similar types of content (e.g. an index of all articles, videos, etc.), -rather organise content by its subject (e.g. everything related to CI goes together) -and cross-link between any related content. - -The table below shows what kind of documentation goes where. - -| Directory | What belongs here | -| --------- | -------------- | -| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here including `/admin`. | -| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface go under `doc/user/admin_area/`. | -| `doc/api/` | API related documentation. | -| `doc/development/` | Documentation related to the development of GitLab. Any styleguides should go here. | -| `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/`| Probably the most visited directory, since `installation.md` is there. Ideally this should go under `doc/administration/`, but it's best to leave it as-is in order to avoid confusion (still debated though). | -| `doc/update/` | Same with `doc/install/`. Should be under `administration/`, but this is a well known location, better leave as-is, at least for now. | -| `doc/topics/` | Indexes per Topic (`doc/topics/topic-name/index.md`): all resources for that topic (user and admin documentation, articles, and third-party docs) | - ---- - -**General rules:** - -1. The correct naming and location of a new document, is a combination - of the relative URL of the document in question and the GitLab Map design - that is used for UX purposes ([source][graffle], [image][gitlab-map]). -1. When creating a new document and it has more than one word in its name, - make sure to use underscores instead of spaces or dashes (`-`). For example, - a proper naming would be `import_projects_from_github.md`. The same rule - applies to images. -1. There are four main directories, `user`, `administration`, `api` and `development`. -1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, - `profile/`, `dashboard/` and `admin_area/`. - 1. `doc/user/project/` should contain all project related documentation. - 1. `doc/user/group/` should contain all group related documentation. - 1. `doc/user/profile/` should contain all profile related documentation. - Every page you would navigate under `/profile` should have its own document, - i.e. `account.md`, `applications.md`, `emails.md`, etc. - 1. `doc/user/dashboard/` should contain all dashboard related documentation. - 1. `doc/user/admin_area/` should contain all admin related documentation - describing what can be achieved by accessing GitLab's admin interface - (_not to be confused with `doc/administration` where server access is - required_). - 1. Every category under `/admin/application_settings` should have its - own document located at `doc/user/admin_area/settings/`. For example, - the **Visibility and Access Controls** category should have a document - located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. -1. The `doc/topics/` directory holds topic-related technical content. Create - `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. - General user- and admin- related documentation, should be placed accordingly. - ---- - -If you are unsure where a document should live, you can ping `@axil` or `@marcia` in your -merge request. +Check the GitLab hanbook for the [writing styles guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). ## Text -- Split up long lines, this makes it much easier to review and edit. Only +- Split up long lines (wrap text), this makes it much easier to review and edit. Only double line breaks are shown as a full line break in [GitLab markdown][gfm]. 80-100 characters is a good line length -- Make sure that the documentation is added in the correct directory and that +- Make sure that the documentation is added in the correct + [directory](writing_documentation.md#documentation-directory-structure) and that there's a link to it somewhere useful - Do not duplicate information - Be brief and clear - Unless there's a logical reason not to, add documents in alphabetical order - Write in US English - Use [single spaces][] instead of double spaces +- Jump a line between different markups (e.g., after every paragraph, hearder, list, etc) +- Capitalize "G" and "L" in GitLab +- Capitalize feature, products, and methods names. E.g.: GitLab Runner, Geo, +Issue Boards, Git, Prometheus, Continuous Integration. ## Formatting @@ -103,6 +42,8 @@ merge request. links shift too, which eventually leads to dead links. If you think it is compelling to add numbers in headings, make sure to at least discuss it with someone in the Merge Request +- [Avoid using symbols and special chars](https://gitlab.com/gitlab-com/gitlab-docs/issues/84) + in headers. Whenever possible, they should be plain and short text. - Avoid adding things that show ephemeral statuses. For example, if a feature is considered beta or experimental, put this info in a note, not in the heading. - When introducing a new document, be careful for the headings to be @@ -121,71 +62,18 @@ merge request. you can use `[Text][identifier]` and at the bottom of the section or the document add: `[identifier]: https://example.com`, in which case, we do encourage you to also add an alternative text: `[identifier]: https://example.com "Alternative text"` that appears when hovering your mouse on a link. - -### Linking to inline docs - -Sometimes it's needed to link to the built-in documentation that GitLab provides -under `/help`. This is normally done in files inside the `app/views/` directory -with the help of the `help_page_path` helper method. - -In its simplest form, the HAML code to generate a link to the `/help` page is: - -```haml -= link_to 'Help page', help_page_path('user/permissions') -``` - -The `help_page_path` contains the path to the document you want to link to with -the following conventions: - -- it is relative to the `doc/` directory in the GitLab repository -- the `.md` extension must be omitted -- it must not end with a slash (`/`) - -Below are some special cases where should be used depending on the context. -You can combine one or more of the following: - -1. **Linking to an anchor link.** Use `anchor` as part of the `help_page_path` - method: - - ```haml - = link_to 'Help page', help_page_path('user/permissions', anchor: 'anchor-link') - ``` - -1. **Opening links in a new tab.** This should be the default behavior: - - ```haml - = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' - ``` - -1. **Linking to a circle icon.** Usually used in settings where a long - description cannot be used, like near checkboxes. You can basically use - any font awesome icon, but prefer the `question-circle`: - - ```haml - = link_to icon('question-circle'), help_page_path('user/permissions') - ``` - -1. **Using a button link.** Useful in places where text would be out of context - with the rest of the page layout: - - ```haml - = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' - ``` - -1. **Using links inline of some text.** - - ```haml - Description to #{link_to 'Help page', help_page_path('user/permissions')}. - ``` - -1. **Adding a period at the end of the sentence.** Useful when you don't want - the period to be part of the link: - - ```haml - = succeed '.' do - Learn more in the - = link_to 'Help page', help_page_path('user/permissions') - ``` +- To link to internal documentation, use relative links, not full URLs. Use `../` to + navigate tp high-level directories, and always add the file name `file.md` at the + end of the link with the `.md` extension, not `.html`. + Example: instead of `[text](../../merge_requests/)`, use + `[text](../../merge_requests/index.md)` or, `[text](../../ci/README.md)`, or, + for anchor links, `[text](../../ci/README.md#examples)`. + Using the markdown extension is necessary for the [`/help`](writing_documentation.md#gitlab-help) + section of GitLab. +- To link from CE to EE-only documentation, use the EE-only doc full URL. +- Use [meaningful anchor texts](https://www.futurehosting.com/blog/links-should-have-meaningful-anchor-text-heres-why/). + E.g., instead of writing something like `Read more about GitLab Issue Boards [here](LINK)`, + write `Read more about [GitLab Issue Boards](LINK)`. ## Images @@ -222,7 +110,7 @@ Inside the document: - Notes should be quoted with the word `Note:` being bold. Use this form: - ``` + ```md >**Note:** This is something to note. ``` @@ -234,25 +122,25 @@ Inside the document: If the note spans across multiple lines it's OK to split the line. -## New features +## Specific sections and terms -New features must be shipped with its accompanying documentation and the doc -reviewed by a technical writer. +To mention and/or reference specific terms in GitLab, please follow the styles +below. -### Mentioning GitLab versions and tiers +### GitLab versions and tiers - Every piece of documentation that comes with a new feature should declare the GitLab version that feature got introduced. Right below the heading add a note: - ``` + ```md > Introduced in GitLab 8.3. ``` - If possible every feature should have a link to the MR, issue, or epic that introduced it. The above note would be then transformed to: - ``` + ```md > [Introduced][ce-1242] in GitLab 8.3. ``` @@ -263,121 +151,41 @@ reviewed by a technical writer. the [paid tier](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers) the feature is available in: - ``` + ```md > [Introduced][ee-1234] in [GitLab Starter](https://about.gitlab.com/products/) 8.3. ``` Otherwise, leave this mention out. -## References - -- **GitLab Restart:** - There are many cases that a restart/reconfigure of GitLab is required. To - avoid duplication, link to the special document that can be found in - [`doc/administration/restart_gitlab.md`][doc-restart]. Usually the text will - read like: - - ``` - Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) - for the changes to take effect. - ``` - If the document you are editing resides in a place other than the GitLab CE/EE - `doc/` directory, instead of the relative link, use the full path: - `http://docs.gitlab.com/ce/administration/restart_gitlab.html`. - Replace `reconfigure` with `restart` where appropriate. - -## Installation guide - -- **Ruby:** - In [step 2 of the installation guide](../install/installation.md#2-ruby), - we install Ruby from source. Whenever there is a new version that needs to - be updated, remember to change it throughout the codeblock and also replace - the sha256sum (it can be found in the [downloads page][ruby-dl] of the Ruby - website). +### GitLab Restart -[ruby-dl]: https://www.ruby-lang.org/en/downloads/ "Ruby download website" - -## Changing document location - -Changing a document's location is not to be taken lightly. Remember that the -documentation is available to all installations under `help/` and not only to -GitLab.com or http://docs.gitlab.com. Make sure this is discussed with the -Documentation team beforehand. - -If you indeed need to change a document's location, do NOT remove the old -document, but rather replace all of its contents with a new line: - -``` -This document was moved to [another location](path/to/new_doc.md). -``` +There are many cases that a restart/reconfigure of GitLab is required. To +avoid duplication, link to the special document that can be found in +[`doc/administration/restart_gitlab.md`][doc-restart]. Usually the text will +read like: -where `path/to/new_doc.md` is the relative path to the root directory `doc/`. + ``` + Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) + for the changes to take effect. + ``` ---- +If the document you are editing resides in a place other than the GitLab CE/EE +`doc/` directory, instead of the relative link, use the full path: +`http://docs.gitlab.com/ce/administration/restart_gitlab.html`. +Replace `reconfigure` with `restart` where appropriate. -For example, if you were to move `doc/workflow/lfs/lfs_administration.md` to -`doc/administration/lfs.md`, then the steps would be: +### Installation guide -1. Copy `doc/workflow/lfs/lfs_administration.md` to `doc/administration/lfs.md` -1. Replace the contents of `doc/workflow/lfs/lfs_administration.md` with: +**Ruby:** +In [step 2 of the installation guide](../install/installation.md#2-ruby), +we install Ruby from source. Whenever there is a new version that needs to +be updated, remember to change it throughout the codeblock and also replace +the sha256sum (it can be found in the [downloads page][ruby-dl] of the Ruby +website). - ``` - This document was moved to [another location](../../administration/lfs.md). - ``` - -1. Find and replace any occurrences of the old location with the new one. - A quick way to find them is to use `git grep`. First go to the root directory - where you cloned the `gitlab-ce` repository and then do: - - ``` - git grep -n "workflow/lfs/lfs_administration" - git grep -n "lfs/lfs_administration" - ``` - -NOTE: **Note:** -If the document being moved has any Disqus comments on it, there are extra steps -to follow documented just [below](#redirections-for-pages-with-disqus-comments). - -Things to note: - -- Since we also use inline documentation, except for the documentation itself, - the document might also be referenced in the views of GitLab (`app/`) which will - render when visiting `/help`, and sometimes in the testing suite (`spec/`). -- The above `git grep` command will search recursively in the directory you run - it in for `workflow/lfs/lfs_administration` and `lfs/lfs_administration` - and will print the file and the line where this file is mentioned. - You may ask why the two greps. Since we use relative paths to link to - documentation, sometimes it might be useful to search a path deeper. -- The `*.md` extension is not used when a document is linked to GitLab's - built-in help page, that's why we omit it in `git grep`. -- Use the checklist on the documentation MR description template. - -### Redirections for pages with Disqus comments - -If the documentation page being relocated already has any Disqus comments, -we need to preserve the Disqus thread. - -Disqus uses an identifier per page, and for docs.gitlab.com, the page identifier -is configured to be the page URL. Therefore, when we change the document location, -we need to preserve the old URL as the same Disqus identifier. - -To do that, add to the frontmatter the variable `redirect_from`, -using the old URL as value. For example, let's say I moved the document -available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, -`https://docs.gitlab.com/my-new-location/index.html`. - -Into the **new document** frontmatter add the following: - -```yaml ---- -redirect_from: 'https://docs.gitlab.com/my-old-location/README.html' ---- -``` - -Note: it is necessary to include the file name in the `redirect_from` URL, -even if it's `index.html` or `README.html`. +[ruby-dl]: https://www.ruby-lang.org/en/downloads/ "Ruby download website" -## Configuration documentation for source and Omnibus installations +### Configuration documentation for source and Omnibus installations GitLab currently officially supports two installation methods: installations from source and Omnibus packages installations. @@ -394,7 +202,7 @@ When there is a list of steps to perform, usually that entails editing the configuration file and reconfiguring/restarting GitLab. In such case, follow the style below as a guide: -```` +```md **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb`: @@ -421,7 +229,7 @@ the style below as a guide: [reconfigure]: path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure [restart]: path/to/administration/restart_gitlab.md#installations-from-source -```` +``` In this case: @@ -433,7 +241,7 @@ In this case: - different highlighting languages are used for each config in the code block - the [references](#references) guide is used for reconfigure/restart -## Fake tokens +### Fake tokens There may be times where a token is needed to demonstrate an API call using cURL or a secret variable used in CI. It is strongly advised not to use real @@ -456,7 +264,7 @@ You can use the following fake tokens as examples. | Health check token | `Tu7BgjR9qeZTEyRzGG2P` | | Request profile token | `7VgpS4Ax5utVD2esNstz` | -## API +### API Here is a list of must-have items. Use them in the exact order that appears on this document. Further explanation is given below. @@ -472,10 +280,10 @@ on this document. Further explanation is given below. - Every method must have a cURL example. - Every method must have a response body (in JSON format). -### Method description +#### Method description Use the following table headers to describe the methods. Attributes should -always be in code blocks using backticks (`). +always be in code blocks using backticks (``` ` ```). ``` | Attribute | Type | Required | Description | @@ -488,7 +296,7 @@ Rendered example: | --------- | ---- | -------- | ----------- | | `user` | string | yes | The GitLab username | -### cURL commands +#### cURL commands - Use `https://gitlab.example.com/api/v4/` as an endpoint. - Wherever needed use this personal access token: `9koXpg98eAheJpvBs5tK`. @@ -505,11 +313,11 @@ Rendered example: | `-X PUT` | Use this method when updating existing objects | | `-X DELETE` | Use this method when removing existing objects | -### cURL Examples +#### cURL Examples Below is a set of [cURL][] examples that you can use in the API documentation. -#### Simple cURL command +##### Simple cURL command Get the details of a group: @@ -517,7 +325,7 @@ Get the details of a group: curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/gitlab-org ``` -#### cURL example with parameters passed in the URL +##### cURL example with parameters passed in the URL Create a new project under the authenticated user's namespace: @@ -525,7 +333,7 @@ Create a new project under the authenticated user's namespace: curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects?name=foo" ``` -#### Post data using cURL's --data +##### Post data using cURL's --data Instead of using `-X POST` and appending the parameters to the URI, you can use cURL's `--data` option. The example below will create a new project `foo` under @@ -535,7 +343,7 @@ the authenticated user's namespace. curl --data "name=foo" --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects" ``` -#### Post data using JSON content +##### Post data using JSON content > **Note:** In this example we create a new group. Watch carefully the single and double quotes. @@ -544,7 +352,7 @@ and double quotes. curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' https://gitlab.example.com/api/v4/groups ``` -#### Post data using form-data +##### Post data using form-data Instead of using JSON or urlencode you can use multipart/form-data which properly handles data encoding: @@ -556,7 +364,7 @@ curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --form "title The above example is run by and administrator and will add an SSH public key titled ssh-key to user's account which has an id of 25. -#### Escape special characters +##### Escape special characters Spaces or slashes (`/`) may sometimes result to errors, thus it is recommended to escape them when possible. In the example below we create a new issue which @@ -569,7 +377,7 @@ curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitl Use `%2F` for slashes (`/`). -#### Pass arrays to API calls +##### Pass arrays to API calls The GitLab API sometimes accepts arrays of strings or integers. For example, to restrict the sign-up e-mail domains of a GitLab instance to `*.example.com` and @@ -584,6 +392,3 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --data "domain [gfm]: http://docs.gitlab.com/ce/user/markdown.html#newlines "GitLab flavored markdown documentation" [ce-1242]: https://gitlab.com/gitlab-org/gitlab-ce/issues/1242 [doc-restart]: ../administration/restart_gitlab.md "GitLab restart documentation" -[ce-3349]: https://gitlab.com/gitlab-org/gitlab-ce/issues/3349 "Documentation restructure" -[graffle]: https://gitlab.com/gitlab-org/gitlab-design/blob/d8d39f4a87b90fb9ae89ca12dc565347b4900d5e/production/resources/gitlab-map.graffle -[gitlab-map]: https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 1eb90c30ebd..fea92e740cb 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -360,27 +360,15 @@ Instead place EE specs in the `ee/spec` folder. ## JavaScript code in `assets/javascripts/` -To separate EE-specific JS-files we can also move the files into an `ee` folder. +To separate EE-specific JS-files we should also move the files into an `ee` folder. For example there can be an `app/assets/javascripts/protected_branches/protected_branches_bundle.js` and an EE counterpart `ee/app/assets/javascripts/protected_branches/protected_branches_bundle.js`. -That way we can create a separate webpack bundle in `webpack.config.js`: - -```javascript - protected_branches: '~/protected_branches', - ee_protected_branches: 'ee/protected_branches/protected_branches_bundle.js', -``` - -With the separate bundle in place, we can decide which bundle to load inside the -view, using the `page_specific_javascript_bundle_tag` helper. - -```haml -- content_for :page_specific_javascripts do - = page_specific_javascript_bundle_tag('protected_branches') -``` +See the frontend guide [performance section](./fe_guide/performance.md) for +information on managing page-specific javascript within EE. ## SCSS code in `assets/stylesheets` diff --git a/doc/development/emails.md b/doc/development/emails.md index 677029b1295..4dbf064fd75 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -60,16 +60,10 @@ See the [Rails guides] for more info. As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. -1. Uncomment the `mail_room` line in your `Procfile`: - - ```yaml - mail_room: bundle exec mail_room -q -c config/mail_room.yml - ``` - -1. Restart GitLab: +1. Run this command in the GitLab root directory to launch `mail_room`: ```sh - bundle exec foreman start + bundle exec mail_room -q -c config/mail_room.yml ``` 1. Verify that everything is configured correctly: diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 12dfc10812b..2280cf79f86 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -14,8 +14,8 @@ support through [webpack][webpack]. We also utilize [webpack][webpack] to handle the bundling, minification, and compression of our assets. -Working with our frontend assets requires Node (v4.3 or greater) and Yarn -(v0.17 or greater). You can find information on how to install these on our +Working with our frontend assets requires Node (v6.0 or greater) and Yarn +(v1.2 or greater). You can find information on how to install these on our [installation guide][install]. [jQuery][jquery] is used throughout the application's JavaScript, with diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 98e43931a02..1320efaf767 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -23,7 +23,7 @@ controlled by the server. 1. The backend code will most likely be using etags. You do not and should not check for status `304 Not Modified`. The browser will transform it for you. -### Lazy Loading +### Lazy Loading Images To improve the time to first render we are using lazy loading for images. This works by setting the actual image source on the `data-src` attribute. After the HTML is rendered and JavaScript is loaded, @@ -47,41 +47,103 @@ properties once, and handle the actual animation with transforms. ## Reducing Asset Footprint -### Page-specific JavaScript +### Universal code -Certain pages may require the use of a third party library, such as [d3][d3] for -the User Activity Calendar and [Chart.js][chartjs] for the Graphs pages. These -libraries increase the page size significantly, and impact load times due to -bandwidth bottlenecks and the browser needing to parse more JavaScript. - -In cases where libraries are only used on a few specific pages, we use -"page-specific JavaScript" to prevent the main `main.js` file from -becoming unnecessarily large. - -Steps to split page-specific JavaScript from the main `main.js`: - -1. Create a directory for the specific page(s), e.g. `graphs/`. -1. In that directory, create a `namespace_bundle.js` file, e.g. `graphs_bundle.js`. -1. Add the new "bundle" file to the list of entry files in `config/webpack.config.js`. - - For example: `graphs: './graphs/graphs_bundle.js',`. -1. Move code reliant on these libraries into the `graphs` directory. -1. In `graphs_bundle.js` add CommonJS `require('./path_to_some_component.js');` statements to load any other files in this directory. Make sure to use relative urls. -1. In the relevant views, add the scripts to the page with the following: - -```haml -- content_for :page_specific_javascripts do - = webpack_bundle_tag 'lib_chart' - = webpack_bundle_tag 'graphs' -``` +Code that is contained within `main.js` and `commons/index.js` are loaded and +run on _all_ pages. **DO NOT ADD** anything to these files unless it is truly +needed _everywhere_. These bundles include ubiquitous libraries like `vue`, +`axios`, and `jQuery`, as well as code for the main navigation and sidebar. +Where possible we should aim to remove modules from these bundles to reduce our +code footprint. + +### Page-specific JavaScript -The above loads `chart.js` and `graphs_bundle.js` for this page only. `chart.js` -is separated from the bundle file so it can be cached separately from the bundle -and reused for other pages that also rely on the library. For an example, see -[this Haml file][page-specific-js-example]. +Webpack has been configured to automatically generate entry point bundles based +on the file structure within `app/assets/javascripts/pages/*`. The directories +within the `pages` directory correspond to Rails controllers and actions. These +auto-generated bundles will be automatically included on the corresponding +pages. + +For example, if you were to visit [gitlab.com/gitlab-org/gitlab-ce/issues](https://gitlab.com/gitlab-org/gitlab-ce/issues), +you would be accessing the `app/controllers/projects/issues_controller.rb` +controller with the `index` action. If a corresponding file exists at +`pages/projects/issues/index/index.js`, it will be compiled into a webpack +bundle and included on the page. + +> **Note:** Previously we had encouraged the use of +> `content_for :page_specific_javascripts` within haml files, along with +> manually generated webpack bundles. However under this new system you should +> not ever need to manually add an entry point to the `webpack.config.js` file. + +> **Tip:** +> If you are unsure what controller and action corresponds to a given page, you +> can find this out by inspecting `document.body.dataset.page` within your +> browser's developer console while on any page within gitlab. + +#### Important Considerations: + +- **Keep Entry Points Lite:** + Page-specific javascript entry points should be as lite as possible. These + files are exempt from unit tests, and should be used primarily for + instantiation and dependency injection of classes and methods that live in + modules outside of the entry point script. Just import, read the DOM, + instantiate, and nothing else. + +- **Entry Points May Be Asynchronous:** + _DO NOT ASSUME_ that the DOM has been fully loaded and available when an + entry point script is run. If you require that some code be run after the + DOM has loaded, you should attach an event handler to the `DOMContentLoaded` + event with: + + ```javascript + import initMyWidget from './my_widget'; + + document.addEventListener('DOMContentLoaded', () => { + initMyWidget(); + }); + ``` + +- **Supporting Module Placement:** + - If a class or a module is _specific to a particular route_, try to locate + it close to the entry point it will be used. For instance, if + `my_widget.js` is only imported within `pages/widget/show/index.js`, you + should place the module at `pages/widget/show/my_widget.js` and import it + with a relative path (e.g. `import initMyWidget from './my_widget';`). + + - If a class or module is _used by multiple routes_, place it within a + shared directory at the closest common parent directory for the entry + points that import it. For example, if `my_widget.js` is imported within + both `pages/widget/show/index.js` and `pages/widget/run/index.js`, then + place the module at `pages/widget/shared/my_widget.js` and import it with + a relative path if possible (e.g. `../shared/my_widget`). + +- **Enterprise Edition Caveats:** + For GitLab Enterprise Edition, page-specific entry points will override their + Community Edition counterparts with the same name, so if + `ee/app/assets/javascripts/pages/foo/bar/index.js` exists, it will take + precedence over `app/assets/javascripts/pages/foo/bar/index.js`. If you want + to minimize duplicate code, you can import one entry point from the other. + This is not done automatically to allow for flexibility in overriding + functionality. ### Code Splitting -> *TODO* flesh out this section once webpack is ready for code-splitting +For any code that does not need to be run immediately upon page load, (e.g. +modals, dropdowns, and other behaviors that can be lazy-loaded), you can split +your module into asynchronous chunks with dynamic import statements. These +imports return a Promise which will be resolved once the script has loaded: + +```javascript +import(/* webpackChunkName: 'emoji' */ '~/emoji') + .then(/* do something */) + .catch(/* report error */) +``` + +Please try to use `webpackChunkName` when generating these dynamic imports as +it will provide a deterministic filename for the chunk which can then be cached +the browser across GitLab versions. + +More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports). ### Minimizing page size @@ -95,7 +157,8 @@ General tips: - Prefer font formats with better compression, e.g. WOFF2 is better than WOFF, which is better than TTF. - Compress and minify assets wherever possible (For CSS/JS, Sprockets and webpack do this for us). - If some functionality can reasonably be achieved without adding extra libraries, avoid them. -- Use page-specific JavaScript as described above to dynamically load libraries that are only needed on certain pages. +- Use page-specific JavaScript as described above to load libraries that are only needed on certain pages. +- Use code-splitting dynamic imports wherever possible to lazy-load code that is not needed initially. - [High Performance Animations][high-perf-animations] ------- @@ -112,8 +175,5 @@ General tips: [pagespeed-insights]: https://developers.google.com/speed/pagespeed/insights/ [google-devtools-profiling]: https://developers.google.com/web/tools/chrome-devtools/profile/?hl=en [browser-diet]: https://browserdiet.com/ -[d3]: https://d3js.org/ -[chartjs]: http://www.chartjs.org/ -[page-specific-js-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/13bb9ed77f405c5f6ee4fdbc964ecf635c9a223f/app/views/projects/graphs/_head.html.haml#L6-8 [high-perf-animations]: https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/ [flip]: https://aerotwist.com/blog/flip-your-animations/ diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index 917d28b48ee..7b5fa6ca42f 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -548,6 +548,57 @@ On those a default key should not be provided. 1. Properties in a Vue Component: Check [order of properties in components rule][vue-order]. +#### `:key` +When using `v-for` you need to provide a *unique* `:key` attribute for each item. + +1. If the elements of the array being iterated have an unique `id` it is advised to use it: + ```html + <div + v-for="item in items" + :key="item.id" + > + <!-- content --> + </div> + ``` + +1. When the elements being iterated don't have a unique id, you can use the array index as the `:key` attribute + ```html + <div + v-for="(item, index) in items" + :key="index" + > + <!-- content --> + </div> + ``` + + +1. When using `v-for` with `template` and there is more than one child element, the `:key` values must be unique. It's advised to use `kebab-case` namespaces. + ```html + <template v-for="(item, index) in items"> + <span :key="`span-${index}`"></span> + <button :key="`button-${index}`"></button> + </template> + ``` + +1. When dealing with nested `v-for` use the same guidelines as above. + ```html + <div + v-for="item in items" + :key="item.id" + > + <span + v-for="element in array" + :key="element.id" + > + <!-- content --> + </span> + </div> + ``` + + +Useful links: +1. [`key`](https://vuejs.org/v2/guide/list.html#key) +1. [Vue Style Guide: Keyed v-for](https://vuejs.org/v2/style-guide/#Keyed-v-for-essential ) #### Vue and Bootstrap 1. Tooltips: Do not rely on `has-tooltip` class name for Vue components diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 093a3ca4407..c1170fa3b13 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -53,13 +53,13 @@ you can find a clear separation of concerns: ``` new_feature ├── components -│ └── component.js.es6 +│ └── component.vue │ └── ... -├── store -│ └── new_feature_store.js.es6 -├── service -│ └── new_feature_service.js.es6 -├── new_feature_bundle.js.es6 +├── stores +│ └── new_feature_store.js +├── services +│ └── new_feature_service.js +├── new_feature_bundle.js ``` _For consistency purposes, we recommend you to follow the same structure._ diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index c0ce49eb40b..856ef882453 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -45,7 +45,7 @@ We basically have 4 types of files: 1. Ruby files: basically Models and Controllers. 1. HAML files: these are the view files. 1. ERB files: used for email templates. -1. JavaScript files: we mostly need to work with VUE JS templates. +1. JavaScript files: we mostly need to work with Vue templates. ### Ruby files diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index b732cc65b73..960eabd5538 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -17,10 +17,13 @@ are very appreciative of the work done by translators and proofreaders! - French - Rémy Coutable - [GitLab](https://gitlab.com/rymai), [Crowdin](https://crowdin.com/profile/rymai) - German +- Indonesian + - Ahmad Naufal Mukhtar - [GitLab](https://gitlab.com/anaufalm), [Crowdin](https://crowdin.com/profile/anaufalm) - Italian - Paolo Falomo - [GitLab](https://gitlab.com/paolofalomo), [Crowdin](https://crowdin.com/profile/paolo.falomo) - Japanese - Korean + - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - Polish - Filip Mech - [GitLab](https://gitlab.com/mehenz), [Crowdin](https://crowdin.com/profile/mehenz) diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 243ac7f0c98..1e060ffd941 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -136,11 +136,14 @@ class MyMigration < ActiveRecord::Migration disable_ddl_transaction! def up - remove_concurrent_index :table_name, :column_name if index_exists?(:table_name, :column_name) + remove_concurrent_index :table_name, :column_name end end ``` +Note that it is not necessary to check if the index exists prior to +removing it. + ## Adding indexes If you need to add a unique index please keep in mind there is the possibility diff --git a/doc/development/new_fe_guide/development/performance.md b/doc/development/new_fe_guide/development/performance.md index 26b07874f0f..244dfb3756f 100644 --- a/doc/development/new_fe_guide/development/performance.md +++ b/doc/development/new_fe_guide/development/performance.md @@ -1,3 +1,16 @@ # Performance -> TODO: Add content +## Monitoring + +We have a performance dashboard available in one of our [grafana instances](https://performance.gprd.gitlab.com/dashboard/db/sitespeed-page-summary?orgId=1). This dashboard automatically aggregates metric data from [sitespeed.io](https://sitespeed.io) every 6 hours. These changes are displayed after a set number of pages are aggregated. + +These pages can be found inside a text file in the gitlab-build-images [repository](https://gitlab.com/gitlab-org/gitlab-build-images) called [gitlab.txt](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/scripts/gitlab.txt) +Any frontend engineer can contribute to this dashboard. They can contribute by adding or removing urls of pages from this text file. Please have a [frontend monitoring expert](https://about.gitlab.com/team) review your changes before assigning to a maintainer of the `gitlab-build-images` project. The changes will go live on the next scheduled run after the changes are merged into `master`. + +There are 3 recommended high impact metrics to review on each page + +* [First visual change](https://developers.google.com/web/tools/lighthouse/audits/first-meaningful-paint) +* [Speed Index](https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index) +* [Visual Complete 95%](https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index) + +For these metrics, lower numbers are better as it means that the website is more performant. diff --git a/doc/development/new_fe_guide/development/security.md b/doc/development/new_fe_guide/development/security.md index debda7de0c6..5bb38f17988 100644 --- a/doc/development/new_fe_guide/development/security.md +++ b/doc/development/new_fe_guide/development/security.md @@ -1,3 +1,14 @@ # Security -> TODO: Add content +## Avoid inline scripts and styles + +Inline scripts and styles should be avoided in almost all cases. In an effort to protect users from [XSS vulnerabilities](https://en.wikipedia.org/wiki/Cross-site_scripting), we will be disabling inline scripts using Content Security Policy. + +## Including external resources + +External fonts, CSS, and JavaScript should never be used with the exception of Google Analytics and Piwik - and only when the instance has enabled it. Assets should always be hosted and served locally from the GitLab instance. Embedded resources via `iframes` should never be used except in certain circumstances such as with ReCaptcha, which cannot be used without an `iframe`. + +## Resources for security testing + +- [Mozilla's HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) +- [Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) diff --git a/doc/development/new_fe_guide/index.md b/doc/development/new_fe_guide/index.md index 08c6a266e7f..78931defa24 100644 --- a/doc/development/new_fe_guide/index.md +++ b/doc/development/new_fe_guide/index.md @@ -19,7 +19,7 @@ Guidance on topics related to development. Learn about all the dependencies that make up our frontend, including some of our own custom built libraries. -## [Style](style/index.md) +## [Style guides](style/index.md) Style guides to keep our code consistent. diff --git a/doc/development/new_fe_guide/principles.md b/doc/development/new_fe_guide/principles.md index 2126d202a7e..0af5f506a91 100644 --- a/doc/development/new_fe_guide/principles.md +++ b/doc/development/new_fe_guide/principles.md @@ -1,3 +1,35 @@ # Principles -> TODO: Add principles +These principles will ensure that your frontend contribution starts off in the right direction. + +## Discuss architecture before implementation + +Discuss your architecture design in an issue before writing code. This helps decrease the review time and also provides good practice for writing and thinking about system design. + +## Be consistent + +There are multiple ways of writing code to accomplish the same results. We should be as consistent as possible in how we write code across our codebases. This will make it more easier us to maintain our code across GitLab. + +## Enhance progressively + +Whenever you see with existing code that does not follow our current style guide, update it proactively. Refrain from changing everything but each merge request should progressively enhance our codebase and reduce technical debt. + +## When to use Vue + +- Use Vue for feature that make use of heavy DOM manipulation +- Use Vue for reusable components + +## When to use jQuery + +- Use jQuery to interact with Bootstrap JavaScript components +- Avoid jQuery when a better alternative exists. We are slowly moving away from it [#43559][jquery-future] + +## Mixing Vue and jQuery + +- Mixing Vue and jQuery is not recommended. +- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it][select2]. +- It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. +- It is not recommended to add new jQuery events for Vue to interact with jQuery. + +[jquery-future]: https://gitlab.com/gitlab-org/gitlab-ce/issues/43559 +[select2]: https://vuejs.org/v2/examples/select2.html diff --git a/doc/development/new_fe_guide/style/html.md b/doc/development/new_fe_guide/style/html.md index 5489def5d6e..2d5b7d048ab 100644 --- a/doc/development/new_fe_guide/style/html.md +++ b/doc/development/new_fe_guide/style/html.md @@ -1,3 +1,53 @@ # HTML style guide -> TODO: Add content +## Buttons + +<a name="button-type"></a><a name="1.1"></a> +- [1.1](#button-type) **Use button type** Button tags requires a `type` attribute according to the [W3C HTML specification][button-type-spec]. + +``` +// bad +<button></button> + +// good +<button type="button"></button> +``` + +<a name="button-role"></a><a name="1.2"></a> +- [1.2](#button-role) **Use button role for non buttons** If an HTML element has an onClick handler but is not a button, it should have `role="button"`. This is more [accessible][button-role-accessible]. + +``` +// bad +<div onClick="doSomething"></div> + +// good +<div role="button" onClick="doSomething"></div> +``` + +## Links + +<a name="blank-links"></a><a name="2.1"></a> +- [2.1](#blank-links) **Use rel for target blank** Use `rel="noopener noreferrer"` whenever your links open in a new window i.e. `target="_blank"`. This prevents [the following][jitbit-target-blank] security vulnerability documented by JitBit + +``` +// bad +<a href="url" target="_blank"></a> + +// good +<a href="url" target="_blank" rel="noopener noreferrer"></a> +``` + +<a name="fake-links"></a><a name="2.2"></a> +- [2.2](#fake-links) **Do not use fake links** Use a button tag if a link only invokes JavaScript click event handlers. This is more semantic. + +``` +// bad +<a class="js-do-something" href="#"></a> + +// good +<button class="js-do-something" type="button"></button> +``` + +[button-type-spec]: https://www.w3.org/TR/2011/WD-html5-20110525/the-button-element.html#dom-button-type +[button-role-accessible]: https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_button_role +[jitbit-target-blank]: https://www.jitbit.com/alexblog/256-targetblank---the-most-underestimated-vulnerability-ever/ diff --git a/doc/development/new_fe_guide/style/index.md b/doc/development/new_fe_guide/style/index.md index d2d576b3b46..335d9e66240 100644 --- a/doc/development/new_fe_guide/style/index.md +++ b/doc/development/new_fe_guide/style/index.md @@ -1,4 +1,4 @@ -# Style +# Style guides ## [HTML style guide](html.md) @@ -7,3 +7,9 @@ ## [JavaScript style guide](javascript.md) ## [Vue style guide](vue.md) + +# Tooling + +## [Prettier](prettier.md) + +Our code is automatically formatted with [Prettier](https://prettier.io) to follow our guidelines. diff --git a/doc/development/new_fe_guide/style/prettier.md b/doc/development/new_fe_guide/style/prettier.md new file mode 100644 index 00000000000..eb18189282b --- /dev/null +++ b/doc/development/new_fe_guide/style/prettier.md @@ -0,0 +1,45 @@ +# Formatting with Prettier + +Our code is automatically formatted with [Prettier](https://prettier.io) to follow our style guides. Prettier is taking care of formatting .js, .vue, and .scss files based on the standard prettier rules. You can find all settings for Prettier in `.prettierrc`. + +## Editor + +The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. + +Please take care that you only let Prettier format the same file types as the global Yarn script does (.js, .vue, and .scss). In VSCode by example you can easily exclude file formats in your settings file: + +``` + "prettier.disableLanguages": [ + "json", + "markdown" + ], +``` + +## Yarn Script + +The following yarn scripts are available to do global formatting: + +``` +yarn prettier-staged-save +``` + +Updates all currently staged files (based on `git diff`) with Prettier and saves the needed changes. + +``` +yarn prettier-staged +``` +Checks all currently staged files (based on `git diff`) with Prettier and log which files would need manual updating to the console. + +``` +yarn prettier-all +``` + +Checks all files with Prettier and logs which files need manual updating to the console. + +``` +yarn prettier-all-save +``` + +Formats all files in the repository with Prettier. (This should only be used to test global rule updates otherwise you would end up with huge MR's). + +The source of these Yarn scripts can be found in `/scripts/frontend/prettier.js`. diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 5b4f6511f04..d10a797a142 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -22,7 +22,7 @@ You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pip It is possible to run end-to-end tests (eventually being run within a [GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering -the `package-qa` manual action, that should be present in a merge request +the `package-and-qa` manual action, that should be present in a merge request widget. Manual action that starts end-to-end tests is also available in merge requests diff --git a/doc/development/writing_documentation.md b/doc/development/writing_documentation.md index 40c21e5355c..d6a13e7483a 100644 --- a/doc/development/writing_documentation.md +++ b/doc/development/writing_documentation.md @@ -1,13 +1,9 @@ -# Writing documentation +# GitLab Documentation guidelines - **General Documentation**: written by the [developers responsible by creating features](#contributing-to-docs). Should be submitted in the same merge request containing code. Feature proposals (by GitLab contributors) should also be accompanied by its respective documentation. They can be later improved by PMs and Technical Writers. - **[Technical Articles](#technical-articles)**: written by any [GitLab Team](https://about.gitlab.com/team/) member, GitLab contributors, or [Community Writers](https://about.gitlab.com/handbook/product/technical-writing/community-writers/). - **Indexes per topic**: initially prepared by the Technical Writing Team, and kept up-to-date by developers and PMs in the same merge request containing code. They gather all resources for that topic in a single page (user and admin documentation, articles, and third-party docs). -## Documentation style guidelines - -All the docs follow the same [styleguide](doc_styleguide.md). - ## Contributing to docs Whenever a feature is changed, updated, introduced, or deprecated, the merge @@ -29,23 +25,16 @@ Whenever you submit a merge request for the documentation, use the documentation Please check the [documentation workflow](https://about.gitlab.com/handbook/product/technical-writing/workflow/) before getting started. -### Documentation directory structure - -The documentation is structured based on the GitLab UI structure itself, -separated by [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), -[`administrator`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`contributor`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development). - -To learn where to place a new document, check the [documentation style guide](doc_styleguide.md#location-and-naming-of-documents). +## Documentation structure -In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, -all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. +- Overview and use cases: what it is, why it is necessary, why one would use it +- Requirements: what do we need to get started +- Tutorial: how to set it up, how to use it -The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have -been deprecated and the majority their docs have been moved to their correct location -in small iterations. Don't create new docs in these folders. +Always link a new document from its topic-related index, otherwise, it will +not be included it in the documentation site search. -To move a document from its location to another directory, read the section -[changing document location](doc_styleguide.md#changing-document-location) of the doc style guide. +_Note: to be extended._ ### Feature overview and use cases @@ -75,16 +64,169 @@ overview there. > **Overview** and **use cases** are required to **every** Enterprise Edition feature, and for every **major** feature present in Community Edition. -### Markdown +## Markdown and styles Currently GitLab docs use Redcarpet as [markdown](../user/markdown.md) engine, but there's an [open discussion](https://gitlab.com/gitlab-com/gitlab-docs/issues/50) for implementing Kramdown in the near future. -### Previewing locally +All the docs follow the [documentation style guidelines](doc_styleguide.md). -To preview your changes to documentation locally, please follow -this [development guide](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md#development). +## Documentation directory structure + +The documentation is structured based on the GitLab UI structure itself, +separated by [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), +[`administrator`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`contributor`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development). + +In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, +all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. + +The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have +been deprecated and the majority their docs have been moved to their correct location +in small iterations. Please don't create new docs in these folders. + +### Location and naming documents + +The documentation hierarchy can be vastly improved by providing a better layout +and organization of directories. + +Having a structured document layout, we will be able to have meaningful URLs +like `docs.gitlab.com/user/project/merge_requests/index.html`. With this pattern, +you can immediately tell that you are navigating a user related documentation +and is about the project and its merge requests. + +Do not create summaries of similar types of content (e.g. an index of all articles, videos, etc.), +rather organize content by its subject (e.g. everything related to CI goes together) +and cross-link between any related content. + +The table below shows what kind of documentation goes where. + +| Directory | What belongs here | +| --------- | -------------- | +| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here including `/admin`. | +| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface go under `doc/user/admin_area/`. | +| `doc/api/` | API related documentation. | +| `doc/development/` | Documentation related to the development of GitLab. Any styleguides should go here. | +| `doc/legal/` | Legal documents about contributing to GitLab. | +| `doc/install/`| Probably the most visited directory, since `installation.md` is there. Ideally this should go under `doc/administration/`, but it's best to leave it as-is in order to avoid confusion (still debated though). | +| `doc/update/` | Same with `doc/install/`. Should be under `administration/`, but this is a well known location, better leave as-is, at least for now. | +| `doc/topics/` | Indexes per Topic (`doc/topics/topic-name/index.md`): all resources for that topic (user and admin documentation, articles, and third-party docs) | + +--- + +**General rules:** + +1. The correct naming and location of a new document, is a combination + of the relative URL of the document in question and the GitLab Map design + that is used for UX purposes ([source][graffle], [image][gitlab-map]). +1. When creating a new document and it has more than one word in its name, + make sure to use underscores instead of spaces or dashes (`-`). For example, + a proper naming would be `import_projects_from_github.md`. The same rule + applies to images. +1. Start a new directory with an `index.md` file. +1. There are four main directories, `user`, `administration`, `api` and `development`. +1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, + `profile/`, `dashboard/` and `admin_area/`. + 1. `doc/user/project/` should contain all project related documentation. + 1. `doc/user/group/` should contain all group related documentation. + 1. `doc/user/profile/` should contain all profile related documentation. + Every page you would navigate under `/profile` should have its own document, + i.e. `account.md`, `applications.md`, `emails.md`, etc. + 1. `doc/user/dashboard/` should contain all dashboard related documentation. + 1. `doc/user/admin_area/` should contain all admin related documentation + describing what can be achieved by accessing GitLab's admin interface + (_not to be confused with `doc/administration` where server access is + required_). + 1. Every category under `/admin/application_settings` should have its + own document located at `doc/user/admin_area/settings/`. For example, + the **Visibility and Access Controls** category should have a document + located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. +1. The `doc/topics/` directory holds topic-related technical content. Create + `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. + General user- and admin- related documentation, should be placed accordingly. + +If you are unsure where a document should live, you can ping `@axil` or `@marcia` in your +merge request. + +### Changing document location + +Changing a document's location is not to be taken lightly. Remember that the +documentation is available to all installations under `help/` and not only to +GitLab.com or http://docs.gitlab.com. Make sure this is discussed with the +Documentation team beforehand. + +If you indeed need to change a document's location, do NOT remove the old +document, but rather replace all of its contents with a new line: + +``` +This document was moved to [another location](path/to/new_doc.md). +``` + +where `path/to/new_doc.md` is the relative path to the root directory `doc/`. + +--- + +For example, if you were to move `doc/workflow/lfs/lfs_administration.md` to +`doc/administration/lfs.md`, then the steps would be: + +1. Copy `doc/workflow/lfs/lfs_administration.md` to `doc/administration/lfs.md` +1. Replace the contents of `doc/workflow/lfs/lfs_administration.md` with: + + ``` + This document was moved to [another location](../../administration/lfs.md). + ``` + +1. Find and replace any occurrences of the old location with the new one. + A quick way to find them is to use `git grep`. First go to the root directory + where you cloned the `gitlab-ce` repository and then do: -### Testing + ``` + git grep -n "workflow/lfs/lfs_administration" + git grep -n "lfs/lfs_administration" + ``` + +NOTE: **Note:** +If the document being moved has any Disqus comments on it, there are extra steps +to follow documented just [below](#redirections-for-pages-with-disqus-comments). + +Things to note: + +- Since we also use inline documentation, except for the documentation itself, + the document might also be referenced in the views of GitLab (`app/`) which will + render when visiting `/help`, and sometimes in the testing suite (`spec/`). +- The above `git grep` command will search recursively in the directory you run + it in for `workflow/lfs/lfs_administration` and `lfs/lfs_administration` + and will print the file and the line where this file is mentioned. + You may ask why the two greps. Since we use relative paths to link to + documentation, sometimes it might be useful to search a path deeper. +- The `*.md` extension is not used when a document is linked to GitLab's + built-in help page, that's why we omit it in `git grep`. +- Use the checklist on the documentation MR description template. + +### Redirections for pages with Disqus comments + +If the documentation page being relocated already has any Disqus comments, +we need to preserve the Disqus thread. + +Disqus uses an identifier per page, and for docs.gitlab.com, the page identifier +is configured to be the page URL. Therefore, when we change the document location, +we need to preserve the old URL as the same Disqus identifier. + +To do that, add to the frontmatter the variable `redirect_from`, +using the old URL as value. For example, let's say I moved the document +available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, +`https://docs.gitlab.com/my-new-location/index.html`. + +Into the **new document** frontmatter add the following: + +```yaml +--- +redirect_from: 'https://docs.gitlab.com/my-old-location/README.html' +--- +``` + +Note: it is necessary to include the file name in the `redirect_from` URL, +even if it's `index.html` or `README.html`. + +## Testing We treat documentation as code, thus have implemented some testing. Currently, the following tests are in place: @@ -103,7 +245,7 @@ Currently, the following tests are in place: Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is essential to avoid them. -### Branch naming +## Branch naming If your contribution contains **only** documentation changes, you can speed up the CI process by following some branch naming conventions. You have three @@ -118,7 +260,7 @@ choices: If your branch name matches any of the above, it will run only the docs tests. If it doesn't, the whole test suite will run (including docs). -### Merge requests for GitLab documentation +## Merge requests for GitLab documentation Before getting started, make sure you read the introductory section "[contributing to docs](#contributing-to-docs)" above and the @@ -137,7 +279,7 @@ frozen or released, use the label `Pick into X.Y` to get it merged into the correct release. Avoid picking into a past release as much as you can, as it increases the work of the release managers. -#### Cherry-picking from CE to EE +### Cherry-picking from CE to EE As we have the `master` branch of CE merged into EE once a day, it's common to run into merge conflicts. To avoid them, we [test for merge conflicts against EE](#testing) @@ -161,7 +303,10 @@ a couple more commits to the EE branch, but ask the reviewer to review the EE me additionally to the CE MR. If there are many EE-only changes though, start a new MR to EE only. -### Previewing the changes live +## Previewing the changes live + +To preview your changes to documentation locally, please follow +this [development guide](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md#development). If you want to preview the doc changes of your merge request live, you can use the manual `review-docs-deploy` job in your merge request. You will need at @@ -221,7 +366,7 @@ working on. If you don't, the remote docs branch won't be removed either, and the server where the Review Apps are hosted will eventually be out of disk space. -#### Technical aspects +### Technical aspects If you want to know the hot details, here's what's really happening: @@ -256,6 +401,74 @@ The following GitLab features are used among others: - [Artifacts](../ci/yaml/README.md#artifacts) - [Specific Runner](../ci/runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) +## GitLab `/help` + +Every GitLab instance includes the documentation, which is available from `/help` +(`http://my-instance.com/help`), e.g., <https://gitlab.com/help>. + +When you're building a new feature, you may need to link the documentation +from GitLab, the application. This is normally done in files inside the +`app/views/` directory with the help of the `help_page_path` helper method. + +In its simplest form, the HAML code to generate a link to the `/help` page is: + +```haml += link_to 'Help page', help_page_path('user/permissions') +``` + +The `help_page_path` contains the path to the document you want to link to with +the following conventions: + +- it is relative to the `doc/` directory in the GitLab repository +- the `.md` extension must be omitted +- it must not end with a slash (`/`) + +Below are some special cases where should be used depending on the context. +You can combine one or more of the following: + +1. **Linking to an anchor link.** Use `anchor` as part of the `help_page_path` + method: + + ```haml + = link_to 'Help page', help_page_path('user/permissions', anchor: 'anchor-link') + ``` + +1. **Opening links in a new tab.** This should be the default behavior: + + ```haml + = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' + ``` + +1. **Linking to a circle icon.** Usually used in settings where a long + description cannot be used, like near checkboxes. You can basically use + any font awesome icon, but prefer the `question-circle`: + + ```haml + = link_to icon('question-circle'), help_page_path('user/permissions') + ``` + +1. **Using a button link.** Useful in places where text would be out of context + with the rest of the page layout: + + ```haml + = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' + ``` + +1. **Using links inline of some text.** + + ```haml + Description to #{link_to 'Help page', help_page_path('user/permissions')}. + ``` + +1. **Adding a period at the end of the sentence.** Useful when you don't want + the period to be part of the link: + + ```haml + = succeed '.' do + Learn more in the + = link_to 'Help page', help_page_path('user/permissions') + ``` + ## General Documentation vs Technical Articles ### General documentation @@ -270,7 +483,7 @@ They are topic-related documentation, written with an user-friendly approach and A technical article guides users and/or admins to achieve certain objectives (within guides and tutorials), or provide an overview of that particular topic or feature (within technical overviews). It can also describe the use, implementation, or integration of third-party tools with GitLab. -They should be placed in a new directory named `/article-title/index.md` under a topic-related folder, and their images should be placed in `/article-title/img/`. For example, a new article on GitLab Pages should be placed in `doc/user/project/pages/article-title/` and a new article on GitLab CI/CD should be placed in `doc/ci/article-title/`. +They should be placed in a new directory named `/article-title/index.md` under a topic-related folder, and their images should be placed in `/article-title/img/`. For example, a new article on GitLab Pages should be placed in `doc/user/project/pages/article-title/` and a new article on GitLab CI/CD should be placed in `doc/ci/examples/article-title/`. #### Types of Technical Articles @@ -324,3 +537,5 @@ date: 2017-02-01 Use the [writing method](https://about.gitlab.com/handbook/product/technical-writing/#writing-method) defined by the Technical Writing team. +[gitlab-map]: https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png +[graffle]: https://gitlab.com/gitlab-org/gitlab-design/blob/d8d39f4a87b90fb9ae89ca12dc565347b4900d5e/production/resources/gitlab-map.graffle diff --git a/doc/downgrade_ee_to_ce/README.md b/doc/downgrade_ee_to_ce/README.md index 75bae324585..f656057e3da 100644 --- a/doc/downgrade_ee_to_ce/README.md +++ b/doc/downgrade_ee_to_ce/README.md @@ -70,7 +70,7 @@ To downgrade an Omnibus installation, it is sufficient to install the Community Edition package on top of the currently installed one. You can do this manually, by directly [downloading the package](https://packages.gitlab.com/gitlab/gitlab-ce) you need, or by adding our CE package repository and following the -[CE installation instructions](https://about.gitlab.com/downloads/). +[CE installation instructions](https://about.gitlab.com/installation/?version=ce). **Source Installation** diff --git a/doc/install/installation.md b/doc/install/installation.md index 6eb767b00b3..1abbfd78738 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -162,13 +162,14 @@ page](https://golang.org/dl). ## 4. Node -Since GitLab 8.17, GitLab requires the use of node >= v4.3.0 to compile -javascript assets, and yarn >= v0.17.0 to manage javascript dependencies. -In many distros the versions provided by the official package repositories -are out of date, so we'll need to install through the following commands: - - # install node v7.x - curl --location https://deb.nodesource.com/setup_7.x | sudo bash - +Since GitLab 8.17, GitLab requires the use of Node to compile javascript +assets, and Yarn to manage javascript dependencies. The current minimum +requirements for these are node >= v6.0.0 and yarn >= v1.2.0. In many distros +the versions provided by the official package repositories are out of date, so +we'll need to install through the following commands: + + # install node v8.x + curl --location https://deb.nodesource.com/setup_8.x | sudo bash - sudo apt-get install -y nodejs curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index cd889e74487..aa9b8777359 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -10,7 +10,7 @@ should be deployed, upgraded, and configured. ## Chart Overview * **[GitLab-Omnibus](gitlab_omnibus.md)**: The best way to run GitLab on Kubernetes today, suited for small deployments. The chart is in beta and will be deprecated by the [cloud native GitLab chart](#cloud-native-gitlab-chart). -* **[Cloud Native GitLab Chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md)**: The next generation GitLab chart, currently in development. Will support large deployments with horizontal scaling of individual GitLab components. +* **[Cloud Native GitLab Chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md)**: The next generation GitLab chart, currently in alpha. Will support large deployments with horizontal scaling of individual GitLab components. * Other Charts * [GitLab Runner Chart](gitlab_runner_chart.md): For deploying just the GitLab Runner. * [Advanced GitLab Installation](gitlab_chart.md): Deprecated, being replaced by the [cloud native GitLab chart](#cloud-native-gitlab-chart). Provides additional deployment options, but provides less functionality out-of-the-box. @@ -35,9 +35,9 @@ By offering individual containers and charts, we will be able to provide a numbe * Potential for rolling updates and canaries within a service, * and plenty more. -This is a large project and will be worked on over the span of multiple releases. For the most up-to-date status and release information, please see our [tracking issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2420). We are planning to launch this chart in beta by the end of 2017. +Presently this chart is available in alpha for testing, and not recommended for production use. -Learn more about the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). +Learn more about the [cloud native GitLab chart here ](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md) and [here [Video]](https://youtu.be/Z6jWR8Z8dv8). ## Other Charts diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 1f46ee4c1ea..e6ccfccd33f 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -464,7 +464,9 @@ bother us. In any case, it is something to keep in mind when deploying GitLab on a production cluster. In order to deploy GitLab on a production cluster, you will need to assign the -GitLab service account to the `anyuid` Security Context. +GitLab service account to the `anyuid` [Security Context Constraints][scc]. + +For OpenShift v3.0, you will need to do this manually: 1. Edit the Security Context: ```sh @@ -477,6 +479,12 @@ GitLab service account to the `anyuid` Security Context. 1. Save and exit the editor +For OpenShift v3.1 and above, you can do: + +```sh +oc adm policy add-scc-to-user anyuid system:serviceaccount:gitlab:gitlab-ce-user +``` + ## Conclusion By now, you should have an understanding of the basic OpenShift Origin concepts @@ -513,3 +521,4 @@ PaaS and managing your applications with the ease of containers. [autoscaling]: https://docs.openshift.org/latest/dev_guide/pod_autoscaling.html "Documentation - Autoscale" [basic-cli]: https://docs.openshift.org/latest/cli_reference/basic_cli_operations.html "Documentation - Basic CLI operations" [openshift-docs]: https://docs.openshift.org "OpenShift documentation" +[scc]: https://docs.openshift.org/latest/admin_guide/manage_scc.html "Documentation - Managing Security Context Constraints"
\ No newline at end of file diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index 36a8844e953..7d73026a6c6 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -15,9 +15,10 @@ Taking the trigger term as `project-name`, the commands are: | `/project-name issue new <title> <shift+return> <description>` | Creates a new issue with title `<title>` and description `<description>` | | `/project-name issue show <id>` | Shows the issue with id `<id>` | | `/project-name issue search <query>` | Shows up to 5 issues matching `<query>` | +| `/project-name issue move <id> to <project>` | Moves issue ID `<id>` to `<project>` | | `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment | -Note that if you are using the [GitLab Slack application](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html) for +Note that if you are using the [GitLab Slack application](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html) for your GitLab.com projects, you need to [add the `gitlab` keyword at the beginning of the command](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html#usage). ## Issue commands diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 5f5ba2b69bc..4dc3adc1441 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -20,7 +20,7 @@ project in an easy and automatic way: 1. [Auto Test](#auto-test) 1. [Auto Code Quality](#auto-code-quality) 1. [Auto SAST (Static Application Security Testing)](#auto-sast) -1. [Auto SAST for Docker images](#auto-sast-for-docker-images) +1. [Auto Container Scanning](#auto-container-scanning) 1. [Auto Review Apps](#auto-review-apps) 1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast) 1. [Auto Deploy](#auto-deploy) @@ -217,7 +217,7 @@ check out. In GitLab Ultimate, any security warnings are also [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html). -### Auto SAST for Docker images +### Auto Container Scanning > Introduced in GitLab 10.4. @@ -228,7 +228,7 @@ created, it's uploaded as an artifact which you can later download and check out. In GitLab Ultimate, any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/sast_docker.html). +[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). ### Auto Review Apps @@ -309,6 +309,18 @@ enable them. You can make use of [environment variables](#helm-chart-variables) to automatically scale your pod replicas. +It's important to note that when a project is deployed to a Kubernetes cluster, +it relies on a Docker image that has been pushed to the +[GitLab Container Registry](../../user/project/container_registry.md). Kubernetes +fetches this image and uses it to run the application. If the project is public, +the image can be accessed by Kubernetes without any authentication, allowing us +to have deployments more usable. If the project is private/internal, the +Registry requires credentials to pull the image. Currently, this is addressed +by providing `CI_JOB_TOKEN` as the password that can be used, but this token will +no longer be valid as soon as the deployment job finishes. This means that +Kubernetes can run the application, but in case it should be restarted or +executed somewhere else, it cannot be accessed again. + ### Auto Monitoring NOTE: **Note:** diff --git a/doc/update/10.5-to-10.6.md b/doc/update/10.5-to-10.6.md index af8343b5958..f5c5c305726 100644 --- a/doc/update/10.5-to-10.6.md +++ b/doc/update/10.5-to-10.6.md @@ -56,8 +56,8 @@ sudo gem install bundler --no-ri --no-rdoc ### 4. Update Node -GitLab now runs [webpack](http://webpack.js.org) to compile frontend assets. -We require a minimum version of node v6.0.0. +GitLab utilizes [webpack](http://webpack.js.org) to compile frontend assets. +This requires a minimum version of node v6.0.0. You can check which version you are running with `node -v`. If you are running a version older than `v6.0.0` you will need to update to a newer version. You @@ -66,8 +66,8 @@ from source at the nodejs.org website. <https://nodejs.org/en/download/> -Since 8.17, GitLab requires the use of yarn `>= v0.17.0` to manage -JavaScript dependencies. +GitLab also requires the use of yarn `>= v1.2.0` to manage JavaScript +dependencies. ```bash curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 914a80bcd6a..a520279c29e 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -25,7 +25,8 @@ The following table depicts the various user permission levels in a project. | Create confidential issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Lock discussions (issues and merge requests) | | | | ✓ | ✓ | +| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | +| Lock merge request discussions | | | ✓ | ✓ | ✓ | | See a list of jobs | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | Download and browse job artifacts | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 661697aaeb7..bd9bcfadb99 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -167,6 +167,17 @@ external IP address with the following procedure. It can be deployed using the In order to publish your web application, you first need to find the external IP address associated to your load balancer. +### Let GitLab fetch the IP address + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17052) in GitLab 10.6. + +If you installed the Ingress [via the **Applications**](#installing-applications), +you should see the Ingress IP address on this same page within a few minutes. +If you don't see this, GitLab might not be able to determine the IP address of +your ingress application in which case you should manually determine it. + +### Manually determining the IP address + If the cluster is on GKE, click on the **Google Kubernetes Engine** link in the **Advanced settings**, or go directly to the [Google Kubernetes Engine dashboard](https://console.cloud.google.com/kubernetes/) @@ -193,6 +204,24 @@ The output is the external IP address of your cluster. This information can then be used to set up DNS entries and forwarding rules that allow external access to your deployed applications. +### Using a static IP + +By default, an ephemeral external IP address is associated to the cluster's load +balancer. If you associate the ephemeral IP with your DNS and the IP changes, +your apps will not be able to be reached, and you'd have to change the DNS +record again. In order to avoid that, you should change it into a static +reserved IP. + +[Read how to promote an ephemeral external IP address in GKE.](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip) + +### Pointing your DNS at the cluster IP + +Once you've set up the static IP, you should associate it to a [wildcard DNS +record](https://en.wikipedia.org/wiki/Wildcard_DNS_record), in order to be able +to reach your apps. This heavily depends on your domain provider, but in case +you aren't sure, just create an A record with a wildcard host like +`*.example.com.`. + ## Setting the environment scope NOTE: **Note:** diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 249463fb86e..fa7e504c4aa 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -2,7 +2,7 @@ > [Introduced][ce-8935] in GitLab 9.0. -GitLab offers powerful integration with [Prometheus] for monitoring key metrics your apps, directly within GitLab. +GitLab offers powerful integration with [Prometheus] for monitoring key metrics of your apps, directly within GitLab. Metrics for each environment are retrieved from Prometheus, and then displayed within the GitLab interface. @@ -12,17 +12,21 @@ There are two ways to setup Prometheus integration, depending on where your apps * For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) * For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). -## Managed Prometheus on Kubernetes +Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-ci-cd-environments). + +## Enabling Prometheus Integration + +### Managed Prometheus on Kubernetes > **Note**: [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916) in GitLab 10.5 GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring of your apps easy. -### Requirements +#### Requirements * A [connected Kubernetes cluster](../clusters/index.md) * Helm Tiller [installed by GitLab](../clusters/index.md#installing-applications) -### Getting started +#### Getting started Once you have a connected Kubernetes cluster with Helm installed, deploying a managed Prometheus is as easy as a single click. @@ -32,7 +36,7 @@ Once you have a connected Kubernetes cluster with Helm installed, deploying a ma  -### About managed Prometheus deployments +#### About managed Prometheus deployments Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/). @@ -45,9 +49,9 @@ CPU and Memory consumption is monitored, but requires [naming conventions](prome The [NGINX Ingress](../clusters/index.md#installing-applications) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates. -## Manual configuration of Prometheus +### Manual configuration of Prometheus -### Requirements +#### Requirements Integration with Prometheus requires the following: @@ -56,7 +60,7 @@ Integration with Prometheus requires the following: 1. Each metric must be have a label to indicate the environment 1. GitLab must have network connectivity to the Prometheus server -### Getting started +#### Getting started Installing and configuring Prometheus to monitor applications is fairly straight forward. @@ -64,7 +68,7 @@ Installing and configuring Prometheus to monitor applications is fairly straight 1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md) 1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config) -### Configuration in GitLab +#### Configuration in GitLab The actual configuration of Prometheus integration within GitLab is very simple. All you will need is the DNS or IP address of the Prometheus server you'd like @@ -83,9 +87,9 @@ to integrate with. Once configured, GitLab will attempt to retrieve performance metrics for any environment which has had a successful deployment. -GitLab will automatically scan the Prometheus server for known metrics and attempt to identify the metrics for a particular environment. The supported metrics and scan process is detailed in our [Prometheus Metric Library documentation](prometheus_library/metrics.html). +GitLab will automatically scan the Prometheus server for metrics from known serves like Kubernetes and NGINX, and attempt to identify individual environment. The supported metrics and scan process is detailed in our [Prometheus Metric Library documentation](prometheus_library/metrics.html). -[Learn more about monitoring environments.](../../../ci/environments.md#monitoring-environments) +You can view the performance dashboard for an environment by [clicking on the monitoring button](../../../ci/environments.md#monitoring-environments). ## Determining the performance impact of a merge @@ -93,7 +97,7 @@ GitLab will automatically scan the Prometheus server for known metrics and attem > GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages. > Requires [Kubernetes](prometheus_library/kubernetes.md) metrics -Developers can view theperformance impact of their changes within the merge +Developers can view the performance impact of their changes within the merge request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot indicates when the current changes were deployed, with up to 30 minutes of performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after @@ -109,7 +113,7 @@ Prometheus server. ## Troubleshooting -If the "Attempting to load performance data" screen continues to appear, it could be due to: +If the "No data found" screen continues to appear, it could be due to: - No successful deployments have occurred to this environment. - Prometheus does not have performance data for this environment, or the metrics diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 550ad4a8b1f..d403d5698a9 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -245,6 +245,17 @@ navigation level. A group-level issue board allows you to view all issues from a boards. When updating milestones and labels for an issue through the sidebar update mechanism, again only group-level objects are available. +## Features per tier + +Different issue board features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: + +| Tier | Number of project issue boards | Board with configuration in project issue boards | Number of group issue boards | Board with configuration in group issue boards | +| --- | --- | --- | --- | --- | +| Libre | 1 | No | 1 | No | +| Starter | Multiple | Yes | 1 | No | +| Premium | Multiple | Yes | Multiple | Yes | +| Ultimate | Multiple | Yes | Multiple | Yes | + ## Tips A few things to remember: diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index f5c748a03b3..5d819998dd9 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -16,19 +16,29 @@ say 'Project Acme', in GitLab is to make the 'Engineering' group the owner of 'P Acme'. But what if 'Project Acme' already belongs to another group, say 'Open Source'? This is where the group sharing feature can be of use. -To share 'Project Acme' with the 'Engineering' group, go to the project settings page for 'Project Acme' and use the left navigation menu to go to the **Settings > Members** section. +To share 'Project Acme' with the 'Engineering' group: - +1. For 'Project Acme' use the left navigation menu to go to **Settings > Members** -Then select the 'Share with group' tab by clicking it. +  -Now you can add the 'Engineering' group with the maximum access level of your choice. Click 'Share' to share it. +1. Select the 'Share with group' tab +1. Add the 'Engineering' group with the maximum access level of your choice +1. Click **Share** to share it - +  -After sharing 'Project Acme' with 'Engineering', the project will be listed on the group dashboard. +1. After sharing 'Project Acme' with 'Engineering', the project will be listed + on the group dashboard - +  + +Note that you can only share a project with: + +- groups for which you have an explicitly defined membership +- groups that contain a nested subgroup or project for which you have an explicitly defined role + +Admins are able to share projects with any group in the system. ## Maximum access level diff --git a/doc/user/project/merge_requests/img/allow_maintainer_push.png b/doc/user/project/merge_requests/img/allow_maintainer_push.png Binary files differindex 1631527071b..91cc399f4ff 100644 --- a/doc/user/project/merge_requests/img/allow_maintainer_push.png +++ b/doc/user/project/merge_requests/img/allow_maintainer_push.png diff --git a/doc/user/project/merge_requests/maintainer_access.md b/doc/user/project/merge_requests/maintainer_access.md index 7feccc28f6b..c9763a3fe02 100644 --- a/doc/user/project/merge_requests/maintainer_access.md +++ b/doc/user/project/merge_requests/maintainer_access.md @@ -1,12 +1,17 @@ # Allow maintainer pushes for merge requests across forks +> [Introduced][ce-17395] in GitLab 10.6. + This feature is available for merge requests across forked projects that are -publicly accessible. It makes it easier for maintainers of projects to collaborate -on merge requests across forks. +publicly accessible. It makes it easier for maintainers of projects to +collaborate on merge requests across forks. -When enabling this feature for a merge request, you give can give members with push access to the target project rights to edit files on the source branch of the merge request. +When enabled for a merge request, members with merge access to the target +branch of the project will be granted write permissions to the source branch +of the merge request. -The feature can only be enabled by users who already have push access to the source project. And only lasts while the merge request is open. +The feature can only be enabled by users who already have push access to the +source project, and only lasts while the merge request is open. Enable this functionality while creating a merge request: diff --git a/doc/user/project/repository/img/jupyter_notebook.png b/doc/user/project/repository/img/jupyter_notebook.png Binary files differnew file mode 100644 index 00000000000..52c5c5aea32 --- /dev/null +++ b/doc/user/project/repository/img/jupyter_notebook.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index e6aede7f46e..ae131d51305 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -53,6 +53,22 @@ To get started with the command line, please read through the Use GitLab's [file finder](../../../workflow/file_finder.md) to search for files in a repository. +### Jupyter Notebook files + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2508) in GitLab 9.1 + +[Jupyter][jupyter] Notebook (previously IPython Notebook) files are used for +interactive computing in many fields and contain a complete record of the +user's sessions and include code, narrative text, equations and rich output. + +When added to a repository, Jupyter Notebooks with a `.ipynb` extension will be +rendered to HTML when viewed. + + + +Interactive features, including JavaScript plots, will not work when viewed in +GitLab. + ## Branches When you submit changes in a new [branch](branches/index.md), you create a new version @@ -158,3 +174,5 @@ Lock your files to prevent any conflicting changes. ## Repository's API You can access your repos via [repository API](../../../api/repositories.md). + +[jupyter]: https://jupyter.org |
