summaryrefslogtreecommitdiff
path: root/doc/ci
diff options
context:
space:
mode:
authorFilipa Lacerda <filipa@gitlab.com>2018-03-19 10:05:08 +0000
committerFilipa Lacerda <filipa@gitlab.com>2018-03-19 10:05:08 +0000
commit2a5f9709614a33143cb0f47a2602cf161ba4b2cd (patch)
treee7b5d209853ac608a4aeb67bb4a7161a2f64f143 /doc/ci
parent024a0b837b6cb93f6ba5171541eadfc64e3897d6 (diff)
parente1739e47c5664c93c66dd58ded59f9d79cd8a426 (diff)
downloadgitlab-ce-2a5f9709614a33143cb0f47a2602cf161ba4b2cd.tar.gz
[ci skip] Merge branch 'master' into 42568-pipeline-empty-state
* master: (156 commits) Make issue boards height calculation clearer and remove magic numbers Rename `package-qa` in docs Copyedit CI services docs Fix dropzone project show Prettify notes. Fix "A copy of Gitlab::Metrics::Methods" have been removed error Add Jupyter Notebook docs Fixed profile notifications not being editable Rename manual job to `package-and-qa` Update vendored gitlab-ci.yml template for auto-devops Docs: refactor doc general guidelines and style guidelines Use Gitaly 0.91.0 Add version available info to integrity check rake task docs Adds the option to override project description on export via API and fixes the project description not being imported Extract constant for LfsPointerFile::VERSION_LINE Setup Faraday middleware before adapter Revert "Merge branch 'update-httparty' into 'master'" Update Code Quality example documentation Fix timeouts loading /admin/projects page OpenShift install docs: Recommend "add-scc-to-user" over "edit scc" ...
Diffstat (limited to 'doc/ci')
-rw-r--r--doc/ci/docker/using_docker_images.md63
-rw-r--r--doc/ci/examples/README.md7
-rw-r--r--doc/ci/examples/code_climate.md22
-rw-r--r--doc/ci/examples/container_scanning.md55
-rw-r--r--doc/ci/examples/sast_docker.md56
-rw-r--r--doc/ci/yaml/README.md8
6 files changed, 142 insertions, 69 deletions
diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md
index fb5bfe26bb0..bc5d3840368 100644
--- a/doc/ci/docker/using_docker_images.md
+++ b/doc/ci/docker/using_docker_images.md
@@ -58,7 +58,7 @@ your job and is linked to the Docker image that the `image` keyword defines.
This allows you to access the service image during build time.
The service image can run any application, but the most common use case is to
-run a database container, eg. `mysql`. It's easier and faster to use an
+run a database container, e.g., `mysql`. It's easier and faster to use an
existing image and run it as an additional container than install `mysql` every
time the project is built.
@@ -83,6 +83,67 @@ So, in order to access your database service you have to connect to the host
named `mysql` instead of a socket or `localhost`. Read more in [accessing the
services](#accessing-the-services).
+### How the health check of services works
+
+Services are designed to provide additional functionality which is **network accessible**.
+It may be a database like MySQL, or Redis, and even `docker:dind` which
+allows you to use Docker in Docker. It can be practically anything that is
+required for the CI/CD job to proceed and is accessed by network.
+
+To make sure this works, the Runner:
+
+1. checks which ports are exposed from the container by default
+1. starts a special container that waits for these ports to be accessible
+
+When the second stage of the check fails, either because there is no opened port in the
+service, or the service was not started properly before the timeout and the port is not
+responding, it prints the warning: `*** WARNING: Service XYZ probably didn't start properly`.
+
+In most cases it will affect the job, but there may be situations when the job
+will still succeed even if that warning was printed. For example:
+
+- The service was started a little after the warning was raised, and the job is
+ not using the linked service from the very beginning. In that case, when the
+ job needed to access the service, it may have been already there waiting for
+ connections.
+- The service container is not providing any networking service, but it's doing
+ something with the job's directory (all services have the job directory mounted
+ as a volume under `/builds`). In that case, the service will do its job, and
+ since the job is not trying to connect to it, it won't fail.
+
+### What services are not for
+
+As it was mentioned before, this feature is designed to provide **network accessible**
+services. A database is the simplest example of such a service.
+
+NOTE: **Note:**
+The services feature is not designed to, and will not add any software from the
+defined `services` image(s) to the job's container.
+
+For example, if you have the following `services` defined in your job, the `php`,
+`node` or `go` commands will **not** be available for your script, and thus
+the job will fail:
+
+```yaml
+job:
+ services:
+ - php:7
+ - node:latest
+ - golang:1.10
+ image: alpine:3.7
+ script:
+ - php -v
+ - node -v
+ - go version
+```
+
+If you need to have `php`, `node` and `go` available for your script, you should
+either:
+
+- choose an existing Docker image that contains all required tools, or
+- create your own Docker image, which will have all the required tools included
+ and use that in your job
+
### Accessing the services
Let's say that you need a Wordpress instance to test some API integration with
diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md
index f69729f602d..c1e258aedca 100644
--- a/doc/ci/examples/README.md
+++ b/doc/ci/examples/README.md
@@ -47,8 +47,11 @@ There's also a collection of repositories with [example projects](https://gitlab
## Static Application Security Testing (SAST)
-- **(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html)
-- [Scan your Docker images for vulnerabilities](sast_docker.md)
+**(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html)
+
+## Container Scanning
+
+[Scan your Docker images for vulnerabilities](container_scanning.md)
## Dynamic Application Security Testing (DAST)
diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md
index d7df53494ed..ec5e5afb8c6 100644
--- a/doc/ci/examples/code_climate.md
+++ b/doc/ci/examples/code_climate.md
@@ -16,18 +16,26 @@ codequality:
- docker:dind
script:
- docker pull codeclimate/codeclimate
- - docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 init
- - docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 analyze -f json > codeclimate.json || true
+ - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
+ - docker run
+ --env SOURCE_CODE="$PWD" \
+ --volume "$PWD":/code \
+ --volume /var/run/docker.sock:/var/run/docker.sock \
+ "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code
artifacts:
paths: [codeclimate.json]
```
-This will create a `codequality` job in your CI pipeline and will allow you to
-download and analyze the report artifact in JSON format.
+The above example will create a `codequality` job in your CI/CD pipeline which
+will scan your source code for code quality issues. The report will be saved
+as an artifact that you can later download and analyze.
-For [GitLab Starter][ee] users, this information can be automatically
-extracted and shown right in the merge request widget. [Learn more on code quality
-diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html).
+TIP: **Tip:**
+Starting with [GitLab Starter][ee] 9.3, this information will
+be automatically extracted and shown right in the merge request widget. To do
+so, the CI/CD job must be named `codequality` and the artifact path must be
+`codeclimate.json`.
+[Learn more on code quality diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html).
[cli]: https://github.com/codeclimate/codeclimate
[dind]: ../docker/using_docker_build.md#use-docker-in-docker-executor
diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md
new file mode 100644
index 00000000000..3437b63748a
--- /dev/null
+++ b/doc/ci/examples/container_scanning.md
@@ -0,0 +1,55 @@
+# Container Scanning with GitLab CI/CD
+
+You can check your Docker images (or more precisely the containers) for known
+vulnerabilities by using [Clair](https://github.com/coreos/clair) and
+[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools
+for Vulnerability Static Analysis for containers.
+
+All you need is a GitLab Runner with the Docker executor (the shared Runners on
+GitLab.com will work fine). You can then add a new job to `.gitlab-ci.yml`,
+called `sast:container`:
+
+```yaml
+sast:container:
+ image: docker:latest
+ variables:
+ DOCKER_DRIVER: overlay2
+ ## Define two new variables based on GitLab's CI/CD predefined variables
+ ## https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables
+ CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG
+ CI_APPLICATION_TAG: $CI_COMMIT_SHA
+ allow_failure: true
+ services:
+ - docker:dind
+ script:
+ - docker run -d --name db arminc/clair-db:latest
+ - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1
+ - apk add -U wget ca-certificates
+ - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG}
+ - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
+ - mv clair-scanner_linux_amd64 clair-scanner
+ - chmod +x clair-scanner
+ - touch clair-whitelist.yml
+ - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-sast-container-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true
+ artifacts:
+ paths: [gl-sast-container-report.json]
+```
+
+The above example will create a `sast:container` job in your CI/CD pipeline, pull
+the image from the [Container Registry](../../user/project/container_registry.md)
+(whose name is defined from the two `CI_APPLICATION_` variables) and scan it
+for possible vulnerabilities. The report will be saved as an artifact that you
+can later download and analyze.
+
+If you want to whitelist some specific vulnerabilities, you can do so by defining
+them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file),
+in our case its named `clair-whitelist.yml`.
+
+TIP: **Tip:**
+Starting with [GitLab Ultimate][ee] 10.4, this information will
+be automatically extracted and shown right in the merge request widget. To do
+so, the CI/CD job must be named `sast:container` and the artifact path must be
+`gl-sast-container-report.json`.
+[Learn more on container scanning results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html).
+
+[ee]: https://about.gitlab.com/products/
diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md
index 57a9c4bcfc1..9f4a63e296d 100644
--- a/doc/ci/examples/sast_docker.md
+++ b/doc/ci/examples/sast_docker.md
@@ -1,55 +1 @@
-# Static Application Security Testing for Docker containers with GitLab CI/CD
-
-You can check your Docker images (or more precisely the containers) for known
-vulnerabilities by using [Clair](https://github.com/coreos/clair) and
-[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools
-for Vulnerability Static Analysis for containers.
-
-All you need is a GitLab Runner with the Docker executor (the shared Runners on
-GitLab.com will work fine). You can then add a new job to `.gitlab-ci.yml`,
-called `sast:container`:
-
-```yaml
-sast:container:
- image: docker:latest
- variables:
- DOCKER_DRIVER: overlay2
- ## Define two new variables based on GitLab's CI/CD predefined variables
- ## https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables
- CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG
- CI_APPLICATION_TAG: $CI_COMMIT_SHA
- allow_failure: true
- services:
- - docker:dind
- script:
- - docker run -d --name db arminc/clair-db:latest
- - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1
- - apk add -U wget ca-certificates
- - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG}
- - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
- - mv clair-scanner_linux_amd64 clair-scanner
- - chmod +x clair-scanner
- - touch clair-whitelist.yml
- - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-sast-container-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true
- artifacts:
- paths: [gl-sast-container-report.json]
-```
-
-The above example will create a `sast:container` job in your CI/CD pipeline, pull
-the image from the [Container Registry](../../user/project/container_registry.md)
-(whose name is defined from the two `CI_APPLICATION_` variables) and scan it
-for possible vulnerabilities. The report will be saved as an artifact that you
-can later download and analyze.
-
-If you want to whitelist some specific vulnerabilities, you can do so by defining
-them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file),
-in our case its named `clair-whitelist.yml`.
-
-TIP: **Tip:**
-Starting with [GitLab Ultimate][ee] 10.4, this information will
-be automatically extracted and shown right in the merge request widget. To do
-so, the CI/CD job must be named `sast:container` and the artifact path must be
-`gl-sast-container-report.json`.
-[Learn more on application security testing results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/sast_docker.html).
-
-[ee]: https://about.gitlab.com/products/
+This document was moved to [another location](./container_scanning.md). \ No newline at end of file
diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md
index ae200f9b6e2..accf6340398 100644
--- a/doc/ci/yaml/README.md
+++ b/doc/ci/yaml/README.md
@@ -966,7 +966,7 @@ tag including only the files that are untracked by Git:
```yaml
job:
artifacts:
- name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
+ name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
untracked: true
```
@@ -975,7 +975,7 @@ To create an archive with a name of the current [stage](#stages) and branch name
```yaml
job:
artifacts:
- name: "${CI_JOB_STAGE}_${CI_COMMIT_REF_NAME}"
+ name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME"
untracked: true
```
@@ -987,7 +987,7 @@ If you use **Windows Batch** to run your shell scripts you need to replace
```yaml
job:
artifacts:
- name: "%CI_JOB_STAGE%_%CI_COMMIT_REF_NAME%"
+ name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%"
untracked: true
```
@@ -997,7 +997,7 @@ If you use **Windows PowerShell** to run your shell scripts you need to replace
```yaml
job:
artifacts:
- name: "$env:CI_JOB_STAGE_$env:CI_COMMIT_REF_NAME"
+ name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME"
untracked: true
```