diff options
Diffstat (limited to 'doc')
47 files changed, 913 insertions, 163 deletions
diff --git a/doc/administration/img/circuitbreaker_config.png b/doc/administration/img/circuitbreaker_config.png Binary files differindex e811d173634..693b2ee9c69 100644 --- a/doc/administration/img/circuitbreaker_config.png +++ b/doc/administration/img/circuitbreaker_config.png diff --git a/doc/administration/img/repository_storages_admin_ui.png b/doc/administration/img/repository_storages_admin_ui.png Binary files differindex 3e76c5b282c..036e708cdac 100644 --- a/doc/administration/img/repository_storages_admin_ui.png +++ b/doc/administration/img/repository_storages_admin_ui.png diff --git a/doc/administration/index.md b/doc/administration/index.md index 60a45426636..0906821d6a3 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -39,6 +39,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. +- [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. #### Customizing GitLab's appearance @@ -85,7 +86,6 @@ created in snippets, wikis, and repos. - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. -server with IMAP authentication on Ubuntu, to be used with Reply by email. - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. [reply by email]: reply_by_email.md diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index ac3a12930c3..896cb93e5ed 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -108,6 +108,7 @@ For source installations the following settings are nested under `artifacts:` an |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Artfacts will be stored| | +| `direct_upload` | Set to true to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. Currently only `Google` provider is supported | `false` | | `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 00a2f3d01b8..c8a3ef80e8f 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -146,6 +146,28 @@ this file. For example: 2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} ``` +Instead of the format above, you can opt to generate JSON logs for +Sidekiq. For example: + +```json +{"severity":"INFO","time":"2018-04-03T22:57:22.071Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"06aeaa3b0aadacf9981f368e","created_at":"2018-04-03T22:57:21.930Z","enqueued_at":"2018-04-03T22:57:21.931Z","pid":10077,"message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec","job_status":"done","duration":0.139,"completed_at":"2018-04-03T22:57:22.071Z"} +``` + +For Omnibus GitLab installations, add the configuration option: + +```ruby +sidekiq['log_format'] = 'json' +``` + +For source installations, edit the `gitlab.yml` and set the Sidekiq +`log_format` configuration option: + +```yaml + ## Sidekiq + sidekiq: + log_format: json +``` + ## `gitlab-shell.log` This file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log` for @@ -206,4 +228,12 @@ is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure was initiated, such as `1509705644.log` +## `sidekiq_exporter.log` + +If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq will +start a Web server and listen to the defined port (default: 3807). Access logs +will be generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for +installations from source. + [repocheck]: repository_checks.md diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index f43c89dad87..3d24812c66a 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -62,7 +62,14 @@ To change the address/port that Prometheus listens on: ``` Replace `localhost:9090` with the address/port you want Prometheus to - listen on. + listen on. If you would like to allow access to Prometheus to hosts other + than `localhost`, leave out the host, or use `0.0.0.0` to allow public access: + + ```ruby + prometheus['listen_address'] = ':9090' + # or + prometheus['listen_address'] = '0.0.0.0:9090' + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index c91ac3012b9..3ae41638ac3 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -1,66 +1,80 @@ -# Plugins +# GitLab Plugin system -**Note:** Plugins must be configured on the filesystem of the GitLab -server. Only GitLab server administrators will be able to complete these tasks. -Please explore [system hooks] or [webhooks] as an option if you do not -have filesystem access. +> Introduced in GitLab 10.6. -Introduced in GitLab 10.6. +With custom plugins, GitLab administrators can introduce custom integrations +without modifying GitLab's source code. -A plugin will run on each event so it's up to you to filter events or projects within a plugin code. You can have as many plugins as you want. Each plugin will be triggered by GitLab asynchronously in case of an event. For a list of events please see [system hooks] documentation. +NOTE: **Note:** +Instead of writing and supporting your own plugin you can make changes +directly to the GitLab source code and contribute back upstream. This way we can +ensure functionality is preserved across versions and covered by tests. + +NOTE: **Note:** +Plugins must be configured on the filesystem of the GitLab server. Only GitLab +server administrators will be able to complete these tasks. Explore +[system hooks] or [webhooks] as an option if you do not have filesystem access. + +A plugin will run on each event so it's up to you to filter events or projects +within a plugin code. You can have as many plugins as you want. Each plugin will +be triggered by GitLab asynchronously in case of an event. For a list of events +see the [system hooks] documentation. ## Setup -Plugins must be placed directly into `plugins` directory, subdirectories will be ignored. -There is an `example` directory inside `plugins` where you can find some basic examples. +The plugins must be placed directly into the `plugins` directory, subdirectories +will be ignored. There is an +[`example` directory inside `plugins`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/plugins/examples) +where you can find some basic examples. Follow the steps below to set up a custom hook: -1. On the GitLab server, navigate to the project's plugin directory. +1. On the GitLab server, navigate to the plugin directory. For an installation from source the path is usually `/home/git/gitlab/plugins/`. For Omnibus installs the path is usually `/opt/gitlab/embedded/service/gitlab-rails/plugins`. -1. Inside the `plugins` directory, create a file with a name of your choice, but without spaces or special characters. +1. Inside the `plugins` directory, create a file with a name of your choice, + without spaces or special characters. 1. Make the hook file executable and make sure it's owned by the git user. -1. Write the code to make the plugin function as expected. Plugin can be - in any language. Ensure the 'shebang' at the top properly reflects the language - type. For example, if the script is in Ruby the shebang will probably be - `#!/usr/bin/env ruby`. -1. The data to the plugin will be provided as JSON on STDIN. It will be exactly same as one for [system hooks] +1. Write the code to make the plugin function as expected. That can be + in any language, and ensure the 'shebang' at the top properly reflects the + language type. For example, if the script is in Ruby the shebang will + probably be `#!/usr/bin/env ruby`. +1. The data to the plugin will be provided as JSON on STDIN. It will be exactly + same as for [system hooks] -That's it! Assuming the plugin code is properly implemented the hook will fire -as appropriate. Plugins file list is updated for each event. There is no need to restart GitLab to apply a new plugin. +That's it! Assuming the plugin code is properly implemented, the hook will fire +as appropriate. The plugins file list is updated for each event, there is no +need to restart GitLab to apply a new plugin. If a plugin executes with non-zero exit code or GitLab fails to execute it, a message will be logged to `plugin.log`. ## Validation -Writing own plugin can be tricky and its easier if you can check it without altering the system. -We provided a rake task you can use with staging environment to test your plugin before using it in production. -The rake task will use a sample data and execute each of plugins. By output you should be able to determine if -system sees your plugin and if it was executed without errors. +Writing your own plugin can be tricky and it's easier if you can check it +without altering the system. A rake task is provided so that you can use it +in a staging environment to test your plugin before using it in production. +The rake task will use a sample data and execute each of plugin. The output +should be enough to determine if the system sees your plugin and if it was +executed without errors. ```bash # Omnibus installations sudo gitlab-rake plugins:validate # Installations from source +cd /home/git/gitlab bundle exec rake plugins:validate RAILS_ENV=production ``` -Example of output can be next: +Example of output: ``` --> bundle exec rake plugins:validate RAILS_ENV=production Validating plugins from /plugins directory * /home/git/gitlab/plugins/save_to_file.clj succeed (zero exit code) * /home/git/gitlab/plugins/save_to_file.rb failure (non-zero exit code) ``` -[hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks [system hooks]: ../system_hooks/system_hooks.md [webhooks]: ../user/project/integrations/webhooks.md -[5073]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5073 -[93]: https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/93 - diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index a82735cc72c..2fa3284b6be 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -65,6 +65,7 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | +| `direct_upload` | Set to true to enable direct upload of Uploads without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. This is beta option as it uses inefficient way of uploading data (via Unicorn). The accelerated uploads gonna be implemented in future releases | `false` | | `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | diff --git a/doc/api/commits.md b/doc/api/commits.md index a6b96ba539f..d1584cf64de 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -412,9 +412,10 @@ Example response: Since GitLab 8.1, this is the new commit status API. -### Get the status of a commit +### List the statuses of a commit -Get the statuses of a commit in a project. +List the statuses of a commit in a project. +The pagination parameters `page` and `per_page` can be used to restrict the list of references. ``` GET /projects/:id/repository/commits/:sha/statuses @@ -538,6 +539,8 @@ Example response: ## List Merge Requests associated with a commit +> [Introduced][ce-18004] in GitLab 10.7. + Get a list of Merge Requests related to the specified commit. ``` @@ -607,3 +610,4 @@ Example response: [ce-6096]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6096 "Multi-file commit" [ce-8047]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8047 [ce-15026]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15026 +[ce-18004]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18004 diff --git a/doc/api/features.md b/doc/api/features.md index 6861dbf00a2..6ee1c36ef5b 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -86,3 +86,11 @@ Example response: ] } ``` + +## Delete a feature + +Removes a feature gate. Response is equal when the gate exists, or doesn't. + +``` +DELETE /features/:name +``` diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md index 3e0683f378d..0d7d0fd9c42 100644 --- a/doc/api/group_badges.md +++ b/doc/api/group_badges.md @@ -1,5 +1,8 @@ # Group badges API +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17082) +in GitLab 10.6. + ## Placeholder tokens Badges support placeholders that will be replaced in real time in both the link and image URL. The allowed placeholders are: @@ -182,7 +185,7 @@ curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/a Example response: ```json -{ +{ "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", diff --git a/doc/api/issues.md b/doc/api/issues.md index a4a51101297..7479c1d2f93 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -100,6 +100,7 @@ Example response: }, "updated_at" : "2016-01-04T15:31:51.081Z", "closed_at" : null, + "closed_by" : null, "id" : 76, "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", "created_at" : "2016-01-04T15:31:51.081Z", @@ -122,6 +123,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## List group issues Get a list of a group's issues. @@ -216,6 +219,7 @@ Example response: "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", "closed_at" : null, + "closed_by" : null, "user_notes_count": 1, "due_date": null, "web_url": "http://example.com/example/example/issues/1", @@ -233,6 +237,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## List project issues Get a list of a project's issues. @@ -326,6 +332,14 @@ Example response: "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", "closed_at" : "2016-01-05T15:31:46.176Z", + "closed_by" : { + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" + }, "user_notes_count": 1, "due_date": "2016-07-22", "web_url": "http://example.com/example/example/issues/1", @@ -343,6 +357,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Single issue Get a single project issue. @@ -409,6 +425,8 @@ Example response: "title" : "Ut commodi ullam eos dolores perferendis nihil sunt.", "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", + "closed_at" : null, + "closed_by" : null, "subscribed": false, "user_notes_count": 1, "due_date": null, @@ -432,6 +450,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## New issue Creates a new project issue. @@ -484,6 +504,7 @@ Example response: "description" : null, "updated_at" : "2016-01-07T12:44:33.959Z", "closed_at" : null, + "closed_by" : null, "milestone" : null, "subscribed" : true, "user_notes_count": 0, @@ -508,6 +529,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Edit issue Updates an existing project issue. This call is also used to mark an issue as @@ -556,6 +579,14 @@ Example response: "description" : null, "updated_at" : "2016-01-07T12:55:16.213Z", "closed_at" : "2016-01-08T12:55:16.213Z", + "closed_by" : { + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" + }, "iid" : 15, "labels" : [ "bug" @@ -587,6 +618,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Delete an issue Only for admins and project owners. Soft deletes the issue in question. @@ -640,6 +673,7 @@ Example response: "created_at": "2016-04-05T21:41:45.652Z", "updated_at": "2016-04-07T12:20:17.596Z", "closed_at": null, + "closed_by": null, "labels": [], "milestone": null, "assignees": [{ @@ -687,6 +721,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Subscribe to an issue Subscribes the authenticated user to an issue to receive notifications. @@ -719,6 +755,7 @@ Example response: "created_at": "2016-04-05T21:41:45.652Z", "updated_at": "2016-04-07T12:20:17.596Z", "closed_at": null, + "closed_by": null, "labels": [], "milestone": null, "assignees": [{ @@ -766,6 +803,9 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Unsubscribe from an issue Unsubscribes the authenticated user from the issue to not receive notifications @@ -807,6 +847,8 @@ Example response: "avatar_url": "http://www.gravatar.com/avatar/3e6f06a86cf27fa8b56f3f74f7615987?s=80&d=identicon", "web_url": "https://gitlab.example.com/keyon" }, + "closed_at": null, + "closed_by": null, "author": { "name": "Vivian Hermann", "username": "orville", @@ -927,6 +969,9 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Set a time estimate for an issue Sets an estimated time of work for this issue. @@ -1112,6 +1157,8 @@ Example response: "assignee": null, "source_project_id": 1, "target_project_id": 1, + "closed_at": null, + "closed_by": null, "labels": [], "work_in_progress": false, "milestone": null, @@ -1206,3 +1253,4 @@ Example response: [ce-13004]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13004 [ce-14016]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14016 +[ce-17042]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17042 diff --git a/doc/api/project_badges.md b/doc/api/project_badges.md index 3f6e348b5b4..94389273e9c 100644 --- a/doc/api/project_badges.md +++ b/doc/api/project_badges.md @@ -1,5 +1,8 @@ # Project badges API +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17082) +in GitLab 10.6. + ## Placeholder tokens Badges support placeholders that will be replaced in real time in both the link and image URL. The allowed placeholders are: @@ -179,7 +182,7 @@ curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/a Example response: ```json -{ +{ "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", "image_url": "https://shields.io/my/badge", "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index de5207fc5e4..d8f61852b11 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -8,6 +8,14 @@ Start a new export. +The endpoint also accepts an `upload` param. This param is a hash that contains +all the necessary information to upload the exported project to a web server or +to any S3-compatible platform. At the moment we only support binary +data file uploads to the final server. + +If the `upload` params is present, `upload[url]` param is required. + (**Note:** This feature was introduced in GitLab 10.7) + ```http POST /projects/:id/export ``` @@ -16,9 +24,12 @@ POST /projects/:id/export | --------- | -------------- | -------- | ---------------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `description` | string | no | Overrides the project description | +| `upload` | hash | no | Hash that contains the information to upload the exported project to a web server | +| `upload[url]` | string | yes | The URL to upload the project | +| `upload[http_method]` | string | no | The HTTP method to upload the exported project. Only `PUT` and `POST` methods allowed. Default is `PUT` | ```console -curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --form "description=Foo Bar" https://gitlab.example.com/api/v4/projects/1/export +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/export --data "description=FooBar&upload[http_method]=PUT&upload[url]=https://example-bucket.s3.eu-west-3.amazonaws.com/backup?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMBJHN2O62W8IELQ%2F20180312%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Date=20180312T110328Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=8413facb20ff33a49a147a0b4abcff4c8487cc33ee1f7e450c46e8f695569dbd" ``` ```json @@ -43,7 +54,11 @@ GET /projects/:id/export curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/export ``` -Status can be one of `none`, `started`, or `finished`. +Status can be one of `none`, `started`, `after_export_action` or `finished`. The +`after_export_action` state represents that the export process has been completed successfully and +the platform is performing some actions on the resulted file. For example, sending +an email notifying the user to download the file, uploading the exported file +to a web server, etc. `_links` are only present when export has finished. @@ -96,6 +111,10 @@ POST /projects/import | `namespace` | integer/string | no | The ID or path of the namespace that the project will be imported to. Defaults to the current user's namespace | | `file` | string | yes | The file to be uploaded | | `path` | string | yes | Name and path for new project | +| `overwrite` | boolean | no | If there is a project with the same path the import will overwrite it. Default to false | +| `override_params` | Hash | no | Supports all fields defined in the [Project API](projects.md)] | + +The override params passed will take precendence over all values defined inside the export file. To upload a file from your filesystem, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. diff --git a/doc/api/projects.md b/doc/api/projects.md index 271ee91dc72..a0cb5aa0820 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -16,6 +16,21 @@ Values for the project visibility level are: * `public`: The project can be cloned without any authentication. +## Project merge method + +There are currently three options for `merge_method` to choose from: + +* `merge`: + A merge commit is created for every merge, and merging is allowed as long as there are no conflicts. + +* `rebase_merge`: + A merge commit is created for every merge, but merging is only allowed if fast-forward merge is possible. + This way you could make sure that if this merge request would build, after merging to target branch it would also build. + +* `ff`: + No merge commits are created and all merges are fast-forwarded, which means that merging is only allowed if the branch could be fast-forwarded. + + ## List all projects Get a list of all visible projects across GitLab for the authenticated user. @@ -94,6 +109,7 @@ GET /projects "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "statistics": { "commit_count": 37, "storage_size": 1038090, @@ -173,6 +189,7 @@ GET /projects "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -278,6 +295,7 @@ GET /users/:user_id/projects "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "statistics": { "commit_count": 37, "storage_size": 1038090, @@ -357,6 +375,7 @@ GET /users/:user_id/projects "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -467,6 +486,7 @@ GET /projects/:id "only_allow_merge_if_all_discussions_are_resolved": false, "printing_merge_requests_link_enabled": true, "request_access_enabled": false, + "merge_method": "merge", "statistics": { "commit_count": 37, "storage_size": 1038090, @@ -550,6 +570,7 @@ POST /projects | `public_jobs` | boolean | no | If `true`, jobs can be viewed by non-project-members | | `only_allow_merge_if_pipeline_succeeds` | boolean | no | Set whether merge requests can only be merged with successful jobs | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | no | Set whether merge requests can only be merged when all the discussions are resolved | +| `merge_method` | string | no | Set the merge method used | | `lfs_enabled` | boolean | no | Enable LFS | | `request_access_enabled` | boolean | no | Allow users to request member access | | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | @@ -586,6 +607,7 @@ POST /projects/user/:user_id | `public_jobs` | boolean | no | If `true`, jobs can be viewed by non-project-members | | `only_allow_merge_if_pipeline_succeeds` | boolean | no | Set whether merge requests can only be merged with successful jobs | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | no | Set whether merge requests can only be merged when all the discussions are resolved | +| `merge_method` | string | no | Set the merge method used | | `lfs_enabled` | boolean | no | Enable LFS | | `request_access_enabled` | boolean | no | Allow users to request member access | | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | @@ -621,6 +643,7 @@ PUT /projects/:id | `public_jobs` | boolean | no | If `true`, jobs can be viewed by non-project-members | | `only_allow_merge_if_pipeline_succeeds` | boolean | no | Set whether merge requests can only be merged with successful jobs | | `only_allow_merge_if_all_discussions_are_resolved` | boolean | no | Set whether merge requests can only be merged when all the discussions are resolved | +| `merge_method` | string | no | Set the merge method used | | `lfs_enabled` | boolean | no | Enable LFS | | `request_access_enabled` | boolean | no | Allow users to request member access | | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | @@ -724,6 +747,7 @@ Example responses: "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "_links": { "self": "http://example.com/api/v4/projects", "issues": "http://example.com/api/v4/projects/1/issues", @@ -801,6 +825,7 @@ Example response: "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "_links": { "self": "http://example.com/api/v4/projects", "issues": "http://example.com/api/v4/projects/1/issues", @@ -877,6 +902,7 @@ Example response: "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "_links": { "self": "http://example.com/api/v4/projects", "issues": "http://example.com/api/v4/projects/1/issues", @@ -971,6 +997,7 @@ Example response: "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "_links": { "self": "http://example.com/api/v4/projects", "issues": "http://example.com/api/v4/projects/1/issues", @@ -1065,6 +1092,7 @@ Example response: "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, + "merge_method": "merge", "_links": { "self": "http://example.com/api/v4/projects", "issues": "http://example.com/api/v4/projects/1/issues", @@ -1344,3 +1372,7 @@ Read more in the [Project members](members.md) documentation. ## Project badges Read more in the [Project Badges](project_badges.md) documentation. + +## Issue and merge request description templates + +The non-default [issue and merge request description templates](../user/project/description_templates.md) are managed inside the project's repository. So you can manage them via the API through the [Repositories API](repositories.md) and the [Repository Files API](repository_files.md).
\ No newline at end of file diff --git a/doc/api/tags.md b/doc/api/tags.md index fa25dc76452..4af096c3c0c 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -42,6 +42,7 @@ Parameters: "description": "Amazing release. Wow" }, "name": "v1.0.0", + "target": "2695effb5807a22ff3d138d593fd856244e155e7", "message": null } ] @@ -73,6 +74,7 @@ Example Response: { "name": "v5.0.0", "message": null, + "target": "60a8ff033665e1207714d6670fcd7b65304ec02f", "commit": { "id": "60a8ff033665e1207714d6670fcd7b65304ec02f", "short_id": "60a8ff03", @@ -132,12 +134,16 @@ Parameters: "description": "Amazing release. Wow" }, "name": "v1.0.0", + "target: "2695effb5807a22ff3d138d593fd856244e155e7", "message": null } ``` The message will be `null` when creating a lightweight tag otherwise it will contain the annotation. +The target will contain the tag objects ID when creating annotated tags, +otherwise it will contain the commit ID when creating lightweight tags. + In case of an error, status code `405` with an explaining error message is returned. diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 58c4a71cef9..b3d9f0bc96c 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -247,10 +247,19 @@ declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is the basis of [Review apps](review_apps/index.md). >**Note:** -The `name` and `url` parameters can use any of the defined CI variables, +The `name` and `url` parameters can use most of the defined CI variables, including predefined, secure variables and `.gitlab-ci.yml` -[`variables`](yaml/README.md#variables). -You however cannot use variables defined under `script` or on the Runner's side. +[`variables`](yaml/README.md#variables). You however cannot use variables +defined under `script` or on the Runner's side. There are other variables that +are unsupported in environment name context: +- `CI_JOB_ID` +- `CI_JOB_TOKEN` +- `CI_BUILD_ID` +- `CI_BUILD_TOKEN` +- `CI_REGISTRY_USER` +- `CI_REGISTRY_PASSWORD` +- `CI_REPOSITORY_URL` +- `CI_ENVIRONMENT_URL` GitLab Runner exposes various [environment variables][variables] when a job runs, and as such, you can use them as environment names. Let's add another job in diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index 42dc6ef36ba..691370d7195 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -1,22 +1,28 @@ # Browser Performance Testing with the Sitespeed.io container -This example shows how to run the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) on your code by using -GitLab CI/CD and [Sitespeed.io](https://www.sitespeed.io) using Docker-in-Docker. +This example shows how to run the +[Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) on +your code by using GitLab CI/CD and [Sitespeed.io](https://www.sitespeed.io) +using Docker-in-Docker. -First, you need a GitLab Runner with the [docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `performance`: +First, you need a GitLab Runner with the +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called +`performance`: ```yaml +performance: stage: performance image: docker:git + variables: + URL: https://example.com services: - docker:dind script: - mkdir gitlab-exporter - - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/10-5/index.js + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/master/index.js - mkdir sitespeed-results - - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results https://my.website.com + - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL - mv sitespeed-results/data/performance.json performance.json artifacts: paths: @@ -24,37 +30,84 @@ Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `performan - sitespeed-results/ ``` -This will create a `performance` job in your CI/CD pipeline and will run Sitespeed.io against the webpage you define. The GitLab plugin for Sitespeed.io is downloaded in order to export key metrics to JSON. The full HTML Sitespeed.io report will also be saved as an artifact, and if you have Pages enabled it can be viewed directly in your browser. For further customization options of Sitespeed.io, including the ability to provide a list of URLs to test, please consult their [documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/). +The above example will: + +1. Create a `performance` job in your CI/CD pipeline and will run + Sitespeed.io against the webpage you defined in `URL`. +1. The [GitLab plugin](https://gitlab.com/gitlab-org/gl-performance) for + Sitespeed.io is downloaded in order to export key metrics to JSON. The full + HTML Sitespeed.io report will also be saved as an artifact, and if you have + [GitLab Pages](../../user/project/pages/index.md) enabled, it can be viewed + directly in your browser. + +For further customization options of Sitespeed.io, including the ability to +provide a list of URLs to test, please consult +[their documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/). -For [GitLab Premium](https://about.gitlab.com/products/) users, key metrics are automatically -extracted and shown right in the merge request widget. Learn more about [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). +TIP: **Tip:** +For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically +extracted and shown right in the merge request widget. Learn more about +[Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ## Performance testing on Review Apps -The above CI YML is great for testing against static environments, and it can be extended for dynamic environments. There are a few extra steps to take to set this up: -1. The `performance` job should run after the environment has started. -1. In the `deploy` job, persist the hostname so it is available to the `performance` job. The same can be done for static environments like staging and production to unify the code path. Saving it as an artifact is as simple as `echo $CI_ENVIRONMENT_URL > environment_url.txt`. -1. In the `performance` job read the artifact into an environment variable, like `$CI_ENVIRONMENT_URL`, and use it to parameterize the test URL's. -1. Now you can run the Sitespeed.io container against the desired hostname and paths. +The above CI YML is great for testing against static environments, and it can +be extended for dynamic environments. There are a few extra steps to take to +set this up: -A simple `performance` job would look like: +1. The `performance` job should run after the dynamic environment has started. +1. In the `review` job, persist the hostname and upload it as an artifact so + it's available to the `performance` job (the same can be done for static + environments like staging and production to unify the code path). Saving it + as an artifact is as simple as `echo $CI_ENVIRONMENT_URL > environment_url.txt` + in your job's `script`. +1. In the `performance` job, read the previous artifact into an environment + variable, like `$CI_ENVIRONMENT_URL`, and use it to parameterize the test + URLs. +1. You can now run the Sitespeed.io container against the desired hostname and + paths. + +Your `.gitlab-ci.yml` file would look like: ```yaml +stages: + - deploy + - performance + +review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_SLUG + url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN + script: + - run_deploy_script + - echo $CI_ENVIRONMENT_URL > environment_url.txt + artifacts: + paths: + - environment_url.txt + only: + - branches + except: + - master + +performance: stage: performance image: docker:git services: - docker:dind + dependencies: + - review script: - export CI_ENVIRONMENT_URL=$(cat environment_url.txt) - mkdir gitlab-exporter - - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/10-5/index.js + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/master/index.js - mkdir sitespeed-results - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results "$CI_ENVIRONMENT_URL" - mv sitespeed-results/data/performance.json performance.json artifacts: paths: - - performance.json - - sitespeed-results/ + - performance.json + - sitespeed-results/ ``` -A complete example can be found in our [Auto DevOps CI YML](https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml).
\ No newline at end of file +A complete example can be found in our [Auto DevOps CI YML](https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml). diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md index 64a759a9a99..92317c77427 100644 --- a/doc/ci/examples/code_climate.md +++ b/doc/ci/examples/code_climate.md @@ -9,11 +9,12 @@ Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `codequali ```yaml codequality: - image: docker:latest + image: docker:stable variables: - DOCKER_DRIVER: overlay + DOCKER_DRIVER: overlay2 + allow_failure: true services: - - docker:dind + - docker:stable-dind script: - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') - docker run --env SOURCE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index 3437b63748a..c58efc7392a 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -11,7 +11,7 @@ called `sast:container`: ```yaml sast:container: - image: docker:latest + image: docker:stable variables: DOCKER_DRIVER: overlay2 ## Define two new variables based on GitLab's CI/CD predefined variables @@ -20,7 +20,7 @@ sast:container: CI_APPLICATION_TAG: $CI_COMMIT_SHA allow_failure: true services: - - docker:dind + - docker:stable-dind script: - docker run -d --name db arminc/clair-db:latest - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index 96de0f5ff5c..8df223ee560 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -14,9 +14,10 @@ called `dast`: ```yaml dast: - image: owasp/zap2docker-stable + image: registry.gitlab.com/gitlab-org/security-products/zaproxy variables: website: "https://example.com" + allow_failure: true script: - mkdir /zap/wrk/ - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true @@ -30,6 +31,28 @@ the tests on the URL defined in the `website` variable (change it to use your own) and finally write the results in the `gl-dast-report.json` file. You can then download and analyze the report artifact in JSON format. +It's also possible to authenticate the user before performing DAST checks: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + login_url: "https://example.com/sign-in" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website \ + --auth-url $login_url \ + --auth-username "john.doe@example.com" \ + --auth-password "john-doe-password" || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + paths: [gl-dast-report.json] +``` +See [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) +to learn more about authentication settings. + TIP: **Tip:** Starting with [GitLab Ultimate][ee] 10.4, this information will be automatically extracted and shown right in the merge request widget. To do diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index b62874ef029..1f9b9d53fc1 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -190,7 +190,7 @@ To start, we create an `Envoy.blade.php` in the root of our app with a simple ta ```php @servers(['web' => 'remote_username@remote_host']) -@task('list', [on => 'web']) +@task('list', ['on' => 'web']) ls -l @endtask ``` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 856d7f264e4..301cccc80a3 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -2,6 +2,11 @@ > Introduced in GitLab 8.8. +NOTE: **Note:** +If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository), +you may need to enable pipeline triggering in your project's +**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. + ## Pipelines A pipeline is a group of [jobs][] that get executed in [stages][](batches). @@ -121,9 +126,8 @@ The basic requirements is that there are two numbers separated with one of the following (you can even use them interchangeably): - a space -- a forward slash (`/`) +- a slash (`/`) - a colon (`:`) -- a dot (`.`) >**Note:** More specifically, [it uses][regexp] this regular expression: `\d+[\s:\/\\]+\d+\s*`. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index f64e868d390..fec0ff87326 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -126,6 +126,11 @@ git push origin master Now if you go to the **Pipelines** page you will see that the pipeline is pending. +NOTE: **Note:** +If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository), +you may need to enable pipeline triggering in your project's +**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. + You can also go to the **Commits** page and notice the little pause icon next to the commit SHA. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index bd4aeb006bd..4a504a98902 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -449,6 +449,107 @@ export CI_REGISTRY_USER="gitlab-ci-token" export CI_REGISTRY_PASSWORD="longalfanumstring" ``` +## Variables expressions + +> Variables expressions were added in GitLab 10.7. + +It is possible to use variables expressions with only / except policies in +`.gitlab-ci.yml`. By using this approach you can limit what jobs are going to +be created within a pipeline after pushing a code to GitLab. + +This is particularly useful in combination with secret variables and triggered +pipeline variables. + +```yaml +deploy: + script: cap staging deploy + environment: staging + only: + variables: + - $RELEASE == "staging" + - $STAGING +``` + +Each expression provided is going to be evaluated before creating a pipeline. + +If any of the conditions in `variables` evaluates to truth when using `only`, +a new job is going to be created. If any of the expressions evaluates to truth +when `except` is being used, a job is not going to be created. + +This follows usual rules for [`only` / `except` policies][builds-policies]. + +### Supported syntax + +Below you can find supported syntax reference: + +1. Equality matching using a string + + > Example: `$VARIABLE == "some value"` + + You can use equality operator `==` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. + +1. Checking for an undefined value + + > Example: `$VARIABLE == null` + + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined. + +1. Checking for an empty variable + + > Example: `$VARIABLE == ""` + + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''`. + +1. Comparing two variables + + > Example: `$VARIABLE_1 == $VARIABLE_2` + + It is possible to compare two variables. This is going to compare values + of these variables. + +1. Variable presence check + + > Example: `$STAGING` + + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. + +### Unsupported predefined variables + +Because GitLab evaluates variables before creating jobs, we do not support a +few variables that depend on persistence layer, like `$CI_JOB_ID`. + +Environments (like `production` or `staging`) are also being created based on +what jobs pipeline consists of, thus some environment-specific variables are +not supported as well. + +We do not support variables containing tokens because of security reasons. + +You can find a full list of unsupported variables below: + +- `CI_JOB_ID` +- `CI_JOB_TOKEN` +- `CI_BUILD_ID` +- `CI_BUILD_TOKEN` +- `CI_REGISTRY_USER` +- `CI_REGISTRY_PASSWORD` +- `CI_REPOSITORY_URL` +- `CI_ENVIRONMENT_URL` + +These variables are also not supported in a contex of a +[dynamic environment name][dynamic-environments]. + [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI secret variables" [eep]: https://about.gitlab.com/products/ "Available only in GitLab Premium" [envs]: ../environments.md @@ -459,3 +560,5 @@ export CI_REGISTRY_PASSWORD="longalfanumstring" [triggered]: ../triggers/README.md [triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger [subgroups]: ../../user/group/subgroups/index.md +[builds-policies]: ../yaml/README.md#only-and-except-complex +[dynamic-environments]: ../environments.md#dynamic-environments diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index c2b06e53c2f..68aa64b3834 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -10,6 +10,11 @@ of your repository and contains definitions of how your project should be built. If you want a quick introduction to GitLab CI, follow our [quick start guide](../quick_start/README.md). +NOTE: **Note:** +If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository), +you may need to enable pipeline triggering in your project's +**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. + ## Jobs The YAML file defines a set of jobs with constraints stating when they should @@ -315,9 +320,14 @@ policy configuration. GitLab now supports both, simple and complex strategies, so it is possible to use an array and a hash configuration scheme. -Two keys are now available: `refs` and `kubernetes`. Refs strategy equals to -simplified only/except configuration, whereas kubernetes strategy accepts only -`active` keyword. +Three keys are now available: `refs`, `kubernetes` and `variables`. +Refs strategy equals to simplified only/except configuration, whereas +kubernetes strategy accepts only `active` keyword. + +`variables` keyword is used to define variables expressions. In other words +you can use predefined variables / secret variables / project / group or +environment-scoped variables to define an expression GitLab is going to +evaluate in order to decide whether a job should be created or not. See the example below. Job is going to be created only when pipeline has been scheduled or runs for a `master` branch, and only if kubernetes service is @@ -332,6 +342,20 @@ job: kubernetes: active ``` +Example of using variables expressions: + +```yaml +deploy: + only: + refs: + - branches + variables: + - $RELEASE == "staging" + - $STAGING +``` + +Learn more about variables expressions on [a separate page][variables-expressions]. + ## `tags` `tags` is used to select specific Runners from the list of all Runners that are @@ -1550,3 +1574,4 @@ CI with various languages. [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 [ce-12909]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12909 [schedules]: ../../user/project/pipelines/schedules.md +[variables-expressions]: ../variables/README.md#variables-expressions diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 1962392a9eb..d5a4acff481 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -44,6 +44,7 @@ the `author` field. GitLab team members **should not**. - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. Example: "Fixed a typo on the search results page. (Jane Smith)" +- Performance improvements **should** have a changelog entry. ## Writing good changelog entries diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 3ba03d2d591..287143d6255 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -644,6 +644,7 @@ information on managing page-specific javascript within EE. To separate EE-specific styles in SCSS files, if a component you're adding styles for is limited to only EE, it is better to have a separate SCSS file in appropriate directory within `app/assets/stylesheets`. +See [backporting changes](#backporting-changes) for instructions on how to merge changes safely. In some cases, this is not entirely possible or creating dedicated SCSS file is an overkill, e.g. a text style of some component is different for EE. In such cases, @@ -683,6 +684,19 @@ to avoid conflicts during CE to EE merge. // EE-specific end ``` +### Backporting changes from EE to CE + +When working in EE-specific features, you might have to tweak a few files that are not EE-specific. Here is a workflow to make sure those changes end up backported safely into CE too. +(This approach does not refer to changes introduced via [csslab](https://gitlab.com/gitlab-org/csslab/).) + +1. **Make your changes in the EE branch.** If possible, keep a separated commit (to be squashed) to help backporting and review. +1. **Open merge request to EE project.** +1. **Apply the changes you made to CE files in a branch of the CE project.** (Tip: Use `patch` with the diff from your commit in EE branch) +1. **Open merge request to CE project**, referring it's a backport of EE changes and link to MR open in EE. +1. Once EE MR is merged, the MR towards CE can be merged. **But not before**. + +**Note:** regarding SCSS, make sure the files living outside `/ee/` don't diverge between CE and EE projects. + ## gitlab-svgs Conflicts in `app/assets/images/icons.json` or `app/assets/images/icons.svg` can diff --git a/doc/development/emails.md b/doc/development/emails.md index 4dbf064fd75..73cac82caf0 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -74,6 +74,24 @@ See the [Rails guides] for more info. 1. Reply by email should now be working. +## Email namespace + +If you need to implement a new feature which requires a new email handler, follow these rules: + + - You must choose a namespace. The namespace cannot contain `/` or `+`, and cannot match `\h{16}`. + - If your feature is related to a project, you will append the namespace **after** the project path, separated by a `+` + - If you have different actions in the namespace, you add the actions **after** the namespace separated by a `+`. The action name cannot contain `/` or `+`, , and cannot match `\h{16}`. + - You will register your handlers in `lib/gitlab/email/handler.rb` + +Therefore, these are the only valid formats for an email handler: + + - `path/to/project+namespace` + - `path/to/project+namespace+action` + - `namespace` + - `namespace+action` + +Please note that `path/to/project` is used in GitLab Premium as handler for the Service Desk feature. + --- [Return to Development documentation](README.md) diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 26abf967dcf..4f9ca1920a5 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -7,6 +7,67 @@ be replaced by Gitaly API calls. Visit the [Gitaly Migration Board](https://gitlab.com/gitlab-org/gitaly/boards/331341) for current status of the migration. +## Developing new Git features + +Starting with Gitlab 10.8, all new Git features should be developed in +Gitaly. + +> This is a new process that is not clearly defined yet. If you want +to contribute a Git feature and you're getting stuck, reach out to the +Gitaly team or `@jacobvosmaer-gitlab`. + +By 'new feature' we mean any method or class in `lib/gitlab/git` that is +called from outside `lib/gitlab/git`. For new methods that are called +from inside `lib/gitlab/git`, see 'Modifying existing Git features' +below. + +There should be no new code that touches Git repositories via +disk access (e.g. Rugged, `git`, `rm -rf`) anywhere outside +`lib/gitlab/git`. + +The process for adding new Gitaly features is: + +- exploration / prototyping +- design and create a new Gitaly RPC [in gitaly-proto](https://gitlab.com/gitlab-org/gitaly-proto) +- release a new version of gitaly-proto +- write implementation and tests for the RPC [in Gitaly](https://gitlab.com/gitlab-org/gitaly), in Go or Ruby +- release a new version of Gitaly +- write client code in gitlab-ce/ee, gitlab-workhorse or gitlab-shell that calls the new Gitaly RPC + +These steps often overlap. It is possible to use an unreleased version +of Gitaly and gitaly-proto during testing and development. + +- See the [Gitaly repo](https://gitlab.com/gitlab-org/gitaly/blob/master/CONTRIBUTING.md#development-and-testing-with-a-custom-gitaly-proto) for instructions on writing server side code with an unreleased protocol. +- See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running gitlab-ce tests with a modified version of Gitaly. +- In GDK run `gdk install` and restart `gdk run` (or `gdk run app`) to use a locally modified Gitaly version for development + +### Gitaly-ruby + +It is possible to implement and test RPC's in Gitaly using Ruby code, +in +[gitaly-ruby](https://gitlab.com/gitlab-org/gitaly/tree/master/ruby). +This should make it easier to contribute for developers who are less +comfortable writing Go code. + +There is documentation for this approach in [the Gitaly +repo](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/ruby_endpoint.md). + +## Modifying existing Git features + +If you modify existing Git features in `lib/gitlab/git` you need to make +sure the changes also work in Gitaly. Because we are still in the +migration process there are a number of subtle pitfalls. Features that +have been migrated have dual implementations (Gitaly and local). The +Gitaly implementation may or may not use a vendored (and therefore +possibly outdated) copy of the local implementation in `lib/gitlab/git`. + +To avoid unexpected problems and conflicts, all changes to +`lib/gitlab/git` need to be approved by a member of the Gitaly team. + +For the time being, while the Gitaly migration is still in progress, +there should be no Enterprise Edition-only Git code in +`lib/gitlab/git`. Also no mixins. + ## Feature Flags Gitaly makes heavy use of [feature flags](feature_flags.md). @@ -99,10 +160,14 @@ end ## Running tests with a locally modified version of Gitaly -Normally, gitlab-ce/ee tests use a local clone of Gitaly in `tmp/tests/gitaly` -pinned at the version specified in GITALY_SERVER_VERSION. If you want -to run tests locally against a modified version of Gitaly you can -replace `tmp/tests/gitaly` with a symlink. +Normally, gitlab-ce/ee tests use a local clone of Gitaly in +`tmp/tests/gitaly` pinned at the version specified in +`GITALY_SERVER_VERSION`. The `GITALY_SERVER_VERSION` file supports +`=my-branch` syntax to use a custom branch in gitlab-org/gitaly. If +you want to run tests locally against a modified version of Gitaly you +can replace `tmp/tests/gitaly` with a symlink. This is much faster +because the `=my-branch` syntax forces a Gitaly re-install each time +you run `rspec`. ```shell rm -rf tmp/tests/gitaly diff --git a/doc/install/installation.md b/doc/install/installation.md index 1abbfd78738..3cf6f7b7ddf 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -133,9 +133,10 @@ Remove the old Ruby 1.8 if present: Download Ruby and compile it: mkdir /tmp/ruby && cd /tmp/ruby - curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.6.tar.gz - echo '4e6a0f828819e15d274ae58485585fc8b7caace0 ruby-2.3.6.tar.gz' | shasum -c - && tar xzf ruby-2.3.6.tar.gz - cd ruby-2.3.6 + curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.7.tar.gz + echo '540996fec64984ab6099e34d2f5820b14904f15a ruby-2.3.7.tar.gz' | shasum -c - && tar xzf ruby-2.3.7.tar.gz + cd ruby-2.3.7 + ./configure --disable-install-rdoc make sudo make install diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md new file mode 100644 index 00000000000..7c9e5bf882e --- /dev/null +++ b/doc/user/admin_area/settings/email.md @@ -0,0 +1,5 @@ +# Email + +## Custom logo + +The logo in the header of some emails can be customized, see the [logo customization section](../../../customization/branded_page_and_email_header.md). diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 633f16a617c..3d38588a9ed 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -32,9 +32,15 @@ When you choose to allow only one of the protocols, a couple of things will happ On top of these UI restrictions, GitLab will deny all Git actions on the protocol not selected. +CAUTION: **Important:** +Starting with [GitLab 10.7][ce-18021], HTTP(s) protocol will be allowed for +git clone/fetch requests done by GitLab Runner from CI/CD Jobs, even if +_Only SSH_ was selected. + > **Note:** Please keep in mind that disabling an access protocol does not actually - block access to the server itself. The ports used for the protocol, be it SSH or - HTTP, will still be accessible. What GitLab does is restrict access on the - application level. +block access to the server itself. The ports used for the protocol, be it SSH or +HTTP, will still be accessible. What GitLab does is restrict access on the +application level. [ce-4696]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4696 +[ce-18021]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18021 diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index d5f77191938..7baccb796c6 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -72,15 +72,23 @@ The maximum size your Git repository is allowed to be including LFS. ## Shared Runners Shared Runners on GitLab.com run in [autoscale mode] and powered by -DigitalOcean. Autoscaling means reduced waiting times to spin up builds, -and isolated VMs for each project, thus maximizing security. +Google Cloud Platform and DigitalOcean. Autoscaling means reduced +waiting times to spin up CI/CD jobs, and isolated VMs for each project, +thus maximizing security. They're free to use for public open source projects and limited to 2000 CI minutes per month per group for private projects. Read about all [GitLab.com plans](https://about.gitlab.com/pricing/). -All your builds run on 2GB (RAM) ephemeral instances, with CoreOS and the latest -Docker Engine installed. The default region of the VMs is NYC. +In case of DigitalOcean based Runners, all your CI/CD jobs run on ephemeral +instances with 2GB of RAM, CoreOS and the latest Docker Engine installed. +Instances provide 2 vCPUs and 60GB of SSD disk space. The default region of the +VMs is NYC1. + +In case of Google Cloud Platform based Runners, all your CI/CD jobs run on +ephemeral instances with 3.75GB of RAM, CoreOS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. Below are the shared Runners settings. @@ -88,52 +96,116 @@ Below are the shared Runners settings. | ----------- | ----------------- | ---------- | | [GitLab Runner] | [Runner versions dashboard][ci_version_dashboard] | - | | Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.1` | - | +| Default Docker image | `ruby:2.5` | - | | `privileged` (run [Docker in Docker]) | `true` | `false` | -[ci_version_dashboard]: https://monitor.gitlab.net/dashboard/db/ci?refresh=5m&orgId=1&panelId=12&fullscreen&from=now-1h&to=now&var-runner_type=All&var-cache_server=All&var-gl_monitor_fqdn=postgres-01.db.prd.gitlab.com&var-has_minutes=yes&var-hanging_droplets_cleaner=All&var-droplet_zero_machines_cleaner=All&var-runner_job_failure_reason=All&theme=light +[ci_version_dashboard]: https://monitor.gitlab.net/dashboard/db/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light ### `config.toml` The full contents of our `config.toml` are: +**DigitalOcean** + ```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + [[runners]] name = "docker-auto-scale" - limit = X request_concurrency = X - url = "https://gitlab.com/ci" + url = "https://gitlab.com/" token = "SHARED_RUNNER_TOKEN" executor = "docker+machine" environment = [ "DOCKER_DRIVER=overlay2" ] + limit = X [runners.docker] - image = "ruby:2.1" + image = "ruby:2.5" privileged = true [runners.machine] - IdleCount = 40 + IdleCount = 20 IdleTime = 1800 + OffPeakPeriods = ["* * * * * sat,sun *"] + OffPeakTimezone = "UTC" + OffPeakIdleCount = 5 + OffPeakIdleTime = 1800 MaxBuilds = 1 + MachineName = "srm-%s" MachineDriver = "digitalocean" - MachineName = "machine-%s-digital-ocean-2gb" MachineOptions = [ - "digitalocean-image=coreos-stable", + "digitalocean-image=X", "digitalocean-ssh-user=core", - "digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN", "digitalocean-region=nyc1", - "digitalocean-size=2gb", + "digitalocean-size=s-2vcpu-2gb", "digitalocean-private-networking", - "digitalocean-userdata=/etc/gitlab-runner/cloudinit.sh", - "engine-registry-mirror=http://IP_TO_OUR_REGISTRY_MIRROR" + "digitalocean-tags=shared_runners,gitlab_com", + "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR", + "digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN", ] [runners.cache] Type = "s3" - ServerAddress = "IP_TO_OUR_CACHE_SERVER" + BucketName = "runner" + Insecure = true + Shared = true + ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER" AccessKey = "ACCESS_KEY" SecretKey = "ACCESS_SECRET_KEY" +``` + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + [runners.machine] + IdleCount = 20 + IdleTime = 1800 + OffPeakPeriods = ["* * * * * sat,sun *"] + OffPeakTimezone = "UTC" + OffPeakIdleCount = 5 + OffPeakIdleTime = 1800 + MaxBuilds = 1 + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR" + ] + [runners.cache] + Type = "s3" BucketName = "runner" + Insecure = true Shared = true + ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER" + AccessKey = "ACCESS_KEY" + SecretKey = "ACCESS_SECRET_KEY" ``` ## Sidekiq diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md new file mode 100644 index 00000000000..c4e59444ef7 --- /dev/null +++ b/doc/user/project/badges.md @@ -0,0 +1,73 @@ +# Badges + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41174) +in GitLab 10.7. + +Badges are a unified way to present condensed pieces of information about your +projects. They consist of a small image and additionally a URL that the image +points to. Examples for badges can be the [pipeline status], [test coverage], +or ways to contact the project maintainers. + + + +## Project badges + +Badges can be added to a project and will then be visible on the project's overview page. +If you find that you have to add the same badges to several projects, you may want to add them at the [group level](#group-badges). + +To add a new badge to a project: + +1. Navigate to your project's **Settings > Badges**. +1. Under "Link", enter the URL that the badges should point to and under + "Badge image URL" the URL of the image that should be displayed. +1. Submit the badge by clicking the **Add badge** button. + +After adding a badge to a project, you can see it in the list below the form. +You can edit it by clicking on the pen icon next to it or to delete it by +clicking on the trash icon. + +Badges associated with a group can only be edited or deleted on the +[group level](#group-badges). + +## Group badges + +Badges can be added to a group and will then be visible on every project's +overview page that's under that group. In this case, they cannot be edited or +deleted on the project level. If you need to have individual badges for each +project, consider adding them on the [project level](#project-badges) or use +[placeholders](#placeholders). + +To add a new badge to a group: + +1. Navigate to your group's **Settings > Project Badges**. +1. Under "Link", enter the URL that the badges should point to and under + "Badge image URL" the URL of the image that should be displayed. +1. Submit the badge by clicking the **Add badge** button. + +After adding a badge to a group, you can see it in the list below the form. +You can edit the badge by clicking on the pen icon next to it or to delete it +by clicking on the trash icon. + +Badges directly associated with a project can be configured on the +[project level](#project-badges). + +## Placeholders + +The URL a badge points to, as well as the image URL, can contain placeholders +which will be evaluated when displaying the badge. The following placeholders +are available: + +- `%{project_path}`: Path of a project including the parent groups +- `%{project_id}`: Database ID associated with a project +- `%{default_branch}`: Default branch name configured for a project's repository +- `%{commit_sha}`: ID of the most recent commit to the default branch of a + project's repository + +## API + +You can also configure badges via the GitLab API. As in the settings, there is +a distinction between endpoints for badges on the +[project level](../../api/project_badges.md) and [group level](../../api/group_badges.md). + +[pipeline status]: pipelines/settings.md#pipeline-status-badge +[test coverage]: pipelines/settings.md#test-coverage-report-badge diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md index 394aa9209e4..9c5e3509046 100644 --- a/doc/user/project/container_registry.md +++ b/doc/user/project/container_registry.md @@ -115,15 +115,16 @@ and [Using the GitLab Container Registry documentation](../../ci/docker/using_do ## Using with private projects -> [Introduced][ce-11845] in GitLab 9.3. +> Personal Access tokens were [introduced][ce-11845] in GitLab 9.3. +> Project Deploy Tokens were [introduced][ce-17894] in GitLab 10.7 If a project is private, credentials will need to be provided for authorization. -The preferred way to do this, is by using [personal access tokens][pat]. -The minimal scope needed is `read_registry`. +The preferred way to do this, is either by using a [personal access tokens][pat] or a [project deploy token][pdt]. +The minimal scope needed for both of them is `read_registry`. Example of using a personal access token: ``` -docker login registry.example.com -u <your_username> -p <your_personal_access_token> +docker login registry.example.com -u <your_username> -p <your_access_token> ``` ## Troubleshooting the GitLab Container Registry @@ -270,5 +271,7 @@ Once the right permissions were set, the error will go away. [ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040 [ce-11845]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845 +[ce-17894]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17894 [docker-docs]: https://docs.docker.com/engine/userguide/intro/ [pat]: ../profile/personal_access_tokens.md +[pdt]: ../project/deploy_tokens/index.md diff --git a/doc/user/project/deploy_tokens/img/deploy_tokens.png b/doc/user/project/deploy_tokens/img/deploy_tokens.png Binary files differnew file mode 100644 index 00000000000..7e2d67a3120 --- /dev/null +++ b/doc/user/project/deploy_tokens/img/deploy_tokens.png diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md new file mode 100644 index 00000000000..86fc58020e8 --- /dev/null +++ b/doc/user/project/deploy_tokens/index.md @@ -0,0 +1,76 @@ +# Deploy Tokens + +> [Introduced][ce-17894] in GitLab 10.7. + +Deploy tokens allow to download (through `git clone`), or read the container registry images of a project without the need of having a user and a password. + +Please note, that the expiration of deploy tokens happens on the date you define, +at midnight UTC and that they can be only managed by [masters](https://docs.gitlab.com/ee/user/permissions.html). + +## Creating a Deploy Token + +You can create as many deploy tokens as you like from the settings of your project: + +1. Log in to your GitLab account. +1. Go to the project you want to create Deploy Tokens for. +1. Go to **Settings** > **Repository** +1. Click on "Expand" on **Deploy Tokens** section +1. Choose a name and optionally an expiry date for the token. +1. Choose the [desired scopes](#limiting-scopes-of-a-deploy-token). +1. Click on **Create deploy token**. +1. Save the deploy token somewhere safe. Once you leave or refresh + the page, **you won't be able to access it again**. + + + +## Revoking a personal access token + +At any time, you can revoke any deploy token by just clicking the +respective **Revoke** button under the 'Active deploy tokens' area. + +## Limiting scopes of a deploy token + +Deploy tokens can be created with two different scopes that allow various +actions that a given token can perform. The available scopes are depicted in +the following table. + +| Scope | Description | +| ----- | ----------- | +| `read_repository` | Allows read-access to the repository through `git clone` | +| `read_registry` | Allows read-access to [container registry] images if a project is private and authorization is required. | + +## Usage + +### Git clone a repository + +To download a repository using a Deploy Token, you just need to: + +1. Create a Deploy Token with `read_repository` as a scope. +2. Take note of your `username` and `token` +3. `git clone` the project using the Deploy Token: + + +```bash +git clone http://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git +``` + +Just replace `<username>` and `<deploy_token>` with the proper values + +### Read container registry images + +To read the container registry images, you'll need to: + +1. Create a Deploy Token with `read_registry` as a scope. +2. Take note of your `username` and `token` +3. Log in to GitLab’s Container Registry using the deploy token: + +``` +docker login registry.example.com -u <username> -p <deploy_token> +``` + +Just replace `<username>` and `<deploy_token>` with the proper values. Then you can simply +pull images from your Container Registry. + +[ce-17894]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17894 +[ce-11845]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845 +[container registry]: ../container_registry.md diff --git a/doc/user/project/img/project_overview_badges.png b/doc/user/project/img/project_overview_badges.png Binary files differnew file mode 100644 index 00000000000..3067a7dfa13 --- /dev/null +++ b/doc/user/project/img/project_overview_badges.png diff --git a/doc/user/project/index.md b/doc/user/project/index.md index f94e93dd7d8..5ce4ebfa811 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -27,6 +27,7 @@ integrated platform - [Protected tags](protected_tags.md): Control over who has permission to create tags, and prevent accidental update or deletion - [Signing commits](gpg_signed_commits/index.md): use GPG to sign your commits + - [Deploy tokens](deploy_tokens/index.md): Manage project-based deploy tokens that allow permanent access to the repository and Container Registry. - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) (**Starter/Premium**): Ask for approval before @@ -73,6 +74,7 @@ website with GitLab Pages - [Cycle Analytics](cycle_analytics.md): Review your development lifecycle - [Syntax highlighting](highlighting.md): An alternative to customize your code blocks, overriding GitLab's default choice of language +- [Badges](badges.md): Badges for the project overview ### Project's integrations diff --git a/doc/user/project/integrations/img/jira_workflow_screenshot.png b/doc/user/project/integrations/img/jira_workflow_screenshot.png Binary files differdeleted file mode 100644 index e62fb202613..00000000000 --- a/doc/user/project/integrations/img/jira_workflow_screenshot.png +++ /dev/null diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index fc527663db0..5933bcedc8b 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -113,7 +113,20 @@ in the table below. | `JIRA API URL` | The base URL to the JIRA instance API. Web URL value will be used if not set. E.g., `https://jira-api.example.com`. | | `Username` | The user name created in [configuring JIRA step](#configuring-jira). | | `Password` |The password of the user created in [configuring JIRA step](#configuring-jira). | -| `Transition ID` | This is the ID of a transition that moves issues to a closed state. You can find this number under JIRA workflow administration ([see screenshot](img/jira_workflow_screenshot.png)). **Closing JIRA issues via commits or Merge Requests won't work if you don't set the ID correctly.** | +| `Transition ID` | This is the ID of a transition that moves issues to the desired state. **Closing JIRA issues via commits or Merge Requests won't work if you don't set the ID correctly.** | + +### Getting a transition ID + +In the most recent JIRA UI, you can no longer see transition IDs in the workflow +administration UI. You can get the ID you need in either of the following ways: + +1. By using the API, with a request like `https://yourcompany.atlassian.net/rest/api/2/issue/ISSUE-123/transitions` + using an issue that is in the appropriate "open" state +1. By mousing over the link for the transition you want and looking for the + "action" parameter in the URL + +Note that the transition ID may vary between workflows (e.g., bug vs. story), +even if the status you are changing to is the same. After saving the configuration, your GitLab project will be able to interact with all JIRA projects in your JIRA instance and you'll see the JIRA link on the GitLab project pages that takes you to the appropriate JIRA project. diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index b4a842f33d6..7eab825fa32 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -240,8 +240,7 @@ Issue Board, that is create/delete lists and drag issues around. >Introduced in GitLab 10.6 Group issue board is analogous to project-level issue board and it is accessible at the group -navigation level. A group-level issue board allows you to view all issues from all projects in that group -(currently, it does not see issues from projects in subgroups). Similarly, you can only filter by group labels for these +navigation level. A group-level issue board allows you to view all issues from all projects in that group or descendant subgroups. Similarly, you can only filter by group labels for these boards. When updating milestones and labels for an issue through the sidebar update mechanism, again only group-level objects are available. diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index dabffaec5fa..a89a1206170 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -9,7 +9,7 @@ Labels allow you to categorize issues or merge requests using descriptive titles In GitLab, you can create project and group labels: - **Project labels** can be assigned to issues or merge requests in that project only. -- **Group labels** can be assigned to any issue or merge request of any project in that group. +- **Group labels** can be assigned to any issue or merge request of any project in that group or subgroup. - In the [future](https://gitlab.com/gitlab-org/gitlab-ce/issues/40915), you will be able to assign group labels to issues and merge reqeusts of projects in [subgroups](../group/subgroups/index.md). ## Creating labels @@ -74,9 +74,9 @@ Every issue and merge request can be assigned any number of labels. The labels a ### Filtering in list pages -From the project issue list page and the project merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels and project labels. +From the project issue list page and the project merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group (including subgroup ancestors) labels and project labels. -From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels and project labels. +From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels (including subgroup ancestors and subgroup descendants) and project labels.  diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 6cead7b9961..14f2e522f01 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -106,7 +106,7 @@ If you want to auto-cancel all pending non-HEAD pipelines on branch, when new pipeline will be created (after your git push or manually from UI), check **Auto-cancel pending pipelines** checkbox and save the changes. -## Badges +## Pipeline Badges In the pipelines settings page you can find pipeline status and test coverage badges for your project. The latest successful pipeline will be used to read diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index dedf102fc37..eb0ac221e30 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -57,11 +57,11 @@ The following items will be exported: - Project configuration including web hooks and services - Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities +- LFS objects The following items will NOT be exported: - Build traces and artifacts -- LFS objects - Container registry images - CI variables - Any encrypted tokens diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md index cac3cb599dd..f824756c10c 100644 --- a/doc/workflow/lfs/lfs_administration.md +++ b/doc/workflow/lfs/lfs_administration.md @@ -19,7 +19,7 @@ There are various configuration options to help GitLab server administrators: * Changing the location of LFS object storage * Setting up AWS S3 compatible object storage -### Omnibus packages +### Configuration for Omnibus installations In `/etc/gitlab/gitlab.rb`: @@ -33,7 +33,7 @@ gitlab_rails['lfs_enabled'] = false gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" ``` -### Installations from source +### Configuration for installations from source In `config/gitlab.yml`: @@ -44,20 +44,21 @@ In `config/gitlab.yml`: storage_path: /mnt/storage/lfs-objects ``` -## Setting up S3 compatible object storage +## Storing the LFS objects in an S3-compatible object storage -> **Note:** [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. -> Available in [GitLab CE][ce] 10.7 +> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core +in 10.7. -It is possible to store LFS objects on remote object storage instead of on a local disk. +It is possible to store LFS objects on a remote object storage which allows you +to offload storage to an external AWS S3 compatible service, freeing up disk +space locally. You can also host your own S3 compatible storage decoupled from +GitLab, with with a service such as [Minio](https://www.minio.io/). -This allows you to offload storage to an external AWS S3 compatible service, freeing up disk space locally. You can also host your own S3 compatible storage decoupled from GitLab, with with a service such as [Minio](https://www.minio.io/). +Object storage currently transfers files first to GitLab, and then on the +object storage in a second stage. This can be done either by using a rake task +to transfer existing objects, or in a background job after each file is received. -Object storage currently transfers files first to GitLab, and then on the object storage in a second stage. This can be done either by using a rake task to transfer existing objects, or in a background job after each file is received. - -### Object Storage Settings - -For source installations the following settings are nested under `lfs:` and then `object_store:`. On omnibus installs they are prefixed by `lfs_object_store_`. +The following general settings are supported. | Setting | Description | Default | |---------|-------------|---------| @@ -68,9 +69,7 @@ For source installations the following settings are nested under `lfs:` and then | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | -#### S3 compatible connection settings - -The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: +The `connection` settings match those provided by [Fog](https://github.com/fog). | Setting | Description | Default | |---------|-------------|---------| @@ -82,8 +81,43 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | | `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | +### S3 for Omnibus installations + +On Omnibus installations, the settings are prefixed by `lfs_object_store_`: + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['lfs_object_store_enabled'] = true + gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" + gitlab_rails['lfs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', + 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', + # The below options configure an S3 compatible host instead of AWS + 'host' => 'localhost', + 'endpoint' => 'http://127.0.0.1:9000', + 'path_style' => true + } + ``` + +1. Save the file and [reconfigure GitLab]s for the changes to take effect. +1. Migrate any existing local LFS objects to the object storage: + + ```bash + gitlab-rake gitlab:lfs:migrate + ``` + + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless + `gitlab_rails['lfs_object_store_background_upload']` is set to false. -### From source +### S3 for installations from source + +For source installations the settings are nested under `lfs:` and then +`object_store:`: 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -108,44 +142,13 @@ The connection settings match those provided by [Fog](https://github.com/fog), a 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: - ```bash - sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production - ``` - - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless - `gitlab_rails['lfs_object_store_background_upload']` is set to false. - -### In Omnibus - -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with - the values you want: - - ```ruby - gitlab_rails['lfs_object_store_enabled'] = true - gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" - gitlab_rails['lfs_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', - 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', - # The below options configure an S3 compatible host instead of AWS - 'host' => 'localhost', - 'endpoint' => 'http://127.0.0.1:9000', - 'path_style' => true - } - ``` - -1. Save the file and [reconfigure GitLab]s for the changes to take effect. -1. Migrate any existing local LFS objects to the object storage: - - ```bash - gitlab-rake gitlab:lfs:migrate - ``` + ```bash + sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production + ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless - `gitlab_rails['lfs_object_store_background_upload']` is set to false. + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless `background_upload` is set to + false. ## Storage statistics |
