diff options
Diffstat (limited to 'doc')
86 files changed, 1226 insertions, 323 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 02de2caf558..8075a40cae7 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -75,6 +75,8 @@ From there, you can see the following actions: - User was removed from project - Project export was downloaded - Project repository was downloaded +- Project was archived +- Project was unarchived ### Instance events **(PREMIUM ONLY)** diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index d43b3718bf9..f5d58db0133 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -471,7 +471,7 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. ``` 1. A certificate keypair is required for GitLab and the Container Registry to - communicate securely. By default omnibus-gitlab will generate one keypair, + communicate securely. By default Omnibus GitLab will generate one keypair, which is saved to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key`. When using a non-bundled Container Registry, you will need to supply a custom certificate key. To do that, add the following to @@ -487,7 +487,7 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. **Note:** The file specified at `registry_key_path` gets populated with the content specified by `internal_key`, each time reconfigure is executed. If - no file is specified, omnibus-gitlab will default it to + no file is specified, Omnibus GitLab will default it to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate it. diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 32462a95a1a..7238d08ab09 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -12,17 +12,17 @@ NOTE: **Note:** Custom Git hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md) Git natively supports hooks that are executed on different actions. -Examples of server-side git hooks include pre-receive, post-receive, and update. +Examples of server-side Git hooks include pre-receive, post-receive, and update. See [Git SCM Server-Side Hooks][hooks] for more information about each hook type. -As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab -administrators can add custom git hooks to any GitLab project. +As of GitLab Shell version 2.2.0 (which requires GitLab 7.5+), GitLab +administrators can add custom Git hooks to any GitLab project. ## Create a custom Git hook for a repository Server-side Git hooks are typically placed in the repository's `hooks` -subdirectory. In GitLab, hook directories are symlinked to the gitlab-shell -`hooks` directory for ease of maintenance between gitlab-shell upgrades. +subdirectory. In GitLab, hook directories are symlinked to the GitLab Shell +`hooks` directory for ease of maintenance between GitLab Shell upgrades. Custom hooks are implemented differently, but the behavior is exactly the same once the hook is created. Follow the steps below to set up a custom hook for a repository: @@ -36,7 +36,7 @@ repository: 1. Inside the new `custom_hooks` directory, create a file with a name matching the hook type. For a pre-receive hook the file name should be `pre-receive` with no extension. -1. Make the hook file executable and make sure it's owned by git. +1. Make the hook file executable and make sure it's owned by Git. 1. Write the code to make the Git hook function as expected. Hooks can be in any language. Ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will probably be @@ -49,17 +49,17 @@ as appropriate. To create a Git hook that applies to all of your repositories in your instance, set a global Git hook. Since all the repositories' `hooks` -directories are symlinked to gitlab-shell's `hooks` directory, adding any hook -to the gitlab-shell `hooks` directory will also apply it to all repositories. Follow +directories are symlinked to GitLab Shell's `hooks` directory, adding any hook +to the GitLab Shell `hooks` directory will also apply it to all repositories. Follow the steps below to properly set up a custom hook for all repositories: 1. On the GitLab server, navigate to the configured custom hook directory. The - default is in the gitlab-shell directory. The gitlab-shell `hook` directory + default is in the GitLab Shell directory. The GitLab Shell `hook` directory for an installation from source the path is usually `/home/git/gitlab-shell/hooks`. For Omnibus installs the path is usually `/opt/gitlab/embedded/service/gitlab-shell/hooks`. To look in a different directory for the global custom hooks, - set `custom_hooks_dir` in the gitlab-shell config. For + set `custom_hooks_dir` in the GitLab Shell config. For Omnibus installations, this can be set in `gitlab.rb`; and in source installations, this can be set in `gitlab-shell/config.yml`. 1. Create a new directory in this location. Depending on your hook, it will be diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index dc4cc401fca..64eca0b00f6 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -122,6 +122,7 @@ production: discover: nameserver: localhost record: secondary.postgresql.service.consul + record_type: A port: 8600 interval: 60 disconnect_timeout: 120 @@ -137,12 +138,16 @@ The following options can be set: | Option | Description | Default | |----------------------|---------------------------------------------------------------------------------------------------|-----------| | `nameserver` | The nameserver to use for looking up the DNS record. | localhost | -| `record` | The A record to look up. This option is required for service discovery to work. | | +| `record` | The record to look up. This option is required for service discovery to work. | | +| `record_type` | Optional record type to look up, this can be either A or SRV (since GitLab 12.3) | A | | `port` | The port of the nameserver. | 8600 | | `interval` | The minimum time in seconds between checking the DNS record. | 60 | | `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | | `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | +If `record_type` is set to `SRV`, GitLab will continue to use a round-robin algorithm +and will ignore the `weight` and `priority` in the record. + The `interval` value specifies the _minimum_ time between checks. If the A record has a TTL greater than this value, then service discovery will honor said TTL. For example, if the TTL of the A record is 90 seconds, then service diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md index d2c52b67e67..5153666705f 100644 --- a/doc/administration/dependency_proxy.md +++ b/doc/administration/dependency_proxy.md @@ -48,7 +48,7 @@ local location or even use object storage. The dependency proxy files for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source -installations under `shared/dependency_proxy/` (relative to the git home directory). +installations under `shared/dependency_proxy/` (relative to the Git home directory). To change the local storage path: **Omnibus GitLab installations** diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 1780e1babe9..9d653d4e09e 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -53,7 +53,7 @@ sudo service ssh restart ## Instructions In order to use the new protocol, clients need to either pass the configuration -`-c protocol.version=2` to the git command, or set it globally: +`-c protocol.version=2` to the Git command, or set it globally: ```sh git config --global protocol.version 2 diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 878f0ef842d..daec5f14e36 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -139,7 +139,7 @@ Git operations in GitLab will result in an API error. NOTE: **Note:** In most or all cases, the storage paths below end in `/repositories` which is -not that case with `path` in `git_data_dirs` of Omnibus GitLab installations. +not the case with `path` in `git_data_dirs` of Omnibus GitLab installations. Check the directory layout on your Gitaly server to be sure. **For Omnibus GitLab** @@ -283,10 +283,6 @@ can read and write to `/mnt/gitlab/storage2`. gitlab_rails['gitaly_token'] = 'abc123secret' ``` - NOTE: **Note:** - In some cases, you'll have to set `path` for each `git_data_dirs` in the - format `'path' => '/mnt/gitlab/<storage name>'`. - 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Tail the logs to see the requests: @@ -304,18 +300,22 @@ can read and write to `/mnt/gitlab/storage2`. storages: default: gitaly_address: tcp://gitaly1.internal:8075 + path: /some/dummy/path storage1: gitaly_address: tcp://gitaly1.internal:8075 + path: /some/dummy/path storage2: gitaly_address: tcp://gitaly2.internal:8075 + path: /some/dummy/path gitaly: token: 'abc123secret' ``` NOTE: **Note:** - In some cases, you'll have to set `path` for each of the `storages` in the - format `path: /mnt/gitlab/<storage name>/repositories`. + `/some/dummy/path` should be set to a local folder that exists, however no + data will be stored in this folder. This will no longer be necessary after + [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Tail the logs to see the requests: @@ -416,18 +416,22 @@ To configure Gitaly with TLS: storages: default: gitaly_address: tls://gitaly1.internal:9999 + path: /some/dummy/path storage1: gitaly_address: tls://gitaly1.internal:9999 + path: /some/dummy/path storage2: gitaly_address: tls://gitaly2.internal:9999 + path: /some/dummy/path gitaly: token: 'abc123secret' ``` NOTE: **Note:** - In some cases, you'll have to set `path` for each of the `storages` in the - format `path: /mnt/gitlab/<storage name>/repositories`. + `/some/dummy/path` should be set to a local folder that exists, however no + data will be stored in this folder. This will no longer be necessary after + [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`: @@ -514,6 +518,47 @@ One current feature of GitLab that still requires a shared directory (NFS) is There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) to eliminate the need for NFS to support GitLab Pages. +## Limiting RPC concurrency + +It can happen that CI clone traffic puts a large strain on your Gitaly +service. The bulk of the work gets done in the SSHUploadPack (for Git +SSH) and PostUploadPack (for Git HTTP) RPC's. To prevent such workloads +from overcrowding your Gitaly server you can set concurrency limits in +Gitaly's configuration file. + +```ruby +# in /etc/gitlab/gitlab.rb + +gitaly['concurrency'] = [ + { + 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", + 'max_per_repo' => 20 + }, + { + 'rpc' => "/gitaly.SSHService/SSHUploadPack", + 'max_per_repo' => 20 + } +] +``` +This will limit the number of in-flight RPC calls for the given RPC's. +The limit is applied per repository. In the example above, each on the +Gitaly server can have at most 20 simultaneous PostUploadPack calls in +flight, and the same for SSHUploadPack. If another request comes in for +a repository that hase used up its 20 slots, that request will get +queued. + +You can observe the behavior of this queue via the Gitaly logs and via +Prometheus. In the Gitaly logs, you can look for the string (or +structured log field) `acquire_ms`. Messages that have this field are +reporting about the concurrency limiter. In Prometheus, look for the +`gitaly_rate_limiting_in_progress`, `gitaly_rate_limiting_queued` and +`gitaly_rate_limiting_seconds` metrics. + +The name of the Prometheus metric is not quite right because this is a +concurrency limiter, not a rate limiter. If a client makes 1000 requests +in a row in a very short timespan, the concurrency will not exceed 1, +and this mechanism (the concurrency limiter) will do nothing. + ## Troubleshooting Gitaly ### Commits, pushes, and clones return a 401 @@ -655,3 +700,33 @@ To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-se on the Gitaly node matches the one on all other nodes. If it doesn't match, update the secrets file on the Gitaly node to match the others, then [reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Command line tools cannot connect to Gitaly + +If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. + +Verify that you can reach Gitaly via TCP: + +```bash +sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] +``` + +If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. + +If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. + +If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: + +```bash +echo $http_proxy +echo $https_proxy +``` + +If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. + +To remove the proxy setting, run the following commands (depending on which variables had values): + +```bash +unset http_proxy +unset https_proxy +```
\ No newline at end of file diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 73a39a6dd35..29915cb3a99 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -151,7 +151,7 @@ Reply by email should now be working. #### Postfix -Example configuration for Postfix mail server. Assumes mailbox incoming@gitlab.example.com. +Example configuration for Postfix mail server. Assumes mailbox `incoming@gitlab.example.com`. Example for Omnibus installs: @@ -218,7 +218,7 @@ incoming_email: #### Gmail -Example configuration for Gmail/G Suite. Assumes mailbox gitlab-incoming@gmail.com. +Example configuration for Gmail/G Suite. Assumes mailbox `gitlab-incoming@gmail.com`. Example for Omnibus installs: diff --git a/doc/administration/index.md b/doc/administration/index.md index 2b25e8efd23..650cb10a64a 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -64,6 +64,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. +- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification emails with S/MIME #### Customizing GitLab's appearance @@ -192,6 +193,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. for troubleshooting Kubernetes-related issues. - Useful links from the Support Team: - [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html). - - [Repairing and recovering broken git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). + - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html). - [Strace zine](https://wizardzines.com/zines/strace/). diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index e1bbabb2878..f9be06c6daf 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -1,8 +1,8 @@ # Issue closing pattern **(CORE ONLY)** >**Note:** -This is the administration documentation. -There is a separate [user documentation] on issue closing pattern. +This is the administration documentation. There is a separate [user documentation](../user/project/issues/managing_issues.md#closing-issues-automatically) +on issue closing pattern. When a commit or merge request resolves one or more issues, it is possible to automatically have these issues closed when the commit or merge request lands @@ -13,8 +13,8 @@ in the project's default branch. In order to change the pattern you need to have access to the server that GitLab is installed on. -The default pattern can be located in [gitlab.yml.example] under the -"Automatic issue closing" section. +The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example) +under the "Automatic issue closing" section. > **Tip:** You are advised to use <http://rubular.com> to test the issue closing pattern. @@ -31,7 +31,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" ``` -1. [Reconfigure] GitLab for the changes to take effect. +1. [Reconfigure](restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. **For installations from source** @@ -42,9 +42,4 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" ``` -1. [Restart] GitLab for the changes to take effect. - -[gitlab.yml.example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example -[reconfigure]: restart_gitlab.md#omnibus-gitlab-reconfigure -[restart]: restart_gitlab.md#installations-from-source -[user documentation]: ../user/project/issues/managing_issues.md#closing-issues-automatically +1. [Restart](restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 60921ca791f..350cd5b7992 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -94,7 +94,7 @@ If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need ### Object Storage Settings -For source installations the following settings are nested under `artifacts:` and then `object_store:`. On omnibus installs they are prefixed by `artifacts_object_store_`. +For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`. | Setting | Description | Default | |---------|-------------|---------| diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index ec26c0b2e7e..4809bee0df6 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,7 +1,7 @@ # GitLab Prometheus metrics >**Note:** -Available since [Omnibus GitLab 9.3][29118]. For +Available since [Omnibus GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-ce/issues/29118). For installations from source you'll have to configure it yourself. To enable the GitLab Prometheus metrics: @@ -9,125 +9,202 @@ To enable the GitLab Prometheus metrics: 1. Log into GitLab as an administrator, and go to the Admin area. 1. Click on the gear, then click on Settings. 1. Find the `Metrics - Prometheus` section, and click `Enable Prometheus Metrics` -1. [Restart GitLab][restart] for the changes to take effect +1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect ## Collecting the metrics GitLab monitors its own internal service metrics, and makes them available at the -`/-/metrics` endpoint. Unlike other [Prometheus] exporters, in order to access -it, the client IP needs to be [included in a whitelist][whitelist]. +`/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, in order to access +it, the client IP needs to be [included in a whitelist](../ip_whitelist.md). For Omnibus and Chart installations, these metrics are automatically enabled and collected as of [GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1702). For source installations or earlier versions, these metrics will need to be enabled manually and collected by a Prometheus server. -## Unicorn Metrics available +## Metrics available The following metrics are available: -| Metric | Type | Since | Description | -|:--------------------------------- |:--------- |:----- |:----------- | -| db_ping_timeout | Gauge | 9.4 | Whether or not the last database ping timed out | -| db_ping_success | Gauge | 9.4 | Whether or not the last database ping succeeded | -| db_ping_latency_seconds | Gauge | 9.4 | Round trip time of the database ping | -| filesystem_access_latency_seconds | Gauge | 9.4 | Latency in accessing a specific filesystem | -| filesystem_accessible | Gauge | 9.4 | Whether or not a specific filesystem is accessible | -| filesystem_write_latency_seconds | Gauge | 9.4 | Write latency of a specific filesystem | -| filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | -| filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | -| filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | -| gitlab_cache_misses_total | Counter | 10.2 | Cache read miss | -| gitlab_cache_operation_duration_seconds | Histogram | 10.2 | Cache access time | -| gitlab_cache_operations_total | Counter | 12.2 | Cache operations by controller/action | -| http_requests_total | Counter | 9.4 | Rack request count | -| http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | -| pipelines_created_total | Counter | 9.4 | Counter of pipelines created | -| rack_uncaught_errors_total | Counter | 9.4 | Rack connections handling uncaught errors count | -| redis_ping_timeout | Gauge | 9.4 | Whether or not the last redis ping timed out | -| redis_ping_success | Gauge | 9.4 | Whether or not the last redis ping succeeded | -| redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | -| user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | -| upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | -| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | -| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | -| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | -| unicorn_workers | Gauge | 12.0 | The number of Unicorn workers | +| Metric | Type | Since | Description | Labels | +|:---------------------------------------------------------------|:----------|-----------------------:|:----------------------------------------------------------------------------------------------------|:----------------------------------------------------| +| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering markdown into HTML when cached output exists | controller, action | +| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering markdown into HTML when cached outupt does not exist | controller, action | +| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | controller, action | +| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | controller, action, operation | +| `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | +| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | controller, action, module, method | +| `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | +| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT | | +| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (gitlab_transaction_* metrics) | | +| `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | +| `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | +| `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | controller, action | +| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | controller, action | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | controller, action | +| `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for api /jobs/request | | +| `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for api /jobs/request | | +| `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for api /jobs/request | | +| `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for api /jobs/request | | +| `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | +| `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for etag cache hit. | endpoint | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for etag cache miss - header missing | endpoint | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for etag cache miss - key not found | endpoint | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for etag middleware accessed | endpoint | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for etag cache miss - resource changed | endpoint | +| `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | +| `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | +| `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | +| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | branch | +| `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | +| `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | +| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for recieved emails | handler | +| `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | +| `gitlab_transaction_event_remove_branch_total` | Counter | 9.4 | Counter when a branch is removed for any repository | | +| `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | +| `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | +| `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of sidekiq exceptions | | +| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | projects_without_jid_count, projects_with_jid_count | +| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for api /jobs/request/:id | | +| `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new redis connections | | +| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | +| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | controller, action | +| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | controller, action, view | +| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | controller, action, view | +| `http_requests_total` | Counter | 9.4 | Rack request count | method | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | method, status | +| `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | +| `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | +| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in | | +| `upload_file_does_not_exist` | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | | +| `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | +| `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | + +## Metrics controlled by a feature flag + +The following metrics can be controlled by feature flags: + +| Metric | Feature Flag | +|:---------------------------------------------------------------|:-------------------------------------------------------------------| +| `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | +| `gitlab_transaction_allocated_memory_bytes` | `prometheus_metrics_transaction_allocated_memory` | +| `gitlab_transaction_event_build_found_total` | `prometheus_transaction_event_build_found_total` | +| `gitlab_transaction_event_build_invalid_total` | `prometheus_transaction_event_build_invalid_total` | +| `gitlab_transaction_event_build_not_found_cached_total` | `prometheus_transaction_event_build_not_found_cached_total` | +| `gitlab_transaction_event_build_not_found_total` | `prometheus_transaction_event_build_not_found_total` | +| `gitlab_transaction_event_change_default_branch_total` | `prometheus_transaction_event_change_default_branch_total` | +| `gitlab_transaction_event_create_repository_total` | `prometheus_transaction_event_create_repository_total` | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | `prometheus_transaction_event_etag_caching_cache_hit_total` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | `prometheus_transaction_event_etag_caching_header_missing_total` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | `prometheus_transaction_event_etag_caching_key_not_found_total` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | `prometheus_transaction_event_etag_caching_middleware_used_total` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | `prometheus_transaction_event_etag_caching_resource_changed_total` | +| `gitlab_transaction_event_fork_repository_total` | `prometheus_transaction_event_fork_repository_total` | +| `gitlab_transaction_event_import_repository_total` | `prometheus_transaction_event_import_repository_total` | +| `gitlab_transaction_event_push_branch_total` | `prometheus_transaction_event_push_branch_total` | +| `gitlab_transaction_event_push_commit_total` | `prometheus_transaction_event_push_commit_total` | +| `gitlab_transaction_event_push_tag_total` | `prometheus_transaction_event_push_tag_total` | +| `gitlab_transaction_event_rails_exception_total` | `prometheus_transaction_event_rails_exception_total` | +| `gitlab_transaction_event_receive_email_total` | `prometheus_transaction_event_receive_email_total` | +| `gitlab_transaction_event_remote_mirrors_failed_total` | `prometheus_transaction_event_remote_mirrors_failed_total` | +| `gitlab_transaction_event_remote_mirrors_finished_total` | `prometheus_transaction_event_remote_mirrors_finished_total` | +| `gitlab_transaction_event_remote_mirrors_running_total` | `prometheus_transaction_event_remote_mirrors_running_total` | +| `gitlab_transaction_event_remove_branch_total` | `prometheus_transaction_event_remove_branch_total` | +| `gitlab_transaction_event_remove_repository_total` | `prometheus_transaction_event_remove_repository_total` | +| `gitlab_transaction_event_remove_tag_total` | `prometheus_transaction_event_remove_tag_total` | +| `gitlab_transaction_event_sidekiq_exception_total` | `prometheus_transaction_event_sidekiq_exception_total` | +| `gitlab_transaction_event_stuck_import_jobs_total` | `prometheus_transaction_event_stuck_import_jobs_total` | +| `gitlab_transaction_event_update_build_total` | `prometheus_transaction_event_update_build_total` | +| `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | ## Sidekiq Metrics available for Geo **(PREMIUM)** Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. -| Metric | Type | Since | Description | Labels | -|:-------------------------------------------- |:------- |:----- |:----------- |:------ | -| geo_db_replication_lag_seconds | Gauge | 10.2 | Database replication lag (seconds) | url -| geo_repositories | Gauge | 10.2 | Total number of repositories available on primary | url -| geo_repositories_synced | Gauge | 10.2 | Number of repositories synced on secondary | url -| geo_repositories_failed | Gauge | 10.2 | Number of repositories failed to sync on secondary | url -| geo_lfs_objects | Gauge | 10.2 | Total number of LFS objects available on primary | url -| geo_lfs_objects_synced | Gauge | 10.2 | Number of LFS objects synced on secondary | url -| geo_lfs_objects_failed | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url -| geo_attachments | Gauge | 10.2 | Total number of file attachments available on primary | url -| geo_attachments_synced | Gauge | 10.2 | Number of attachments synced on secondary | url -| geo_attachments_failed | Gauge | 10.2 | Number of attachments failed to sync on secondary | url -| geo_last_event_id | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url -| geo_last_event_timestamp | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url -| geo_cursor_last_event_id | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url -| geo_cursor_last_event_timestamp | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url -| geo_status_failed_total | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url -| geo_last_successful_status_check_timestamp | Gauge | 10.2 | Last timestamp when the status was successfully updated | url -| geo_lfs_objects_synced_missing_on_primary | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url -| geo_job_artifacts_synced_missing_on_primary | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url -| geo_attachments_synced_missing_on_primary | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url -| geo_repositories_checksummed_count | Gauge | 10.7 | Number of repositories checksummed on primary | url -| geo_repositories_checksum_failed_count | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url -| geo_wikis_checksummed_count | Gauge | 10.7 | Number of wikis checksummed on primary | url -| geo_wikis_checksum_failed_count | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url -| geo_repositories_verified_count | Gauge | 10.7 | Number of repositories verified on secondary | url -| geo_repositories_verification_failed_count | Gauge | 10.7 | Number of repositories failed to verify on secondary | url -| geo_repositories_checksum_mismatch_count | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url -| geo_wikis_verified_count | Gauge | 10.7 | Number of wikis verified on secondary | url -| geo_wikis_verification_failed_count | Gauge | 10.7 | Number of wikis failed to verify on secondary | url -| geo_wikis_checksum_mismatch_count | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url -| geo_repositories_checked_count | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url -| geo_repositories_checked_failed_count | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url -| geo_repositories_retrying_verification_count | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url -| geo_wikis_retrying_verification_count | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url +| Metric | Type | Since | Description | Labels | +|:---------------------------------------------- |:------- |:----- |:----------- |:------ | +| `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | url | +| `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | url | +| `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | url | +| `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | url | +| `geo_lfs_objects` | Gauge | 10.2 | Total number of LFS objects available on primary | url | +| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of LFS objects synced on secondary | url | +| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url | +| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | url | +| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | url | +| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | url | +| `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url | +| `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url | +| `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url | +| `geo_cursor_last_event_timestamp` | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url | +| `geo_status_failed_total` | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url | +| `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | url | +| `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url | +| `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url | +| `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url | +| `geo_repositories_checksummed_count` | Gauge | 10.7 | Number of repositories checksummed on primary | url | +| `geo_repositories_checksum_failed_count` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url | +| `geo_wikis_checksummed_count` | Gauge | 10.7 | Number of wikis checksummed on primary | url | +| `geo_wikis_checksum_failed_count` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url | +| `geo_repositories_verified_count` | Gauge | 10.7 | Number of repositories verified on secondary | url | +| `geo_repositories_verification_failed_count` | Gauge | 10.7 | Number of repositories failed to verify on secondary | url | +| `geo_repositories_checksum_mismatch_count` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url | +| `geo_wikis_verified_count` | Gauge | 10.7 | Number of wikis verified on secondary | url | +| `geo_wikis_verification_failed_count` | Gauge | 10.7 | Number of wikis failed to verify on secondary | url | +| `geo_wikis_checksum_mismatch_count` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url | +| `geo_repositories_checked_count` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url | +| `geo_repositories_checked_failed_count` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url | +| `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url | +| `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url | ### Ruby metrics Some basic Ruby runtime metrics are available: -| Metric | Type | Since | Description | -|:-------------------------------------- |:--------- |:----- |:----------- | -| ruby_gc_duration_seconds_total | Counter | 11.1 | Time spent by Ruby in GC | -| ruby_gc_stat_... | Gauge | 11.1 | Various metrics from [GC.stat] | -| ruby_file_descriptors | Gauge | 11.1 | File descriptors per process | -| ruby_memory_bytes | Gauge | 11.1 | Memory usage by process | -| ruby_sampler_duration_seconds_total | Counter | 11.1 | Time spent collecting stats | -| ruby_process_cpu_seconds_total | Gauge | 12.0 | Total amount of CPU time per process | -| ruby_process_max_fds | Gauge | 12.0 | Maximum number of open file descriptors per process | -| ruby_process_resident_memory_bytes | Gauge | 12.0 | Memory usage by process, measured in bytes | -| ruby_process_start_time_seconds | Gauge | 12.0 | UNIX timestamp of process start time | +| Metric | Type | Since | Description | +|:------------------------------------ |:--------- |:----- |:----------- | +| `ruby_gc_duration_seconds` | Counter | 11.1 | Time spent by Ruby in GC | +| `ruby_gc_stat_...` | Gauge | 11.1 | Various metrics from [GC.stat] | +| `ruby_file_descriptors` | Gauge | 11.1 | File descriptors per process | +| `ruby_memory_bytes` | Gauge | 11.1 | Memory usage by process | +| `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | +| `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | +| `ruby_process_max_fds` | Gauge | 12.0 | Maximum number of open file descriptors per process | +| `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process, measured in bytes | +| `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | -[GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat +[GC.stat]: https://ruby-doc.org/core-2.6.3/GC.html#method-c-stat + +## Unicorn Metrics + +Unicorn specific metrics, when Unicorn is used. + +| Metric | Type | Since | Description | +|:-----------------------------|:------|:------|:---------------------------------------------------| +| `unicorn_active_connections` | Gauge | 11.0 | The number of active Unicorn connections (workers) | +| `unicorn_queued_connections` | Gauge | 11.0 | The number of queued Unicorn connections | +| `unicorn_workers` | Gauge | 12.0 | The number of Unicorn workers | ## Puma Metrics **(EXPERIMENTAL)** -When Puma is used instead of Unicorn, following metrics are available: - -| Metric | Type | Since | Description | -|:-------------------------------------------- |:------- |:----- |:----------- | -| puma_workers | Gauge | 12.0 | Total number of workers | -| puma_running_workers | Gauge | 12.0 | Number of booted workers | -| puma_stale_workers | Gauge | 12.0 | Number of old workers | -| puma_running | Gauge | 12.0 | Number of running threads | -| puma_queued_connections | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | -| puma_active_connections | Gauge | 12.0 | Number of threads processing a request | -| puma_pool_capacity | Gauge | 12.0 | Number of requests the worker is capable of taking right now | -| puma_max_threads | Gauge | 12.0 | Maximum number of worker threads | -| puma_idle_threads | Gauge | 12.0 | Number of spawned threads which are not processing a request | -| rack_state_total | Gauge | 12.0 | Number of requests in a given rack state | -| puma_killer_terminations_total | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | +When Puma is used instead of Unicorn, the following metrics are available: + +| Metric | Type | Since | Description | +|:---------------------------------------------- |:------- |:----- |:----------- | +| `puma_workers` | Gauge | 12.0 | Total number of workers | +| `puma_running_workers` | Gauge | 12.0 | Number of booted workers | +| `puma_stale_workers` | Gauge | 12.0 | Number of old workers | +| `puma_running` | Gauge | 12.0 | Number of running threads | +| `puma_queued_connections` | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | +| `puma_active_connections` | Gauge | 12.0 | Number of threads processing a request | +| `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | +| `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | +| `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | +| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | ## Metrics shared directory @@ -144,9 +221,3 @@ If GitLab is installed using Omnibus and `tmpfs` is available then metrics directory will be automatically configured. [← Back to the main Prometheus page](index.md) - -[29118]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29118 -[Prometheus]: https://prometheus.io -[restart]: ../../restart_gitlab.md#omnibus-gitlab-restart -[whitelist]: ../ip_whitelist.md -[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/packages.md b/doc/administration/packages.md index c0f8777a8c0..1628b6d6f91 100644 --- a/doc/administration/packages.md +++ b/doc/administration/packages.md @@ -55,7 +55,7 @@ local location or even use object storage. The packages for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/packages/` and for source -installations under `shared/packages/` (relative to the git homedir). +installations under `shared/packages/` (relative to the Git homedir). To change the local storage path: **Omnibus GitLab installations** diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index 52f1e7cde28..92a4d56ca63 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -39,7 +39,7 @@ Follow the steps below to set up a custom hook: 1. Inside the `plugins` directory, create a file with a name of your choice, without spaces or special characters. -1. Make the hook file executable and make sure it's owned by the git user. +1. Make the hook file executable and make sure it's owned by the Git user. 1. Write the code to make the plugin function as expected. That can be in any language, and ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 6d1ffbffefb..56cd23b2eb8 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -18,7 +18,7 @@ The instructions make the assumption that you will be using the email address `i sudo apt-get install postfix ``` - When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches gitlab.example.com`. + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. 1. Install the `mailutils` package. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 05a7cb006a3..ab911c1cf0e 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -3,7 +3,7 @@ > [Introduced][ce-3232] in GitLab 8.7. It is OFF by default because it still causes too many false alarms. -Git has a built-in mechanism, [git fsck][git-fsck], to verify the +Git has a built-in mechanism, [`git fsck`][git-fsck], to verify the integrity of all data committed to a repository. GitLab administrators can trigger such a check for a project via the project page under the admin panel. The checks run asynchronously so it may take a few minutes diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 1669f8b128c..3ee8673b297 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -82,19 +82,17 @@ by another folder with the next 2 characters. They are both stored in a special > [Introduced](https://gitlab.com/gitlab-org/gitaly/issues/1606) in GitLab 12.1. -Forks of public projects are deduplicated by creating a third repository, the object pool, containing the objects from the source project. Using `objects/info/alternates`, the source project and forks use the object pool for shared objects. Objects are moved from the source project to the object pool when housekeeping is run on the source project. +Forks of public projects are deduplicated by creating a third repository, the +object pool, containing the objects from the source project. Using +`objects/info/alternates`, the source project and forks use the object pool for +shared objects. Objects are moved from the source project to the object pool +when housekeeping is run on the source project. ```ruby # object pool paths "@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" ``` -Object pools can be disabled using the `object_pools` feature flag, and can be -disabled for individual projects by executing -`Feature.disable(:object_pools, Project.find(<id>))`. Disabling object pools -will not change existing deduplicated forks, but will prevent new forks from -being deduplicated. - DANGER: **Danger:** Do not run `git prune` or `git gc` in pool repositories! This can cause data loss in "real" repositories that depend on the pool in @@ -118,7 +116,7 @@ If you do have any existing integration, you may want to do a small rollout firs to validate. You can do so by specifying a range with the operation. This is an example of how to limit the rollout to Project IDs 50 to 100, running in -an Omnibus Gitlab installation: +an Omnibus GitLab installation: ```bash sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 @@ -139,7 +137,7 @@ To schedule a complete rollback, see the [rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. The rollback task also supports specifying a range of Project IDs. Here is an example -of limiting the rollout to Project IDs 50 to 100, in an Omnibus Gitlab installation: +of limiting the rollout to Project IDs 50 to 100, in an Omnibus GitLab installation: ```bash sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 @@ -185,7 +183,7 @@ CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in ##### LFS Objects -LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following git own implementation: +LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following Git own implementation: ```ruby "shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md new file mode 100644 index 00000000000..b2e3bf8487b --- /dev/null +++ b/doc/administration/smime_signing_email.md @@ -0,0 +1,49 @@ +# Signing outgoing email with S/MIME + +Notification emails sent by Gitlab can be signed with S/MIME for improved +security. + +> **Note:** +Please be aware that S/MIME certificates and TLS/SSL certificates are not the +same and are used for different purposes: TLS creates a secure channel, whereas +S/MIME signs and/or encrypts the message itself + +## Enable S/MIME signing + +This setting must be explicitly enabled and a single pair of key and certificate +files must be provided in `gitlab.rb` or `gitlab.yml` if you are using Omnibus +GitLab or installed GitLab from source respectively: + +```yaml +email_smime: + enabled: true + key_file: /etc/pki/smime/private/gitlab.key + cert_file: /etc/pki/smime/certs/gitlab.crt +``` + +- Both files must be provided PEM-encoded. +- The key file must be unencrypted so that Gitlab can read it without user + intervention. + +NOTE: **Note:** Be mindful of the access levels for your private keys and visibility to +third parties. + +### How to convert S/MIME PKCS#12 / PFX format to PEM encoding + +Typically S/MIME certificates are handled in binary PKCS#12 format (`.pfx` or `.p12` +extensions), which contain the following in a single encrypted file: + +- Server certificate +- Intermediate certificates (if any) +- Private key + +In order to export the required files in PEM encoding from the PKCS#12 file, +the `openssl` command can be used: + +```bash +#-- Extract private key in PEM encoding (no password, unencrypted) +$ openssl pkcs12 -in gitlab.p12 -nocerts -nodes -out gitlab.key + +#-- Extract certificates in PEM encoding (full certs chain including CA) +$ openssl pkcs12 -in gitlab.p12 -nokeys -out gitlab.crt +``` diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 238c522a0ee..260af333e8e 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -83,12 +83,22 @@ and they will assist you with any issues you are having. kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn ``` -- It is not possible to get all the logs via Kubectl at once, like with `gitlab-ctl tail`, - but a number of third-party tools can be used to do it: +- Tail and follow all pods that share a label (in this case, `unicorn`): - - [Kubetail](https://github.com/johanhaleby/kubetail) - - [kail: kubernetes tail](https://github.com/boz/kail) - - [stern](https://github.com/wercker/stern) + ```bash + # all containers in the unicorn pods + kubectl logs -f -l app=unicorn --all-containers=true --max-log-requests=50 + + # only the unicorn containers in all unicorn pods + kubectl logs -f -l app=unicorn -c unicorn --max-log-requests=50 + ``` + +- One can stream logs from all containers at once, similar to the Omnibus + command `gitlab-ctl tail`: + + ```bash + kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100 + ``` - Check all events in the `gitlab` namespace (the namespace name can be different if you specified a different one when deploying the helm chart): diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 9e9d12f23bb..a4bed72b965 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -57,7 +57,7 @@ This configuration relies on valid AWS credentials to be configured already. ## Object Storage Settings -For source installations the following settings are nested under `uploads:` and then `object_store:`. On omnibus installs they are prefixed by `uploads_object_store_`. +For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`. | Setting | Description | Default | |---------|-------------|---------| diff --git a/doc/api/dependencies.md b/doc/api/dependencies.md index 5767d3572dd..5296d4e316f 100644 --- a/doc/api/dependencies.md +++ b/doc/api/dependencies.md @@ -5,7 +5,8 @@ This API is in an alpha stage and considered unstable. The response payload may be subject to change or breakage across GitLab releases. -Every call to this endpoint requires authentication. To perform this call, user should be authorized to read +Every call to this endpoint requires authentication. To perform this call, user should be authorized to read repository. +To see vulnerabilities in response, user should be authorized to read [Project Security Dashboard](../user/application_security/security_dashboard/index.md#project-security-dashboard). ## List project dependencies @@ -17,8 +18,8 @@ supported by Gemnasium. ``` GET /projects/:id/dependencies -GET /projects/:id/vulnerabilities?package_manager=maven -GET /projects/:id/vulnerabilities?package_manager=yarn,bundler +GET /projects/:id/dependencies?package_manager=maven +GET /projects/:id/dependencies?package_manager=yarn,bundler ``` | Attribute | Type | Required | Description | @@ -38,13 +39,18 @@ Example response: "name": "rails", "version": "5.0.1", "package_manager": "bundler", - "dependency_file_path": "Gemfile.lock" + "dependency_file_path": "Gemfile.lock", + "vulnerabilities": [{ + "name": "DDoS", + "severity": "unknown" + }] }, { "name": "hanami", "version": "1.3.1", "package_manager": "bundler", - "dependency_file_path": "Gemfile.lock" + "dependency_file_path": "Gemfile.lock", + "vulnerabilities": [] } ] ``` diff --git a/doc/api/epics.md b/doc/api/epics.md index 3036b3c2364..87ae0c48199 100644 --- a/doc/api/epics.md +++ b/doc/api/epics.md @@ -65,6 +65,8 @@ Example response: "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/4", + "reference": "&4", "author": { "id": 10, "name": "Lu Mayer", @@ -118,6 +120,8 @@ Example response: "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/5", + "reference": "&5", "author":{ "id": 7, "name": "Pamella Huel", @@ -182,6 +186,8 @@ Example response: "title": "Epic", "description": "Epic description", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/6", + "reference": "&6", "author": { "name" : "Alexandra Bashirian", "avatar_url" : null, @@ -247,6 +253,8 @@ Example response: "title": "New Title", "description": "Epic description", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/6", + "reference": "&6", "author": { "name" : "Alexandra Bashirian", "avatar_url" : null, diff --git a/doc/api/events.md b/doc/api/events.md index 6dca8e52f69..1cd7047b867 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -70,7 +70,7 @@ Parameters: Example request: -``` +```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/events?target_type=issue&action=created&after=2017-01-31&before=2017-03-01 ``` @@ -275,7 +275,7 @@ Parameters: Example request: -``` +```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:project_id/events?target_type=issue&action=created&after=2017-01-31&before=2017-03-01 ``` @@ -343,8 +343,8 @@ Example response: "username": "root", "id": 1, "state": "active", - "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/root" + "avatar_url": "https://gitlab.example.com/uploads/user/avatar/1/fox_avatar.png", + "web_url": "https://gitlab.example.com/root" }, "created_at": "2015-12-04T10:33:56.698Z", "system": false, @@ -357,8 +357,8 @@ Example response: "username": "root", "id": 1, "state": "active", - "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/root" + "avatar_url": "https://gitlab.example.com/uploads/user/avatar/1/fox_avatar.png", + "web_url": "https://gitlab.example.com/root" }, "author_username": "root" } diff --git a/doc/api/features.md b/doc/api/features.md index 6ecd4ec14b9..e8d0c7c942b 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -60,8 +60,8 @@ POST /features/:name | `value` | integer/string | yes | `true` or `false` to enable/disable, or an integer for percentage of time | | `feature_group` | string | no | A Feature group name | | `user` | string | no | A GitLab username | -| `group` | string | no | A GitLab group's path, for example 'gitlab-org' | -| `project` | string | no | A projects path, for example 'gitlab-org/gitlab-ce' | +| `group` | string | no | A GitLab group's path, for example `gitlab-org` | +| `project` | string | no | A projects path, for example `gitlab-org/gitlab-ce` | Note that you can enable or disable a feature for a `feature_group`, a `user`, a `group`, and a `project` in a single API call. diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index d0b33ab467f..5eba7f038ed 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -111,7 +111,7 @@ PUT /geo_nodes/:id |-----------------------------|---------|-----------|---------------------------------------------------------------------------| | `id` | integer | yes | The ID of the Geo node. | | `enabled` | boolean | no | Flag indicating if the Geo node is enabled. | -| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in gitlab.rb, otherwise it must match `external_url`. | +| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in `gitlab.rb`, otherwise it must match `external_url`. | | `url` | string | yes | The user-facing URL of the Geo node. | | `internal_url` | string | no | The URL defined on the primary node that secondary nodes should use to contact it. Returns `url` if not set.| | `files_max_capacity` | integer | no | Control the maximum concurrency of LFS/attachment backfill for this secondary node. | diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 2d3bec4ff67..d99a4c37d72 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -109,6 +109,7 @@ The API can be explored interactively using the [GraphiQL IDE](../index.md#graph | `visibility` | String | | | `lfsEnabled` | Boolean | | | `requestAccessEnabled` | Boolean | | +| `rootStorageStatistics` | RootStorageStatistics | The aggregated storage statistics. Only available if the namespace has no parent | | `userPermissions` | GroupPermissions! | Permissions for the current user on the resource | | `webUrl` | String! | | | `avatarUrl` | String | | @@ -453,6 +454,17 @@ The API can be explored interactively using the [GraphiQL IDE](../index.md#graph | `exists` | Boolean! | | | `tree` | Tree | | +### RootStorageStatistics + +| Name | Type | Description | +| --- | ---- | ---------- | +| `storageSize` | Int! | The total storage in Bytes | +| `repositorySize` | Int! | The git repository size in Bytes | +| `lfsObjectsSize` | Int! | The LFS objects size in Bytes | +| `buildArtifactsSize` | Int! | The CI artifacts size in Bytes | +| `packagesSize` | Int! | The packages size in Bytes | +| `wikiSize` | Int! | The wiki size in Bytes | + ### Submodule | Name | Type | Description | diff --git a/doc/api/issues.md b/doc/api/issues.md index 4f2b4a966c9..cadc9291489 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -49,7 +49,7 @@ GET /issues?confidential=true | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `iids[]` | integer array | no | Return only the issues having the given `iid` | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | | `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | @@ -198,7 +198,7 @@ GET /groups/:id/issues?confidential=true | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search group issues against their `title` and `description` | | `created_after` | datetime | no | Return issues created on or after the given time | @@ -284,7 +284,6 @@ Example response: "award_emoji":"http://example.com/api/v4/projects/4/issues/41/award_emoji", "project":"http://example.com/api/v4/projects/4" }, - "subscribed": false, "task_completion_status":{ "count":0, "completed_count":0 @@ -347,7 +346,7 @@ GET /projects/:id/issues?confidential=true | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search project issues against their `title` and `description` | | `created_after` | datetime | no | Return issues created on or after the given time | diff --git a/doc/api/labels.md b/doc/api/labels.md index 5db0edcf14d..fde1d861cf6 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -137,8 +137,9 @@ DELETE /projects/:id/labels | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the label | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `label_id` | integer | yes (or `name`) | The id of the existing label | +| `name` | string | yes (or `label_id`) | The name of the existing label | ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels?name=bug" @@ -156,7 +157,8 @@ PUT /projects/:id/labels | Attribute | Type | Required | Description | | --------------- | ------- | --------------------------------- | ------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the existing label | +| `label_id` | integer | yes (or `name`) | The id of the existing label | +| `name` | string | yes (or `label_id`) | The name of the existing label | | `new_name` | string | yes if `color` is not provided | The new name of the label | | `color` | string | yes if `new_name` is not provided | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | | `description` | string | no | The new description of the label | diff --git a/doc/api/lint.md b/doc/api/lint.md index 79f5e629c7f..dacd3f4c493 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -1,4 +1,4 @@ -# Validate the .gitlab-ci.yml (API) +# Validate the `.gitlab-ci.yml` (API) > [Introduced][ce-5953] in GitLab 8.12. @@ -10,7 +10,7 @@ POST /ci/lint | Attribute | Type | Required | Description | | ---------- | ------- | -------- | -------- | -| `content` | string | yes | the .gitlab-ci.yaml content| +| `content` | string | yes | the `.gitlab-ci.yaml` content| ```bash curl --header "Content-Type: application/json" https://gitlab.example.com/api/v4/ci/lint --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' diff --git a/doc/api/projects.md b/doc/api/projects.md index 373607f8f4b..cf28ea84704 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -852,10 +852,10 @@ Get the users list of a project. GET /projects/:id/users ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `search` | string | no | Search for specific users | -| `skip_users` | array[int] | no | Filter out users with the specified IDs | +| Attribute | Type | Required | Description | +| ------------ | ------------- | -------- | ----------- | +| `search` | string | no | Search for specific users | +| `skip_users` | integer array | no | Filter out users with the specified IDs | ```json [ @@ -2037,13 +2037,13 @@ Read more in the [Project Badges](project_badges.md) documentation. The non-default [issue and merge request description templates](../user/project/description_templates.md) are managed inside the project's repository. So you can manage them via the API through the [Repositories API](repositories.md) and the [Repository Files API](repository_files.md). -## Download snapshot of a git repository +## Download snapshot of a Git repository > Introduced in GitLab 10.7 This endpoint may only be accessed by an administrative user. -Download a snapshot of the project (or wiki, if requested) git repository. This +Download a snapshot of the project (or wiki, if requested) Git repository. This snapshot is always in uncompressed [tar](https://en.wikipedia.org/wiki/Tar_(computing)) format. diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index b292c9dd7de..513dc996c91 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -246,7 +246,7 @@ error message. Possible causes for a failed commit include: user tried to make an empty commit; - the branch was updated by a Git push while the file edit was in progress. -Currently gitlab-shell has a boolean return code, preventing GitLab from specifying the error. +Currently GitLab Shell has a boolean return code, preventing GitLab from specifying the error. ## Delete existing file in repository diff --git a/doc/api/settings.md b/doc/api/settings.md index 4b5b2b924d7..710b63c9a2f 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -232,7 +232,7 @@ are listed in the descriptions of the relevant settings. | `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `geo_status_timeout` | integer | no | **(PREMIUM)** The amount of seconds after which a request to get a secondary node status will time out. | -| `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | +| `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for Git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | | `gravatar_enabled` | boolean | no | Enable Gravatar. | @@ -244,7 +244,7 @@ are listed in the descriptions of the relevant settings. | `hide_third_party_offers` | boolean | no | Do not display offers from third parties within GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | -| `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable git housekeeping. | +| `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable Git housekeeping. | | `housekeeping_full_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | | `housekeeping_gc_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which `git gc` is run. | | `housekeeping_incremental_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | diff --git a/doc/api/tags.md b/doc/api/tags.md index af86ba961f4..1d874fea1f8 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -112,7 +112,7 @@ Parameters: - `tag_name` (required) - The name of a tag - `ref` (required) - Create tag using commit SHA, another tag name, or branch name. - `message` (optional) - Creates annotated tag. -- `release_description` (optional) - Add release notes to the git tag and store it in the GitLab database. +- `release_description` (optional) - Add release notes to the Git tag and store it in the GitLab database. ```json { @@ -166,7 +166,7 @@ Parameters: ## Create a new release -Add release notes to the existing git tag. If there +Add release notes to the existing Git tag. If there already exists a release for the given tag, status code `409` is returned. ``` diff --git a/doc/ci/README.md b/doc/ci/README.md index 94da8354f0b..4be13204227 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -174,7 +174,7 @@ been necessary. These are: #### 12.0 -- [Use refspec to clone/fetch git +- [Use refspec to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/issues/4069). - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4070). diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index cb8d383f7d9..61f260cb70d 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -205,5 +205,5 @@ Some features are not implemented yet. For example, support for environments. - `stage` - `allow_failure` - `only` and `except` -- `when` +- `when` (only with `on_success`, `on_failure`, and `always` values) - `extends` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index ed8d0e3bc35..eaa6efc526d 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -377,6 +377,10 @@ This functionality is only available: - For users with at least Developer access. - If the the stage contains [manual actions](#manual-actions-from-pipeline-graphs). +## Most Recent Pipeline + +There's a link to the latest pipeline for the last commit of a given branch at `/project/pipelines/[branch]/latest`. Also, `/project/pipelines/latest` will redirect you to the latest pipeline for the last commit on the project's default branch. + ## Security on protected branches A strict security model is enforced when pipelines are executed on diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 89a61b2a9e3..663fa1395a6 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -100,6 +100,7 @@ The following table lists available parameters for jobs: | [`stage`](#stage) | Defines a job stage (default: `test`). | | [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | | [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | +| [`rules`](#rules) | List of coniditions to evaluate and determine selected attributes of a build and whether or not it is created. May not be used alongside `only`/`except`. | [`tags`](#tags) | List of tags which are used to select Runner. | | [`allow_failure`](#allow_failure) | Allow job to fail. Failed job doesn't contribute to commit status. | | [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | @@ -690,6 +691,125 @@ In the scenario above, if a merge request is created or updated that changes either files in `service-one` directory or the `Dockerfile`, GitLab creates and triggers the `docker build service one` job. +### `rules` + +Using `rules` allows for a list of individual rule objects to be evaluated +*in order*, until one matches and dynamically provides attributes to the job. + +Available rule clauses include: + +- `if` (similar to [`only:variables`](#onlyvariablesexceptvariables)). +- `changes` (same as [`only:changes`](#onlychangesexceptchanges)). + +For example, using `if`: + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH == "master"' # This rule will be evaluated + when: always + - if: '$VAR =~ /pattern/' # This rule will only be evaluated if the first does not match + when: manual + - when: on_success # A Rule entry with no conditional clauses evaluates to true. If neither of the first two Rules match, this one will and set job:when to "on_success" +``` + +If the first rule does not match, further rules will be evaluated sequentially +until a match is found. The above configuration will specify that `job` should +be built and run for every pipeline on merge requests targeting `master`, +regardless of the status of other builds. + +#### `rules:if` + +`rules:if` differs slightly from `only:variables` by accepting only a single +expression string, rather than an array of them. Any set of expressions to be +evaluated should be conjoined into a single expression using `&&` or `||`. For example: + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH == "master"' # This rule will be evaluated + when: always + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH =~ /^feature/' # This rule will only be evaluated if the target branch is not "master" + when: manual + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH' # If neither of the first two match but the simple presence does, we set to "on_success" by default +``` + +If none of the provided rules match, the job will be set to `when:never`, and +not included in the pipeline. If `rules:when` is not included in the configuration +at all, the behavior defaults to `job:when`, which continues to default to +`on_success`. + +#### `rules:changes` + +`changes` works exactly the same way as [`only`/`except`](#onlychangesexceptchanges), +accepting an array of paths. The following configuration configures a job to be +run manually if `Dockerfile` has changed OR `$VAR == "string value"`. Otherwise +it is set to `when:on_success` by the last rule, where 0 clauses evaluate as +vacuously true. + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + when: manual + - if: '$VAR == "string value"' + when: manual # Will include the job and set to when:manual if the expression evaluates to true, after the `changes:` rule fails to match. + - when: on_success # If neither of the first rules match, set to on_success + +``` + +#### Complex Rule Clauses + +To conjoin `if` and `changes` clauses with an AND, use them in the same rule. +Here we run the job manually if `Dockerfile` or any file in `docker/scripts/` +has changed AND `$VAR == "string value"`. Otherwise, the job will not be +included in the pipeline. + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$VAR == "string value"' + changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + - docker/scripts/* + when: manual + # - when: never would be redundant here, this is implied any time rules are listed. +``` + +The only clauses currently available are `if` and `changes`. Keywords such as +`branches` or `refs` that are currently available for `only`/`except` are not +yet available in `rules` as they are being individually considered for their +usage and behavior in the newer context. + +#### Permitted attributes + +The only job attributes currently set by `rules` are `when` and `start_in`, if +`when` is set to `delayed`. A job will be included in a pipeline if `when` is +evaluated to any value except `never`. + +Delayed jobs require a `start_in` value, so rule objects do as well. For example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: # Will include the job and delay 3 hours when the Dockerfile has changed + - Dockerfile + when: delayed + start_in: '3 hours' + - when: on_success # Otherwise include the job and set to run normally + +``` + +Additional Job configuration may be added to rules in the future, if something +useful isn't available, please open an issue on +[Gitlab CE](https://www.gitlab.com/gitlab-org/gitlab-ce/issues). + ### `tags` `tags` is used to select specific Runners from the list of all Runners that are diff --git a/doc/customization/libravatar.md b/doc/customization/libravatar.md index 1c3bf877fa1..6d96528f760 100644 --- a/doc/customization/libravatar.md +++ b/doc/customization/libravatar.md @@ -14,7 +14,7 @@ server. ## Configuration -In the [gitlab.yml gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set +In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set the configuration options as follows: ### For HTTP @@ -46,7 +46,7 @@ For example, you host a service on `http://libravatar.example.com` and the `http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` -### Omnibus-gitlab example +### Omnibus GitLab example In `/etc/gitlab/gitlab.rb`: diff --git a/doc/customization/system_header_and_footer_messages.md b/doc/customization/system_header_and_footer_messages.md index 15830be4e8a..bd2de3e201c 100644 --- a/doc/customization/system_header_and_footer_messages.md +++ b/doc/customization/system_header_and_footer_messages.md @@ -8,7 +8,7 @@ Navigate to the **Admin** area and go to the **Appearance** page. Under **System header and footer** insert your header message and/or footer message. Both background and font color of the header and footer are customizable. -You can also apply the header and footer messages to gitlab emails, +You can also apply the header and footer messages to GitLab emails, by checking the **Enable header and footer in emails** checkbox. Note that color settings will only be applied within the app interface and not to emails diff --git a/doc/development/README.md b/doc/development/README.md index 09aa64a7bf1..3912a828dec 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -65,6 +65,7 @@ description: 'Learn how to contribute to GitLab.' - [Repository mirroring](repository_mirroring.md) - [Git LFS](lfs.md) - [Developing against interacting components or features](interacting_components.md) +- [File uploads](uploads.md) ## Performance guides @@ -116,6 +117,7 @@ description: 'Learn how to contribute to GitLab.' ## Case studies - [Database case study: Filtering by label](filtering_by_label.md) +- [Database case study: Namespaces storage statistics](namespaces_storage_statistics.md) ## Integration guides diff --git a/doc/development/changelog.md b/doc/development/changelog.md index bd07a01e782..814624c7586 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -35,6 +35,7 @@ the `author` field. GitLab team members **should not**. - Any user-facing change **should** have a changelog entry. Example: "GitLab now uses system fonts for all text." +- Any docs-only changes **should not** have a changelog entry. - Any change behind a feature flag **should not** have a changelog entry. The entry should be added [in the merge request removing the feature flags](feature_flags/development.md). - A fix for a regression introduced and then fixed in the same release (i.e., fixing a bug introduced during a monthly release candidate) **should not** diff --git a/doc/development/documentation/feature-change-workflow.md b/doc/development/documentation/feature-change-workflow.md index ac93ada5a4b..00c76fe0f1b 100644 --- a/doc/development/documentation/feature-change-workflow.md +++ b/doc/development/documentation/feature-change-workflow.md @@ -69,7 +69,7 @@ To follow a consistent workflow every month, documentation changes involve the Product Managers, the developer who shipped the feature, and the technical writer for the DevOps stage. Each role is described below. -The Documentation items in the GitLab CE/EE [Feature Proposal issue template](https://gitlab.com/gitlab-org/gitlab-ce/raw/template-improvements-for-documentation/.gitlab/issue_templates/Feature%20proposal.md) +The Documentation items in the GitLab CE/EE [Feature Proposal issue template](https://gitlab.com/gitlab-org/gitlab-ce/raw/master/.gitlab/issue_templates/Feature%20proposal.md) and default merge request template will assist you with following this process. ### Product Manager role diff --git a/doc/development/emails.md b/doc/development/emails.md index e6af075a282..5676c3b32f4 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -5,6 +5,10 @@ To view rendered emails "sent" in your development instance, visit [`/rails/letter_opener`](http://localhost:3000/rails/letter_opener). +Please note that [S/MIME signed](../administration/smime_signing_email.md) emails +[cannot be currently previewed](https://github.com/fgrehm/letter_opener_web/issues/96) with +`letter_opener`. + ## Mailer previews Rails provides a way to preview our mailer templates in HTML and plaintext using diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 475d1c1611e..44af2b020a4 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -2,6 +2,8 @@ We use the [CarrierWave] gem to handle file upload, store and retrieval. +File uploads should be accelerated by workhorse, for details please refer to [uploads development documentation](uploads.md). + There are many places where file uploading is used, according to contexts: - System diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index 5ce59891afa..8d32d0314ae 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -8,30 +8,6 @@ storage disk use. To counteract this problem, we are adding Git object deduplication for forks to GitLab. In this document, we will describe how GitLab implements Git object deduplication. -## Enabling Git object deduplication via feature flags - -As of GitLab 12.0, Git object deduplication in GitLab is still behind a -feature flag. In this document, you can read about the effects of -enabling the feature. Also, note that Git object deduplication is -limited to forks of public projects on hashed repository storage. - -You can enable deduplication globally by setting the `object_pools` -feature flag to `true`: - -``` {.ruby} -Feature.enable(:object_pools) -``` - -Or just for forks of a specific project: - -``` {.ruby} -fork_parent = Project.find(MY_PROJECT_ID) -Feature.enable(:object_pools, fork_parent) -``` - -To check if a project uses Git object deduplication, look in a Rails -console if `project.pool_repository` is present. - ## Pool repositories ### Understanding Git alternates diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md new file mode 100644 index 00000000000..2c7e5935435 --- /dev/null +++ b/doc/development/namespaces_storage_statistics.md @@ -0,0 +1,178 @@ +# Database case study: Namespaces storage statistics + +## Introduction + +On [Storage and limits management for groups](https://gitlab.com/groups/gitlab-org/-/epics/886), +we want to facilitate a method for easily viewing the amount of +storage consumed by a group, and allow easy management. + +## Proposal + +1. Create a new ActiveRecord model to hold the namespaces' statistics in an aggregated form (only for root namespaces). +1. Refresh the statistics in this model every time a project belonging to this namespace is changed. + +## Problem + +In GitLab, we update the project storage statistics through a +[callback](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/project.rb#L90) +every time the project is saved. + +The summary of those statistics per namespace is then retrieved +by [`Namespaces#with_statistics`](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/namespace.rb#L70) scope. Analyzing this query we noticed that: + +- It takes up to `1.2` seconds for namespaces with over `15k` projects. +- It can't be analyzed with [ChatOps](chatops_on_gitlabcom.md), as it times out. + +Additionally, the pattern that is currently used to update the project statistics +(the callback) doesn't scale adequately. It is currently one of the largest +[database queries transactions on production](https://gitlab.com/gitlab-org/gitlab-ce/issues/62488) +that takes the most time overall. We can't add one more query to it as +it will increase the transaction's length. + +Because of all of the above, we can't apply the same pattern to store +and update the namespaces statistics, as the `namespaces` table is one +of the largest tables on GitLab.com. Therefore we needed to find a performant and +alternative method. + +## Attempts + +### Attempt A: PostgreSQL materialized view + +Model can be updated through a refresh strategy based on a project routes SQL and a [materialized view](https://www.postgresql.org/docs/9.6/rules-materializedviews.html): + +```sql +SELECT split_part("rs".path, '/', 1) as root_path, + COALESCE(SUM(ps.storage_size), 0) AS storage_size, + COALESCE(SUM(ps.repository_size), 0) AS repository_size, + COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, + COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, + COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size +FROM "projects" + INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' + INNER JOIN project_statistics ps ON ps.project_id = projects.id +GROUP BY root_path +``` + +We could then execute the query with: + +```sql +REFRESH MATERIALIZED VIEW root_namespace_storage_statistics; +``` + +While this implied a single query update (and probably a fast one), it has some downsides: + +- Materialized views syntax varies from PostgreSQL and MySQL. While this feature was worked on, MySQL was still supported by GitLab. +- Rails does not have native support for materialized views. We'd need to use a specialized gem to take care of the management of the database views, which implies additional work. + +### Attempt B: An update through a CTE + +Similar to Attempt A: Model update done through a refresh strategy with a [Common Table Expression](https://www.postgresql.org/docs/9.1/queries-with.html) + +```sql +WITH refresh AS ( + SELECT split_part("rs".path, '/', 1) as root_path, + COALESCE(SUM(ps.storage_size), 0) AS storage_size, + COALESCE(SUM(ps.repository_size), 0) AS repository_size, + COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, + COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, + COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size + FROM "projects" + INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' + INNER JOIN project_statistics ps ON ps.project_id = projects.id + GROUP BY root_path) +UPDATE namespace_storage_statistics +SET storage_size = refresh.storage_size, + repository_size = refresh.repository_size, + wiki_size = refresh.wiki_size, + lfs_objects_size = refresh.lfs_objects_size, + build_artifacts_size = refresh.build_artifacts_size, + packages_size = refresh.packages_size +FROM refresh + INNER JOIN routes rs ON rs.path = refresh.root_path AND rs.source_type = 'Namespace' +WHERE namespace_storage_statistics.namespace_id = rs.source_id +``` + +Same benefits and downsides as attempt A. + +### Attempt C: Get rid of the model and store the statistics on Redis + +We could get rid of the model that stores the statistics in aggregated form and instead use a Redis Set. +This would be the [boring solution](https://about.gitlab.com/handbook/values/#boring-solutions) and the fastest one +to implement, as GitLab already includes Redis as part of its [Architecture](architecture.md#redis). + +The downside of this approach is that Redis does not provide the same persistence/consistency guarantees as PostgreSQL, +and this is information we can't afford to lose in a Redis failure. + +### Attempt D: Tag the root namespace and its child namespaces + +Directly relate the root namespace to its child namespaces, so +whenever a namespace is created without a parent, this one is tagged +with the root namespace ID: + +| id | root_id | parent_id +|:---|:--------|:---------- +| 1 | 1 | NULL +| 2 | 1 | 1 +| 3 | 1 | 2 + +To aggregate the statistics inside a namespace we'd execute something like: + +```sql +SELECT COUNT(...) +FROM projects +WHERE namespace_id IN ( + SELECT id + FROM namespaces + WHERE root_id = X +) +``` + +Even though this approach would make aggregating much easier, it has some major downsides: + +- We'd have to migrate **all namespaces** by adding and filling a new column. Because of the size of the table, dealing with time/cost will not be great. The background migration will take approximately `153h`, see <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29772>. +- Background migration has to be shipped one release before, delaying the functionality by another milestone. + +### Attempt E (final): Update the namespace storage statistics in async way + +This approach consists of keep using the incremental statistics updates we currently already have, +but we refresh them through Sidekiq jobs and in different transactions: + +1. Create a second table (`namespace_aggregation_schedules`) with two columns `id` and `namespace_id`. +1. Whenever the statistics of a project changes, insert a row into `namespace_aggregation_schedules` + - We don't insert a new row if there's already one related to the root namespace. + - Keeping in mind the length of the transaction that involves updating `project_statistics`(<https://gitlab.com/gitlab-org/gitlab-ce/issues/62488>), the insertion should be done in a different transaction and through a Sidekiq Job. +1. After inserting the row, we schedule another worker to be executed async at two different moments: + - One enqueued for immediate execution and another one scheduled in `1.5h` hours. + - We only schedule the jobs, if we can obtain a `1.5h` lease on Redis on a key based on the root namespace ID. + - If we can't obtain the lease, it indicates there's another aggregation already in progress, or scheduled in no more than `1.5h`. +1. This worker will: + - Update the root namespace storage statistics by querying all the namespaces through a service. + - Delete the related `namespace_aggregation_schedules` after the update. +1. Another Sidekiq job is also included to traverse any remaining rows on the `namespace_aggregation_schedules` table and schedule jobs for every pending row. + - This job is scheduled with cron to run every night (UTC). + +This implementation has the following benefits: + +- All the updates are done async, so we're not increasing the length of the transactions for `project_statistics`. +- We're doing the update in a single SQL query. +- It is compatible with PostgreSQL and MySQL. +- No background migration required. + +The only downside of this approach is that namespaces' statistics are updated up to `1.5` hours after the change is done, +which means there's a time window in which the statistics are inaccurate. Because we're still not +[enforcing storage limits](https://gitlab.com/gitlab-org/gitlab-ce/issues/30421), this is not a major problem. + +## Conclusion + +Updating the storage statistics asynchronously, was the less problematic and +performant approach of aggregating the root namespaces. + +All the details regarding this use case can be found on: + +- <https://gitlab.com/gitlab-org/gitlab-ce/issues/62214> +- Merge Request with the implementation: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28996> + +Performance of the namespace storage statistics were measured in staging and production (GitLab.com). All results were posted +on <https://gitlab.com/gitlab-org/gitlab-ce/issues/64092>: No problem has been reported so far. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index d6b944a3e74..3ae3ce183d9 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -45,11 +45,11 @@ Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests -#### Using the `package-and-qa` job +#### Using the `package-and-qa-manual` job It is possible to run end-to-end tests for a merge request, eventually being run in a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project, -by triggering the `package-and-qa` manual action in the `test` stage (not +by triggering the `package-and-qa-manual` manual action in the `test` stage (not available for forks). **This runs end-to-end tests against a custom Omnibus package built from your @@ -71,7 +71,7 @@ graph LR B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 subgraph "gitlab-ce/ee pipeline" - A1[`test` stage<br>`package-and-qa` job] + A1[`test` stage<br>`package-and-qa-manual` job] end subgraph "omnibus-gitlab pipeline" @@ -79,7 +79,7 @@ subgraph "omnibus-gitlab pipeline" end subgraph "gitlab-qa pipeline" - A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br>and post the result on the original commit tested| A1 + A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa-manual` job<br>and post the result on the original commit tested| A1 end ``` diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 47e58a425fd..850ea6b60ac 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -40,7 +40,7 @@ the time it would take to build packages and test everything. That is why when someone changes `t.text_field :login` to `t.text_field :username` in the _new session_ view we won't know about this change until our GitLab QA nightly pipeline fails, or until someone triggers -`package-and-qa` action in their merge request. +`package-and-qa-manual` action in their merge request. Obviously such a change would break all tests. We call this problem a _fragile tests problem_. diff --git a/doc/development/uploads.md b/doc/development/uploads.md new file mode 100644 index 00000000000..681ce9d9fe8 --- /dev/null +++ b/doc/development/uploads.md @@ -0,0 +1,270 @@ +# Uploads development documentation + +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. +To prevent occupying a ruby process on I/O operations, we process the upload in workhorse, where is cheaper. +This process can also directly upload to object storage. + +## The problem description + +The following graph explains machine boundaries in a scalable GitLab installation. Without any workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->workhorse + + subgraph "web or API fleet" + workhorse-- 2 -->rails + end + rails-- "3 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +We have three challenges here: performance, availability, and scalability. + +### Performance + +Rails process are expensive in terms of both CPU and memory. Ruby [global interpreter lock](https://en.wikipedia.org/wiki/Global_interpreter_lock) adds to cost too because the ruby process will spend time on I/O operations on step 3 causing incoming requests to pile up. + +In order to improve this, [workhorse disk acceleration](#workhorse-disk-acceleration) was implemented. With this, Rails no longer deals with writing uploaded files to disk. + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->workhorse + + subgraph "web or API fleet" + workhorse-- "3 (without files)" -->rails + end + workhorse -- "2 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +### Availability + +There's also an availability problem in this setup, NFS is a [single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure). + +To address this problem an HA object storage can be used and it's supported by [workhorse object storage acceleration](#workhorse-object-storage-acceleration) + +### Scalability + +Scaling NFS is outside of our support scope, and NFS is not a part of cloud native installations. + +All features that require sidekiq and do not use object storage acceleration won't work without NFS. In Kubernetes, machine boundaries translate to PODs, and in this case the uploaded file will be written into the POD private disk. Since sidekiq POD cannot reach into other pods, the operation will fail to read it. + +## How to select the proper level of acceleration? + +Selecting the proper acceleration is a tradeoff between speed of development and operational costs. + +We can identify three major use-cases for an upload: + +1. **storage:** if we are uploading for storing a file (i.e. artifacts, packages, discussion attachments). In this case [object storage acceleration](#workhorse-object-storage-acceleration) is the proper level as it's the less resource-intensive operation. Additional information can be found on [File Storage in GitLab](file_storage.md). +1. **in-controller/synchronous processing:** if we allow processing **small files** synchronously, using [disk acceleration](#workhorse-disk-acceleration) may speed up development. +1. **sidekiq/asynchronous processing:** Async processing must implement [object storage acceleration](#workhorse-object-storage-acceleration), the reason being that it's the only way to support Cloud Native deployments without a shared NFS. + +For more details about currently broken feature see [epic &1802](https://gitlab.com/groups/gitlab-org/-/epics/1802). + +### Handling repository uploads + +Some features involves git repository uploads without using a regular git client. +Some examples are uploading a repository file from the web interface and [design management](../user/project/issues/design_management.md). + +Those uploads requires the rails controller to act as a git client in lieu of the user. +Those operation falls into _in-controller/synchronous processing_ category, but we have no warranties on the file size. + +In case of a LFS upload, the file pointer is committed synchronously, but file upload to object storage is performed asynchronously with sidekiq. + +## Upload encodings + +By upload encoding we mean how the file is included within the incoming request. + +We have three kinds of file encoding in our uploads: + +1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. +1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. +1. <i class="fa fa-times-circle"></i> **JSON**: some JSON API uploads files as base64 encoded strings. This requires [gitlab-workhorse#226](https://gitlab.com/gitlab-org/gitlab-workhorse/issues/226) to be implemented. + +## Uploading technologies + +By uploading technologies we mean how all the involved services interact with each other. + +GitLab supports 3 kinds of uploading technologies, here follows a brief description with a sequence diagram for each one. Diagrams are not meant to be exhaustive. + +### Regular rails upload + +This is the default kind of upload, and it's most expensive in terms of resources. + +In this case, workhorse is unaware of files being uploaded and acts as a regular proxy. + +When a multipart request reaches the rails application, `Rack::Multipart` leaves behind tempfiles in `/tmp` and uses valuable Ruby process time to copy files around. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + + activate c + c ->>+w: POST /some/url/upload + w->>+r: POST /some/url/upload + + r->>r: save the incoming file on /tmp + r->>r: read the file for processing + + r-->>-c: request result + deactivate c + deactivate w +``` + +### Workhorse disk acceleration + +This kind of upload avoids wasting resources caused by handling upload writes to `/tmp` in rails. + +This optimization is not active by default on REST API requests. + +When enabled, Workhorse looks for files in multipart MIME requests, uploading +any it finds to a temporary file on shared storage. The MIME data in the request +is replaced with the path to the corresponding file before it is forwarded to +Rails. + +To prevent abuse of this feature, Workhorse signs the modified request with a +special header, stating which entries it modified. Rails will ignore any +unsigned path entries. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant s as NFS + + activate c + c ->>+w: POST /some/url/upload + + w->>+s: save the incoming file on a temporary location + s-->>-w: + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+s: read file + s-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +### Workhorse object storage acceleration + +This is the more advanced acceleration technique we have in place. + +Workhorse asks rails for temporary pre-signed object storage URLs and directly uploads to object storage. + +In this setup an extra rails route needs to be implemented in order to handle authorization, +you can see an example of this in [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/controllers/projects/lfs_storage_controller.rb) +and [its routes](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0/config/routes/git_http.rb#L31-32). + +**note:** this will fallback to _Workhorse disk acceleration_ when object storage is not enabled in the gitlab instance. The answer to the `/authorize` call will only contain a file system path. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant os as Object Storage + + activate c + c ->>+w: POST /some/url/upload + + w ->>+r: POST /some/url/upload/authorize + Note over w,r: this request has an empty body + r-->>-w: presigned OS URL + + w->>+os: PUT file + Note over w,os: file is stored on a temporary location. Rails select the destination + os-->>-w: + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + r->>+os: move object to final destination + os-->>-r: + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+os: get object + os-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +## What does the `direct_upload` setting mean? + +[Object storage setting](../administration/uploads.md#object-storage-settings) allows instance administators to enable `direct_upload`, this in an option that only affects the behavior of [workhorse object storage acceleration](#workhorse-object-storage-acceleration). + +This option affect the response to the `/authorize` call. When not enabled, the API response will not contain presigned URLs and workhorse will write the file the shared disk, on the path is provided by rails, acting like object storage was disabled. + +Once the request reachs rails, it will schedule an object storage upload as a sidekiq job. diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index 5de173abbff..a289b90b81b 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -102,7 +102,7 @@ files to your local computer, automatically preserving the Git connection with t remote repository. You can either clone it via HTTPS or [SSH](../ssh/README.md). If you chose to clone -it via HTTPS, you'll have to enter your credentials every time you pull and push. You can read more about credential storage in the [Git Credentials documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). With SSH, you enter your credentials only once. +it via HTTPS, you'll have to enter your credentials every time you pull and push. You can read more about credential storage in the [Git Credentials documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). With SSH, you enter your credentials only once. You can find both paths (HTTPS and SSH) by navigating to your project's landing page and clicking **Clone**. GitLab will prompt you with both paths, from which you can copy @@ -157,7 +157,7 @@ git pull <REMOTE> <name-of-branch> When you clone a repository, `REMOTE` is typically `origin`. This is where the repository was cloned from, and it indicates the SSH or HTTPS URL of the repository on the remote server. `<name-of-branch>` is usually `master`, but it may be any existing -branch. You can create additional named remotes and branches as necessary. +branch. You can create additional named remotes and branches as necessary. You can learn more on how Git manages remote repositories in the [Git Remote documentation](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes). @@ -169,7 +169,7 @@ To view your remote repositories, type: git remote -v ``` -The `-v` flag stands for verbose. +The `-v` flag stands for verbose. ### Add a remote repository diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 018c273c51a..f7ba7c16a9e 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -86,6 +86,7 @@ Please see the table below for some examples: | 9.4.5 | 8.13.4 | `8.13.4` -> `8.17.7` -> `9.4.5` | `8.17.7` is the last version in version `8` | | 10.1.4 | 8.13.4 | `8.13.4 -> 8.17.7 -> 9.5.10 -> 10.1.4` | `8.17.7` is the last version in version `8`, `9.5.10` is the last version in version `9` | | 11.3.4 | 8.13.4 | `8.13.4` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.3.4` | `8.17.7` is the last version in version `8`, `9.5.10` is the last version in version `9`, `10.8.7` is the last version in version `10` | +| 12.0.2 | 11.3.4 | `11.3.4` -> `11.11.x` -> `12.0.2` | `11.11.x` is the last version in version `11` More information about the release procedures can be found in our [release documentation](https://gitlab.com/gitlab-org/release/docs). You may also want to read our diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index b4b7d711d5a..19486052dd9 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -923,6 +923,29 @@ backup beforehand. UPDATE ci_runners SET token = null, token_encrypted = null; ``` +#### Reset pending pipeline jobs + +1. Enter the DB console: + + For Omnibus GitLab packages: + + ```sh + sudo gitlab-rails dbconsole + ``` + + For installations from source: + + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` + +1. Clear all the tokens for pending jobs: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token = null, token_encrypted = null; + ``` + A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted, GitLab can be brought back into working order, and the lost data can be manually replaced. diff --git a/doc/security/README.md b/doc/security/README.md index 5d498ac7602..e3fb07c69c2 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -5,6 +5,7 @@ type: index # Security +- [Password storage](password_storage.md) - [Password length limits](password_length_limits.md) - [Restrict SSH key technologies and minimum length](ssh_keys_restrictions.md) - [Rate limits](rate_limits.md) diff --git a/doc/security/password_storage.md b/doc/security/password_storage.md new file mode 100644 index 00000000000..f4e32f96f7b --- /dev/null +++ b/doc/security/password_storage.md @@ -0,0 +1,13 @@ +--- +type: reference +--- + +# Password Storage + +GitLab stores user passwords in a hashed format, to prevent passwords from being visible. + +GitLab uses the [Devise](https://github.com/plataformatec/devise) authentication library, which handles the hashing of user passwords. Password hashes are created with the following attributes: + +- **Hashing**: the [bcrypt](https://en.wikipedia.org/wiki/Bcrypt) hashing function is used to generate the hash of the provided password. This is a strong, industry-standard cryptographic hashing function. +- **Stretching**: Password hashes are [stretched](https://en.wikipedia.org/wiki/Key_stretching) to harden against brute-force attacks. GitLab uses a stretching factor of 10 by default. +- **Salting**: A [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)) is added to each password to harden against pre-computed hash and dictionary attacks. Each salt is randomly generated for each password, so that no two passwords share a salt, to further increase security. diff --git a/doc/user/admin_area/abuse_reports.md b/doc/user/admin_area/abuse_reports.md index 0c5d2f81e25..cd8dc7bcbf6 100644 --- a/doc/user/admin_area/abuse_reports.md +++ b/doc/user/admin_area/abuse_reports.md @@ -2,7 +2,7 @@ type: reference, howto --- -# Abuse reports +# Abuse reports **(CORE ONLY)** View and resolve abuse reports from GitLab users. diff --git a/doc/user/admin_area/broadcast_messages.md b/doc/user/admin_area/broadcast_messages.md index 01b6558bdbe..b0491499f88 100644 --- a/doc/user/admin_area/broadcast_messages.md +++ b/doc/user/admin_area/broadcast_messages.md @@ -2,7 +2,7 @@ type: reference, howto --- -# Broadcast Messages +# Broadcast Messages **(CORE ONLY)** GitLab can display messages to all users of a GitLab instance in a banner that appears in the UI. diff --git a/doc/user/admin_area/diff_limits.md b/doc/user/admin_area/diff_limits.md index 4063c40a751..9fe4b50a991 100644 --- a/doc/user/admin_area/diff_limits.md +++ b/doc/user/admin_area/diff_limits.md @@ -2,7 +2,7 @@ type: reference --- -# Diff limits administration +# Diff limits administration **(CORE ONLY)** You can set a maximum size for display of diff files (patches). diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index 35e7b6fb541..52f24c602df 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -2,7 +2,7 @@ type: concepts, howto --- -# Health Check +# Health Check **(CORE ONLY)** > - Liveness and readiness probes were [introduced][ce-10416] in GitLab 9.1. > - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and was @@ -21,28 +21,63 @@ traffic until the system is ready or restart the container as needed. To access monitoring resources, the requesting client IP needs to be included in a whitelist. For details, see [how to add IPs to a whitelist for the monitoring endpoints](../../../administration/monitoring/ip_whitelist.md). -## Using the endpoints +## Using the endpoints locally With default whitelist settings, the probes can be accessed from localhost using the following URLs: -- `http://localhost/-/health` -- `http://localhost/-/readiness` -- `http://localhost/-/liveness` +```text +GET http://localhost/-/health +``` + +```text +GET http://localhost/-/readiness +``` + +```text +GET http://localhost/-/liveness +``` + +## Health -The first endpoint, `health`, only checks whether the application server is running. It does not verify the database or other services are running. A successful response will return a 200 status code with the following message: +Checks whether the application server is running. It does not verify the database or other services are running. + +```text +GET /-/health +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/health' +``` + +Example response: ```text GitLab OK ``` -The readiness and liveness probes will provide a report of system health in JSON format. +## Readiness + +The readiness probe checks whether the Gitlab instance is ready to use. It checks the dependent services (Database, Redis, Gitaly etc.) and gives a status for each. + +```text +GET /-/readiness +``` + +Example request: -`readiness` probe example output: +```sh +curl 'https://gitlab.example.com/-/readiness' +``` + +Example response: ```json { "db_check":{ - "status":"ok" + "status":"failed", + "message": "unexpected Db check result: 0" }, "redis_check":{ "status":"ok" @@ -65,7 +100,23 @@ The readiness and liveness probes will provide a report of system health in JSON } ``` -`liveness` probe example output: +## Liveness + +The liveness probe checks whether the application server is alive. Unlike the [`health`](#health) check, this check hits the database. + +```text +GET /-/liveness +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/liveness' +``` + +Example response: + +On success, the endpoint will return a valid successful HTTP status code, and a response like below. ```json { @@ -90,10 +141,7 @@ The readiness and liveness probes will provide a report of system health in JSON } ``` -## Status - -On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint -will return a valid successful HTTP status code, and a `success` message. +On failure, the endpoint will return a `500` HTTP status code. ## Access token (Deprecated) diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 6faab685b26..5e385b7216d 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -2,7 +2,7 @@ type: reference --- -# Account and limit settings +# Account and limit settings **(CORE ONLY)** ## Max attachment size diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 490163c1816..ddf989d0181 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -2,7 +2,7 @@ type: reference --- -# Email +# Email **(CORE ONLY)** You can customize some of the content in emails sent from your GitLab instance. diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index 1b1bcbcd6e8..aea717e806d 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -2,7 +2,7 @@ type: reference --- -# Sign-up restrictions +# Sign-up restrictions **(CORE ONLY)** You can use sign-up restrictions to require user email confirmation, as well as to blacklist or whitelist email addresses belonging to specific domains. diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index a1bce5a6c69..baf219ac9c7 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -2,7 +2,7 @@ type: reference --- -# Enforce accepting Terms of Service +# Enforce accepting Terms of Service **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18570) > in [GitLab Core](https://about.gitlab.com/pricing/) 10.8 diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index d3c9cf7d8ff..ca26147b287 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -2,7 +2,7 @@ type: reference --- -# Third party offers +# Third party offers **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20379) > in [GitLab Core](https://about.gitlab.com/pricing/) 11.1 diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index 516600c9d99..efac7e699f3 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -2,7 +2,7 @@ type: reference --- -# Usage statistics +# Usage statistics **(CORE ONLY)** GitLab Inc. will periodically collect information about your instance in order to perform various actions. diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index e3a495750f2..b9d93bf3671 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -2,7 +2,7 @@ type: reference --- -# User and IP rate limits +# User and IP rate limits **(CORE ONLY)** Rate limiting is a common technique used to improve the security and durability of a web application. For more details, see diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index bf59f49b993..1e2f5705728 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -2,7 +2,7 @@ type: reference --- -# Visibility and access controls +# Visibility and access controls **(CORE ONLY)** GitLab allows administrators to: diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 86491c7d74e..7b631a5a1cd 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -94,6 +94,36 @@ If you want to whitelist some specific vulnerabilities, you can do so by definin them in a YAML file named `clair-whitelist.yml`. Read more in the [Clair documentation](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file). +## Example + +The following is a sample `.gitlab-ci.yml` that will build your Docker Image, push it to the container registry and run Container Scanning. + +```yaml +variables: + DOCKER_DRIVER: overlay2 + +services: + - docker:stable-dind + +stages: + - build + - test + +include: + - template: Container-Scanning.gitlab-ci.yml + +build: + image: docker:stable + stage: build + variables: + IMAGE: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA + script: + - docker info + - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker build -t $IMAGE . + - docker push $IMAGE +``` + ## Security Dashboard The Security Dashboard is a good place to get an overview of all the security @@ -125,4 +155,4 @@ docker: Error response from daemon: failed to copy xattrs: failed to set xattr " This is a result of a bug in Docker which is now [fixed](https://github.com/containerd/continuity/pull/138 "fs: add WithAllowXAttrErrors CopyOpt"). To prevent the error, ensure the Docker version that the Runner is using is `18.09.03` or higher. For more information, see -[issue #10241](https://gitlab.com/gitlab-org/gitlab-ee/issues/10241 "Investigate why Container Scanning is not working with NFS mounts"). +[issue #10241](https://gitlab.com/gitlab-org/gitlab-ee/issues/10241 "Investigate why Container Scanning is not working with NFS mounts").
\ No newline at end of file diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 3148ec63c79..b40392e12d5 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -141,22 +141,22 @@ dependency_scanning: Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings) using environment variables. -| Environment variable | Function | -|-------------------------------- |----------| -| `DS_ANALYZER_IMAGES` | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). | -| `DS_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | -| `DS_ANALYZER_IMAGE_TAG` | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). | -| `DS_PYTHON_VERSION` | Version of Python. If set to 2, dependencies are installed using Python 2.7 instead of Python 3.6. ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12296) in GitLab 12.1)| -| `DS_PIP_DEPENDENCY_PATH` | Path to load Python pip dependencies from. ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12412) in GitLab 12.2) | -| `DS_DEFAULT_ANALYZERS` | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). | -| `DS_DISABLE_REMOTE_CHECKS` | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). | -| `DS_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to `0` to disable). | -| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | -| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | -| `DS_PULL_ANALYZER_IMAGE_TIMEOUT` | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | -| `DS_RUN_ANALYZER_TIMEOUT` | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | -| `PIP_INDEX_URL` | Base URL of Python Package Index (default `https://pypi.org/simple`). | -| `PIP_EXTRA_INDEX_URL` | Array of [extra URLs](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-extra-index-url) of package indexes to use in addition to `PIP_INDEX_URL`. Comma separated. | +| Environment variable | Description | Example usage | +|-------------------------------- |-------------| | +| `DS_ANALYZER_IMAGES` | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). | | +| `DS_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | | +| `DS_ANALYZER_IMAGE_TAG` | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). | | +| `DS_PYTHON_VERSION` | Version of Python. If set to 2, dependencies are installed using Python 2.7 instead of Python 3.6. ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12296) in GitLab 12.1)| | +| `DS_PIP_DEPENDENCY_PATH` | Path to load Python pip dependencies from. ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12412) in GitLab 12.2) | | +| `DS_DEFAULT_ANALYZERS` | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). | | +| `DS_DISABLE_REMOTE_CHECKS` | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). | | +| `DS_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to `0` to disable). | | +| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | `DS_EXCLUDED_PATHS=doc,spec` | +| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | | +| `DS_PULL_ANALYZER_IMAGE_TIMEOUT` | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | | +| `DS_RUN_ANALYZER_TIMEOUT` | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | | +| `PIP_INDEX_URL` | Base URL of Python Package Index (default `https://pypi.org/simple`). | | +| `PIP_EXTRA_INDEX_URL` | Array of [extra URLs](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-extra-index-url) of package indexes to use in addition to `PIP_INDEX_URL`. Comma separated. | | ## Reports JSON format @@ -276,8 +276,8 @@ it highlighted: Here is the description of the report file structure nodes and their meaning. All fields are mandatory to be present in the report JSON unless stated otherwise. Presence of optional fields depends on the underlying analyzers being used. -| Report JSON node | Function | -|------------------------------------------------------|----------| +| Report JSON node | Description | +|------------------------------------------------------|-------------| | `version` | Report syntax version used to generate this JSON. | | `vulnerabilities` | Array of vulnerability objects. | | `vulnerabilities[].category` | Where this vulnerability belongs (SAST, Dependency Scanning etc.). For Dependency Scanning, it will always be `dependency_scanning`. | diff --git a/doc/user/application_security/license_management/index.md b/doc/user/application_security/license_management/index.md index 912f2f0e209..44b2671930e 100644 --- a/doc/user/application_security/license_management/index.md +++ b/doc/user/application_security/license_management/index.md @@ -169,9 +169,9 @@ If you still need to run tests during `mvn install`, add `-DskipTests=false` to > [Introduced](https://gitlab.com/gitlab-org/security-products/license-management/merge_requests/36) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. -License Compliance uses Python 2.7 and pip 10.0 by default. -If your project requires Python 3, you can switch to Python 3.5 and pip 19.1 -by setting the `LM_PYTHON_VERSION` environment variable to `3`. +License Compliance uses Python 3.5 and pip 19.1 by default. +If your project requires Python 2, you can switch to Python 2.7 and pip 10.0 +by setting the `LM_PYTHON_VERSION` environment variable to `2`. ```yaml include: @@ -179,7 +179,7 @@ include: license_management: variables: - LM_PYTHON_VERSION: 3 + LM_PYTHON_VERSION: 2 ``` ## Project policies for License Compliance diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index f730a25a9fc..a1bd00f34e3 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -128,7 +128,7 @@ custom analyzer can scan the source code. | Start column | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | | End column | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | | External id (e.g. CVE) | 𐄂 | 𐄂 | ⚠ | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | -| URLs | ✓ | 𐄂 | ✓ | 𐄂 | ⚠ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| URLs | ✓ | 𐄂 | ✓ | 𐄂 | ⚠ | 𐄂 | ⚠ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | | Internal doc/explanation | ✓ | ⚠ | ✓ | 𐄂 | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | | Solution | ✓ | 𐄂 | 𐄂 | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | | Confidence | 𐄂 | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 2f15d997b5b..3eead6ccd3f 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -160,17 +160,21 @@ The following are Docker image-related variables. Some analyzers make it possible to filter out vulnerabilities under a given threshold. -| `SAST_BANDIT_EXCLUDED_PATHS` | - | comma-separated list of paths to exclude from scan. Uses Python's [`fnmatch` syntax](https://docs.python.org/2/library/fnmatch.html) | -| `SAST_BRAKEMAN_LEVEL` | 1 | Ignore Brakeman vulnerabilities under given confidence level. Integer, 1=Low 3=High. | -| `SAST_FLAWFINDER_LEVEL` | 1 | Ignore Flawfinder vulnerabilities under given risk level. Integer, 0=No risk, 5=High risk. | -| `SAST_GITLEAKS_ENTROPY_LEVEL` | 8.0 | Minimum entropy for secret detection. Float, 0.0 = low, 8.0 = high. | -| `SAST_GOSEC_LEVEL` | 0 | Ignore gosec vulnerabilities under given confidence level. Integer, 0=Undefined, 1=Low, 2=Medium, 3=High. | -| `SAST_EXCLUDED_PATHS` | - | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | +| Environment variable | Default value | Description | Example usage | +|----------------------|---------------|-------------|---| +| `SAST_BANDIT_EXCLUDED_PATHS` | - | comma-separated list of paths to exclude from scan. Uses Python's [`fnmatch` syntax](https://docs.python.org/2/library/fnmatch.html) | | +| `SAST_BRAKEMAN_LEVEL` | 1 | Ignore Brakeman vulnerabilities under given confidence level. Integer, 1=Low 3=High. | | +| `SAST_FLAWFINDER_LEVEL` | 1 | Ignore Flawfinder vulnerabilities under given risk level. Integer, 0=No risk, 5=High risk. | | +| `SAST_GITLEAKS_ENTROPY_LEVEL` | 8.0 | Minimum entropy for secret detection. Float, 0.0 = low, 8.0 = high. | | +| `SAST_GOSEC_LEVEL` | 0 | Ignore gosec vulnerabilities under given confidence level. Integer, 0=Undefined, 1=Low, 2=Medium, 3=High. | | +| `SAST_EXCLUDED_PATHS` | - | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | `SAST_EXCLUDED_PATHS=doc,spec` | ### Timeouts The following variables configure timeouts. +| Environment variable | Default value | Description | +|----------------------|---------------|-------------| | `SAST_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | 2m | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". For example, "300ms", "1.5h" or "2h45m". | | `SAST_PULL_ANALYZER_IMAGE_TIMEOUT` | 5m | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". For example, "300ms", "1.5h" or "2h45m". | | `SAST_RUN_ANALYZER_TIMEOUT` | 20m | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". For example, "300ms", "1.5h" or "2h45m".| diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index d21a325d401..af37cc896ad 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -43,13 +43,15 @@ Host gitlab.com Below are the settings for [GitLab Pages]. -| Setting | GitLab.com | Default | -| ----------------------- | ---------------- | ------------- | -| Domain name | `gitlab.io` | - | -| IP address | `35.185.44.232` | - | -| Custom domains support | yes | no | -| TLS certificates support| yes | no | +| Setting | GitLab.com | Default | +| --------------------------- | ---------------- | ------------- | +| Domain name | `gitlab.io` | - | +| IP address | `35.185.44.232` | - | +| Custom domains support | yes | no | +| TLS certificates support | yes | no | +| Maximum size (uncompressed) | 1G | 100M | +NOTE: **Note:** The maximum size of your Pages site is regulated by the artifacts maximum size which is part of [GitLab CI/CD](#gitlab-cicd). @@ -59,7 +61,7 @@ Below are the current settings regarding [GitLab CI/CD](../../ci/README.md). | Setting | GitLab.com | Default | | ----------- | ----------------- | ------------- | -| Artifacts maximum size | 1G | 100M | +| Artifacts maximum size (uncompressed) | 1G | 100M | | Artifacts [expiry time](../../ci/yaml/README.md#artifactsexpire_in) | kept forever | deleted after 30 days unless otherwise specified | ## Repository size limit diff --git a/doc/user/group/index.md b/doc/user/group/index.md index d1d4f3740b0..864f1a397d5 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -342,7 +342,7 @@ underlying projects, issues, etc, by IP address. This can help ensure that particular content doesn't leave the premises, while not blocking off access to the entire instance. -Add whitelisted IP subnet using CIDR notation to the group settings and anyone +Add one or more whitelisted IP subnets using CIDR notation in comma separated format to the group settings and anyone coming from a different IP address won't be able to access the restricted content. diff --git a/doc/user/index.md b/doc/user/index.md index c93f64cd528..27e75189fc3 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -53,7 +53,7 @@ With GitLab Enterprise Edition, you can also: - Improve collaboration with [Merge Request Approvals](project/merge_requests/index.md#merge-request-approvals-starter), [Multiple Assignees for Issues](project/issues/multiple_assignees_for_issues.md), - and [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards-starter). + and [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards). - Create formal relationships between issues with [Related Issues](project/issues/related_issues.md). - Use [Burndown Charts](project/milestones/burndown_charts.md) to track progress during a sprint or while working on a new version of their software. - Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Global Search](search/advanced_global_search.md) and [Advanced Syntax Search](search/advanced_search_syntax.md) for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 932e7fd10b2..64c4066683b 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -17,7 +17,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow - - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **(STARTER)** + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards): Allow your teams to create their own workflows (Issue Boards) for the same project - [Repositories](repository/index.md): Host your code in a fully integrated platform - [Branches](repository/branches/index.md): use Git branching strategies to @@ -34,7 +34,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow - - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **(STARTER)** + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards): Allow your teams to create their own workflows (Issue Boards) for the same project - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - [Merge Request Approvals](merge_requests/merge_request_approvals.md): Ask for approval before diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 9fc0ade809e..a80332adc46 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -194,7 +194,7 @@ The following tables outline the details of expected properties. | Property | Type | Required | Description | | ------ | ------ | ------ | ------- | -| `type` | enum | no, defaults to `area-chart` | Specifies the chart type to use. | +| `type` | enum | no, defaults to `area-chart` | Specifies the chart type to use, can be `area-chart` or `line-chart` | | `title` | string | yes | Heading for the panel. | | `y_label` | string | no, but highly encouraged | Y-Axis label for the panel. | | `weight` | number | no, defaults to order in file | Order to appear within the grouping. Lower number means higher priority, which will be higher on the page. Numbers do not need to be consecutive. | diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index eaca5f8cfb8..519c02cf0ad 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -13,7 +13,7 @@ keeping everything in the same place, so that you don't need to jump between different platforms to organize your workflow. With GitLab Issue Boards, you organize your issues in lists that correspond to -their assigned labels, visualizing issues designed as cards throughout that lists. +their assigned labels, visualizing issues designed as cards throughout those lists. You define your process and GitLab organizes it for you. You add your labels then create the corresponding list to pull in your existing issues. When @@ -27,12 +27,9 @@ Issue Boards** (version introduced in GitLab 8.11 - August 2016). ### Advanced features of Issue Boards -With [GitLab Starter](https://about.gitlab.com/pricing/), you can create -[multiple issue boards](#multiple-issue-boards-starter) for a given project. **(STARTER)** - -With [GitLab Premium](https://about.gitlab.com/pricing/), you can also create multiple -issue boards for your groups, and add lists for [assignees](#assignee-lists-premium) and -[milestones](#milestone-lists-premium). **(PREMIUM)** +- Create multiple issue boards per project. +- Create multiple issue boards per group. **(PREMIUM)** +- Add lists for [assignees](#assignee-lists-premium) and [milestones](#milestone-lists-premium). **(PREMIUM)** Check all the [advanced features of Issue Boards](#gitlab-enterprise-features-for-issue-boards) below. @@ -58,8 +55,7 @@ You create issues, host code, perform reviews, build, test, and deploy from one single platform. Issue Boards help you to visualize and manage the entire process _in_ GitLab. -With [Multiple Issue Boards](#use-cases-for-multiple-issue-boards), available -only in [different tiers of GitLab Enterprise Edition](#gitlab-enterprise-features-for-issue-boards), +With [Multiple Issue Boards](#use-cases-for-multiple-issue-boards), you go even further, as you can not only keep yourself and your project organized from a broader perspective with one Issue Board per project, but also allow your team members to organize their own workflow by creating @@ -88,7 +84,7 @@ If we have the labels "**backend**", "**frontend**", "**staging**", and "**production**", and an Issue Board with a list for each, we can: - Visualize the entire flow of implementations since the - beginning of the development lifecycle until deployed to production + beginning of the development life cycle until deployed to production - Prioritize the issues in a list by moving them vertically - Move issues between lists to organize them according to the labels you've set - Add multiple issues to lists in the board by selecting one or more existing issues @@ -97,8 +93,7 @@ If we have the labels "**backend**", "**frontend**", "**staging**", and ### Use cases for Multiple Issue Boards -With [Multiple Issue Boards](#multiple-issue-boards-starter), available only in -[GitLab Enterprise Edition](https://about.gitlab.com/pricing/), +With [Multiple Issue Boards](#multiple-issue-boards), each team can have their own board to organize their workflow individually. #### Scrum team @@ -159,13 +154,14 @@ Issue Board, that is, create or delete lists and drag issues from one list to an GitLab Issue Boards are available on GitLab Core and GitLab.com Free, but some advanced functionalities are only present in higher tiers: GitLab.com Bronze, Silver, or Gold, or GitLab self-managed Starter, Premium, and Ultimate, as described -on the following sections. +in the following sections. For a collection of [features per tier](#summary-of-features-per-tier), check the summary below. -### Multiple Issue Boards **(STARTER)** +### Multiple Issue Boards -> Introduced in [GitLab Enterprise Edition 8.13](https://about.gitlab.com/2016/10/22/gitlab-8-13-released/#multiple-issue-boards-ee). +> - Multiple Issue Boards per project [moved](https://gitlab.com/gitlab-org/gitlab-ce/issues/53811) to [GitLab Core](https://about.gitlab.com/pricing/) in GitLab 12.1. +> - Multiple Issue Boards per group is available in [GitLab Premium Edition](https://about.gitlab.com/pricing/). Multiple Issue Boards, as the name suggests, allow for more than one Issue Board for a given project or group. This is great for large projects with more than one team @@ -184,10 +180,6 @@ These are shortcuts to your last 4 visited boards. When you're revisiting an issue board in a project or group with multiple boards, GitLab will automatically load the last board you visited. -NOTE: **Note:** -The Multiple Issue Boards feature is available for -**projects in GitLab Starter Edition** and for **groups in GitLab Premium Edition**. - ### Configurable Issue Boards **(STARTER)** > Introduced in [GitLab Starter Edition 10.2](https://about.gitlab.com/2017/11/22/gitlab-10-2-released/#issue-boards-configuration). diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png Binary files differdeleted file mode 100644 index 6a09412533f..00000000000 --- a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png Binary files differnew file mode 100644 index 00000000000..32b9a3b9ce4 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index 5161b25de99..a0432414bcf 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -83,7 +83,7 @@ request approval rules: 1. Give the approval rule a name that describes the set of approvers selected. 1. Click **Add approvers** to submit the new rule. -  +  ## Multiple approval rules **(PREMIUM)** diff --git a/doc/workflow/forking/branch_select.png b/doc/workflow/forking/branch_select.png Binary files differindex 77236137190..0ea5410f832 100644 --- a/doc/workflow/forking/branch_select.png +++ b/doc/workflow/forking/branch_select.png diff --git a/doc/workflow/forking/merge_request.png b/doc/workflow/forking/merge_request.png Binary files differindex 407ddfb4799..43851203f3f 100644 --- a/doc/workflow/forking/merge_request.png +++ b/doc/workflow/forking/merge_request.png diff --git a/doc/workflow/forking_workflow.md b/doc/workflow/forking_workflow.md index 869a0a621c3..82c2709143d 100644 --- a/doc/workflow/forking_workflow.md +++ b/doc/workflow/forking_workflow.md @@ -10,8 +10,7 @@ document more information about using branches to work together. Forking a project is in most cases a two-step process. -1. Click on the fork button located in the middle of the page or a project's - home page right next to the stars button. +1. Click on the fork button located located in between the star and clone buttons on the project's home page.  @@ -25,7 +24,7 @@ Forking a project is in most cases a two-step process. **Note:** If the namespace you chose to fork the project to has another project with the same path name, you will be presented with a warning that the forking - could not be completed. Try to resolve the error and repeat the forking + could not be completed. Try to resolve the error before repeating the forking process.  @@ -44,7 +43,7 @@ create the [merge request](merge_requests.md).  You can then assign the merge request to someone to have them review -your changes. Upon pressing the 'Accept Merge Request' button, your +your changes. Upon pressing the 'Submit Merge Request' button, your changes will be added to the repository and branch you're merging into.  diff --git a/doc/workflow/img/forking_workflow_choose_namespace.png b/doc/workflow/img/forking_workflow_choose_namespace.png Binary files differindex b34b12090a1..eb023ca85f2 100644 --- a/doc/workflow/img/forking_workflow_choose_namespace.png +++ b/doc/workflow/img/forking_workflow_choose_namespace.png diff --git a/doc/workflow/img/forking_workflow_fork_button.png b/doc/workflow/img/forking_workflow_fork_button.png Binary files differindex 941d5363c35..7fb07529b6d 100644 --- a/doc/workflow/img/forking_workflow_fork_button.png +++ b/doc/workflow/img/forking_workflow_fork_button.png diff --git a/doc/workflow/img/forking_workflow_path_taken_error.png b/doc/workflow/img/forking_workflow_path_taken_error.png Binary files differindex df938da5677..ef62d0ab6a9 100644 --- a/doc/workflow/img/forking_workflow_path_taken_error.png +++ b/doc/workflow/img/forking_workflow_path_taken_error.png |
