diff options
Diffstat (limited to 'doc')
193 files changed, 2883 insertions, 1540 deletions
diff --git a/doc/README.md b/doc/README.md index 8ce5d2e240a..f12c06199c2 100644 --- a/doc/README.md +++ b/doc/README.md @@ -358,7 +358,7 @@ The following documentation relates to the DevOps **Secure** stage: | [Dependency Scanning](user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | | [Dynamic Application Security Testing (DAST)](user/application_security/dast/index.md) **(ULTIMATE)** | Analyze running web applications for known vulnerabilities. | | [Group Security Dashboard](user/application_security/security_dashboard/index.md) **(ULTIMATE)** | View vulnerabilities in all the projects in a group and its subgroups. | -| [License Management](user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | +| [License Compliance](user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | | [Project Security Dashboard](user/application_security/security_dashboard/index.md) **(ULTIMATE)** | View the latest security reports for your project. | | [Static Application Security Testing (SAST)](user/application_security/sast/index.md) **(ULTIMATE)** | Analyze source code for known vulnerabilities. | diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 02de2caf558..8075a40cae7 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -75,6 +75,8 @@ From there, you can see the following actions: - User was removed from project - Project export was downloaded - Project repository was downloaded +- Project was archived +- Project was unarchived ### Instance events **(PREMIUM ONLY)** diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index d0adeb89543..f5d58db0133 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -137,6 +137,15 @@ otherwise you will run into conflicts. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Validate using: + + ```sh + openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:443 > cacert.pem + ``` + +NOTE: **Note:** +If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. + **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and @@ -163,9 +172,13 @@ docker login gitlab.example.com:4567 If the Registry is configured to use its own domain, you will need a TLS certificate for that specific domain (e.g., `registry.example.com`) or maybe -a wildcard certificate if hosted under a subdomain of your existing GitLab +a wildcard certificate if hosted under a subdomain of your existing GitLab domain (e.g., `registry.gitlab.example.com`). +NOTE: **Note:** +As well as manually generated SSL certificates (explained here), certificates automatically +generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html#host-services). + Let's assume that you want the container Registry to be accessible at `https://registry.gitlab.example.com`. @@ -458,7 +471,7 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. ``` 1. A certificate keypair is required for GitLab and the Container Registry to - communicate securely. By default omnibus-gitlab will generate one keypair, + communicate securely. By default Omnibus GitLab will generate one keypair, which is saved to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key`. When using a non-bundled Container Registry, you will need to supply a custom certificate key. To do that, add the following to @@ -474,7 +487,7 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. **Note:** The file specified at `registry_key_path` gets populated with the content specified by `internal_key`, each time reconfigure is executed. If - no file is specified, omnibus-gitlab will default it to + no file is specified, Omnibus GitLab will default it to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate it. diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 32462a95a1a..7238d08ab09 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -12,17 +12,17 @@ NOTE: **Note:** Custom Git hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md) Git natively supports hooks that are executed on different actions. -Examples of server-side git hooks include pre-receive, post-receive, and update. +Examples of server-side Git hooks include pre-receive, post-receive, and update. See [Git SCM Server-Side Hooks][hooks] for more information about each hook type. -As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab -administrators can add custom git hooks to any GitLab project. +As of GitLab Shell version 2.2.0 (which requires GitLab 7.5+), GitLab +administrators can add custom Git hooks to any GitLab project. ## Create a custom Git hook for a repository Server-side Git hooks are typically placed in the repository's `hooks` -subdirectory. In GitLab, hook directories are symlinked to the gitlab-shell -`hooks` directory for ease of maintenance between gitlab-shell upgrades. +subdirectory. In GitLab, hook directories are symlinked to the GitLab Shell +`hooks` directory for ease of maintenance between GitLab Shell upgrades. Custom hooks are implemented differently, but the behavior is exactly the same once the hook is created. Follow the steps below to set up a custom hook for a repository: @@ -36,7 +36,7 @@ repository: 1. Inside the new `custom_hooks` directory, create a file with a name matching the hook type. For a pre-receive hook the file name should be `pre-receive` with no extension. -1. Make the hook file executable and make sure it's owned by git. +1. Make the hook file executable and make sure it's owned by Git. 1. Write the code to make the Git hook function as expected. Hooks can be in any language. Ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will probably be @@ -49,17 +49,17 @@ as appropriate. To create a Git hook that applies to all of your repositories in your instance, set a global Git hook. Since all the repositories' `hooks` -directories are symlinked to gitlab-shell's `hooks` directory, adding any hook -to the gitlab-shell `hooks` directory will also apply it to all repositories. Follow +directories are symlinked to GitLab Shell's `hooks` directory, adding any hook +to the GitLab Shell `hooks` directory will also apply it to all repositories. Follow the steps below to properly set up a custom hook for all repositories: 1. On the GitLab server, navigate to the configured custom hook directory. The - default is in the gitlab-shell directory. The gitlab-shell `hook` directory + default is in the GitLab Shell directory. The GitLab Shell `hook` directory for an installation from source the path is usually `/home/git/gitlab-shell/hooks`. For Omnibus installs the path is usually `/opt/gitlab/embedded/service/gitlab-shell/hooks`. To look in a different directory for the global custom hooks, - set `custom_hooks_dir` in the gitlab-shell config. For + set `custom_hooks_dir` in the GitLab Shell config. For Omnibus installations, this can be set in `gitlab.rb`; and in source installations, this can be set in `gitlab-shell/config.yml`. 1. Create a new directory in this location. Depending on your hook, it will be diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md index d2c52b67e67..5153666705f 100644 --- a/doc/administration/dependency_proxy.md +++ b/doc/administration/dependency_proxy.md @@ -48,7 +48,7 @@ local location or even use object storage. The dependency proxy files for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source -installations under `shared/dependency_proxy/` (relative to the git home directory). +installations under `shared/dependency_proxy/` (relative to the Git home directory). To change the local storage path: **Omnibus GitLab installations** diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index d44e141b66b..ba95843b0b0 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,4 +1,4 @@ -# Disaster Recovery **(PREMIUM ONLY)** +# Disaster Recovery (Geo) **(PREMIUM ONLY)** Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 5aedb665935..dbd466b906d 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,4 +1,4 @@ -# Geo Replication **(PREMIUM ONLY)** +# Replication (Geo) **(PREMIUM ONLY)** > - Introduced in GitLab Enterprise Edition 8.9. > - Using Geo in combination with @@ -6,7 +6,7 @@ > is considered **Generally Available** (GA) in > [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. -Geo is the solution for widely distributed development teams. +Replication with Geo is the solution for widely distributed development teams. ## Overview diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 1780e1babe9..9d653d4e09e 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -53,7 +53,7 @@ sudo service ssh restart ## Instructions In order to use the new protocol, clients need to either pass the configuration -`-c protocol.version=2` to the git command, or set it globally: +`-c protocol.version=2` to the Git command, or set it globally: ```sh git config --global protocol.version 2 diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 8e10de3d592..1ee8dcd21fd 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -64,6 +64,7 @@ The following list depicts what the network architecture of Gitaly is: topology. - A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. - A Gitaly server hosts one or more storages. +- A GitLab server can use one or more Gitaly servers. - Gitaly addresses must be specified in such a way that they resolve correctly for ALL Gitaly clients. - Gitaly clients are: Unicorn, Sidekiq, gitlab-workhorse, @@ -77,14 +78,16 @@ The following list depicts what the network architecture of Gitaly is: - Authentication is done through a static token which is shared among the Gitaly and GitLab Rails nodes. -Below we describe how to configure a Gitaly server at address -`gitaly.internal:8075` with secret token `abc123secret`. We assume -your GitLab installation has two repository storages, `default` and -`storage1`. +Below we describe how to configure two Gitaly servers one at +`gitaly1.internal` and the other at `gitaly2.internal` +with secret token `abc123secret`. We assume +your GitLab installation has three repository storages: `default`, +`storage1` and `storage2`. ### 1. Installation -First install Gitaly using either Omnibus GitLab or install it from source: +First install Gitaly on each Gitaly server using either +Omnibus GitLab or install it from source: - For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page but @@ -119,7 +122,7 @@ Configure a token on the instance that runs the GitLab Rails application. ### 3. Gitaly server configuration -Next, on the Gitaly server, you need to configure storage paths, enable +Next, on the Gitaly servers, you need to configure storage paths, enable the network listener and configure the token. NOTE: **Note:** if you want to reduce the risk of downtime when you enable @@ -175,15 +178,29 @@ Check the directory layout on your Gitaly server to be sure. gitaly['listen_addr'] = "0.0.0.0:8075" gitaly['auth_token'] = 'abc123secret' + # To use TLS for Gitaly you need to add + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + + For `gitaly1.internal`: + + ``` gitaly['storage'] = [ { 'name' => 'default' }, { 'name' => 'storage1' }, ] + ``` + + For `gitaly2.internal`: - # To use TLS for Gitaly you need to add - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "path/to/cert.pem" - gitaly['key_path'] = "path/to/key.pem" + ``` + gitaly['storage'] = [ + { 'name' => 'storage2' }, + ] ``` NOTE: **Note:** @@ -206,13 +223,26 @@ Check the directory layout on your Gitaly server to be sure. [auth] token = 'abc123secret' + ``` + +1. Append the following to `/home/git/gitaly/config.toml` for each respective server: + + For `gitaly1.internal`: + ```toml [[storage]] name = 'default' [[storage]] name = 'storage1' ``` + + For `gitaly2.internal`: + + ```toml + [[storage]] + name = 'storage2' + ``` NOTE: **Note:** In some cases, you'll have to set `path` for each `[[storage]]` in the @@ -231,9 +261,13 @@ then all Gitaly requests will fail. Additionally, you need to [disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). -We assume that your Gitaly server can be reached at -`gitaly.internal:8075` from your GitLab server, and that Gitaly can read and -write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. +We assume that your `gitaly1.internal` Gitaly server can be reached at +`gitaly1.internal:8075` from your GitLab server, and that Gitaly server +can read and write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. + +We assume also that your `gitaly2.internal` Gitaly server can be reached at +`gitaly2.internal:8075` from your GitLab server, and that Gitaly server +can read and write to `/mnt/gitlab/storage2`. **For Omnibus GitLab** @@ -241,8 +275,9 @@ write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) gitlab_rails['gitaly_token'] = 'abc123secret' @@ -264,10 +299,13 @@ write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. repositories: storages: default: - gitaly_address: tcp://gitaly.internal:8075 + gitaly_address: tcp://gitaly1.internal:8075 path: /some/dummy/path storage1: - gitaly_address: tcp://gitaly.internal:8075 + gitaly_address: tcp://gitaly1.internal:8075 + path: /some/dummy/path + storage2: + gitaly_address: tcp://gitaly2.internal:8075 path: /some/dummy/path gitaly: @@ -349,8 +387,9 @@ To configure Gitaly with TLS: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, }) gitlab_rails['gitaly_token'] = 'abc123secret' @@ -376,11 +415,14 @@ To configure Gitaly with TLS: repositories: storages: default: + gitaly_address: tls://gitaly1.internal:9999 path: /some/dummy/path - gitaly_address: tls://gitaly.internal:9999 storage1: + gitaly_address: tls://gitaly1.internal:9999 + path: /some/dummy/path + storage2: + gitaly_address: tls://gitaly2.internal:9999 path: /some/dummy/path - gitaly_address: tls://gitaly.internal:9999 gitaly: token: 'abc123secret' @@ -617,3 +659,33 @@ To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-se on the Gitaly node matches the one on all other nodes. If it doesn't match, update the secrets file on the Gitaly node to match the others, then [reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Command line tools cannot connect to Gitaly + +If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. + +Verify that you can reach Gitaly via TCP: + +```bash +sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] +``` + +If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. + +If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. + +If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: + +```bash +echo $http_proxy +echo $https_proxy +``` + +If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. + +To remove the proxy setting, run the following commands (depending on which variables had values): + +```bash +unset http_proxy +unset https_proxy +```
\ No newline at end of file diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index 9e9f604317a..f11d27487d1 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -60,11 +60,21 @@ for details on managing SSL certificates and configuring Nginx. ### Basic ports -| LB Port | Backend Port | Protocol | -| ------- | ------------ | --------------- | -| 80 | 80 | HTTP [^1] | -| 443 | 443 | TCP or HTTPS [^1] [^2] | -| 22 | 22 | TCP | +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. ### GitLab Pages Ports @@ -72,12 +82,19 @@ If you're using GitLab Pages with custom domain support you will need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the -[GitLab Pages documentation][gitlab-pages] for more information. +[GitLab Pages documentation](../pages/index.md) for more information. -| LB Port | Backend Port | Protocol | -| ------- | ------------ | -------- | -| 80 | Varies [^3] | HTTP | -| 443 | Varies [^3] | TCP [^4] | +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. ### Alternate SSH Port @@ -86,7 +103,7 @@ it may be helpful to configure an alternate SSH hostname that allows users to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address compared to the other GitLab HTTP configuration above. -Configure DNS for an alternate SSH hostname such as altssh.gitlab.example.com. +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | LB Port | Backend Port | Protocol | | ------- | ------------ | -------- | @@ -101,24 +118,6 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) -[^1]: [Web terminal](../../ci/environments.md#web-terminals) support requires - your load balancer to correctly handle WebSocket connections. When using - HTTP or HTTPS proxying, this means your load balancer must be configured - to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the - [web terminal](../integration/terminal.md) integration guide for - more details. -[^2]: When using HTTPS protocol for port 443, you will need to add an SSL - certificate to the load balancers. If you wish to terminate SSL at the - GitLab application server instead, use TCP protocol. -[^3]: The backend port for GitLab Pages depends on the - `gitlab_pages['external_http']` and `gitlab_pages['external_https']` - setting. See [GitLab Pages documentation][gitlab-pages] for more details. -[^4]: Port 443 for GitLab Pages should always use the TCP protocol. Users can - configure custom domains with custom SSL, which would not be possible - if SSL was terminated at the load balancer. - -[gitlab-pages]: ../pages/index.md - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 73a39a6dd35..29915cb3a99 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -151,7 +151,7 @@ Reply by email should now be working. #### Postfix -Example configuration for Postfix mail server. Assumes mailbox incoming@gitlab.example.com. +Example configuration for Postfix mail server. Assumes mailbox `incoming@gitlab.example.com`. Example for Omnibus installs: @@ -218,7 +218,7 @@ incoming_email: #### Gmail -Example configuration for Gmail/G Suite. Assumes mailbox gitlab-incoming@gmail.com. +Example configuration for Gmail/G Suite. Assumes mailbox `gitlab-incoming@gmail.com`. Example for Omnibus installs: diff --git a/doc/administration/index.md b/doc/administration/index.md index 2b25e8efd23..650cb10a64a 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -64,6 +64,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. +- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification emails with S/MIME #### Customizing GitLab's appearance @@ -192,6 +193,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. for troubleshooting Kubernetes-related issues. - Useful links from the Support Team: - [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html). - - [Repairing and recovering broken git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). + - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html). - [Strace zine](https://wizardzines.com/zines/strace/). diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index ab6a4646a71..825435deff9 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -14,4 +14,3 @@ Additionally you will be contacted by our team for further review which should h [6995]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6995 [ee]: https://about.gitlab.com/pricing/ - diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index e1bbabb2878..66ab3ef8d81 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -13,7 +13,7 @@ in the project's default branch. In order to change the pattern you need to have access to the server that GitLab is installed on. -The default pattern can be located in [gitlab.yml.example] under the +The default pattern can be located in [`gitlab.yml.example`] under the "Automatic issue closing" section. > **Tip:** diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 2b624f8de77..350cd5b7992 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -94,7 +94,7 @@ If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need ### Object Storage Settings -For source installations the following settings are nested under `artifacts:` and then `object_store:`. On omnibus installs they are prefixed by `artifacts_object_store_`. +For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`. | Setting | Description | Default | |---------|-------------|---------| @@ -115,7 +115,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | -| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 306d611f6bf..9030c941cad 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -88,7 +88,7 @@ Introduced in GitLab 10.0, this file lives in It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} ``` This entry above shows an access to an internal endpoint to check whether an @@ -284,13 +284,16 @@ Introduced in GitLab 11.3. This file lives in `/var/log/gitlab/gitlab-rails/impo Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for installations from source. -## `auth.log` +## `auth.log` Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for installations from source. -It logs information whenever [Rack Attack] registers an abusive request. +This log records: + +- Information whenever [Rack Attack] registers an abusive request. +- Requests over the [Rate Limit] on raw endpoints. NOTE: **Note:** From [%12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/62756), user id and username are available on this log. @@ -334,3 +337,4 @@ installations from source. [repocheck]: repository_checks.md [Rack Attack]: ../security/rack_attack.md +[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 0b065082ded..d52d865cec5 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -90,7 +90,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | -| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md index 8e33cea6217..d445b68721d 100644 --- a/doc/administration/monitoring/gitlab_instance_administration_project/index.md +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -27,7 +27,7 @@ If that's not the case or if you have an external Prometheus instance or an HA s you should [configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). -## Taking action on Prometheus alerts **[ULTIMATE]** +## Taking action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) to the Prometheus config in order for GitLab to receive notifications of any alerts. diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 9f671e0db11..c32edb60f9d 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -5,7 +5,7 @@ 1. Grab the profiling token from **Monitoring > Requests Profiles** admin page (highlighted in a blue in the image below).  -1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where <mode> can be `execution` or `memory`) to the request you want to profile. You can use: +1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where `<mode>` can be `execution` or `memory`) to the request you want to profile. You can use: - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. - `curl`. For example, `curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' https://gitlab.example.com/group/project`. 1. Once request is finished (which will take a little longer than usual), you can diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index ec26c0b2e7e..0605fb76e2f 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -19,38 +19,106 @@ it, the client IP needs to be [included in a whitelist][whitelist]. For Omnibus and Chart installations, these metrics are automatically enabled and collected as of [GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1702). For source installations or earlier versions, these metrics will need to be enabled manually and collected by a Prometheus server. -## Unicorn Metrics available +## Metrics available The following metrics are available: -| Metric | Type | Since | Description | -|:--------------------------------- |:--------- |:----- |:----------- | -| db_ping_timeout | Gauge | 9.4 | Whether or not the last database ping timed out | -| db_ping_success | Gauge | 9.4 | Whether or not the last database ping succeeded | -| db_ping_latency_seconds | Gauge | 9.4 | Round trip time of the database ping | -| filesystem_access_latency_seconds | Gauge | 9.4 | Latency in accessing a specific filesystem | -| filesystem_accessible | Gauge | 9.4 | Whether or not a specific filesystem is accessible | -| filesystem_write_latency_seconds | Gauge | 9.4 | Write latency of a specific filesystem | -| filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | -| filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | -| filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | -| gitlab_cache_misses_total | Counter | 10.2 | Cache read miss | -| gitlab_cache_operation_duration_seconds | Histogram | 10.2 | Cache access time | -| gitlab_cache_operations_total | Counter | 12.2 | Cache operations by controller/action | -| http_requests_total | Counter | 9.4 | Rack request count | -| http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | -| pipelines_created_total | Counter | 9.4 | Counter of pipelines created | -| rack_uncaught_errors_total | Counter | 9.4 | Rack connections handling uncaught errors count | -| redis_ping_timeout | Gauge | 9.4 | Whether or not the last redis ping timed out | -| redis_ping_success | Gauge | 9.4 | Whether or not the last redis ping succeeded | -| redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | -| user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | -| upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | -| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | -| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | -| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | -| unicorn_workers | Gauge | 12.0 | The number of Unicorn workers | +| Metric | Type | Since | Description | Labels | +|:-------------------------------------------------------------|:----------|-----------------------:|:----------------------------------------------------------------------------------------------------|:----------------------------------------------------| +| gitlab_banzai_cached_render_real_duration_seconds | Histogram | 9.4 | Duration of rendering markdown into HTML when cached output exists | controller, action | +| gitlab_banzai_cacheless_render_real_duration_seconds | Histogram | 9.4 | Duration of rendering markdown into HTML when cached outupt does not exist | controller, action | +| gitlab_cache_misses_total | Counter | 10.2 | Cache read miss | controller, action | +| gitlab_cache_operation_duration_seconds | Histogram | 10.2 | Cache access time | | +| gitlab_cache_operations_total | Counter | 12.2 | Cache operations by controller/action | controller, action, operation | +| gitlab_database_transaction_seconds | Histogram | 12.1 | Time spent in database transactions, in seconds | | +| gitlab_method_call_duration_seconds | Histogram | 10.2 | Method calls real duration | controller, action, module, method | +| gitlab_rails_queue_duration_seconds | Histogram | 9.4 | Measures latency between gitlab-workhorse forwarding a request to Rails | | +| gitlab_sql_duration_seconds | Histogram | 10.2 | SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT | | +| gitlab_transaction_allocated_memory_bytes | Histogram | 10.2 | Allocated memory for all transactions (gitlab_transaction_* metrics) | | +| gitlab_transaction_cache_<key>_count_total | Counter | 10.2 | Counter for total Rails cache calls (per key) | | +| gitlab_transaction_cache_<key>_duration_total | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | +| gitlab_transaction_cache_count_total | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | +| gitlab_transaction_cache_duration_total | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | +| gitlab_transaction_cache_read_hit_count_total | Counter | 10.2 | Counter for cache hits for Rails cache calls | controller, action | +| gitlab_transaction_cache_read_miss_count_total | Counter | 10.2 | Counter for cache misses for Rails cache calls | controller, action | +| gitlab_transaction_duration_seconds | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | controller, action | +| gitlab_transaction_event_build_found_total | Counter | 9.4 | Counter for build found for api /jobs/request | | +| gitlab_transaction_event_build_invalid_total | Counter | 9.4 | Counter for build invalid due to concurrency conflict for api /jobs/request | | +| gitlab_transaction_event_build_not_found_cached_total | Counter | 9.4 | Counter for cached response of build not found for api /jobs/request | | +| gitlab_transaction_event_build_not_found_total | Counter | 9.4 | Counter for build not found for api /jobs/request | | +| gitlab_transaction_event_change_default_branch_total | Counter | 9.4 | Counter when default branch is changed for any repository | | +| gitlab_transaction_event_create_repository_total | Counter | 9.4 | Counter when any repository is created | | +| gitlab_transaction_event_etag_caching_cache_hit_total | Counter | 9.4 | Counter for etag cache hit. | endpoint | +| gitlab_transaction_event_etag_caching_header_missing_total | Counter | 9.4 | Counter for etag cache miss - header missing | endpoint | +| gitlab_transaction_event_etag_caching_key_not_found_total | Counter | 9.4 | Counter for etag cache miss - key not found | endpoint | +| gitlab_transaction_event_etag_caching_middleware_used_total | Counter | 9.4 | Counter for etag middleware accessed | endpoint | +| gitlab_transaction_event_etag_caching_resource_changed_total | Counter | 9.4 | Counter for etag cache miss - resource changed | endpoint | +| gitlab_transaction_event_fork_repository_total | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | +| gitlab_transaction_event_import_repository_total | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | +| gitlab_transaction_event_push_branch_total | Counter | 9.4 | Counter for all branch pushes | | +| gitlab_transaction_event_push_commit_total | Counter | 9.4 | Counter for commits | branch | +| gitlab_transaction_event_push_tag_total | Counter | 9.4 | Counter for tag pushes | | +| gitlab_transaction_event_rails_exception_total | Counter | 9.4 | Counter for number of rails exceptions | | +| gitlab_transaction_event_receive_email_total | Counter | 9.4 | Counter for recieved emails | handler | +| gitlab_transaction_event_remote_mirrors_failed_total | Counter | 10.8 | Counter for failed remote mirrors | | +| gitlab_transaction_event_remote_mirrors_finished_total | Counter | 10.8 | Counter for finished remote mirrors | | +| gitlab_transaction_event_remote_mirrors_running_total | Counter | 10.8 | Counter for running remote mirrors | | +| gitlab_transaction_event_remove_branch_total | Counter | 9.4 | Counter when a branch is removed for any repository | | +| gitlab_transaction_event_remove_repository_total | Counter | 9.4 | Counter when a repository is removed | | +| gitlab_transaction_event_remove_tag_total | Counter | 9.4 | Counter when a tag is remove for any repository | | +| gitlab_transaction_event_sidekiq_exception_total | Counter | 9.4 | Counter of sidekiq exceptions | | +| gitlab_transaction_event_stuck_import_jobs_total | Counter | 9.4 | Count of stuck import jobs | projects_without_jid_count, projects_with_jid_count | +| gitlab_transaction_event_update_build_total | Counter | 9.4 | Counter for update build for api /jobs/request/:id | | +| gitlab_transaction_new_redis_connections_total | Counter | 9.4 | Counter for new redis connections | | +| gitlab_transaction_queue_duration_total | Counter | 9.4 | Duration jobs were enqueued before processing | | +| gitlab_transaction_rails_queue_duration_total | Counter | 9.4 | Measures latency between gitlab-workhorse forwarding a request to Rails | controller, action | +| gitlab_transaction_view_duration_total | Counter | 9.4 | Duration for views | controller, action, view | +| gitlab_view_rendering_duration_seconds | Histogram | 10.2 | Duration for views (histogram) | controller, action, view | +| http_requests_total | Counter | 9.4 | Rack request count | method | +| http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | method, status | +| pipelines_created_total | Counter | 9.4 | Counter of pipelines created | | +| rack_uncaught_errors_total | Counter | 9.4 | Rack connections handling uncaught errors count | | +| user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | | +| upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | | +| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | +| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | + +## Metrics controlled by a feature flag + +The following metrics can be controlled by feature flags: + +| Metric | Feature Flag | +|:-------------------------------------------------------------|:-----------------------------------------------------------------| +| gitlab_method_call_duration_seconds | prometheus_metrics_method_instrumentation | +| gitlab_transaction_allocated_memory_bytes | prometheus_metrics_transaction_allocated_memory | +| gitlab_transaction_event_build_found_total | prometheus_transaction_event_build_found_total | +| gitlab_transaction_event_build_invalid_total | prometheus_transaction_event_build_invalid_total | +| gitlab_transaction_event_build_not_found_cached_total | prometheus_transaction_event_build_not_found_cached_total | +| gitlab_transaction_event_build_not_found_total | prometheus_transaction_event_build_not_found_total | +| gitlab_transaction_event_change_default_branch_total | prometheus_transaction_event_change_default_branch_total | +| gitlab_transaction_event_create_repository_total | prometheus_transaction_event_create_repository_total | +| gitlab_transaction_event_etag_caching_cache_hit_total | prometheus_transaction_event_etag_caching_cache_hit_total | +| gitlab_transaction_event_etag_caching_header_missing_total | prometheus_transaction_event_etag_caching_header_missing_total | +| gitlab_transaction_event_etag_caching_key_not_found_total | prometheus_transaction_event_etag_caching_key_not_found_total | +| gitlab_transaction_event_etag_caching_middleware_used_total | prometheus_transaction_event_etag_caching_middleware_used_total | +| gitlab_transaction_event_etag_caching_resource_changed_total | prometheus_transaction_event_etag_caching_resource_changed_total | +| gitlab_transaction_event_fork_repository_total | prometheus_transaction_event_fork_repository_total | +| gitlab_transaction_event_import_repository_total | prometheus_transaction_event_import_repository_total | +| gitlab_transaction_event_push_branch_total | prometheus_transaction_event_push_branch_total | +| gitlab_transaction_event_push_commit_total | prometheus_transaction_event_push_commit_total | +| gitlab_transaction_event_push_tag_total | prometheus_transaction_event_push_tag_total | +| gitlab_transaction_event_rails_exception_total | prometheus_transaction_event_rails_exception_total | +| gitlab_transaction_event_receive_email_total | prometheus_transaction_event_receive_email_total | +| gitlab_transaction_event_remote_mirrors_failed_total | prometheus_transaction_event_remote_mirrors_failed_total | +| gitlab_transaction_event_remote_mirrors_finished_total | prometheus_transaction_event_remote_mirrors_finished_total | +| gitlab_transaction_event_remote_mirrors_running_total | prometheus_transaction_event_remote_mirrors_running_total | +| gitlab_transaction_event_remove_branch_total | prometheus_transaction_event_remove_branch_total | +| gitlab_transaction_event_remove_repository_total | prometheus_transaction_event_remove_repository_total | +| gitlab_transaction_event_remove_tag_total | prometheus_transaction_event_remove_tag_total | +| gitlab_transaction_event_sidekiq_exception_total | prometheus_transaction_event_sidekiq_exception_total | +| gitlab_transaction_event_stuck_import_jobs_total | prometheus_transaction_event_stuck_import_jobs_total | +| gitlab_transaction_event_update_build_total | prometheus_transaction_event_update_build_total | +| gitlab_view_rendering_duration_seconds | prometheus_metrics_view_instrumentation | ## Sidekiq Metrics available for Geo **(PREMIUM)** @@ -99,17 +167,27 @@ Some basic Ruby runtime metrics are available: | Metric | Type | Since | Description | |:-------------------------------------- |:--------- |:----- |:----------- | -| ruby_gc_duration_seconds_total | Counter | 11.1 | Time spent by Ruby in GC | +| ruby_gc_duration_seconds | Counter | 11.1 | Time spent by Ruby in GC | | ruby_gc_stat_... | Gauge | 11.1 | Various metrics from [GC.stat] | | ruby_file_descriptors | Gauge | 11.1 | File descriptors per process | | ruby_memory_bytes | Gauge | 11.1 | Memory usage by process | -| ruby_sampler_duration_seconds_total | Counter | 11.1 | Time spent collecting stats | +| ruby_sampler_duration_seconds | Counter | 11.1 | Time spent collecting stats | | ruby_process_cpu_seconds_total | Gauge | 12.0 | Total amount of CPU time per process | | ruby_process_max_fds | Gauge | 12.0 | Maximum number of open file descriptors per process | | ruby_process_resident_memory_bytes | Gauge | 12.0 | Memory usage by process, measured in bytes | | ruby_process_start_time_seconds | Gauge | 12.0 | UNIX timestamp of process start time | -[GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat +[GC.stat]: https://ruby-doc.org/core-2.6.3/GC.html#method-c-stat + +## Unicorn Metrics + +Unicorn specific metrics, when Unicorn is used. + +| Metric | Type | Since | Description | +|:---------------------------|:------|:------|:---------------------------------------------------| +| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | +| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +| unicorn_workers | Gauge | 12.0 | The number of Unicorn workers | ## Puma Metrics **(EXPERIMENTAL)** @@ -126,7 +204,6 @@ When Puma is used instead of Unicorn, following metrics are available: | puma_pool_capacity | Gauge | 12.0 | Number of requests the worker is capable of taking right now | | puma_max_threads | Gauge | 12.0 | Maximum number of worker threads | | puma_idle_threads | Gauge | 12.0 | Number of spawned threads which are not processing a request | -| rack_state_total | Gauge | 12.0 | Number of requests in a given rack state | | puma_killer_terminations_total | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | ## Metrics shared directory diff --git a/doc/administration/packages.md b/doc/administration/packages.md index c0f8777a8c0..1628b6d6f91 100644 --- a/doc/administration/packages.md +++ b/doc/administration/packages.md @@ -55,7 +55,7 @@ local location or even use object storage. The packages for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/packages/` and for source -installations under `shared/packages/` (relative to the git homedir). +installations under `shared/packages/` (relative to the Git homedir). To change the local storage path: **Omnibus GitLab installations** diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index 4cf3c607dae..92a4d56ca63 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -39,7 +39,7 @@ Follow the steps below to set up a custom hook: 1. Inside the `plugins` directory, create a file with a name of your choice, without spaces or special characters. -1. Make the hook file executable and make sure it's owned by the git user. +1. Make the hook file executable and make sure it's owned by the Git user. 1. Write the code to make the plugin function as expected. That can be in any language, and ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will @@ -112,4 +112,4 @@ Validating plugins from /plugins directory [system hooks]: ../system_hooks/system_hooks.md [webhooks]: ../user/project/integrations/webhooks.md -[highly available]: ./high_availability/README.md
\ No newline at end of file +[highly available]: ./high_availability/README.md diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 406f7e8a034..56cd23b2eb8 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -18,7 +18,7 @@ The instructions make the assumption that you will be using the email address `i sudo apt-get install postfix ``` - When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches gitlab.example.com`. + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. 1. Install the `mailutils` package. @@ -331,7 +331,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo a logout ``` -## Done! +## Done If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email] guide to configure GitLab. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 05a7cb006a3..ab911c1cf0e 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -3,7 +3,7 @@ > [Introduced][ce-3232] in GitLab 8.7. It is OFF by default because it still causes too many false alarms. -Git has a built-in mechanism, [git fsck][git-fsck], to verify the +Git has a built-in mechanism, [`git fsck`][git-fsck], to verify the integrity of all data committed to a repository. GitLab administrators can trigger such a check for a project via the project page under the admin panel. The checks run asynchronously so it may take a few minutes diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 1669f8b128c..142bcc65561 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -118,7 +118,7 @@ If you do have any existing integration, you may want to do a small rollout firs to validate. You can do so by specifying a range with the operation. This is an example of how to limit the rollout to Project IDs 50 to 100, running in -an Omnibus Gitlab installation: +an Omnibus GitLab installation: ```bash sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 @@ -139,7 +139,7 @@ To schedule a complete rollback, see the [rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. The rollback task also supports specifying a range of Project IDs. Here is an example -of limiting the rollout to Project IDs 50 to 100, in an Omnibus Gitlab installation: +of limiting the rollout to Project IDs 50 to 100, in an Omnibus GitLab installation: ```bash sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 @@ -185,7 +185,7 @@ CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in ##### LFS Objects -LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following git own implementation: +LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following Git own implementation: ```ruby "shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md new file mode 100644 index 00000000000..9f719088f25 --- /dev/null +++ b/doc/administration/smime_signing_email.md @@ -0,0 +1,49 @@ +# Signing outgoing email with S/MIME + +Notification emails sent by Gitlab can be signed with S/MIME for improved +security. + +> **Note:** +Please be aware that S/MIME certificates and TLS/SSL certificates are not the +same and are used for different purposes: TLS creates a secure channel, whereas +S/MIME signs and/or encrypts the message itself + +## Enable S/MIME signing + +This setting must be explicitly enabled and a single pair of key and certificate +files must be provided in `gitlab.rb` or `gitlab.yml` if you are using Omnibus +GitLab or installed GitLab from source respectively: + +```yaml +email_smime: + enabled: true + key_file: /etc/pki/smime/private/gitlab.key + cert_file: /etc/pki/smime/certs/gitlab.crt +``` + +- Both files must be provided PEM-encoded. +- The key file must be unencrypted so that Gitlab can read it without user + intervention. + +NOTE: **Note:** Be mindful of the access levels for your private keys and visibility to +third parties. + +### How to convert S/MIME PKCS#12 / PFX format to PEM encoding + +Typically S/MIME certificates are handled in binary PKCS#12 format (`.pfx` or `.p12` +extensions), which contain the following in a single encrypted file: + +- Server certificate +- Intermediate certificates (if any) +- Private key + +In order to export the required files in PEM encoding from the PKCS#12 file, +the `openssl` command can be used: + +```bash +#-- Extract private key in PEM encoding (no password, unencrypted) +$ openssl pkcs12 -in gitlab.p12 -nocerts -nodes -out gitlab.key + +#-- Extract certificates in PEM encoding (full certs chain including CA) +$ openssl pkcs12 -in gitlab.p12 -nokeys -out gitlab.crt +``` diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 95cdb1508fa..260af333e8e 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -29,7 +29,7 @@ and they will assist you with any issues you are having. ```bash # for minikube: minikube dashboard —url - # for non-local installations if access via kubectl is configured: + # for non-local installations if access via Kubectl is configured: kubectl proxy ``` @@ -49,7 +49,7 @@ and they will assist you with any issues you are having. - What to do with pods in `CrashLoopBackoff` status: - Check logs via Kubernetes dashboard. - - Check logs via `kubectl`: + - Check logs via Kubectl: ```bash kubectl logs <unicorn pod> -c dependencies @@ -72,7 +72,7 @@ and they will assist you with any issues you are having. This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) for details. -## Gitlab-specific kubernetes information +## GitLab-specific kubernetes information - Minimal config that can be used to test a Kubernetes helm chart can be found [here](https://gitlab.com/charts/gitlab/issues/620). @@ -83,12 +83,22 @@ and they will assist you with any issues you are having. kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn ``` -- It is not possible to get all the logs via `kubectl` at once, like with `gitlab-ctl tail`, - but a number of third-party tools can be used to do it: +- Tail and follow all pods that share a label (in this case, `unicorn`): - - [Kubetail](https://github.com/johanhaleby/kubetail) - - [kail: kubernetes tail](https://github.com/boz/kail) - - [stern](https://github.com/wercker/stern) + ```bash + # all containers in the unicorn pods + kubectl logs -f -l app=unicorn --all-containers=true --max-log-requests=50 + + # only the unicorn containers in all unicorn pods + kubectl logs -f -l app=unicorn -c unicorn --max-log-requests=50 + ``` + +- One can stream logs from all containers at once, similar to the Omnibus + command `gitlab-ctl tail`: + + ```bash + kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100 + ``` - Check all events in the `gitlab` namespace (the namespace name can be different if you specified a different one when deploying the helm chart): @@ -131,7 +141,7 @@ and they will assist you with any issues you are having. - Check the output of `kubectl get events -w --all-namespaces`. - Check the logs of pods within `gitlab-managed-apps` namespace. - On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed - via helm chart, kubernetes.log can be found inside the sidekiq pod. + via Helm Chart, `kubernetes.log` can be found inside the sidekiq pod. - How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: @@ -142,19 +152,19 @@ and they will assist you with any issues you are having. kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo ``` -- How to connect to a GitLab postgres database: +- How to connect to a GitLab Postgres database: ```bash kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p ``` -- How to get info about helm installation status: +- How to get info about Helm installation status: ```bash helm status name-of-installation ``` -- How to update GitLab installed using helm chart: +- How to update GitLab installed using Helm Chart: ```bash helm repo upgrade @@ -179,25 +189,25 @@ and they will assist you with any issues you are having. helm upgrade <release name> <chart path> -f gitlab.yaml ``` -## Installation of minimal GitLab config via minukube on macOS +## Installation of minimal GitLab config via Minukube on macOS This section is based on [Developing for Kubernetes with Minikube](https://gitlab.com/charts/gitlab/blob/master/doc/minikube/index.md) and [Helm](https://gitlab.com/charts/gitlab/blob/master/doc/helm/index.md). Refer to those documents for details. -- Install kubectl via Homebrew: +- Install Kubectl via Homebrew: ```bash brew install kubernetes-cli ``` -- Install minikube via Homebrew: +- Install Minikube via Homebrew: ```bash brew cask install minikube ``` -- Start minikube and configure it. If minikube cannot start, try running `minikube delete && minikube start` +- Start Minikube and configure it. If Minikube cannot start, try running `minikube delete && minikube start` and repeat the steps: ```bash @@ -206,7 +216,7 @@ to those documents for details. minikube addons enable kube-dns ``` -- Install helm via Homebrew and initialize it: +- Install Helm via Homebrew and initialize it: ```bash brew install kubernetes-helm @@ -219,7 +229,7 @@ to those documents for details. - Find the IP address in the output of `minikube ip` and update the yaml file with this IP address. -- Install the GitLab helm chart: +- Install the GitLab Helm Chart: ```bash helm repo add gitlab https://charts.gitlab.io diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 7067958ecb4..9b016c64e29 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -169,3 +169,121 @@ The PostgreSQL wiki has details on the query you can run to see blocking queries. The query is different based on PostgreSQL version. See [Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for the query details. + +## Managing Sidekiq queues + +It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) +to perform a number of troubleshoting on Sidekiq. + +These are the administrative commands and it should only be used if currently +admin interface is not suitable due to scale of installation. + +All this commands should be run using `gitlab-rails console`. + +### View the queue size + +```ruby +Sidekiq::Queue.new("pipeline_processing:build_queue").size +``` + +### Enumerate all enqueued jobs + +```ruby +queue = Sidekiq::Queue.new("chaos:chaos_sleep") +queue.each do |job| + # job.klass # => 'MyWorker' + # job.args # => [1, 2, 3] + # job.jid # => jid + # job.queue # => chaos:chaos_sleep + # job["retry"] # => 3 + # job.item # => { + # "class"=>"Chaos::SleepWorker", + # "args"=>[1000], + # "retry"=>3, + # "queue"=>"chaos:chaos_sleep", + # "backtrace"=>true, + # "queue_namespace"=>"chaos", + # "jid"=>"39bc482b823cceaf07213523", + # "created_at"=>1566317076.266069, + # "correlation_id"=>"c323b832-a857-4858-b695-672de6f0e1af", + # "enqueued_at"=>1566317076.26761}, + # } + + # job.delete if job.jid == 'abcdef1234567890' +end +``` + +### Enumerate currently running jobs + +```ruby +workers = Sidekiq::Workers.new +workers.each do |process_id, thread_id, work| + # process_id is a unique identifier per Sidekiq process + # thread_id is a unique identifier per thread + # work is a Hash which looks like: + # {"queue"=>"chaos:chaos_sleep", + # "payload"=> + # { "class"=>"Chaos::SleepWorker", + # "args"=>[1000], + # "retry"=>3, + # "queue"=>"chaos:chaos_sleep", + # "backtrace"=>true, + # "queue_namespace"=>"chaos", + # "jid"=>"b2a31e3eac7b1a99ff235869", + # "created_at"=>1566316974.9215662, + # "correlation_id"=>"e484fb26-7576-45f9-bf21-b99389e1c53c", + # "enqueued_at"=>1566316974.9229589}, + # "run_at"=>1566316974}], +end +``` + +### Remove sidekiq jobs for given parameters (destructive) + +```ruby +# for jobs like this: +# RepositoryImportWorker.new.perform_async(100) +id_list = [100] + +queue = Sidekiq::Queue.new('repository_import') +queue.each do |job| + job.delete if id_list.include?(job.args[0]) +end +``` + +### Remove specific job ID (destructive) + +```ruby +queue = Sidekiq::Queue.new('repository_import') +queue.each do |job| + job.delete if job.jid == 'my-job-id' +end +``` + +## Canceling running jobs (destructive) + +> Introduced in GitLab 12.3. + +This is highly risky operation and use it as last resort. +Doing that might result in data corruption, as the job +is interrupted mid-execution and it is not guaranteed +that proper rollback of transactions is implemented. + +```ruby +Gitlab::SidekiqMonitor.cancel_job('job-id') +``` + +> This requires the Sidekiq to be run with `SIDEKIQ_MONITOR_WORKER=1` +> environment variable. + +To perform of the interrupt we use `Thread.raise` which +has number of drawbacks, as mentioned in [Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): + +> This is where the implications get interesting, and terrifying. This means that an exception can get raised: +> +> * during a network request (ok, as long as the surrounding code is prepared to catch Timeout::Error) +> * during the cleanup for the network request +> * during a rescue block +> * while creating an object to save to the database afterwards +> * in any of your code, regardless of whether it could have possibly raised an exception before +> +> Nobody writes code to defend against an exception being raised on literally any line. That’s not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that’s unlikely :) diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 99f1c386183..a4bed72b965 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -57,7 +57,7 @@ This configuration relies on valid AWS credentials to be configured already. ## Object Storage Settings -For source installations the following settings are nested under `uploads:` and then `object_store:`. On omnibus installs they are prefixed by `uploads_object_store_`. +For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`. | Setting | Description | Default | |---------|-------------|---------| @@ -78,7 +78,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | -| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -220,7 +220,7 @@ _The uploads are stored by default in openstack_temp_url_key: OPENSTACK_TEMP_URL_KEY openstack_auth_url: 'https://auth.cloud.ovh.net/v2.0/' openstack_region: DE1 - openstack_tenant: 'TENANT_ID' + openstack_tenant: 'TENANT_ID' ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. diff --git a/doc/api/README.md b/doc/api/README.md index b7ee710b87a..33394d46a2d 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -77,11 +77,12 @@ authentication is not provided. For those cases where it is not required, this will be mentioned in the documentation for each individual endpoint. For example, the [`/projects/:id` endpoint](projects.md). -There are three ways to authenticate with the GitLab API: +There are four ways to authenticate with the GitLab API: 1. [OAuth2 tokens](#oauth2-tokens) 1. [Personal access tokens](#personal-access-tokens) 1. [Session cookie](#session-cookie) +1. [GitLab CI job token](#gitlab-ci-job-token-premium) **(PREMIUM)** For admins who want to authenticate with the API as a specific user, or who want to build applications or scripts that do so, two options are available: @@ -151,6 +152,14 @@ The primary user of this authentication method is the web frontend of GitLab its which can use the API as the authenticated user to get a list of their projects, for example, without needing to explicitly pass an access token. +### GitLab CI job token **(PREMIUM)** + +With a few API endpoints you can use a [GitLab CI job token](../user/project/new_ci_build_permissions_model.md#job-token) +to authenticate with the API: + +- [Get job artifacts](jobs.md#get-job-artifacts) +- [Pipeline triggers](pipeline_triggers.md) + ### Impersonation tokens > [Introduced][ce-9099] in GitLab 9.0. Needs admin permissions. diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index b32f11464ef..9af5430f1c8 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -67,7 +67,7 @@ The following API resources are available in the project context: | [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | | [Services](services.md) | `/projects/:id/services` | | [Tags](tags.md) | `/projects/:id/repository/tags` | -| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` (also available for groups) | +| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` | [Wikis](wikis.md) | `/projects/:id/wikis` | ## Group resources diff --git a/doc/api/boards.md b/doc/api/boards.md index 08ec1d832df..b848d7788cd 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -466,7 +466,7 @@ Example response: ## Delete a board list -Only for admins and project owners. Soft deletes the board list in question. +Only for admins and project owners. Deletes the board list in question. ``` DELETE /projects/:id/boards/:board_id/lists/:list_id diff --git a/doc/api/dependencies.md b/doc/api/dependencies.md index 015ffbe60f6..5296d4e316f 100644 --- a/doc/api/dependencies.md +++ b/doc/api/dependencies.md @@ -5,20 +5,21 @@ This API is in an alpha stage and considered unstable. The response payload may be subject to change or breakage across GitLab releases. -Every call to this endpoint requires authentication. To perform this call, user should be authorized to read -[Project Security Dashboard](../user/application_security/security_dashboard/index.md#project-security-dashboard). +Every call to this endpoint requires authentication. To perform this call, user should be authorized to read repository. +To see vulnerabilities in response, user should be authorized to read +[Project Security Dashboard](../user/application_security/security_dashboard/index.md#project-security-dashboard). ## List project dependencies -Get a list of project dependencies. This API partially mirroring +Get a list of project dependencies. This API partially mirroring [Dependency List](../user/application_security/dependency_list/index.md) feature. This list can be generated only for [languages and package managers](../user/application_security/dependency_scanning/index.md#supported-languages-and-package-managers) -supported by Gemnasium. +supported by Gemnasium. ``` GET /projects/:id/dependencies -GET /projects/:id/vulnerabilities?package_manager=maven -GET /projects/:id/vulnerabilities?package_manager=yarn,bundler +GET /projects/:id/dependencies?package_manager=maven +GET /projects/:id/dependencies?package_manager=yarn,bundler ``` | Attribute | Type | Required | Description | @@ -38,13 +39,18 @@ Example response: "name": "rails", "version": "5.0.1", "package_manager": "bundler", - "dependency_file_path": "Gemfile.lock" + "dependency_file_path": "Gemfile.lock", + "vulnerabilities": [{ + "name": "DDoS", + "severity": "unknown" + }] }, { "name": "hanami", "version": "1.3.1", "package_manager": "bundler", - "dependency_file_path": "Gemfile.lock" + "dependency_file_path": "Gemfile.lock", + "vulnerabilities": [] } ] ``` diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index 94351e1a300..df8cf06fc4a 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -203,6 +203,7 @@ Example response: "created_at" : "2015-08-29T12:44:31.550Z" } ``` + ## Adding deploy keys to multiple projects If you want to easily add the same deploy key to multiple projects in the same diff --git a/doc/api/discussions.md b/doc/api/discussions.md index b4a2d0b15f6..12dbba78291 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -160,7 +160,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab Adds a new note to the thread. This can also [create a thread from a single comment](../user/discussions/#start-a-thread-by-replying-to-a-standard-comment). **WARNING** -Notes can be added to other items than comments (system notes, etc.) making them threads. +Notes can be added to other items than comments (system notes, etc.) making them threads. ``` POST /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes diff --git a/doc/api/epics.md b/doc/api/epics.md index 3036b3c2364..87ae0c48199 100644 --- a/doc/api/epics.md +++ b/doc/api/epics.md @@ -65,6 +65,8 @@ Example response: "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/4", + "reference": "&4", "author": { "id": 10, "name": "Lu Mayer", @@ -118,6 +120,8 @@ Example response: "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/5", + "reference": "&5", "author":{ "id": 7, "name": "Pamella Huel", @@ -182,6 +186,8 @@ Example response: "title": "Epic", "description": "Epic description", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/6", + "reference": "&6", "author": { "name" : "Alexandra Bashirian", "avatar_url" : null, @@ -247,6 +253,8 @@ Example response: "title": "New Title", "description": "Epic description", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/6", + "reference": "&6", "author": { "name" : "Alexandra Bashirian", "avatar_url" : null, diff --git a/doc/api/features.md b/doc/api/features.md index 6ecd4ec14b9..e8d0c7c942b 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -60,8 +60,8 @@ POST /features/:name | `value` | integer/string | yes | `true` or `false` to enable/disable, or an integer for percentage of time | | `feature_group` | string | no | A Feature group name | | `user` | string | no | A GitLab username | -| `group` | string | no | A GitLab group's path, for example 'gitlab-org' | -| `project` | string | no | A projects path, for example 'gitlab-org/gitlab-ce' | +| `group` | string | no | A GitLab group's path, for example `gitlab-org` | +| `project` | string | no | A projects path, for example `gitlab-org/gitlab-ce` | Note that you can enable or disable a feature for a `feature_group`, a `user`, a `group`, and a `project` in a single API call. diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index d0b33ab467f..5eba7f038ed 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -111,7 +111,7 @@ PUT /geo_nodes/:id |-----------------------------|---------|-----------|---------------------------------------------------------------------------| | `id` | integer | yes | The ID of the Geo node. | | `enabled` | boolean | no | Flag indicating if the Geo node is enabled. | -| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in gitlab.rb, otherwise it must match `external_url`. | +| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in `gitlab.rb`, otherwise it must match `external_url`. | | `url` | string | yes | The user-facing URL of the Geo node. | | `internal_url` | string | no | The URL defined on the primary node that secondary nodes should use to contact it. Returns `url` if not set.| | `files_max_capacity` | integer | no | Control the maximum concurrency of LFS/attachment backfill for this secondary node. | diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 2d3bec4ff67..d99a4c37d72 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -109,6 +109,7 @@ The API can be explored interactively using the [GraphiQL IDE](../index.md#graph | `visibility` | String | | | `lfsEnabled` | Boolean | | | `requestAccessEnabled` | Boolean | | +| `rootStorageStatistics` | RootStorageStatistics | The aggregated storage statistics. Only available if the namespace has no parent | | `userPermissions` | GroupPermissions! | Permissions for the current user on the resource | | `webUrl` | String! | | | `avatarUrl` | String | | @@ -453,6 +454,17 @@ The API can be explored interactively using the [GraphiQL IDE](../index.md#graph | `exists` | Boolean! | | | `tree` | Tree | | +### RootStorageStatistics + +| Name | Type | Description | +| --- | ---- | ---------- | +| `storageSize` | Int! | The total storage in Bytes | +| `repositorySize` | Int! | The git repository size in Bytes | +| `lfsObjectsSize` | Int! | The LFS objects size in Bytes | +| `buildArtifactsSize` | Int! | The CI artifacts size in Bytes | +| `packagesSize` | Int! | The packages size in Bytes | +| `wikiSize` | Int! | The wiki size in Bytes | + ### Submodule | Name | Type | Description | diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md index 4d10f83720b..99b522a7ae9 100644 --- a/doc/api/group_boards.md +++ b/doc/api/group_boards.md @@ -536,7 +536,7 @@ Example response: ## Delete a group issue board list -Only for admins and group owners. Soft deletes the board list in question. +Only for admins and group owners. Deletes the board list in question. ``` DELETE /groups/:id/boards/:board_id/lists/:list_id diff --git a/doc/api/issues.md b/doc/api/issues.md index 96a547551f1..8313dd2c3bd 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -49,7 +49,7 @@ GET /issues?confidential=true | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `iids[]` | integer array | no | Return only the issues having the given `iid` | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | | `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | @@ -136,7 +136,6 @@ Example response: "award_emoji":"http://example.com/api/v4/projects/1/issues/76/award_emoji", "project":"http://example.com/api/v4/projects/1" }, - "subscribed": false, "task_completion_status":{ "count":0, "completed_count":0 @@ -199,7 +198,7 @@ GET /groups/:id/issues?confidential=true | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search group issues against their `title` and `description` | | `created_after` | datetime | no | Return issues created on or after the given time | @@ -348,7 +347,7 @@ GET /projects/:id/issues?confidential=true | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search project issues against their `title` and `description` | | `created_after` | datetime | no | Return issues created on or after the given time | @@ -441,7 +440,6 @@ Example response: "award_emoji":"http://example.com/api/v4/projects/4/issues/41/award_emoji", "project":"http://example.com/api/v4/projects/4" }, - "subscribed": false, "task_completion_status":{ "count":0, "completed_count":0 @@ -790,7 +788,7 @@ the `weight` parameter: ## Delete an issue -Only for admins and project owners. Soft deletes the issue in question. +Only for admins and project owners. Deletes the issue in question. ``` DELETE /projects/:id/issues/:issue_iid diff --git a/doc/api/labels.md b/doc/api/labels.md index 5db0edcf14d..fde1d861cf6 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -137,8 +137,9 @@ DELETE /projects/:id/labels | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the label | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `label_id` | integer | yes (or `name`) | The id of the existing label | +| `name` | string | yes (or `label_id`) | The name of the existing label | ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels?name=bug" @@ -156,7 +157,8 @@ PUT /projects/:id/labels | Attribute | Type | Required | Description | | --------------- | ------- | --------------------------------- | ------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the existing label | +| `label_id` | integer | yes (or `name`) | The id of the existing label | +| `name` | string | yes (or `label_id`) | The name of the existing label | | `new_name` | string | yes if `color` is not provided | The new name of the label | | `color` | string | yes if `new_name` is not provided | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | | `description` | string | no | The new description of the label | diff --git a/doc/api/lint.md b/doc/api/lint.md index 79f5e629c7f..dacd3f4c493 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -1,4 +1,4 @@ -# Validate the .gitlab-ci.yml (API) +# Validate the `.gitlab-ci.yml` (API) > [Introduced][ce-5953] in GitLab 8.12. @@ -10,7 +10,7 @@ POST /ci/lint | Attribute | Type | Required | Description | | ---------- | ------- | -------- | -------- | -| `content` | string | yes | the .gitlab-ci.yaml content| +| `content` | string | yes | the `.gitlab-ci.yaml` content| ```bash curl --header "Content-Type: application/json" https://gitlab.example.com/api/v4/ci/lint --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index cc95689a65f..bf83bfd7e6a 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -23,36 +23,6 @@ GET /projects/:id/approvals ```json { - "approvers": [ - { - "user": { - "id": 5, - "name": "John Doe6", - "username": "user5", - "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" - } - } - ], - "approver_groups": [ - { - "group": { - "id": 1, - "name": "group1", - "path": "group1", - "description": "", - "visibility": "public", - "lfs_enabled": false, - "avatar_url": null, - "web_url": "http://localhost/groups/group1", - "request_access_enabled": false, - "full_name": "group1", - "full_path": "group1", - "parent_id": null, - "ldap_cn": null, - "ldap_access": null - } - } - ], "approvals_before_merge": 2, "reset_approvals_on_push": true, "disable_overriding_approvers_per_merge_request": false @@ -75,7 +45,7 @@ POST /projects/:id/approvals | Attribute | Type | Required | Description | | ------------------------------------------------ | ------- | -------- | --------------------------------------------------------------------------------------------------- | | `id` | integer | yes | The ID of a project | -| `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged | +| `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged. Deprecated in 12.0 in favor of Approval Rules API. | | `reset_approvals_on_push` | boolean | no | Reset approvals on a new push | | `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | | `merge_requests_author_approval` | boolean | no | Allow/Disallow authors from self approving merge requests; `true` means authors cannot self approve | @@ -83,20 +53,68 @@ POST /projects/:id/approvals ```json { - "approvers": [ - { - "user": { + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false, + "merge_requests_author_approval": false, + "merge_requests_disable_committers_approval": false +} +``` + +### Get project-level rules + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can request information about a project's approval rules using the following endpoint: + +``` +GET /projects/:id/approval_rules +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | + +```json +[ + { + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { "id": 5, - "name": "John Doe6", - "username": "user5", - "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" } - } - ], - "approver_groups": [ - { - "group": { - "id": 1, + ], + "approvals_required": 3, + "users": [ + { + "id": 5, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, "name": "group1", "path": "group1", "description": "", @@ -111,18 +129,187 @@ POST /projects/:id/approvals "ldap_cn": null, "ldap_access": null } + ], + "contains_hidden_groups": false + } +] +``` + +### Create project-level rule + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can create project approval rules using the following endpoint: + +``` +POST /projects/:id/approval_rules +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `name` | string | yes | The name of the approval rule | +| `approvals_required` | integer | yes | The number of required approvals for this rule | +| `users` | integer | no | The ids of users as approvers | +| `groups` | integer | no | The ids of groups as approvers | + +```json +{ + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" } ], - "approvals_before_merge": 2, - "reset_approvals_on_push": true, - "disable_overriding_approvers_per_merge_request": false, - "merge_requests_author_approval": false, - "merge_requests_disable_committers_approval": false + "approvals_required": 1, + "users": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + ], + "contains_hidden_groups": false +} +``` + +### Update project-level rule + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can update project approval rules using the following endpoint: + +``` +PUT /projects/:id/approval_rules/:approval_rule_id +``` + +**Important:** Approvers and groups not in the `users`/`groups` param will be **removed** + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `approval_rule_id` | integer | yes | The ID of a approval rule | +| `name` | string | yes | The name of the approval rule | +| `approvals_required` | integer | yes | The number of required approvals for this rule | +| `users` | integer | no | The ids of users as approvers | +| `groups` | integer | no | The ids of groups as approvers | + +```json +{ + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" + } + ], + "approvals_required": 1, + "users": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + ], + "contains_hidden_groups": false } ``` +### Delete project-level rule + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can delete project approval rules using the following endpoint: + +``` +DELETE /projects/:id/approval_rules/:approval_rule_id +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `approval_rule_id` | integer | yes | The ID of a approval rule + ### Change allowed approvers +>**Note:** This API endpoint has been deprecated. Please use Approval Rule API instead. >**Note:** This API endpoint is only available on 10.6 Starter and above. If you are allowed to, you can change approvers and approver groups using @@ -227,8 +414,6 @@ GET /projects/:id/merge_requests/:merge_request_iid/approvals } } ], - "approvers": [], - "approver_groups": [] } ``` @@ -249,7 +434,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/approvals |----------------------|---------|----------|--------------------------------------------| | `id` | integer | yes | The ID of a project | | `merge_request_iid` | integer | yes | The IID of MR | -| `approvals_required` | integer | yes | Approvals required before MR can be merged | +| `approvals_required` | integer | yes | Approvals required before MR can be merged. Deprecated in 12.0 in favor of Approval Rules API. | ```json { @@ -264,14 +449,13 @@ POST /projects/:id/merge_requests/:merge_request_iid/approvals "merge_status": "cannot_be_merged", "approvals_required": 2, "approvals_left": 2, - "approved_by": [], - "approvers": [], - "approver_groups": [] + "approved_by": [] } ``` ### Change allowed approvers for Merge Request +>**Note:** This API endpoint has been deprecated. Please use Approval Rule API instead. >**Note:** This API endpoint is only available on 10.6 Starter and above. If you are allowed to, you can change approvers and approver groups using @@ -401,8 +585,6 @@ does not match, the response code will be `409`. } } ], - "approvers": [], - "approver_groups": [] } ``` diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 1ade46efb1c..49ed4968b0d 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1127,7 +1127,7 @@ the `approvals_before_merge` parameter: ## Delete a merge request -Only for admins and project owners. Soft deletes the merge request in question. +Only for admins and project owners. Deletes the merge request in question. ``` DELETE /projects/:id/merge_requests/:merge_request_iid diff --git a/doc/api/projects.md b/doc/api/projects.md index 70df44ec0fd..9f392418153 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -855,6 +855,7 @@ GET /projects/:id/users | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `search` | string | no | Search for specific users | +| `skip_users` | array[int] | no | Filter out users with the specified IDs | ```json [ @@ -2036,13 +2037,13 @@ Read more in the [Project Badges](project_badges.md) documentation. The non-default [issue and merge request description templates](../user/project/description_templates.md) are managed inside the project's repository. So you can manage them via the API through the [Repositories API](repositories.md) and the [Repository Files API](repository_files.md). -## Download snapshot of a git repository +## Download snapshot of a Git repository > Introduced in GitLab 10.7 This endpoint may only be accessed by an administrative user. -Download a snapshot of the project (or wiki, if requested) git repository. This +Download a snapshot of the project (or wiki, if requested) Git repository. This snapshot is always in uncompressed [tar](https://en.wikipedia.org/wiki/Tar_(computing)) format. diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index 9309306ba05..ffb4a70168b 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -185,6 +185,7 @@ Example response: { "access_level": 30, "access_level_description": "Developers + Maintainers" + } ], "unprotect_access_levels": [ { @@ -217,6 +218,7 @@ Example response: "user_id": null, "group_id": null, "access_level_description": "Developers + Maintainers" + } ], "unprotect_access_levels": [ { @@ -232,7 +234,7 @@ Example response: ### Example with user / group level access **(STARTER)** Elements in the `allowed_to_push` / `allowed_to_merge` / `allowed_to_unprotect` array should take the -form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). These access levels allow [more granular control over protected branch access](../user/project/protected_branches.md#restricting-push-and-merge-access-to-certain-users-starter) and were [added to the API in ][ee-3516] in GitLab 10.3 EE. +form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). These access levels allow [more granular control over protected branch access](../user/project/protected_branches.md#restricting-push-and-merge-access-to-certain-users-starter) and were [added to the API in](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3516) in GitLab 10.3 EE. ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&allowed_to_push%5B%5D%5Buser_id%5D=1' @@ -242,29 +244,29 @@ Example response: ```json { - "name":"*-stable", + "name": "*-stable", "push_access_levels": [ { - "access_level":null, - "user_id":1, - "group_id":null, - "access_level_description":"Administrator" + "access_level": null, + "user_id": 1, + "group_id": null, + "access_level_description": "Administrator" } ], "merge_access_levels": [ { - "access_level":40, - "user_id":null, - "group_id":null, - "access_level_description":"Maintainers" + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" } ], "unprotect_access_levels": [ { - "access_level":40, - "user_id":null, - "group_id":null, - "access_level_description":"Maintainers" + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" } ] } @@ -286,5 +288,3 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" 'https://git | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the branch | - -[ee-3516]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3516 "ProtectedBranches API handles per user/group granularity" diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index b292c9dd7de..513dc996c91 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -246,7 +246,7 @@ error message. Possible causes for a failed commit include: user tried to make an empty commit; - the branch was updated by a Git push while the file edit was in progress. -Currently gitlab-shell has a boolean return code, preventing GitLab from specifying the error. +Currently GitLab Shell has a boolean return code, preventing GitLab from specifying the error. ## Delete existing file in repository diff --git a/doc/api/settings.md b/doc/api/settings.md index 248d19461f6..710b63c9a2f 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -232,7 +232,7 @@ are listed in the descriptions of the relevant settings. | `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `geo_status_timeout` | integer | no | **(PREMIUM)** The amount of seconds after which a request to get a secondary node status will time out. | -| `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | +| `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for Git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | | `gravatar_enabled` | boolean | no | Enable Gravatar. | @@ -244,7 +244,7 @@ are listed in the descriptions of the relevant settings. | `hide_third_party_offers` | boolean | no | Do not display offers from third parties within GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | -| `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable git housekeeping. | +| `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable Git housekeeping. | | `housekeeping_full_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | | `housekeeping_gc_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which `git gc` is run. | | `housekeeping_incremental_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | @@ -322,7 +322,7 @@ are listed in the descriptions of the relevant settings. | `version_check_enabled` | boolean | no | Let GitLab inform you when an update is available. | | `local_markdown_version` | integer | no | Increase this value when any cached markdown should be invalidated. | | `snowplow_enabled` | boolean | no | Enable snowplow tracking. | -| `snowplow_collector_hostname` | string | required by: `snowplow_enabled` | The Snowplow collector hostname. (e.g. `snowplow.trx.gitlab.net`) | +| `snowplow_collector_hostname` | string | required by: `snowplow_enabled` | The Snowplow collector hostname. (e.g. `snowplow.trx.gitlab.net`) | | `snowplow_site_id` | string | no | The Snowplow site name / application id. (e.g. `gitlab`) | | `snowplow_cookie_domain` | string | no | The Snowplow cookie domain. (e.g. `.gitlab.com`) | | `geo_node_allowed_ips` | string | yes | **(PREMIUM)** Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | diff --git a/doc/api/tags.md b/doc/api/tags.md index af86ba961f4..1d874fea1f8 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -112,7 +112,7 @@ Parameters: - `tag_name` (required) - The name of a tag - `ref` (required) - Create tag using commit SHA, another tag name, or branch name. - `message` (optional) - Creates annotated tag. -- `release_description` (optional) - Add release notes to the git tag and store it in the GitLab database. +- `release_description` (optional) - Add release notes to the Git tag and store it in the GitLab database. ```json { @@ -166,7 +166,7 @@ Parameters: ## Create a new release -Add release notes to the existing git tag. If there +Add release notes to the existing Git tag. If there already exists a release for the given tag, status code `409` is returned. ``` diff --git a/doc/ci/README.md b/doc/ci/README.md index ca9d0aa61bd..4be13204227 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -131,7 +131,7 @@ Its feature set is listed on the table below according to DevOps stages. | **Secure** || | [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities.| | [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | -| [License Management](../user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | +| [License Compliance](../user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | | [Security Test reports](../user/project/merge_requests/index.md#security-reports-ultimate) **(ULTIMATE)** | Check for app vulnerabilities. | ## Examples @@ -174,7 +174,7 @@ been necessary. These are: #### 12.0 -- [Use refspec to clone/fetch git +- [Use refspec to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/issues/4069). - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4070). diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index e54be9f3bd9..1c38c08b7cb 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -65,7 +65,7 @@ as quickly as possible. Relationships are defined between jobs using the [`needs:` keyword](../yaml/README.md#needs). Note that `needs:` also works with the [parallel](../yaml/README.md#parallel) keyword, -giving your powerful options for parallelization within your pipeline. +giving you powerful options for parallelization within your pipeline. ## Limitations diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index d5056568dff..489791141ed 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -495,14 +495,6 @@ that runner. ## Define an image from a private Container Registry -> **Notes:** -> -> - This feature requires GitLab Runner **1.8** or higher -> - For GitLab Runner versions **>= 0.6, <1.8** there was a partial -> support for using private registries, which required manual configuration -> of credentials on runner's host. We recommend to upgrade your Runner to -> at least version **1.8** if you want to use private registries. - To access private container registries, the GitLab Runner process can use: - [Statically defined credentials](#using-statically-defined-credentials). That is, a username and password for a specific registry. @@ -525,6 +517,17 @@ it's provided as an environment variable. This is because GitLab Runnner uses ** `config.toml` configuration and doesn't interpolate **ANY** environment variables at runtime. +### Requirements and limitations + +- This feature requires GitLab Runner **1.8** or higher. +- For GitLab Runner versions **>= 0.6, <1.8** there was a partial + support for using private registries, which required manual configuration + of credentials on runner's host. We recommend to upgrade your Runner to + at least version **1.8** if you want to use private registries. +- Not available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html), + follow <https://gitlab.com/gitlab-org/gitlab-runner/issues/2673> for + details. + ### Using statically-defined credentials There are two approaches that you can take in order to access a diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index ad07c662965..126e12e460f 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -35,10 +35,9 @@ get out of WIP status or resolve merge conflicts as soon as possible. ## Requirements and limitations -Pipelines for merged results require: +Pipelines for merged results require a [GitLab Runner][runner] 11.9 or newer. -- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. +[runner]: https://gitlab.com/gitlab-org/gitlab-runner In addition, pipelines for merged results have the following limitations: diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index cb8d383f7d9..61f260cb70d 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -205,5 +205,5 @@ Some features are not implemented yet. For example, support for environments. - `stage` - `allow_failure` - `only` and `except` -- `when` +- `when` (only with `on_success`, `on_failure`, and `always` values) - `extends` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index ed8d0e3bc35..eaa6efc526d 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -377,6 +377,10 @@ This functionality is only available: - For users with at least Developer access. - If the the stage contains [manual actions](#manual-actions-from-pipeline-graphs). +## Most Recent Pipeline + +There's a link to the latest pipeline for the last commit of a given branch at `/project/pipelines/[branch]/latest`. Also, `/project/pipelines/latest` will redirect you to the latest pipeline for the last commit on the project's default branch. + ## Security on protected branches A strict security model is enforced when pipelines are executed on diff --git a/doc/ci/quick_start/img/build_log.png b/doc/ci/quick_start/img/build_log.png Binary files differindex 2bf0992c50e..16698629edc 100644 --- a/doc/ci/quick_start/img/build_log.png +++ b/doc/ci/quick_start/img/build_log.png diff --git a/doc/ci/quick_start/img/builds_status.png b/doc/ci/quick_start/img/builds_status.png Binary files differindex 58978e23978..b4aeeb988d2 100644 --- a/doc/ci/quick_start/img/builds_status.png +++ b/doc/ci/quick_start/img/builds_status.png diff --git a/doc/ci/quick_start/img/pipelines_status.png b/doc/ci/quick_start/img/pipelines_status.png Binary files differindex 06d1559f5d2..39a77a26b25 100644 --- a/doc/ci/quick_start/img/pipelines_status.png +++ b/doc/ci/quick_start/img/pipelines_status.png diff --git a/doc/ci/quick_start/img/runners_activated.png b/doc/ci/quick_start/img/runners_activated.png Binary files differindex cd83c1a7e4c..ac09e1d0137 100644 --- a/doc/ci/quick_start/img/runners_activated.png +++ b/doc/ci/quick_start/img/runners_activated.png diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 8474d4ef66e..269bd5c3428 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -88,7 +88,7 @@ visit the project you want to make the Runner work for in GitLab: ## Registering a group Runner -Creating a group Runner requires Maintainer permissions for the group. To create a +Creating a group Runner requires Owner permissions for the group. To create a group Runner visit the group you want to make the Runner work for in GitLab: 1. Go to **Settings > CI/CD** to obtain the token @@ -124,9 +124,9 @@ To lock/unlock a Runner: ## Assigning a Runner to another project -If you are Maintainer on a project where a specific Runner is assigned to, and the +If you are an Owner on a project where a specific Runner is assigned to, and the Runner is not [locked only to that project](#locking-a-specific-runner-from-being-enabled-for-other-projects), -you can enable the Runner also on any other project where you have Maintainer permissions. +you can enable the Runner also on any other project where you have Owner permissions. To enable/disable a Runner in your project: @@ -250,7 +250,7 @@ When you [register a Runner][register], its default behavior is to **only pick** [tagged jobs](../yaml/README.md#tags). NOTE: **Note:** -Maintainer [permissions](../../user/permissions.md) are required to change the +Owner [permissions](../../user/permissions.md) are required to change the Runner settings. To make a Runner pick untagged jobs: diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index d9f022a7125..b6aebd3bd78 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -76,7 +76,7 @@ to access it. This is where an SSH key pair comes in handy. ## without extra base64 encoding. ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556 ## - - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null + - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - ## ## Create the SSH directory and give it the right permissions diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index d741482b662..5a15b907da0 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -94,7 +94,10 @@ This means that the value of the variable will be hidden in job logs, though it must match certain requirements to do so: - The value must be in a single line. -- The value must only consist of characters from the Base64 alphabet ([RFC4648](https://tools.ietf.org/html/rfc4648)) with the addition of `@` and `:`. +- The value must only consist of characters from the Base64 alphabet (RFC4648). + + [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-ce/issues/63043) + and newer, `@` and `:` are also valid values. - The value must be at least 8 characters long. - The value must not use variables. @@ -509,7 +512,7 @@ Below you can find supported syntax reference: 1. Checking for an empty variable Examples: - + - `$VARIABLE == ""` - `$VARIABLE != ""` (introduced in GitLab 11.11) diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 2be93433b36..89a61b2a9e3 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -518,10 +518,24 @@ Four keys are available: - `changes` - `kubernetes` -If you use multiple keys under `only` or `except`, they act as an AND. The logic is: +If you use multiple keys under `only` or `except`, the keys will be evaluated as a +single conjoined expression. That is: + +- `only:` means "include this job if all of the conditions match". +- `except:` means "exclude this job if any of the conditions match". + +With `only`, individual keys are logically joined by an AND: > (any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active) +`except` is implemented as a negation of this complete expression: + +> NOT((any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active)) + +This, more intuitively, means the keys join by an OR. A functionally equivalent expression: + +> (any of refs) OR (any of variables) OR (any of changes) OR (if kubernetes is active) + #### `only:refs`/`except:refs` > `refs` policy introduced in GitLab 10.0. @@ -1568,7 +1582,7 @@ dashboards. The `license_management` report collects [Licenses](../../user/project/merge_requests/license_management.md) as artifacts. -The collected License Management report will be uploaded to GitLab as an artifact and will +The collected License Compliance report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. @@ -1721,7 +1735,7 @@ This example creates three paths of execution: 1. If `needs:` is set to point to a job that is not instantiated because of `only/except` rules or otherwise does not exist, the job will fail. -1. Note that one day one of the launch, we are temporarily limiting the +1. Note that on day one of the launch, we are temporarily limiting the maximum number of jobs that a single job can need in the `needs:` array. Track our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541) for details on the current limit. @@ -1734,8 +1748,8 @@ This example creates three paths of execution: in the first stage (see [gitlab-ce#65504](https://gitlab.com/gitlab-org/gitlab-ce/issues/65504)). 1. If `needs:` refers to a job that is marked as `parallel:`. the current job will depend on all parallel jobs created. -1. `needs:` is similar to `dependencies:` in that needs to use jobs from - prior stages, this means that it is impossible to create circular +1. `needs:` is similar to `dependencies:` in that it needs to use jobs from + prior stages, meaning it is impossible to create circular dependencies or depend on jobs in the current stage (see [gitlab-ce#65505](https://gitlab.com/gitlab-org/gitlab-ce/issues/65505)). 1. Related to the above, stages must be explicitly defined for all jobs that have the keyword `needs:` or are referred to by one. diff --git a/doc/customization/libravatar.md b/doc/customization/libravatar.md index 1c3bf877fa1..6d96528f760 100644 --- a/doc/customization/libravatar.md +++ b/doc/customization/libravatar.md @@ -14,7 +14,7 @@ server. ## Configuration -In the [gitlab.yml gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set +In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set the configuration options as follows: ### For HTTP @@ -46,7 +46,7 @@ For example, you host a service on `http://libravatar.example.com` and the `http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` -### Omnibus-gitlab example +### Omnibus GitLab example In `/etc/gitlab/gitlab.rb`: diff --git a/doc/customization/system_header_and_footer_messages.md b/doc/customization/system_header_and_footer_messages.md index 15830be4e8a..bd2de3e201c 100644 --- a/doc/customization/system_header_and_footer_messages.md +++ b/doc/customization/system_header_and_footer_messages.md @@ -8,7 +8,7 @@ Navigate to the **Admin** area and go to the **Appearance** page. Under **System header and footer** insert your header message and/or footer message. Both background and font color of the header and footer are customizable. -You can also apply the header and footer messages to gitlab emails, +You can also apply the header and footer messages to GitLab emails, by checking the **Enable header and footer in emails** checkbox. Note that color settings will only be applied within the app interface and not to emails diff --git a/doc/development/README.md b/doc/development/README.md index 6281bb809ff..3912a828dec 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -5,7 +5,7 @@ description: 'Learn how to contribute to GitLab.' # Contributor and Development Docs -## Get started! +## Get started - Set up GitLab's development environment with [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/README.md) - [GitLab contributing guide](contributing/index.md) @@ -65,6 +65,7 @@ description: 'Learn how to contribute to GitLab.' - [Repository mirroring](repository_mirroring.md) - [Git LFS](lfs.md) - [Developing against interacting components or features](interacting_components.md) +- [File uploads](uploads.md) ## Performance guides @@ -116,6 +117,7 @@ description: 'Learn how to contribute to GitLab.' ## Case studies - [Database case study: Filtering by label](filtering_by_label.md) +- [Database case study: Namespaces storage statistics](namespaces_storage_statistics.md) ## Integration guides diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 0866d3baeeb..61576236c96 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -51,7 +51,7 @@ allowed. – <https://github.com/ruby-grape/grape#declared> -### Exclude params from parent namespaces! +### Exclude params from parent namespaces > By default `declared(params)`includes parameters that were defined in all parent namespaces. @@ -64,7 +64,7 @@ In most cases you will want to exclude params from the parent namespaces: declared(params, include_parent_namespaces: false) ``` -### When to use `declared(params)`? +### When to use `declared(params)` You should always use `declared(params)` when you pass the params hash as arguments to a method call. diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 87405bc2fec..5cb2ddf6e52 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -20,7 +20,7 @@ A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a w We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/). -The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. +The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. @@ -511,7 +511,15 @@ To summarize here's the [directory structure of the `git` user home directory](. ps aux | grep '^git' ``` -GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). +GitLab has several components to operate. It requires a persistent database +(PostgreSQL) and redis database, and uses Apache httpd or Nginx to proxypass +Unicorn. All these components should run as different system users to GitLab +(e.g., `postgres`, `redis` and `www-data`, instead of `git`). + +As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server +running on port `8080` by default). Under the GitLab user there are normally 4 +processes: `unicorn_rails master` (1 process), `unicorn_rails worker` +(2 processes), `sidekiq` (1 process). ### Repository access @@ -554,12 +562,9 @@ $ /etc/init.d/nginx Usage: nginx {start|stop|restart|reload|force-reload|status|configtest} ``` -Persistent database (one of the following) +Persistent database ``` -/etc/init.d/mysqld -Usage: /etc/init.d/mysqld {start|stop|status|restart|condrestart|try-restart|reload|force-reload} - $ /etc/init.d/postgresql Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..] ``` @@ -597,11 +602,6 @@ PostgreSQL - `/var/log/postgresql/*` -MySQL - -- `/var/log/mysql/*` -- `/var/log/mysql.*` - ### GitLab specific config files GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include: diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index 98b8a48abf4..158606aa6a2 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -173,13 +173,13 @@ Now, every time you create an MR for CE and EE: ## How we run the Automatic CE->EE merge at GitLab -At GitLab, we use the [Merge Train](https://gitlab.com/gitlab-org/merge-train) -project to keep our [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee) -repository updated with commits from +At GitLab, we use the [Merge Train](https://gitlab.com/gitlab-org/merge-train) +project to keep our [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee) +repository updated with commits from [gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce). We have a mirror of the [Merge Train](https://gitlab.com/gitlab-org/merge-train) -project [configured](https://ops.gitlab.net/gitlab-org/merge-train) to run an +project [configured](https://ops.gitlab.net/gitlab-org/merge-train) to run an automatic CE->EE merge job every twenty minutes as a scheduled CI job. The [configured](https://ops.gitlab.net/gitlab-org/merge-train) Merge Train project is only accessible to authorized GitLab staff. @@ -200,7 +200,7 @@ code. ### Why merge automatically? As we work towards continuous deployments and a single repository for both CE -and EE, we need to first make sure that all CE changes make their way into CE as +and EE, we need to first make sure that all CE changes make their way into EE as fast as possible. Past experiences and data have shown that periodic CE to EE merge requests do not scale, and often take a very long time to complete. For example, [in this diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index a38794c49af..b2e3ef7bf63 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -60,37 +60,18 @@ Subject labels are always all-lowercase. ## Team labels -**Important**: Most of the team labels will be soon deprecated in favor of [Group labels](#group-labels). +**Important**: Most of the historical team labels (e.g. Manage, Plan etc.) are +now deprecated in favor of [Group labels](#group-labels) and [Stage labels](#stage-labels). Team labels specify what team is responsible for this issue. Assigning a team label makes sure issues get the attention of the appropriate people. -The team labels planned for deprecation are: - -- ~Configure -- ~Create -- ~Defend -- ~Distribution -- ~Ecosystem -- ~Geo -- ~Gitaly -- ~Growth -- ~Manage -- ~Memory -- ~Monitor -- ~Plan -- ~Release -- ~Secure -- ~Verify - -The following team labels are **true** teams per our [organization structure](https://about.gitlab.com/company/team/structure/#organizational-structure) which will remain post deprecation. +The current team labels are: - ~Delivery - ~Documentation - -The descriptions on the [labels page](https://gitlab.com/gitlab-org/gitlab-ce/-/labels) explain what falls under the -responsibility of each team. +- ~Quality Team labels are always capitalized so that they show up as the first label for any issue. @@ -120,13 +101,13 @@ and thus are mutually exclusive. You can find the groups listed in the [Product Stages, Groups, and Categories][product-categories] page. -We use the term group to map down product requirements from our product stages. +We use the term group to map down product requirements from our product stages. As a team needs some way to collect the work their members are planning to be assigned to, we use the `~group::` labels to do so. Normally there is a 1:1 relationship between Stage labels and Group labels. In the spirit of "Everyone can contribute", any issue can be picked up by any group, depending on current priorities. For example, an issue labeled ~"devops::create" may be picked up by the ~"group::access" group. -We also use stage and group labels to help quantify our [throughput](https://about.gitlab.com/handbook/engineering/management/throughput). +We also use stage and group labels to help quantify our [throughput](https://about.gitlab.com/handbook/engineering/management/throughput). Please read [Stage and Group labels in Throughtput](https://about.gitlab.com/handbook/engineering/management/throughput/#stage-and-group-labels-in-throughput) for more information on how the labels are used in this context. [structure-groups]: https://about.gitlab.com/company/team/structure/#groups diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index bfce7488a8d..4776c8348d4 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -179,4 +179,3 @@ By default, the Jaeger search UI is available at <http://localhost:16686/search> TIP: **Tip:** Don't forget that you will need to generate traces by using the application before they appear in the Jaeger UI. - diff --git a/doc/development/documentation/feature-change-workflow.md b/doc/development/documentation/feature-change-workflow.md index ac93ada5a4b..00c76fe0f1b 100644 --- a/doc/development/documentation/feature-change-workflow.md +++ b/doc/development/documentation/feature-change-workflow.md @@ -69,7 +69,7 @@ To follow a consistent workflow every month, documentation changes involve the Product Managers, the developer who shipped the feature, and the technical writer for the DevOps stage. Each role is described below. -The Documentation items in the GitLab CE/EE [Feature Proposal issue template](https://gitlab.com/gitlab-org/gitlab-ce/raw/template-improvements-for-documentation/.gitlab/issue_templates/Feature%20proposal.md) +The Documentation items in the GitLab CE/EE [Feature Proposal issue template](https://gitlab.com/gitlab-org/gitlab-ce/raw/master/.gitlab/issue_templates/Feature%20proposal.md) and default merge request template will assist you with following this process. ### Product Manager role diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 2217dedccd3..a732a94b7c4 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -945,7 +945,7 @@ export default { - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin -##### Example: +##### Example ```javascript import mixin from 'ee_else_ce/path/mixin'; @@ -976,7 +976,7 @@ For regular JS files, the approach is similar. 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: -#### Example: +#### Example ```javascript import eeCode from 'ee_else_ce/ee_code'; diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 635895051bc..090e5235619 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -148,26 +148,49 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/ - Searches can have their own analyzers. Remember to check when editing analyzers - `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches -## Architecture +## Zero downtime reindexing with multiple indices -GitLab uses `elasticsearch-rails` for handling communication with Elasticsearch server. However, in order to achieve zero-downtime deployment during schema changes, an extra abstraction layer is built to allow: +Currently GitLab can only handle a single version of setting. Any setting/schema changes would require reindexing everything from scratch. Since reindexing can take a long time, this can cause search functionality downtime. -* Indexing (writes) to multiple indexes, with different mappings -* Switching to different index for searches (reads) on the fly +To avoid downtime, GitLab is working to support multiple indices that +can function at the same time. Whenever the schema changes, the admin +will be able to create a new index and reindex to it, while searches +continue to go to the older, stable index. Any data updates will be +forwarded to both indices. Once the new index is ready, an admin can +mark it active, which will direct all searches to it, and remove the old +index. -Currently we are on the process of migrating models to this new design (e.g. `Snippet`), and it is hardwired to work with a single version for now. +This is also helpful for migrating to new servers, e.g. moving to/from AWS. -Traditionally, `elasticsearch-rails` provides class and instance level `__elasticsearch__` proxy methods. If you call `Issue.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`, and if you call `Issue.first.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. These proxy objects would talk to Elasticsearch server directly. +Currently we are on the process of migrating to this new design. Everything is hardwired to work with one single version for now. -In the new design, `__elasticsearch__` instead represents one extra layer of proxy. It would keep multiple versions of the actual proxy objects, and it would forward read and write calls to the proxy of the intended version. +### Architecture -The `elasticsearch-rails`'s way of specifying each model's mappings and other settings is to create a module for the model to include. However in the new design, each model would have its own corresponding subclassed proxy object, where the settings reside in. For example, snippet related setting in the past reside in `SnippetsSearch` module, but in the new design would reside in `SnippetClassProxy` (which is a subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy`). This reduces namespace pollution in model classes. +The traditional setup, provided by `elasticsearch-rails`, is to communicate through its internal proxy classes. Developers would write model-specific logic in a module for the model to include in (e.g. `SnippetsSearch`). The `__elasticsearch__` methods would return a proxy object, e.g.: + +- `Issue.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy` +- `Issue.first.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. + +These proxy objects would talk to Elasticsearch server directly (see top half of the diagram). + + + +In the planned new design, each model would have a pair of corresponding subclassed proxy objects, in which model-specific logic is located. For example, `Snippet` would have `SnippetClassProxy` and `SnippetInstanceProxy` (being subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy` and `Elasticsearch::Model::Proxy::InstanceMethodsProxy`, respectively). + +`__elasticsearch__` would represent another layer of proxy object, keeping track of multiple actual proxy objects. It would forward method calls to the appropriate index. For example: + +- `model.__elasticsearch__.search` would be forwarded to the one stable index, since it is a read operation. +- `model.__elasticsearch__.update_document` would be forwarded to all indices, to keep all indices up-to-date. The global configurations per version are now in the `Elastic::(Version)::Config` class. You can change mappings there. ### Creating new version of schema -Currently GitLab would still work with a single version of setting. Once it is implemented, multiple versions of setting can exists in different folders (e.g. `ee/lib/elastic/v12p1` and `ee/lib/elastic/v12p3`). To keep a continuous git history, the latest version lives under the `/latest` folder, but is aliased as the latest version. +NOTE: **Note:** this is not applicable yet as multiple indices functionality is not fully implemented. + +Folders like `ee/lib/elastic/v12p1` contain snapshots of search logic from different versions. To keep a continuous git history, the latest version lives under `ee/lib/elastic/latest`, but its classes are aliased under an actual version (e.g. `ee/lib/elastic/v12p3`). When referencing these classes, never use the `Latest` namespace directly, but use the actual version (e.g. `V12p3`). + +The version name basically follows GitLab's release version. If setting is changed in 12.3, we will create a new namespace called `V12p3` (p stands for "point"). Raise an issue if there is a need to name a version differently. If the current version is `v12p1`, and we need to create a new version for `v12p3`, the steps are as follows: @@ -176,7 +199,7 @@ If the current version is `v12p1`, and we need to create a new version for `v12p 1. Delete `v12p1` folder 1. Copy the entire folder of `latest` as `v12p1` 1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1` -1. Make changes to `Latest` as needed +1. Make changes to files under the `latest` folder as needed ## Troubleshooting diff --git a/doc/development/emails.md b/doc/development/emails.md index e6af075a282..edec0f86989 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -5,6 +5,10 @@ To view rendered emails "sent" in your development instance, visit [`/rails/letter_opener`](http://localhost:3000/rails/letter_opener). +Please note that [S/MIME signed](../administration/smime_signing_email.md) emails +[cannot be currently previewed](https://github.com/fgrehm/letter_opener_web/issues/96) with +`letter_opener`. + ## Mailer previews Rails provides a way to preview our mailer templates in HTML and plaintext using diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index 09b4a3c3d96..6e7cf523f36 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -38,7 +38,7 @@ Advantages over [`spyOn()`]: - no need to create response objects - does not allow call through (which we want to avoid) -- simple API to test error cases +- simple API to test error cases - provides `replyOnce()` to allow for different responses We have also decided against using [axios interceptors] because they are not suitable for mocking. diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index d3fa350b847..125b11afcd0 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -21,31 +21,31 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. **Never Ever EVER** disable eslint globally for a file ```javascript - // bad - /* eslint-disable */ + // bad + /* eslint-disable */ - // better - /* eslint-disable some-rule, some-other-rule */ + // better + /* eslint-disable some-rule, some-other-rule */ - // best - // nothing :) + // best + // nothing :) ``` 1. If you do need to disable a rule for a single violation, try to do it as locally as possible ```javascript - // bad - /* eslint-disable no-new */ + // bad + /* eslint-disable no-new */ - import Foo from 'foo'; + import Foo from 'foo'; - new Foo(); + new Foo(); - // better - import Foo from 'foo'; + // better + import Foo from 'foo'; - // eslint-disable-next-line no-new - new Foo(); + // eslint-disable-next-line no-new + new Foo(); ``` 1. There are few rules that we need to disable due to technical debt. Which are: @@ -56,16 +56,16 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ followed by any global declarations, then a blank newline prior to any imports or code. ```javascript - // bad - /* global Foo */ - /* eslint-disable no-new */ - import Bar from './bar'; + // bad + /* global Foo */ + /* eslint-disable no-new */ + import Bar from './bar'; - // good - /* eslint-disable no-new */ - /* global Foo */ + // good + /* eslint-disable no-new */ + /* global Foo */ - import Bar from './bar'; + import Bar from './bar'; ``` 1. **Never** disable the `no-undef` rule. Declare globals with `/* global Foo */` instead. @@ -73,23 +73,23 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. When declaring multiple globals, always use one `/* global [name] */` line per variable. ```javascript - // bad - /* globals Flash, Cookies, jQuery */ + // bad + /* globals Flash, Cookies, jQuery */ - // good - /* global Flash */ - /* global Cookies */ - /* global jQuery */ + // good + /* global Flash */ + /* global Cookies */ + /* global jQuery */ ``` 1. Use up to 3 parameters for a function or class. If you need more accept an Object instead. ```javascript - // bad - fn(p1, p2, p3, p4) {} + // bad + fn(p1, p2, p3, p4) {} - // good - fn(options) {} + // good + fn(options) {} ``` #### Modules, Imports, and Exports @@ -97,32 +97,32 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. Use ES module syntax to import modules ```javascript - // bad - const SomeClass = require('some_class'); + // bad + const SomeClass = require('some_class'); - // good - import SomeClass from 'some_class'; + // good + import SomeClass from 'some_class'; - // bad - module.exports = SomeClass; + // bad + module.exports = SomeClass; - // good - export default SomeClass; + // good + export default SomeClass; ``` Import statements are following usual naming guidelines, for example object literals use camel case: ```javascript - // some_object file - export default { - key: 'value', - }; + // some_object file + export default { + key: 'value', + }; - // bad - import ObjectLiteral from 'some_object'; + // bad + import ObjectLiteral from 'some_object'; - // good - import objectLiteral from 'some_object'; + // good + import objectLiteral from 'some_object'; ``` 1. Relative paths: when importing a module in the same directory, a child @@ -171,22 +171,22 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. Avoid adding to the global namespace. ```javascript - // bad - window.MyClass = class { /* ... */ }; + // bad + window.MyClass = class { /* ... */ }; - // good - export default class MyClass { /* ... */ } + // good + export default class MyClass { /* ... */ } ``` 1. Side effects are forbidden in any script which contains export ```javascript - // bad - export default class MyClass { /* ... */ } + // bad + export default class MyClass { /* ... */ } - document.addEventListener("DOMContentLoaded", function(event) { - new MyClass(); - } + document.addEventListener("DOMContentLoaded", function(event) { + new MyClass(); + } ``` #### Data Mutation and Pure functions @@ -257,17 +257,17 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ `.reduce` or `.filter` ```javascript - const users = [ { name: 'Foo' }, { name: 'Bar' } ]; + const users = [ { name: 'Foo' }, { name: 'Bar' } ]; - // bad - users.forEach((user, index) => { - user.id = index; - }); + // bad + users.forEach((user, index) => { + user.id = index; + }); - // good - const usersWithId = users.map((user, index) => { - return Object.assign({}, user, { id: index }); - }); + // good + const usersWithId = users.map((user, index) => { + return Object.assign({}, user, { id: index }); + }); ``` #### Parse Strings into Numbers @@ -275,14 +275,14 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. `parseInt()` is preferable over `Number()` or `+` ```javascript - // bad - +'10' // 10 + // bad + +'10' // 10 - // good - Number('10') // 10 + // good + Number('10') // 10 - // better - parseInt('10', 10); + // better + parseInt('10', 10); ``` #### CSS classes used for JavaScript @@ -290,15 +290,15 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. If the class is being used in Javascript it needs to be prepend with `js-` ```html - // bad - <button class="add-user"> - Add User - </button> + // bad + <button class="add-user"> + Add User + </button> - // good - <button class="js-add-user"> - Add User - </button> + // good + <button class="js-add-user"> + Add User + </button> ``` ### Vue.js @@ -315,41 +315,41 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. Use a function in the bundle file to instantiate the Vue component: ```javascript - // bad - class { - init() { - new Component({}) - } + // bad + class { + init() { + new Component({}) } + } - // good - document.addEventListener('DOMContentLoaded', () => new Vue({ - el: '#element', - components: { - componentName - }, - render: createElement => createElement('component-name'), - })); + // good + document.addEventListener('DOMContentLoaded', () => new Vue({ + el: '#element', + components: { + componentName + }, + render: createElement => createElement('component-name'), + })); ``` 1. Do not use a singleton for the service or the store ```javascript - // bad - class Store { - constructor() { - if (!this.prototype.singleton) { - // do something - } + // bad + class Store { + constructor() { + if (!this.prototype.singleton) { + // do something } } + } - // good - class Store { - constructor() { - // do something - } + // good + class Store { + constructor() { + // do something } + } ``` 1. Use `.vue` for Vue templates. Do not use `%template` in HAML. @@ -360,36 +360,36 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. **Reference Naming**: Use PascalCase for their instances: ```javascript - // bad - import cardBoard from 'cardBoard.vue' + // bad + import cardBoard from 'cardBoard.vue' - components: { - cardBoard, - }; + components: { + cardBoard, + }; - // good - import CardBoard from 'cardBoard.vue' + // good + import CardBoard from 'cardBoard.vue' - components: { - CardBoard, - }; + components: { + CardBoard, + }; ``` 1. **Props Naming:** Avoid using DOM component prop names. 1. **Props Naming:** Use kebab-case instead of camelCase to provide props in templates. ```javascript - // bad - <component class="btn"> + // bad + <component class="btn"> - // good - <component css-class="btn"> + // good + <component css-class="btn"> - // bad - <component myProp="prop" /> + // bad + <component myProp="prop" /> - // good - <component my-prop="prop" /> + // good + <component my-prop="prop" /> ``` [#34371]: https://gitlab.com/gitlab-org/gitlab-ce/issues/34371 @@ -401,37 +401,37 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. With more than one attribute, all attributes should be on a new line: ```javascript - // bad - <component v-if="bar" - param="baz" /> + // bad + <component v-if="bar" + param="baz" /> - <button class="btn">Click me</button> + <button class="btn">Click me</button> - // good - <component - v-if="bar" - param="baz" - /> + // good + <component + v-if="bar" + param="baz" + /> - <button class="btn"> - Click me - </button> + <button class="btn"> + Click me + </button> ``` 1. The tag can be inline if there is only one attribute: ```javascript - // good - <component bar="bar" /> + // good + <component bar="bar" /> - // good - <component - bar="bar" - /> + // good + <component + bar="bar" + /> - // bad - <component - bar="bar" /> + // bad + <component + bar="bar" /> ``` #### Quotes @@ -439,15 +439,15 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. Always use double quotes `"` inside templates and single quotes `'` for all other JS. ```javascript - // bad - template: ` - <button :class='style'>Button</button> - ` + // bad + template: ` + <button :class='style'>Button</button> + ` - // good - template: ` - <button :class="style">Button</button> - ` + // good + template: ` + <button :class="style">Button</button> + ` ``` #### Props @@ -455,37 +455,37 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. Props should be declared as an object ```javascript - // bad - props: ['foo'] - - // good - props: { - foo: { - type: String, - required: false, - default: 'bar' - } + // bad + props: ['foo'] + + // good + props: { + foo: { + type: String, + required: false, + default: 'bar' } + } ``` 1. Required key should always be provided when declaring a prop ```javascript - // bad - props: { - foo: { - type: String, - } + // bad + props: { + foo: { + type: String, } + } - // good - props: { - foo: { - type: String, - required: false, - default: 'bar' - } + // good + props: { + foo: { + type: String, + required: false, + default: 'bar' } + } ``` 1. Default key should be provided if the prop is not required. @@ -493,30 +493,30 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. On those a default key should not be provided. ```javascript - // good - props: { - foo: { - type: String, - required: false, - } + // good + props: { + foo: { + type: String, + required: false, } + } - // good - props: { - foo: { - type: String, - required: false, - default: 'bar' - } + // good + props: { + foo: { + type: String, + required: false, + default: 'bar' } + } - // good - props: { - foo: { - type: String, - required: true - } + // good + props: { + foo: { + type: String, + required: true } + } ``` #### Data @@ -524,17 +524,17 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. `data` method should always be a function ```javascript - // bad - data: { - foo: 'foo' - } + // bad + data: { + foo: 'foo' + } - // good - data() { - return { - foo: 'foo' - }; - } + // good + data() { + return { + foo: 'foo' + }; + } ``` #### Directives @@ -542,31 +542,31 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. Shorthand `@` is preferable over `v-on` ```javascript - // bad - <component v-on:click="eventHandler"/> + // bad + <component v-on:click="eventHandler"/> - // good - <component @click="eventHandler"/> + // good + <component @click="eventHandler"/> ``` 1. Shorthand `:` is preferable over `v-bind` ```javascript - // bad - <component v-bind:class="btn"/> + // bad + <component v-bind:class="btn"/> - // good - <component :class="btn"/> + // good + <component :class="btn"/> ``` 1. Shorthand `#` is preferable over `v-slot` ```javascript - // bad - <template v-slot:header></template> + // bad + <template v-slot:header></template> - // good - <template #header></template> + // good + <template #header></template> ``` #### Closing tags @@ -574,11 +574,11 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. Prefer self closing component tags ```javascript - // bad - <component></component> + // bad + <component></component> - // good - <component /> + // good + <component /> ``` #### Ordering @@ -610,48 +610,48 @@ When using `v-for` you need to provide a *unique* `:key` attribute for each item 1. If the elements of the array being iterated have an unique `id` it is advised to use it: ```html - <div - v-for="item in items" - :key="item.id" - > - <!-- content --> - </div> + <div + v-for="item in items" + :key="item.id" + > + <!-- content --> + </div> ``` 1. When the elements being iterated don't have a unique id, you can use the array index as the `:key` attribute ```html - <div - v-for="(item, index) in items" - :key="index" - > - <!-- content --> - </div> + <div + v-for="(item, index) in items" + :key="index" + > + <!-- content --> + </div> ``` 1. When using `v-for` with `template` and there is more than one child element, the `:key` values must be unique. It's advised to use `kebab-case` namespaces. ```html - <template v-for="(item, index) in items"> - <span :key="`span-${index}`"></span> - <button :key="`button-${index}`"></button> - </template> + <template v-for="(item, index) in items"> + <span :key="`span-${index}`"></span> + <button :key="`button-${index}`"></button> + </template> ``` 1. When dealing with nested `v-for` use the same guidelines as above. ```html - <div - v-for="item in items" - :key="item.id" + <div + v-for="item in items" + :key="item.id" + > + <span + v-for="element in array" + :key="element.id" > - <span - v-for="element in array" - :key="element.id" - > - <!-- content --> - </span> - </div> + <!-- content --> + </span> + </div> ``` Useful links: @@ -664,19 +664,19 @@ Useful links: 1. Tooltips: Do not rely on `has-tooltip` class name for Vue components ```javascript - // bad - <span - class="has-tooltip" - title="Some tooltip text"> - Text - </span> + // bad + <span + class="has-tooltip" + title="Some tooltip text"> + Text + </span> - // good - <span - v-tooltip - title="Some tooltip text"> - Text - </span> + // good + <span + v-tooltip + title="Some tooltip text"> + Text + </span> ``` 1. Tooltips: When using a tooltip, include the tooltip directive, `./app/assets/javascripts/vue_shared/directives/tooltip.js` @@ -684,13 +684,13 @@ Useful links: 1. Don't change `data-original-title`. ```javascript - // bad - <span data-original-title="tooltip text">Foo</span> + // bad + <span data-original-title="tooltip text">Foo</span> - // good - <span title="tooltip text">Foo</span> + // good + <span title="tooltip text">Foo</span> - $('span').tooltip('_fixTitle'); + $('span').tooltip('_fixTitle'); ``` ### The Javascript/Vue Accord diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 9eeaee4482f..557d3132d71 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -313,7 +313,7 @@ export default { 1. Do not call a mutation directly. Always use an action to commit a mutation. Doing so will keep consistency throughout the application. From Vuex docs: - > why don't we just call store.commit('action') directly? Well, remember that mutations must be synchronous? Actions aren't. We can perform asynchronous operations inside an action. + > Why don't we just call store.commit('action') directly? Well, remember that mutations must be synchronous? Actions aren't. We can perform asynchronous operations inside an action. ```javascript // component.vue diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 475d1c1611e..44af2b020a4 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -2,6 +2,8 @@ We use the [CarrierWave] gem to handle file upload, store and retrieval. +File uploads should be accelerated by workhorse, for details please refer to [uploads development documentation](uploads.md). + There are many places where file uploading is used, according to contexts: - System diff --git a/doc/development/hash_indexes.md b/doc/development/hash_indexes.md index e6c1b3590b1..417ea18e22f 100644 --- a/doc/development/hash_indexes.md +++ b/doc/development/hash_indexes.md @@ -1,6 +1,6 @@ # Hash Indexes -Both PostgreSQL and MySQL support hash indexes besides the regular btree +PostgreSQL supports hash indexes besides the regular btree indexes. Hash indexes however are to be avoided at all costs. While they may _sometimes_ provide better performance the cost of rehashing can be very high. More importantly: at least until PostgreSQL 10.0 hash indexes are not diff --git a/doc/development/img/elasticsearch_architecture.svg b/doc/development/img/elasticsearch_architecture.svg new file mode 100644 index 00000000000..2f38f9b04ee --- /dev/null +++ b/doc/development/img/elasticsearch_architecture.svg @@ -0,0 +1 @@ +<svg version="1.2" width="210mm" height="297mm" viewBox="0 0 21000 29700" preserveAspectRatio="xMidYMid" fill-rule="evenodd" stroke-width="28.222" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><defs class="ClipPathGroup"><clipPath id="a" clipPathUnits="userSpaceOnUse"><path d="M0 0h21000v29700H0z"/></clipPath></defs><g class="SlideGroup"><g class="Slide" clip-path="url(#a)"><g class="Page"><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 5575h3051v1651H1975z"/><path fill="#FFF" d="M3500 7200H2000V5600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7200H2000V5600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="6311"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="6785"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1475 3975h4051v3551H1475z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7500H1500V4000h4000v3500H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="1788" y="5048"><tspan>ApplicationSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 4675h8051v701H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v-650h4000"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 5325h8051v1101H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v1050h4000"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 2875h4951v4951H1075z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3550 7800H1100V2900h4900v4900H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="1946" y="3514"><tspan fill="#C9211E">SnippetsSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 12175h3051v1651H1975z"/><path fill="#FFF" d="M3500 13800H2000v-1600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 13800H2000v-1600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="12911"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="13385"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 10775h4951v3251H1075z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000H1100v-3200h4900v3200H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2511" y="11461"><tspan>Application</tspan></tspan><tspan class="TextPosition" x="1933" y="11935"><tspan>VersionedSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v7451H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v7400h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 14075h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 14900h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="14648"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 13075h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 15200h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="13731"><tspan fill="#C9211E">V12p1::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 14575h3051v1851H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 16400H8000v-1800h3000v1800H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="15411"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8429" y="15885"><tspan>ClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 16875h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 17700h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="17448"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 15875h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 18000h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="16531"><tspan fill="#C9211E">V12p2::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 14125h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 15475h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v1450h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v1551H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v1500h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 19975h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 20800h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="20548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 18975h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 21100h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="19631"><tspan fill="#C9211E">V12p1::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 20275h3051v2251H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 22500H8000v-2200h3000v2200H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="21311"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8154" y="21785"><tspan>InstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 22775h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 23600h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="23348"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 21775h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 23900h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="22431"><tspan fill="#C9211E">V12p2::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 20025h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 21375h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v1450h937"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 1600h10697v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="2233"><tspan>Standard elasticsearch-rails setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 9300h7683v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="9933"><tspan>GitLab multi-indices setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3400 21300h4821v1197H3400z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4250" y="21840"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="3651" y="22264"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3380 15400h4821v1197H3380z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4512" y="15940"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="3631" y="16364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 3500h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="10132" y="4040"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="9251" y="4464"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 6400h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="9850" y="6940"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="9251" y="7364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 25175h2051v851H1975z"/><path fill="none" stroke="#999" stroke-width="50" d="M3000 26000H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2634" y="25748"><tspan fill="gray">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 25200h7101v726H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="25710"><tspan>elasticsearch-rails’ internal class</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 26400h8601v1200H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="26910"><tspan>where model-specific logic is</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 26275h2051v851H1975z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3000 27100H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="2613" y="26848"><tspan fill="#C9211E">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4900 17289h5901v2312H4900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="7236" y="17748"><tspan fill="gray">Write operations like </tspan></tspan><tspan class="TextPosition" x="5323" y="18159"><tspan fill="gray">indexing/updating are forwarded </tspan></tspan><tspan class="TextPosition" x="8024" y="18570"><tspan fill="gray">to all instances.</tspan></tspan></tspan><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="5501" y="18981"><tspan fill="gray">Read operations are forwarded </tspan></tspan><tspan class="TextPosition" x="7126" y="19392"><tspan fill="gray">to specified instance.</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 15769h1422v2691h-1422z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1429 0 934-1618 1119-2337"/><path fill="#999" d="M12206 15769l-460 293 267 217 193-510z"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 18429h1528v2862h-1528z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1509 0 970 1782 1200 2526"/><path fill="#999" d="M12312 21290l-227-496-252 235 479 261z"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M1800 24000h7101v807H1800z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="494" font-weight="700"><tspan class="TextPosition" x="2050" y="24574"><tspan>Legend</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 4275h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 5100h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14737" y="4848"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 5975h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 6800h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14462" y="6548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g></g></g></g></svg>
\ No newline at end of file diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index 5f95cf3707c..777d372ec60 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -1,6 +1,6 @@ # Instrumenting Ruby Code -GitLab Performance Monitoring allows instrumenting of both methods and custom +[GitLab Performance Monitoring](../administration/monitoring/performance/index.md) allows instrumenting of both methods and custom blocks of Ruby code. Method instrumentation is the primary form of instrumentation with block-based instrumentation only being used when we want to drill down to specific regions of code within a method. diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index 74d52d808e2..5e6dc8d460a 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -9,8 +9,8 @@ when making _backend_ changes that might involve multiple features or [component ## Uploads -GitLab supports uploads to [object storage]. That means every feature and -change that affects uploads should also be tested against [object storage], +GitLab supports uploads to [object storage]. That means every feature and +change that affects uploads should also be tested against [object storage], which is _not_ enabled by default in [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit). When working on a related feature, make sure to enable and test it diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 8c3408eb6e2..cb4c2d8967b 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -8,4 +8,4 @@ In April 2019, Francisco Javier López hosted a [Deep Dive] on GitLab's [Git LFS [Git LFS]: ../workflow/lfs/manage_large_binaries_with_git_lfs.html [recording on YouTube]: https://www.youtube.com/watch?v=Yyxwcksr0Qc [Google Slides]: https://docs.google.com/presentation/d/1E-aw6-z0rYd0346YhIWE7E9A65zISL9iIMAOq2zaw9E/edit -[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf
\ No newline at end of file +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md new file mode 100644 index 00000000000..b3285de4705 --- /dev/null +++ b/doc/development/namespaces_storage_statistics.md @@ -0,0 +1,178 @@ +# Database case study: Namespaces storage statistics + +## Introduction + +On [Storage and limits management for groups](https://gitlab.com/groups/gitlab-org/-/epics/886), +we want to facilitate a method for easily viewing the amount of +storage consumed by a group, and allow easy management. + +## Proposal + +1. Create a new ActiveRecord model to hold the namespaces' statistics in an aggregated form (only for root namespaces). +1. Refresh the statistics in this model every time a project belonging to this namespace is changed. + +## Problem + +In GitLab, we update the project storage statistics through a +[callback](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/project.rb#L90) +every time the project is saved. + +The summary of those statistics per namespace is then retrieved +by [`Namespaces#with_statistics`](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/namespace.rb#L70) scope. Analyzing this query we noticed that: + +* It takes up to `1.2` seconds for namespaces with over `15k` projects. +* It can't be analyzed with [ChatOps](chatops_on_gitlabcom.md), as it times out. + +Additionally, the pattern that is currently used to update the project statistics +(the callback) doesn't scale adequately. It is currently one of the largest +[database queries transactions on production](https://gitlab.com/gitlab-org/gitlab-ce/issues/62488) +that takes the most time overall. We can't add one more query to it as +it will increase the transaction's length. + +Because of all of the above, we can't apply the same pattern to store +and update the namespaces statistics, as the `namespaces` table is one +of the largest tables on GitLab.com. Therefore we needed to find a performant and +alternative method. + +## Attempts + +### Attempt A: PostgreSQL materialized view + +Model can be updated through a refresh strategy based on a project routes SQL and a [materialized view](https://www.postgresql.org/docs/9.6/rules-materializedviews.html): + +```sql +SELECT split_part("rs".path, '/', 1) as root_path, + COALESCE(SUM(ps.storage_size), 0) AS storage_size, + COALESCE(SUM(ps.repository_size), 0) AS repository_size, + COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, + COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, + COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size +FROM "projects" + INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' + INNER JOIN project_statistics ps ON ps.project_id = projects.id +GROUP BY root_path +``` + +We could then execute the query with: + +```sql +REFRESH MATERIALIZED VIEW root_namespace_storage_statistics; +``` + +While this implied a single query update (and probably a fast one), it has some downsides: + +* Materialized views syntax varies from PostgreSQL and MySQL. While this feature was worked on, MySQL was still supported by GitLab. +* Rails does not have native support for materialized views. We'd need to use a specialized gem to take care of the management of the database views, which implies additional work. + +### Attempt B: An update through a CTE + +Similar to Attempt A: Model update done through a refresh strategy with a [Common Table Expression](https://www.postgresql.org/docs/9.1/queries-with.html) + +```sql +WITH refresh AS ( + SELECT split_part("rs".path, '/', 1) as root_path, + COALESCE(SUM(ps.storage_size), 0) AS storage_size, + COALESCE(SUM(ps.repository_size), 0) AS repository_size, + COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, + COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, + COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size + FROM "projects" + INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' + INNER JOIN project_statistics ps ON ps.project_id = projects.id + GROUP BY root_path) +UPDATE namespace_storage_statistics +SET storage_size = refresh.storage_size, + repository_size = refresh.repository_size, + wiki_size = refresh.wiki_size, + lfs_objects_size = refresh.lfs_objects_size, + build_artifacts_size = refresh.build_artifacts_size, + packages_size = refresh.packages_size +FROM refresh + INNER JOIN routes rs ON rs.path = refresh.root_path AND rs.source_type = 'Namespace' +WHERE namespace_storage_statistics.namespace_id = rs.source_id +``` + +Same benefits and downsides as attempt A. + +### Attempt C: Get rid of the model and store the statistics on Redis + +We could get rid of the model that stores the statistics in aggregated form and instead use a Redis Set. +This would be the [boring solution](https://about.gitlab.com/handbook/values/#boring-solutions) and the fastest one +to implement, as GitLab already includes Redis as part of its [Architecture](architecture.md#redis). + +The downside of this approach is that Redis does not provide the same persistence/consistency guarantees as PostgreSQL, +and this is information we can't afford to lose in a Redis failure. + +### Attempt D: Tag the root namespace and its child namespaces + +Directly relate the root namespace to its child namespaces, so +whenever a namespace is created without a parent, this one is tagged +with the root namespace ID: + +| id | root_id | parent_id +|:---|:--------|:---------- +| 1 | 1 | NULL +| 2 | 1 | 1 +| 3 | 1 | 2 + +To aggregate the statistics inside a namespace we'd execute something like: + +```sql +SELECT COUNT(...) +FROM projects +WHERE namespace_id IN ( + SELECT id + FROM namespaces + WHERE root_id = X +) +``` + +Even though this approach would make aggregating much easier, it has some major downsides: + +* We'd have to migrate **all namespaces** by adding and filling a new column. Because of the size of the table, dealing with time/cost will not be great. The background migration will take approximately `153h`, see <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29772>. +* Background migration has to be shipped one release before, delaying the functionality by another milestone. + +### Attempt E (final): Update the namespace storage statistics in async way + +This approach consists of keep using the incremental statistics updates we currently already have, +but we refresh them through Sidekiq jobs and in different transactions: + +1. Create a second table (`namespace_aggregation_schedules`) with two columns `id` and `namespace_id`. +1. Whenever the statistics of a project changes, insert a row into `namespace_aggregation_schedules` + - We don't insert a new row if there's already one related to the root namespace. + - Keeping in mind the length of the transaction that involves updating `project_statistics`(<https://gitlab.com/gitlab-org/gitlab-ce/issues/62488>), the insertion should be done in a different transaction and through a Sidekiq Job. +1. After inserting the row, we schedule another worker to be executed async at two different moments: + - One enqueued for immediate execution and another one scheduled in `1.5h` hours. + - We only schedule the jobs, if we can obtain a `1.5h` lease on Redis on a key based on the root namespace ID. + - If we can't obtain the lease, it indicates there's another aggregation already in progress, or scheduled in no more than `1.5h`. +1. This worker will: + - Update the root namespace storage statistics by querying all the namespaces through a service. + - Delete the related `namespace_aggregation_schedules` after the update. +1. Another Sidekiq job is also included to traverse any remaining rows on the `namespace_aggregation_schedules` table and schedule jobs for every pending row. + - This job is scheduled with cron to run every night (UTC). + +This implementation has the following benefits: + +* All the updates are done async, so we're not increasing the length of the transactions for `project_statistics`. +* We're doing the update in a single SQL query. +* It is compatible with PostgreSQL and MySQL. +* No background migration required. + +The only downside of this approach is that namespaces' statistics are updated up to `1.5` hours after the change is done, +which means there's a time window in which the statistics are inaccurate. Because we're still not +[enforcing storage limits](https://gitlab.com/gitlab-org/gitlab-ce/issues/30421), this is not a major problem. + +## Conclusion + +Updating the storage statistics asynchronously, was the less problematic and +performant approach of aggregating the root namespaces. + +All the details regarding this use case can be found on: + +* <https://gitlab.com/gitlab-org/gitlab-ce/issues/62214> +* Merge Request with the implementation: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28996> + +Performance of the namespace storage statistics were measured in staging and production (GitLab.com). All results were posted +on <https://gitlab.com/gitlab-org/gitlab-ce/issues/64092>: No problem has been reported so far. diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md index f7ea496d935..e0d413b748b 100644 --- a/doc/development/new_fe_guide/development/testing.md +++ b/doc/development/new_fe_guide/development/testing.md @@ -1,361 +1,6 @@ -# Overview of Frontend Testing +--- +redirect_to: '../../testing_guide/frontend_testing.md' +--- -Tests relevant for frontend development can be found at the following places: +This document was moved to [another location](../../testing_guide/frontend_testing.md). -- `spec/javascripts/` which are run by Karma (command: `yarn karma`) and contain - - [frontend unit tests](#frontend-unit-tests) - - [frontend component tests](#frontend-component-tests) - - [frontend integration tests](#frontend-integration-tests) -- `spec/frontend/` which are run by Jest (command: `yarn jest`) and contain - - [frontend unit tests](#frontend-unit-tests) - - [frontend component tests](#frontend-component-tests) - - [frontend integration tests](#frontend-integration-tests) -- `spec/features/` which are run by RSpec and contain - - [feature tests](#feature-tests) - -All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-ce/issues/52483)). - -In addition there were feature tests in `features/` run by Spinach in the past. -These have been removed from our codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-ce/issues/23036)). - -See also: - -- [Old testing guide](../../testing_guide/frontend_testing.html). -- [Notes on testing Vue components](../../fe_guide/vue.html#testing-vue-components). - -## Frontend unit tests - -Unit tests are on the lowest abstraction level and typically test functionality that is not directly perceivable by a user. - -### When to use unit tests - -<details> - <summary>exported functions and classes</summary> - Anything that is exported can be reused at various places in a way you have no control over. - Therefore it is necessary to document the expected behavior of the public interface with tests. -</details> - -<details> - <summary>Vuex actions</summary> - Any Vuex action needs to work in a consistent way independent of the component it is triggered from. -</details> - -<details> - <summary>Vuex mutations</summary> - For complex Vuex mutations it helps to identify the source of a problem by separating the tests from other parts of the Vuex store. -</details> - -### When *not* to use unit tests - -<details> - <summary>non-exported functions or classes</summary> - Anything that is not exported from a module can be considered private or an implementation detail and doesn't need to be tested. -</details> - -<details> - <summary>constants</summary> - Testing the value of a constant would mean to copy it. - This results in extra effort without additional confidence that the value is correct. -</details> - -<details> - <summary>Vue components</summary> - Computed properties, methods, and lifecycle hooks can be considered an implementation detail of components and don't need to be tested. - They are implicitly covered by component tests. - The <a href="https://vue-test-utils.vuejs.org/guides/#getting-started">official Vue guidelines</a> suggest the same. -</details> - -### What to mock in unit tests - -<details> - <summary>state of the class under test</summary> - Modifying the state of the class under test directly rather than using methods of the class avoids side-effects in test setup. -</details> - -<details> - <summary>other exported classes</summary> - Every class needs to be tested in isolation to prevent test scenarios from growing exponentially. -</details> - -<details> - <summary>single DOM elements if passed as parameters</summary> - For tests that only operate on single DOM elements rather than a whole page, creating these elements is cheaper than loading a whole HTML fixture. -</details> - -<details> - <summary>all server requests</summary> - When running frontend unit tests, the backend may not be reachable. - Therefore all outgoing requests need to be mocked. -</details> - -<details> - <summary>asynchronous background operations</summary> - Background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. -</details> - -### What *not* to mock in unit tests - -<details> - <summary>non-exported functions or classes</summary> - Everything that is not exported can be considered private to the module and will be implicitly tested via the exported classes / functions. -</details> - -<details> - <summary>methods of the class under test</summary> - By mocking methods of the class under test, the mocks will be tested and not the real methods. -</details> - -<details> - <summary>utility functions (pure functions, or those that only modify parameters)</summary> - If a function has no side effects because it has no state, it is safe to not mock it in tests. -</details> - -<details> - <summary>full HTML pages</summary> - Loading the HTML of a full page slows down tests, so it should be avoided in unit tests. -</details> - -## Frontend component tests - -Component tests cover the state of a single component that is perceivable by a user depending on external signals such as user input, events fired from other components, or application state. - -### When to use component tests - -- Vue components - -### When *not* to use component tests - -<details> - <summary>Vue applications</summary> - Vue applications may contain many components. - Testing them on a component level requires too much effort. - Therefore they are tested on frontend integration level. -</details> - -<details> - <summary>HAML templates</summary> - HAML templates contain only Markup and no frontend-side logic. - Therefore they are not complete components. -</details> - -### What to mock in component tests - -<details> - <summary>DOM</summary> - Operating on the real DOM is significantly slower than on the virtual DOM. -</details> - -<details> - <summary>properties and state of the component under test</summary> - Similarly to testing classes, modifying the properties directly (rather than relying on methods of the component) avoids side-effects. -</details> - -<details> - <summary>Vuex store</summary> - To avoid side effects and keep component tests simple, Vuex stores are replaced with mocks. -</details> - -<details> - <summary>all server requests</summary> - Similar to unit tests, when running component tests, the backend may not be reachable. - Therefore all outgoing requests need to be mocked. -</details> - -<details> - <summary>asynchronous background operations</summary> - Similar to unit tests, background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. -</details> - -<details> - <summary>child components</summary> - Every component is tested individually, so child components are mocked. - See also <a href="https://vue-test-utils.vuejs.org/api/#shallowmount">shallowMount()</a> -</details> - -### What *not* to mock in component tests - -<details> - <summary>methods or computed properties of the component under test</summary> - By mocking part of the component under test, the mocks will be tested and not the real component. -</details> - -<details> - <summary>functions and classes independent from Vue</summary> - All plain JavaScript code is already covered by unit tests and needs not to be mocked in component tests. -</details> - -## Frontend integration tests - -Integration tests cover the interaction between all components on a single page. -Their abstraction level is comparable to how a user would interact with the UI. - -### When to use integration tests - -<details> - <summary>page bundles (<code>index.js</code> files in <code>app/assets/javascripts/pages/</code>)</summary> - Testing the page bundles ensures the corresponding frontend components integrate well. -</details> - -<details> - <summary>Vue applications outside of page bundles</summary> - Testing Vue applications as a whole ensures the corresponding frontend components integrate well. -</details> - -### What to mock in integration tests - -<details> - <summary>HAML views (use fixtures instead)</summary> - Rendering HAML views requires a Rails environment including a running database which we cannot rely on in frontend tests. -</details> - -<details> - <summary>all server requests</summary> - Similar to unit and component tests, when running component tests, the backend may not be reachable. - Therefore all outgoing requests need to be mocked. -</details> - -<details> - <summary>asynchronous background operations that are not perceivable on the page</summary> - Background operations that affect the page need to be tested on this level. - All other background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. -</details> - -### What *not* to mock in integration tests - -<details> - <summary>DOM</summary> - Testing on the real DOM ensures our components work in the environment they are meant for. - Part of this will be delegated to <a href="https://gitlab.com/gitlab-org/quality/team-tasks/issues/45">cross-browser testing</a>. -</details> - -<details> - <summary>properties or state of components</summary> - On this level, all tests can only perform actions a user would do. - For example to change the state of a component, a click event would be fired. -</details> - -<details> - <summary>Vuex stores</summary> - When testing the frontend code of a page as a whole, the interaction between Vue components and Vuex stores is covered as well. -</details> - -## Feature tests - -In contrast to [frontend integration tests](#frontend-integration-tests), feature tests make requests against the real backend instead of using fixtures. -This also implies that database queries are executed which makes this category significantly slower. - -See also the [RSpec testing guidelines](../../testing_guide/best_practices.md#rspec). - -### When to use feature tests - -- use cases that require a backend and cannot be tested using fixtures -- behavior that is not part of a page bundle but defined globally - -### Relevant notes - -A `:js` flag is added to the test to make sure the full environment is loaded. - -``` -scenario 'successfully', :js do - sign_in(create(:admin)) -end -``` - -The steps of each test are written using capybara methods ([documentation](https://www.rubydoc.info/gems/capybara)). - -Bear in mind <abbr title="XMLHttpRequest">XHR</abbr> calls might require you to use `wait_for_requests` in between steps, like so: - -```rspec -find('.form-control').native.send_keys(:enter) - -wait_for_requests - -expect(page).not_to have_selector('.card') -``` - -## Test helpers - -### Vuex Helper: `testAction` - -We have a helper available to make testing actions easier, as per [official documentation](https://vuex.vuejs.org/guide/testing.html): - -``` -testAction( - actions.actionName, // action - { }, // params to be passed to action - state, // state - [ - { type: types.MUTATION}, - { type: types.MUTATION_1, payload: {}}, - ], // mutations committed - [ - { type: 'actionName', payload: {}}, - { type: 'actionName1', payload: {}}, - ] // actions dispatched - done, -); -``` - -Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/javascripts/ide/stores/actions_spec.js). - -### Vue Helper: `mountComponent` - -To make mounting a Vue component easier and more readable, we have a few helpers available in `spec/helpers/vue_mount_component_helper`. - -- `createComponentWithStore` -- `mountComponentWithStore` - -Examples of usage: - -``` -beforeEach(() => { - vm = createComponentWithStore(Component, store); - - vm.$store.state.currentBranchId = 'master'; - - vm.$mount(); -}, -``` - -``` -beforeEach(() => { - vm = mountComponentWithStore(Component, { - el: '#dummy-element', - store, - props: { badge }, - }); -}, -``` - -Don't forget to clean up: - -``` -afterEach(() => { - vm.$destroy(); -}); -``` - -## Testing with older browsers - -Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or Browserstack using the following steps: - -### Browserstack - -[Browserstack](https://www.browserstack.com/) allows you to test more than 1200 mobile devices and browsers. -You can use it directly through the [live app](https://www.browserstack.com/live) or you can install the [chrome extension](https://chrome.google.com/webstore/detail/browserstack/nkihdmlheodkdfojglpcjjmioefjahjb) for easy access. -You can find the credentials on 1Password, under `frontendteam@gitlab.com`. - -### Firefox - -#### macOS - -You can download any older version of Firefox from the releases FTP server, <https://ftp.mozilla.org/pub/firefox/releases/> - -1. From the website, select a version, in this case `50.0.1`. -1. Go to the mac folder. -1. Select your preferred language, you will find the dmg package inside, download it. -1. Drag and drop the application to any other folder but the `Applications` folder. -1. Rename the application to something like `Firefox_Old`. -1. Move the application to the `Applications` folder. -1. Open up a terminal and run `/Applications/Firefox_Old.app/Contents/MacOS/firefox-bin -profilemanager` to create a new profile specific to that Firefox version. -1. Once the profile has been created, quit the app, and run it again like normal. You now have a working older Firefox version. diff --git a/doc/development/new_fe_guide/modules/dirty_submit.md b/doc/development/new_fe_guide/modules/dirty_submit.md index 6c03958b463..217743ea395 100644 --- a/doc/development/new_fe_guide/modules/dirty_submit.md +++ b/doc/development/new_fe_guide/modules/dirty_submit.md @@ -1,7 +1,6 @@ # Dirty Submit -> [Introduced][ce-21115] in GitLab 11.3. -> [dirty_submit][dirty-submit] +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21115) in GitLab 11.3. ## Summary @@ -9,6 +8,9 @@ Prevent submitting forms with no changes. Currently handles `input`, `textarea` and `select` elements. +Also, see [the code](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/dirty_submit/) +within the GitLab project. + ## Usage ```js @@ -18,6 +20,3 @@ new DirtySubmitForm(document.querySelector('form')); // or new DirtySubmitForm(document.querySelectorAll('form')); ``` - -[ce-21115]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21115 -[dirty-submit]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/dirty_submit/
\ No newline at end of file diff --git a/doc/development/new_fe_guide/style/javascript.md b/doc/development/new_fe_guide/style/javascript.md index 802ebd12d92..b742d567f41 100644 --- a/doc/development/new_fe_guide/style/javascript.md +++ b/doc/development/new_fe_guide/style/javascript.md @@ -192,4 +192,4 @@ rules only if you are invoking/instantiating existing code modules. - [class-method-use-this](http://eslint.org/docs/rules/class-methods-use-this) > Note: Disable these rules on a per line basis. This makes it easier to refactor - in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`. +> in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`. diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index a6b60149ea4..3787e2ef187 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -36,6 +36,13 @@ it "avoids N+1 database queries" do end ``` +## Use request specs instead of controller specs + +Use a [request spec](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/spec/requests) when writing a N+1 test on the controller level. + +Controller specs should not be used to write N+1 tests as the controller is only initialized once per example. +This could lead to false successes where subsequent "requests" could have queries reduced (e.g. because of memoization). + ## Finding the source of the query It may be useful to identify the source of the queries by looking at the call backtrace. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 67f36eb1ab4..e9d6cfe00b2 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -9,7 +9,7 @@ bundle exec rake setup ``` The `setup` task is an alias for `gitlab:setup`. -This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database. +This tasks calls `db:reset` to create the database, and calls `db:seed_fu` to seed the database. Note: `db:setup` calls `db:seed` but this does nothing. ### Seeding issues for all or a given project @@ -216,4 +216,3 @@ bundle exec rake routes Since these take some time to create, it's often helpful to save the output to a file for quick reference. - diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index f8c33ff2b85..dc51bf80e92 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -8,4 +8,4 @@ In December 2018, Tiago Botelho hosted a [Deep Dive] on GitLab's [Pull Repositor [Pull Repository Mirroring functionality]: ../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter [recording on YouTube]: https://www.youtube.com/watch?v=sSZq0fpdY-Y [Google Slides]: https://docs.google.com/presentation/d/17BTT6M6RyNRckV4wTt-dr07nIfBvD325_xVBoLtSoPM/edit?usp=sharing -[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf
\ No newline at end of file +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf diff --git a/doc/development/sha1_as_binary.md b/doc/development/sha1_as_binary.md index 3151cc29bbc..6c4252ec634 100644 --- a/doc/development/sha1_as_binary.md +++ b/doc/development/sha1_as_binary.md @@ -2,7 +2,7 @@ Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string requires at least 40 bytes, an additional byte to store the encoding, and -perhaps more space depending on the internals of PostgreSQL and MySQL. +perhaps more space depending on the internals of PostgreSQL. On the other hand, if one were to store a SHA1 as binary one would only need 20 bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index ae7f2154682..0809f8b1a0a 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -24,7 +24,8 @@ Having said all of the above, we recommend staying away from shell scripts as much as possible. A language like Ruby or Python (if required for consistency with codebases that we leverage) is almost always a better choice. The high-level interpreted languages have more readable syntax, offer much more -mature capabilities for unit-testing, linting, and error reporting. +mature capabilities for unit-testing, linting, and error reporting. + Use shell scripts only if there's a strong restriction on project's dependencies size or any other requirements that are more important in a particular case. @@ -48,12 +49,12 @@ that is: This section describes the tools that should be made a mandatory part of a project's CI pipeline if it contains shell scripts. These tools -automate shell code formatting, checking for errors or vulnerabilities, etc. +automate shell code formatting, checking for errors or vulnerabilities, etc. ### Linting We're using the [ShellCheck](https://www.shellcheck.net/) utility in its default configuration to lint our -shell scripts. +shell scripts. All projects with shell scripts should use this GitLab CI/CD job: @@ -98,7 +99,7 @@ NOTE: **Note:** This is a work in progress. It is an [ongoing effort](https://gitlab.com/gitlab-org/gitlab-ce/issues/64016) to evaluate different tools for the -automated testing of shell scripts (like [BATS](https://github.com/sstephenson/bats)). +automated testing of shell scripts (like [BATS](https://github.com/sstephenson/bats)). ## Code Review diff --git a/doc/development/sql.md b/doc/development/sql.md index a256fd46c09..2584dcfb4ca 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -15,14 +15,11 @@ FROM issues WHERE title LIKE 'WIP:%'; ``` -On PostgreSQL the `LIKE` statement is case-sensitive. On MySQL this depends on -the case-sensitivity of the collation, which is usually case-insensitive. To -perform a case-insensitive `LIKE` on PostgreSQL you have to use `ILIKE` instead. -This statement in turn isn't supported on MySQL. +On PostgreSQL the `LIKE` statement is case-sensitive. To perform a case-insensitive +`LIKE` you have to use `ILIKE` instead. -To work around this problem you should write `LIKE` queries using Arel instead -of raw SQL fragments as Arel automatically uses `ILIKE` on PostgreSQL and `LIKE` -on MySQL. This means that instead of this: +To handle this automatically you should use `LIKE` queries using Arel instead +of raw SQL fragments, as Arel automatically uses `ILIKE` on PostgreSQL. ```ruby Issue.where('title LIKE ?', 'WIP:%') @@ -45,7 +42,7 @@ table = Issue.arel_table Issue.where(table[:title].matches('WIP:%').or(table[:foo].matches('WIP:%'))) ``` -For PostgreSQL this produces: +On PostgreSQL, this produces: ```sql SELECT * @@ -53,18 +50,10 @@ FROM issues WHERE (title ILIKE 'WIP:%' OR foo ILIKE 'WIP:%') ``` -In turn for MySQL this produces: - -```sql -SELECT * -FROM issues -WHERE (title LIKE 'WIP:%' OR foo LIKE 'WIP:%') -``` - ## LIKE & Indexes -Neither PostgreSQL nor MySQL use any indexes when using `LIKE` / `ILIKE` with a -wildcard at the start. For example, this will not use any indexes: +PostgreSQL won't use any indexes when using `LIKE` / `ILIKE` with a wildcard at +the start. For example, this will not use any indexes: ```sql SELECT * @@ -75,9 +64,8 @@ WHERE title ILIKE '%WIP:%'; Because the value for `ILIKE` starts with a wildcard the database is not able to use an index as it doesn't know where to start scanning the indexes. -MySQL provides no known solution to this problem. Luckily PostgreSQL _does_ -provide a solution: trigram GIN indexes. These indexes can be created as -follows: +Luckily, PostgreSQL _does_ provide a solution: trigram GIN indexes. These +indexes can be created as follows: ```sql CREATE INDEX [CONCURRENTLY] index_name_here diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 9d6792e9139..f30a83a4c71 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -15,16 +15,6 @@ manifest themselves within our code. When designing our tests, take time to revi our test design. We can find some helpful heuristics documented in the Handbook in the [Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section. -## Run tests against MySQL - -By default, tests are only run against PostgreSQL, but you can run them on -demand against MySQL by following one of the following conventions: - -| Convention | Valid example | -|:----------------------|:-----------------------------| -| Include `mysql` in your branch name | `enhance-mysql-support` | -| Include `[run mysql]` in your commit message | `Fix MySQL support<br><br>[run mysql]` | - ## Test speed GitLab has a massive test suite that, without [parallelization], can take hours @@ -455,6 +445,19 @@ complexity of RSpec expectations.They should be placed under a certain type of specs only (e.g. features, requests etc.) but shouldn't be if they apply to multiple type of specs. +#### `be_like_time` + +Time returned from a database can differ in precision from time objects +in Ruby, so we need flexible tolerances when comparing in specs. We can +use `be_like_time` to compare that times are within one second of each +other. + +Example: + +```ruby +expect(metrics.merged_at).to be_like_time(time) +``` + #### `have_gitlab_http_status` Prefer `have_gitlab_http_status` over `have_http_status` because the former diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 87d48726268..d9f66a827de 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -39,7 +39,6 @@ slowest test files and try to improve them. ## CI setup -- On CE and EE, the test suite runs both PostgreSQL and MySQL. - Rails logging to `log/test.log` is disabled by default in CI [for performance reasons][logging]. To override this setting, provide the `RAILS_ENABLE_TEST_LOG` environment variable. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index d6b944a3e74..3ae3ce183d9 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -45,11 +45,11 @@ Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests -#### Using the `package-and-qa` job +#### Using the `package-and-qa-manual` job It is possible to run end-to-end tests for a merge request, eventually being run in a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project, -by triggering the `package-and-qa` manual action in the `test` stage (not +by triggering the `package-and-qa-manual` manual action in the `test` stage (not available for forks). **This runs end-to-end tests against a custom Omnibus package built from your @@ -71,7 +71,7 @@ graph LR B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 subgraph "gitlab-ce/ee pipeline" - A1[`test` stage<br>`package-and-qa` job] + A1[`test` stage<br>`package-and-qa-manual` job] end subgraph "omnibus-gitlab pipeline" @@ -79,7 +79,7 @@ subgraph "omnibus-gitlab pipeline" end subgraph "gitlab-qa pipeline" - A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br>and post the result on the original commit tested| A1 + A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa-manual` job<br>and post the result on the original commit tested| A1 end ``` diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 47e58a425fd..850ea6b60ac 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -40,7 +40,7 @@ the time it would take to build packages and test everything. That is why when someone changes `t.text_field :login` to `t.text_field :username` in the _new session_ view we won't know about this change until our GitLab QA nightly pipeline fails, or until someone triggers -`package-and-qa` action in their merge request. +`package-and-qa-manual` action in their merge request. Obviously such a change would break all tests. We call this problem a _fragile tests problem_. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 931cbc51cae..eb0bf6fc563 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -35,8 +35,8 @@ Once a test is in quarantine, there are 3 choices: Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: -- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE) -- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only) +- `rspec-pg-quarantine` (CE & EE) +- `rspec-pg-quarantine-ee` (EE only) ## Automatic retries and flaky tests detection diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 2985278cc92..7dc89a3fcdb 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -232,7 +232,7 @@ module. GitLab has a custom `spyOnDependency` method which utilizes [babel-plugin-rewire](https://github.com/speedskater/babel-plugin-rewire) to achieve this. It can be used like so: -```js +```javascript // my_module.js import { visitUrl } from '~/lib/utils/url_utility'; @@ -241,7 +241,7 @@ export default function doSomething() { } ``` -```js +```javascript // my_module_spec.js import doSomething from '~/my_module'; @@ -593,6 +593,517 @@ end [capybara]: https://github.com/teamcapybara/capybara [jasmine]: https://jasmine.github.io/ +## Overview of Frontend Testing Levels + +Tests relevant for frontend development can be found at the following places: + +- `spec/javascripts/` which are run by Karma (command: `yarn karma`) and contain + - [frontend unit tests](#frontend-unit-tests) + - [frontend component tests](#frontend-component-tests) + - [frontend integration tests](#frontend-integration-tests) +- `spec/frontend/` which are run by Jest (command: `yarn jest`) and contain + - [frontend unit tests](#frontend-unit-tests) + - [frontend component tests](#frontend-component-tests) + - [frontend integration tests](#frontend-integration-tests) +- `spec/features/` which are run by RSpec and contain + - [feature tests](#feature-tests) + +All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-ce/issues/52483)). + +In addition, there used to be feature tests in `features/`, run by Spinach. +These were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-ce/issues/23036)). + +See also [Notes on testing Vue components](../fe_guide/vue.html#testing-vue-components). + +### Frontend unit tests + +Unit tests are on the lowest abstraction level and typically test functionality that is not directly perceivable by a user. + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class plain tested; + class Vuex tested; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use unit tests + +<details> + <summary>exported functions and classes</summary> + Anything that is exported can be reused at various places in a way you have no control over. + Therefore it is necessary to document the expected behavior of the public interface with tests. +</details> + +<details> + <summary>Vuex actions</summary> + Any Vuex action needs to work in a consistent way independent of the component it is triggered from. +</details> + +<details> + <summary>Vuex mutations</summary> + For complex Vuex mutations it helps to identify the source of a problem by separating the tests from other parts of the Vuex store. +</details> + +#### When *not* to use unit tests + +<details> + <summary>non-exported functions or classes</summary> + Anything that is not exported from a module can be considered private or an implementation detail and doesn't need to be tested. +</details> + +<details> + <summary>constants</summary> + Testing the value of a constant would mean to copy it. + This results in extra effort without additional confidence that the value is correct. +</details> + +<details> + <summary>Vue components</summary> + Computed properties, methods, and lifecycle hooks can be considered an implementation detail of components and don't need to be tested. + They are implicitly covered by component tests. + The <a href="https://vue-test-utils.vuejs.org/guides/#getting-started">official Vue guidelines</a> suggest the same. +</details> + +#### What to mock in unit tests + +<details> + <summary>state of the class under test</summary> + Modifying the state of the class under test directly rather than using methods of the class avoids side-effects in test setup. +</details> + +<details> + <summary>other exported classes</summary> + Every class needs to be tested in isolation to prevent test scenarios from growing exponentially. +</details> + +<details> + <summary>single DOM elements if passed as parameters</summary> + For tests that only operate on single DOM elements rather than a whole page, creating these elements is cheaper than loading a whole HTML fixture. +</details> + +<details> + <summary>all server requests</summary> + When running frontend unit tests, the backend may not be reachable. + Therefore all outgoing requests need to be mocked. +</details> + +<details> + <summary>asynchronous background operations</summary> + Background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. +</details> + +#### What *not* to mock in unit tests + +<details> + <summary>non-exported functions or classes</summary> + Everything that is not exported can be considered private to the module and will be implicitly tested via the exported classes / functions. +</details> + +<details> + <summary>methods of the class under test</summary> + By mocking methods of the class under test, the mocks will be tested and not the real methods. +</details> + +<details> + <summary>utility functions (pure functions, or those that only modify parameters)</summary> + If a function has no side effects because it has no state, it is safe to not mock it in tests. +</details> + +<details> + <summary>full HTML pages</summary> + Loading the HTML of a full page slows down tests, so it should be avoided in unit tests. +</details> + +### Frontend component tests + +Component tests cover the state of a single component that is perceivable by a user depending on external signals such as user input, events fired from other components, or application state. + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class Vue tested; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use component tests + +- Vue components + +#### When *not* to use component tests + +<details> + <summary>Vue applications</summary> + Vue applications may contain many components. + Testing them on a component level requires too much effort. + Therefore they are tested on frontend integration level. +</details> + +<details> + <summary>HAML templates</summary> + HAML templates contain only Markup and no frontend-side logic. + Therefore they are not complete components. +</details> + +#### What to mock in component tests + +<details> + <summary>DOM</summary> + Operating on the real DOM is significantly slower than on the virtual DOM. +</details> + +<details> + <summary>properties and state of the component under test</summary> + Similarly to testing classes, modifying the properties directly (rather than relying on methods of the component) avoids side-effects. +</details> + +<details> + <summary>Vuex store</summary> + To avoid side effects and keep component tests simple, Vuex stores are replaced with mocks. +</details> + +<details> + <summary>all server requests</summary> + Similar to unit tests, when running component tests, the backend may not be reachable. + Therefore all outgoing requests need to be mocked. +</details> + +<details> + <summary>asynchronous background operations</summary> + Similar to unit tests, background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. +</details> + +<details> + <summary>child components</summary> + Every component is tested individually, so child components are mocked. + See also <a href="https://vue-test-utils.vuejs.org/api/#shallowmount">shallowMount()</a> +</details> + +#### What *not* to mock in component tests + +<details> + <summary>methods or computed properties of the component under test</summary> + By mocking part of the component under test, the mocks will be tested and not the real component. +</details> + +<details> + <summary>functions and classes independent from Vue</summary> + All plain JavaScript code is already covered by unit tests and needs not to be mocked in component tests. +</details> + +### Frontend integration tests + +Integration tests cover the interaction between all components on a single page. +Their abstraction level is comparable to how a user would interact with the UI. + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class plain tested; + class Vue tested; + class Vuex tested; + class GraphQL tested; + class browser tested; + linkStyle 0,1,2,3,4,5,6 stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use integration tests + +<details> + <summary>page bundles (<code>index.js</code> files in <code>app/assets/javascripts/pages/</code>)</summary> + Testing the page bundles ensures the corresponding frontend components integrate well. +</details> + +<details> + <summary>Vue applications outside of page bundles</summary> + Testing Vue applications as a whole ensures the corresponding frontend components integrate well. +</details> + +#### What to mock in integration tests + +<details> + <summary>HAML views (use fixtures instead)</summary> + Rendering HAML views requires a Rails environment including a running database which we cannot rely on in frontend tests. +</details> + +<details> + <summary>all server requests</summary> + Similar to unit and component tests, when running component tests, the backend may not be reachable. + Therefore all outgoing requests need to be mocked. +</details> + +<details> + <summary>asynchronous background operations that are not perceivable on the page</summary> + Background operations that affect the page need to be tested on this level. + All other background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. +</details> + +#### What *not* to mock in integration tests + +<details> + <summary>DOM</summary> + Testing on the real DOM ensures our components work in the environment they are meant for. + Part of this will be delegated to <a href="https://gitlab.com/gitlab-org/quality/team-tasks/issues/45">cross-browser testing</a>. +</details> + +<details> + <summary>properties or state of components</summary> + On this level, all tests can only perform actions a user would do. + For example to change the state of a component, a click event would be fired. +</details> + +<details> + <summary>Vuex stores</summary> + When testing the frontend code of a page as a whole, the interaction between Vue components and Vuex stores is covered as well. +</details> + +### Feature tests + +In contrast to [frontend integration tests](#frontend-integration-tests), feature tests make requests against the real backend instead of using fixtures. +This also implies that database queries are executed which makes this category significantly slower. + +See also the [RSpec testing guidelines](../testing_guide/best_practices.md#rspec). + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class backend tested; + class plain tested; + class Vue tested; + class Vuex tested; + class GraphQL tested; + class browser tested; + linkStyle 0,1,2,3,4,5,6,7,8,9,10 stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use feature tests + +- Use cases that require a backend and cannot be tested using fixtures. +- Behavior that is not part of a page bundle but defined globally. + +#### Relevant notes + +A `:js` flag is added to the test to make sure the full environment is loaded. + +```ruby +scenario 'successfully', :js do + sign_in(create(:admin)) +end +``` + +The steps of each test are written using capybara methods ([documentation](https://www.rubydoc.info/gems/capybara)). + +Bear in mind <abbr title="XMLHttpRequest">XHR</abbr> calls might require you to use `wait_for_requests` in between steps, like so: + +```ruby +find('.form-control').native.send_keys(:enter) + +wait_for_requests + +expect(page).not_to have_selector('.card') +``` + +## Test helpers + +### Vuex Helper: `testAction` + +We have a helper available to make testing actions easier, as per [official documentation](https://vuex.vuejs.org/guide/testing.html): + +```javascript +testAction( + actions.actionName, // action + { }, // params to be passed to action + state, // state + [ + { type: types.MUTATION}, + { type: types.MUTATION_1, payload: {}}, + ], // mutations committed + [ + { type: 'actionName', payload: {}}, + { type: 'actionName1', payload: {}}, + ] // actions dispatched + done, +); +``` + +Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/javascripts/ide/stores/actions_spec.js). + +### Vue Helper: `mountComponent` + +To make mounting a Vue component easier and more readable, we have a few helpers available in `spec/helpers/vue_mount_component_helper`: + +- `createComponentWithStore` +- `mountComponentWithStore` + +Examples of usage: + +```javascript +beforeEach(() => { + vm = createComponentWithStore(Component, store); + + vm.$store.state.currentBranchId = 'master'; + + vm.$mount(); +}); +``` + +```javascript +beforeEach(() => { + vm = mountComponentWithStore(Component, { + el: '#dummy-element', + store, + props: { badge }, + }); +}); +``` + +Don't forget to clean up: + +```javascript +afterEach(() => { + vm.$destroy(); +}); +``` + +## Testing with older browsers + +Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or Browserstack using the following steps: + +### Browserstack + +[Browserstack](https://www.browserstack.com/) allows you to test more than 1200 mobile devices and browsers. +You can use it directly through the [live app](https://www.browserstack.com/live) or you can install the [chrome extension](https://chrome.google.com/webstore/detail/browserstack/nkihdmlheodkdfojglpcjjmioefjahjb) for easy access. +You can find the credentials on 1Password, under `frontendteam@gitlab.com`. + +### Firefox + +#### macOS + +You can download any older version of Firefox from the releases FTP server, <https://ftp.mozilla.org/pub/firefox/releases/>: + +1. From the website, select a version, in this case `50.0.1`. +1. Go to the mac folder. +1. Select your preferred language, you will find the dmg package inside, download it. +1. Drag and drop the application to any other folder but the `Applications` folder. +1. Rename the application to something like `Firefox_Old`. +1. Move the application to the `Applications` folder. +1. Open up a terminal and run `/Applications/Firefox_Old.app/Contents/MacOS/firefox-bin -profilemanager` to create a new profile specific to that Firefox version. +1. Once the profile has been created, quit the app, and run it again like normal. You now have a working older Firefox version. + --- [Return to Testing documentation](index.md) diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 7843fc4c874..11449712a04 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -255,8 +255,8 @@ that a machine will hit the "too many mount points" problem in the future. thousands of unused Docker images.** > We have to start somewhere and improve later. Also, we're using the - CNG-mirror project to store these Docker images so that we can just wipe out - the registry at some point, and use a new fresh, empty one. + > CNG-mirror project to store these Docker images so that we can just wipe out + > the registry at some point, and use a new fresh, empty one. **How do we secure this from abuse? Apps are open to the world so we need to find a way to limit it to only us.** diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index e1ce4d3b7d1..0090c84cbf0 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -63,10 +63,9 @@ They're useful to test permissions, redirections, what view is rendered etc. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | -| `app/controllers/` | `spec/controllers/` | RSpec | | +| `app/controllers/` | `spec/controllers/` | RSpec | For N+1 tests, use [request specs](../query_recorder.md#use-request-specs-instead-of-controller-specs) | | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | -| `lib/ci/api/` | `spec/requests/ci/api/` | RSpec | | | `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | ### About controller tests diff --git a/doc/development/uploads.md b/doc/development/uploads.md new file mode 100644 index 00000000000..234539bb673 --- /dev/null +++ b/doc/development/uploads.md @@ -0,0 +1,271 @@ +# Uploads development documentation + +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. +To prevent occupying a ruby process on I/O operations, we process the upload in workhorse, where is cheaper. +This process can also directly upload to object storage. + +## The problem description + +The following graph explains machine boundaries in a scalable GitLab installation. Without any workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->workhorse + + subgraph "web or API fleet" + workhorse-- 2 -->rails + end + rails-- "3 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +We have three challenges here: performance, availability, and scalability. + +### Performance + +Rails process are expensive in terms of both CPU and memory. Ruby [global interpreter lock](https://en.wikipedia.org/wiki/Global_interpreter_lock) adds to cost too because the ruby process will spend time on I/O operations on step 3 causing incoming requests to pile up. + +In order to improve this, [workhorse disk acceleration](#workhorse-disk-acceleration) was implemented. With this, Rails no longer deals with writing uploaded files to disk. + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->workhorse + + subgraph "web or API fleet" + workhorse-- "3 (without files)" -->rails + end + workhorse -- "2 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +### Availability + +There's also an availability problem in this setup, NFS is a [single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure). + +To address this problem an HA object storage can be used and it's supported by [workhorse object storage acceleration](#workhorse-object-storage-acceleration) + +### Scalability + +Scaling NFS is outside of our support scope, and NFS is not a part of cloud native installations. + +All features that require sidekiq and do not use object storage acceleration won't work without NFS. In Kubernetes, machine boundaries translate to PODs, and in this case the uploaded file will be written into the POD private disk. Since sidekiq POD cannot reach into other pods, the operation will fail to read it. + +## How to select the proper level of acceleration? + +Selecting the proper acceleration is a tradeoff between speed of development and operational costs. + +We can identify three major use-cases for an upload: + +1. **storage:** if we are uploading for storing a file (i.e. artifacts, packages, discussion attachments). In this case [object storage acceleration](#workhorse-object-storage-acceleration) is the proper level as it's the less resource-intensive operation. Additional information can be found on [File Storage in GitLab](file_storage.md). +1. **in-controller/synchronous processing:** if we allow processing **small files** synchronously, using [disk acceleration](#workhorse-disk-acceleration) may speed up development. +1. **sidekiq/asynchronous processing:** Async processing must implement [object storage acceleration](#workhorse-object-storage-acceleration), the reason being that it's the only way to support Cloud Native deployments without a shared NFS. + +For more details about currently broken feature see [epic &1802](https://gitlab.com/groups/gitlab-org/-/epics/1802). + +### Handling repository uploads + +Some features involves git repository uploads without using a regular git client. +Some examples are uploading a repository file from the web interface and [design management](../user/project/issues/design_management.md). + +Those uploads requires the rails controller to act as a git client in lieu of the user. +Those operation falls into _in-controller/synchronous processing_ category, but we have no warranties on the file size. + +In case of a LFS upload, the file pointer is committed synchronously, but file upload to object storage is performed asynchronously with sidekiq. + +## Upload encodings + +By upload encoding we mean how the file is included within the incoming request. + +We have three kinds of file encoding in our uploads: + +1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. +1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. +1. <i class="fa fa-times-circle"></i> **JSON**: some JSON API uploads files as base64 encoded strings. This requires [gitlab-workhorse#226](https://gitlab.com/gitlab-org/gitlab-workhorse/issues/226) to be implemented. + +## Uploading technologies + +By uploading technologies we mean how all the involved services interact with each other. + +GitLab supports 3 kinds of uploading technologies, here follows a brief description with a sequence diagram for each one. Diagrams are not meant to be exhaustive. + +### Regular rails upload + +This is the default kind of upload, and it's most expensive in terms of resources. + +In this case, workhorse is unaware of files being uploaded and acts as a regular proxy. + +When a multipart request reaches the rails application, `Rack::Multipart` leaves behind tempfiles in `/tmp` and uses valuable Ruby process time to copy files around. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + + activate c + c ->>+w: POST /some/url/upload + w->>+r: POST /some/url/upload + + r->>r: save the incoming file on /tmp + r->>r: read the file for processing + + r-->>-c: request result + deactivate c + deactivate w +``` + +### Workhorse disk acceleration + +This kind of upload avoids wasting resources caused by handling upload writes to `/tmp` in rails. + +This optimization is not active by default on REST API requests. + +When enabled, Workhorse looks for files in multipart MIME requests, uploading +any it finds to a temporary file on shared storage. The MIME data in the request +is replaced with the path to the corresponding file before it is forwarded to +Rails. + +To prevent abuse of this feature, Workhorse signs the modified request with a +special header, stating which entries it modified. Rails will ignore any +unsigned path entries. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant s as NFS + + activate c + c ->>+w: POST /some/url/upload + + w->>+s: save the incoming file on a temporary location + s-->>-w: + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+s: read file + s-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +### Workhorse object storage acceleration + +This is the more advanced acceleration technique we have in place. + +Workhorse asks rails for temporary pre-signed object storage URLs and directly uploads to object storage. + +In this setup an extra rails route needs to be implemented in order to handle authorization, +you can see an example of this in [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/controllers/projects/lfs_storage_controller.rb) +and [its routes](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0/config/routes/git_http.rb#L31-32). + +**note:** this will fallback to _Workhorse disk acceleration_ when object storage is not enabled in the gitlab instance. The answer to the `/authorize` call will only contain a file system path. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant os as Object Storage + + activate c + c ->>+w: POST /some/url/upload + + w ->>+r: POST /some/url/upload/authorize + Note over w,r: this request has an empty body + r-->>-w: presigned OS URL + + w->>+os: PUT file + Note over w,os: file is stored on a temporary location. Rails select the destination + os-->>-w: + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + r->>+os: move object to final destination + os-->>-r: + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+os: get object + os-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +## What does the `direct_upload` setting mean? + +[Object storage setting](../administration/uploads.md#object-storage-settings) allows instance administators to enable `direct_upload`, this in an option that only affects the behavior of [workhorse object storage acceleration](#workhorse-object-storage-acceleration). + +This option affect the response to the `/authorize` call. When not enabled, the API response will not contain presigned URLs and workhorse will write the file the shared disk, on the path is provided by rails, acting like object storage was disabled. + +Once the request reachs rails, it will schedule an object storage upload as a sidekiq job. + diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index ccec6f7d719..6b4995aebe2 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -1,15 +1,15 @@ # Verifying Database Capabilities -Sometimes certain bits of code may only work on a certain database and/or +Sometimes certain bits of code may only work on a certain database version. While we try to avoid such code as much as possible sometimes it is necessary to add database (version) specific behaviour. To facilitate this we have the following methods that you can use: -- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used -- `Gitlab::Database.mysql?`: returns `true` if MySQL is being used +- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used. + You can normally just assume this is the case. - `Gitlab::Database.version`: returns the PostgreSQL version number as a string - in the format `X.Y.Z`. This method does not work for MySQL + in the format `X.Y.Z`. This allows you to write code such as: diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index f0da1cc2ddc..944bf5900c5 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -7,9 +7,8 @@ downtime. ## Adding Columns -On PostgreSQL you can safely add a new column to an existing table as long as it -does **not** have a default value. For example, this query would not require -downtime: +You can safely add a new column to an existing table as long as it does **not** +have a default value. For example, this query would not require downtime: ```sql ALTER TABLE projects ADD COLUMN random_value int; @@ -27,11 +26,6 @@ This requires updating every single row in the `projects` table so that indexes in a table. This in turn acquires enough locks on the table for it to effectively block any other queries. -As of MySQL 5.6 adding a column to a table is still quite an expensive -operation, even when using `ALGORITHM=INPLACE` and `LOCK=NONE`. This means -downtime _may_ be required when modifying large tables as otherwise the -operation could potentially take hours to complete. - Adding a column with a default value _can_ be done without requiring downtime when using the migration helper method `Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works @@ -311,8 +305,7 @@ migrations](background_migrations.md#cleaning-up). ## Adding Indexes Adding indexes is an expensive process that blocks INSERT and UPDATE queries for -the duration. When using PostgreSQL one can work around this by using the -`CONCURRENTLY` option: +the duration. You can work around this by using the `CONCURRENTLY` option: ```sql CREATE INDEX CONCURRENTLY index_name ON projects (column_name); @@ -336,17 +329,9 @@ end Note that `add_concurrent_index` can not be reversed automatically, thus you need to manually define `up` and `down`. -When running this on PostgreSQL the `CONCURRENTLY` option mentioned above is -used. On MySQL this method produces a regular `CREATE INDEX` query. - -MySQL doesn't really have a workaround for this. Supposedly it _can_ create -indexes without the need for downtime but only for variable width columns. The -details on this are a bit sketchy. Since it's better to be safe than sorry one -should assume that adding indexes requires downtime on MySQL. - ## Dropping Indexes -Dropping an index does not require downtime on both PostgreSQL and MySQL. +Dropping an index does not require downtime. ## Adding Tables @@ -370,7 +355,7 @@ transaction this means this approach would require downtime. GitLab allows you to work around this by using `Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method -ensures that when PostgreSQL is used no downtime is needed. +ensures that no downtime is needed. ## Removing Foreign Keys diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index 3e3f96fb31f..5de173abbff 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -63,8 +63,8 @@ git config --global user.email You'll need to do this only once, since you are using the `--global` option. It tells Git to always use this information for anything you do on that system. If you want -to override this with a different username or email address for specific projects, -you can run the command without the `--global` option when you’re in that project. +to override this with a different username or email address for specific projects or repositories, +you can run the command without the `--global` option when you’re in that project, and that will default to `--local`. You can read more on how Git manages configurations in the [Git Config](https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration) documentation. ## Check your information @@ -102,8 +102,7 @@ files to your local computer, automatically preserving the Git connection with t remote repository. You can either clone it via HTTPS or [SSH](../ssh/README.md). If you chose to clone -it via HTTPS, you'll have to enter your credentials every time you pull and push. -With SSH, you enter your credentials only once. +it via HTTPS, you'll have to enter your credentials every time you pull and push. You can read more about credential storage in the [Git Credentials documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). With SSH, you enter your credentials only once. You can find both paths (HTTPS and SSH) by navigating to your project's landing page and clicking **Clone**. GitLab will prompt you with both paths, from which you can copy @@ -152,13 +151,15 @@ to get the main branch code, or the branch name of the branch you are currently in. ```bash -git pull REMOTE <name-of-branch> +git pull <REMOTE> <name-of-branch> ``` -When you first clone a repository, REMOTE is typically `origin`. This is where the +When you clone a repository, `REMOTE` is typically `origin`. This is where the repository was cloned from, and it indicates the SSH or HTTPS URL of the repository on the remote server. `<name-of-branch>` is usually `master`, but it may be any existing -branch. +branch. You can create additional named remotes and branches as necessary. + +You can learn more on how Git manages remote repositories in the [Git Remote documentation](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes). ### View your remote repositories @@ -168,6 +169,8 @@ To view your remote repositories, type: git remote -v ``` +The `-v` flag stands for verbose. + ### Add a remote repository To add a link to a remote repository: @@ -186,7 +189,7 @@ following (spaces won't be recognized in the branch name, so you will need to us hyphen or underscore): ```bash -git checkout -b <name-of-branch>> +git checkout -b <name-of-branch> ``` ### Work on an existing branch @@ -238,7 +241,7 @@ git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" ``` NOTE: **Note:** -The `.` character typically means _all_ in Git. +The `.` character means _all file changes in the current directory and all subdirectories_. ### Send changes to GitLab.com diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 543a222bd25..c5939dc6856 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -407,7 +407,7 @@ on any cloud service you choose. ## Where to next? -Check out our other [Technical Articles](../../articles/index.md) or browse the [GitLab Documentation][GitLab-Docs](../../README.md) to learn more about GitLab. +Check out our other [Technical Articles](../../articles/index.md) or browse the [GitLab Documentation](../../README.md) to learn more about GitLab. ### Useful links diff --git a/doc/install/installation.md b/doc/install/installation.md index 295d9804497..6d64b8a4936 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -789,7 +789,7 @@ nginx: configuration file /etc/nginx/nginx.conf test failed` sudo service nginx restart ``` -## Done! +## Post-install ### Double-check Application Status diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 6755640d39c..d865f977799 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -35,7 +35,7 @@ and [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF For a real use case, read the blog post [Continuous integration: From Jenkins to GitLab using Docker](https://about.gitlab.com/2017/07/27/docker-my-precious/). -NOTE: **Moving from a traditional CI plug-in to a single application for the entire software development lifecycle can decrease hours spent on maintaining toolchains by 10% or more.** +NOTE: **Moving from a traditional CI plug-in to a single application for the entire software development lifecycle can decrease hours spent on maintaining toolchains by 10% or more.** Visit the ['GitLab vs. Jenkins' comparison page](https://about.gitlab.com/devops-tools/jenkins-vs-gitlab.html) to learn how our built-in CI compares to Jenkins. ## Requirements diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md index f4119b1d1ce..ad59592c45f 100644 --- a/doc/integration/oauth2_generic.md +++ b/doc/integration/oauth2_generic.md @@ -12,7 +12,7 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc 1. Strategy parses user information from the response, using a **configurable** format 1. GitLab finds or creates the returned user and logs them in -## Limitations of this Strategy: +## Limitations of this Strategy - It can only be used for Single Sign on, and will not provide any other access granted by any OAuth provider (importing projects or users, etc) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 9947c19a667..2b8379141c3 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -227,7 +227,7 @@ sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ ``` -### 5. Import the CI data into GitLab. +### 5. Import the CI data into GitLab This step will delete any existing CI data on your GitLab server. There should be no CI data yet because you turned CI on the GitLab server off earlier. diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 018c273c51a..f7ba7c16a9e 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -86,6 +86,7 @@ Please see the table below for some examples: | 9.4.5 | 8.13.4 | `8.13.4` -> `8.17.7` -> `9.4.5` | `8.17.7` is the last version in version `8` | | 10.1.4 | 8.13.4 | `8.13.4 -> 8.17.7 -> 9.5.10 -> 10.1.4` | `8.17.7` is the last version in version `8`, `9.5.10` is the last version in version `9` | | 11.3.4 | 8.13.4 | `8.13.4` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.3.4` | `8.17.7` is the last version in version `8`, `9.5.10` is the last version in version `9`, `10.8.7` is the last version in version `10` | +| 12.0.2 | 11.3.4 | `11.3.4` -> `11.11.x` -> `12.0.2` | `11.11.x` is the last version in version `11` More information about the release procedures can be found in our [release documentation](https://gitlab.com/gitlab-org/release/docs). You may also want to read our diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index 0142e5075cc..dc6ee9b2503 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -6,7 +6,7 @@ type: reference GitLab allows [Owners](../user/permissions.md) to set a projects' visibility as **public**, **internal** or **private**. These visibility levels affect who can see the project in the -public access directory (`/public` under your GitLab instance), like at [https://gitlab.com/public](). +public access directory (`/public` under your GitLab instance), like at <https://gitlab.com/public> ## Visibility of projects diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 51f04df27c7..b4b7d711d5a 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -978,4 +978,3 @@ If this happens, check the following: 1. Confirm there is sufficent diskspace for the gzip operation. 1. If NFS is being used, check if the mount option `timeo` is set. The default is `600`, and changing this to smaller values have resulted in this error. - diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 832078d23cb..f84d29cca9a 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -1,6 +1,10 @@ # Cleanup -## Remove garbage from filesystem. Important! Data loss! +## Remove garbage from filesystem + +DANGER: **Danger:** +The commands below will remove data permanently from your GitLab instance. Only use +these commands if you are 100% certain that it is safe to delete this data. Remove namespaces(dirs) from all repository storage paths if they don't exist in GitLab database. diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index 8f65fab366e..c76180a242e 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -11,7 +11,7 @@ ## How to use -### Create a new folder to import your Git repositories from. +### Create a new folder to import your Git repositories from The new folder needs to have git user ownership and read/write/execute access for git user and its group: @@ -19,7 +19,7 @@ The new folder needs to have git user ownership and read/write/execute access fo sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group ``` -### Copy your bare repositories inside this newly created folder: +### Copy your bare repositories inside this newly created folder - Any .git repositories found on any of the subfolders will be imported as projects - Groups will be created as needed, these could be nested folders. Example: @@ -38,7 +38,7 @@ sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date> If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`. -### Run the command below depending on your type of installation: +### Run the command below depending on your type of installation #### Omnibus Installation diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index cc1166a04cc..d9a042ec8fe 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -1,6 +1,6 @@ # Webhooks administration **(CORE ONLY)** -## Add a webhook for **ALL** projects: +## Add a webhook for **ALL** projects ```sh # omnibus-gitlab @@ -9,7 +9,7 @@ sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production ``` -## Add a webhook for projects in a given **NAMESPACE**: +## Add a webhook for projects in a given **NAMESPACE** ```sh # omnibus-gitlab @@ -18,7 +18,7 @@ sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acm bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ``` -## Remove a webhook from **ALL** projects using: +## Remove a webhook from **ALL** projects using ```sh # omnibus-gitlab @@ -27,7 +27,7 @@ sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production ``` -## Remove a webhook from projects in a given **NAMESPACE**: +## Remove a webhook from projects in a given **NAMESPACE** ```sh # omnibus-gitlab @@ -36,7 +36,7 @@ sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ``` -## List **ALL** webhooks: +## List **ALL** webhooks ```sh # omnibus-gitlab @@ -45,7 +45,7 @@ sudo gitlab-rake gitlab:web_hook:list bundle exec rake gitlab:web_hook:list RAILS_ENV=production ``` -## List the webhooks from projects in a given **NAMESPACE**: +## List the webhooks from projects in a given **NAMESPACE** ```sh # omnibus-gitlab diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index c80f2f264b2..80088da77a0 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -30,4 +30,3 @@ similarly mitigated by a rate limit. This method of rate limiting is cumbersome, but has some advantages. It allows throttling of specific paths, and is also integrated into Git and container registry requests. See [Rack Attack initializer](rack_attack.md). - diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 9c1258fa1aa..26d5221acc5 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -65,7 +65,7 @@ project in a simple and automatic way: 1. [Auto Code Quality](#auto-code-quality-starter) **(STARTER)** 1. [Auto SAST (Static Application Security Testing)](#auto-sast-ultimate) **(ULTIMATE)** 1. [Auto Dependency Scanning](#auto-dependency-scanning-ultimate) **(ULTIMATE)** -1. [Auto License Management](#auto-license-management-ultimate) **(ULTIMATE)** +1. [Auto License Compliance](#auto-license-compliance-ultimate) **(ULTIMATE)** 1. [Auto Container Scanning](#auto-container-scanning-ultimate) **(ULTIMATE)** 1. [Auto Review Apps](#auto-review-apps) 1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast-ultimate) **(ULTIMATE)** @@ -401,13 +401,13 @@ check out. Any security warnings are also shown in the merge request widget. Read more about [Dependency Scanning](../../user/application_security/dependency_scanning/index.md). -### Auto License Management **(ULTIMATE)** +### Auto License Compliance **(ULTIMATE)** > Introduced in [GitLab Ultimate][ee] 11.0. -License Management uses the -[License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) -to search the project dependencies for their license. The Auto License Management stage +License Compliance uses the +[License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +to search the project dependencies for their license. The Auto License Compliance stage will be skipped on licenses other than Ultimate. Once the @@ -415,7 +415,7 @@ report is created, it's uploaded as an artifact which you can later download and check out. Any licenses are also shown in the merge request widget. Read more how -[License Management works](../../user/application_security/license_management/index.md). +[License Compliance works](../../user/application_security/license_management/index.md). ### Auto Container Scanning **(ULTIMATE)** @@ -584,6 +584,55 @@ Unless you have a `Dockerfile` in your repo, your image is built with Herokuish, and you must prefix commands run in these images with `/bin/herokuish procfile exec` to replicate the environment where your application will run. +#### Workers + +Some web applications need to run extra deployments for "worker processes". For +example, it is common in a Rails application to have a separate worker process +to run background tasks like sending emails. + +The [default Helm chart](https://gitlab.com/gitlab-org/charts/auto-deploy-app) +used in Auto Deploy [has support for running worker +processes](https://gitlab.com/gitlab-org/charts/auto-deploy-app/merge_requests/9). + +In order to run a worker, you'll need to ensure that it is able to respond to +the standard health checks, which expect a successful HTTP response on port +`5000`. For [Sidekiq](https://github.com/mperham/sidekiq), you could make use of +the [`sidekiq_alive` gem](https://rubygems.org/gems/sidekiq_alive) to do this. + +In order to work with Sidekiq, you'll also need to ensure your deployments have +access to a Redis instance. Auto DevOps won't deploy this for you so you'll +need to: + +- Maintain your own Redis instance. +- Set a CI variable `K8S_SECRET_REDIS_URL`, which the URL of this instance to + ensure it's passed into your deployments. + +Once you have configured your worker to respond to health checks, you will +need to configure a CI variable `HELM_UPGRADE_EXTRA_ARGS` with the value +`--values helm-values.yaml`. + +Then you can, for example, run a Sidekiq worker for your Rails application +by adding a file named `helm-values.yaml` to your repository with the following +content: + +```yml +workers: + sidekiq: + replicaCount: 1 + command: + - /bin/herokuish + - procfile + - exec + - sidekiq + preStopCommand: + - /bin/herokuish + - procfile + - exec + - sidekiqctl + - quiet + terminationGracePeriodSeconds: 60 +``` + ### Auto Monitoring See the [requirements](#requirements) for Auto Monitoring to enable this stage. diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 7ab59b80374..35a5aff6a60 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -167,7 +167,7 @@ In the **test** stage, GitLab runs various checks on the application: - The `sast` job runs static analysis on the current code to check for potential security issues and is allowed to fail([Auto SAST](index.md#auto-sast-ultimate)) **(ULTIMATE)** - The `license_management` job searches the application's dependencies to determine each of their - licenses and is allowed to fail ([Auto License Management](index.md#auto-license-management-ultimate)) **(ULTIMATE)** + licenses and is allowed to fail ([Auto License Compliance](index.md#auto-license-compliance-ultimate)) **(ULTIMATE)** NOTE: **Note:** As you might have noticed, all jobs except `test` are allowed to fail in the diff --git a/doc/university/README.md b/doc/university/README.md index b17f40b91a5..25f77906e68 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -57,7 +57,7 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres 1. [Migrating from SVN](../user/project/import/svn.md) 1. [Migrating from Fogbugz](../user/project/import/fogbugz.md) -### 1.6. GitLab Inc. +### 1.6. The GitLab team 1. [About GitLab](https://about.gitlab.com/about/) 1. [GitLab Direction](https://about.gitlab.com/direction/) diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 0506d992d4b..6a41c6aec32 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -41,7 +41,7 @@ sudo -u git -H git checkout -- Gemfile.lock db/schema.rb locale sudo -u git -H git checkout LATEST_TAG -b LATEST_TAG ``` -### 3. Install libs, migrations, etc. +### 3. Install libs, migrations, etc ```bash cd /home/git/gitlab diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index bea5bcd9dd7..49e2fb2568e 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -54,7 +54,7 @@ sudo -u git -H git remote add -f ee https://gitlab.com/gitlab-org/gitlab-ee.git sudo -u git -H git checkout EE_BRANCH ``` -### 3. Install libs, migrations, etc. +### 3. Install libs, migrations, etc ```sh cd /home/git/gitlab diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index df35638cba2..f6b64dcf694 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -313,7 +313,7 @@ For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload ``` -### 13. Install libs, migrations, etc. +### 13. Install libs, migrations, etc ```bash cd /home/git/gitlab diff --git a/doc/user/admin_area/abuse_reports.md b/doc/user/admin_area/abuse_reports.md index 0c5d2f81e25..cd8dc7bcbf6 100644 --- a/doc/user/admin_area/abuse_reports.md +++ b/doc/user/admin_area/abuse_reports.md @@ -2,7 +2,7 @@ type: reference, howto --- -# Abuse reports +# Abuse reports **(CORE ONLY)** View and resolve abuse reports from GitLab users. diff --git a/doc/user/admin_area/broadcast_messages.md b/doc/user/admin_area/broadcast_messages.md index 01b6558bdbe..b0491499f88 100644 --- a/doc/user/admin_area/broadcast_messages.md +++ b/doc/user/admin_area/broadcast_messages.md @@ -2,7 +2,7 @@ type: reference, howto --- -# Broadcast Messages +# Broadcast Messages **(CORE ONLY)** GitLab can display messages to all users of a GitLab instance in a banner that appears in the UI. diff --git a/doc/user/admin_area/diff_limits.md b/doc/user/admin_area/diff_limits.md index 4063c40a751..9fe4b50a991 100644 --- a/doc/user/admin_area/diff_limits.md +++ b/doc/user/admin_area/diff_limits.md @@ -2,7 +2,7 @@ type: reference --- -# Diff limits administration +# Diff limits administration **(CORE ONLY)** You can set a maximum size for display of diff files (patches). diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index 35e7b6fb541..a3f4bc774ce 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -2,7 +2,7 @@ type: concepts, howto --- -# Health Check +# Health Check **(CORE ONLY)** > - Liveness and readiness probes were [introduced][ce-10416] in GitLab 9.1. > - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and was @@ -21,28 +21,71 @@ traffic until the system is ready or restart the container as needed. To access monitoring resources, the requesting client IP needs to be included in a whitelist. For details, see [how to add IPs to a whitelist for the monitoring endpoints](../../../administration/monitoring/ip_whitelist.md). -## Using the endpoints +## Using the endpoints locally With default whitelist settings, the probes can be accessed from localhost using the following URLs: -- `http://localhost/-/health` -- `http://localhost/-/readiness` -- `http://localhost/-/liveness` +```text +GET http://localhost/-/health +``` +```text +GET http://localhost/-/readiness +``` +```text +GET http://localhost/-/liveness +```text +GET http://localhost/-/health +``` + +```text +GET http://localhost/-/readiness +``` -The first endpoint, `health`, only checks whether the application server is running. It does not verify the database or other services are running. A successful response will return a 200 status code with the following message: +```text +GET http://localhost/-/liveness +``` + +## Health + +Checks whether the application server is running. It does not verify the database or other services are running. + +```text +GET /-/health +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/health' +``` + +Example response: ```text GitLab OK ``` -The readiness and liveness probes will provide a report of system health in JSON format. +## Readiness + +The readiness probe checks whether the Gitlab instance is ready to use. It checks the dependent services (Database, Redis, Gitaly etc.) and gives a status for each. + +```text +GET /-/readiness +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/readiness' +``` -`readiness` probe example output: +Example response: ```json { "db_check":{ - "status":"ok" + "status":"failed", + "message": "unexpected Db check result: 0" }, "redis_check":{ "status":"ok" @@ -65,7 +108,23 @@ The readiness and liveness probes will provide a report of system health in JSON } ``` -`liveness` probe example output: +## Liveness + +The liveness probe checks whether the application server is alive. Unlike the [`health`](#health) check, this check hits the database. + +```text +GET /-/liveness +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/liveness' +``` + +Example response: + +On success, the endpoint will return a valid successful HTTP status code, and a response like below. ```json { @@ -90,10 +149,7 @@ The readiness and liveness probes will provide a report of system health in JSON } ``` -## Status - -On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint -will return a valid successful HTTP status code, and a `success` message. +On failure, the endpoint will return a `500` HTTP status code. ## Access token (Deprecated) diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 6faab685b26..5e385b7216d 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -2,7 +2,7 @@ type: reference --- -# Account and limit settings +# Account and limit settings **(CORE ONLY)** ## Max attachment size diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 490163c1816..ddf989d0181 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -2,7 +2,7 @@ type: reference --- -# Email +# Email **(CORE ONLY)** You can customize some of the content in emails sent from your GitLab instance. diff --git a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md index b2d56be154b..8e53a6995fb 100644 --- a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md +++ b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md @@ -9,7 +9,7 @@ type: reference This setting allows you to rate limit the requests to raw endpoints, defaults to `300` requests per minute. It can be modified in **Admin Area > Network > Performance Optimization**. -For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/controllers/application_controller.rb` will be blocked. +For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/controllers/application_controller.rb` will be blocked. Access to the raw file will be released after 1 minute.  @@ -18,3 +18,5 @@ This limit is: - Applied independently per project, per commit and per file path. - Not applied per IP address. - Active by default. To disable, set the option to `0`. + +Requests over the rate limit are logged into `auth.log`. diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index 1b1bcbcd6e8..aea717e806d 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -2,7 +2,7 @@ type: reference --- -# Sign-up restrictions +# Sign-up restrictions **(CORE ONLY)** You can use sign-up restrictions to require user email confirmation, as well as to blacklist or whitelist email addresses belonging to specific domains. diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index a1bce5a6c69..baf219ac9c7 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -2,7 +2,7 @@ type: reference --- -# Enforce accepting Terms of Service +# Enforce accepting Terms of Service **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18570) > in [GitLab Core](https://about.gitlab.com/pricing/) 10.8 diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index d3c9cf7d8ff..ca26147b287 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -2,7 +2,7 @@ type: reference --- -# Third party offers +# Third party offers **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20379) > in [GitLab Core](https://about.gitlab.com/pricing/) 11.1 diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index 516600c9d99..efac7e699f3 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -2,7 +2,7 @@ type: reference --- -# Usage statistics +# Usage statistics **(CORE ONLY)** GitLab Inc. will periodically collect information about your instance in order to perform various actions. diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index e3a495750f2..b9d93bf3671 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -2,7 +2,7 @@ type: reference --- -# User and IP rate limits +# User and IP rate limits **(CORE ONLY)** Rate limiting is a common technique used to improve the security and durability of a web application. For more details, see diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index bf59f49b993..1e2f5705728 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -2,7 +2,7 @@ type: reference --- -# Visibility and access controls +# Visibility and access controls **(CORE ONLY)** GitLab allows administrators to: diff --git a/doc/user/analytics/cycle_analytics.md b/doc/user/analytics/cycle_analytics.md new file mode 100644 index 00000000000..b7389c8689d --- /dev/null +++ b/doc/user/analytics/cycle_analytics.md @@ -0,0 +1,182 @@ +# Cycle Analytics + +> - Introduced prior to GitLab 12.2 at the project level. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2 at the group level (enabled by feature flag `analytics`). + +Cycle Analytics measures the time spent to go from an [idea to production] - also known +as cycle time - for each of your projects. Cycle Analytics displays the median time for an idea to +reach production, along with the time typically spent in each DevOps stage along the way. + +Cycle Analytics is useful in order to quickly determine the velocity of a given +project. It points to bottlenecks in the development process, enabling management +to uncover, triage, and identify the root cause of slowdowns in the software development life cycle. + +Cycle Analytics is tightly coupled with the [GitLab flow] and +calculates a separate median for each stage. + +## Overview + +Cycle Analytics is available: + +- From GitLab 12.2, at the group level in the analytics workspace at + **Analytics > Cycle Analytics**. **(PREMIUM)** + + In the future, multiple groups will be selectable which will effectively make this an + instance-level feature. + + NOTE: **Note:** + Requires the [analytics workspace](index.md) to be enabled. + +- At the project level via **Project > Cycle Analytics**. + +There are seven stages that are tracked as part of the Cycle Analytics calculations. + +- **Issue** (Tracker) + - Time to schedule an issue (by milestone or by adding it to an issue board) +- **Plan** (Board) + - Time to first commit +- **Code** (IDE) + - Time to create a merge request +- **Test** (CI) + - Time it takes GitLab CI/CD to test your code +- **Review** (Merge Request/MR) + - Time spent on code review +- **Staging** (Continuous Deployment) + - Time between merging and deploying to production +- **Production** (Total) + - Total lifecycle time; i.e. the velocity of the project or team + +## How the data is measured + +Cycle Analytics records cycle time and data based on the project issues with the +exception of the staging and production stages, where only data deployed to +production are measured. + +Specifically, if your CI is not set up and you have not defined a `production` +or `production/*` [environment], then you will not have any data for those stages. + +Each stage of Cycle Analytics is further described in the table below. + +| **Stage** | **Description** | +| --------- | --------------- | +| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list](../project/issue_board.md#creating-a-new-list) created for it. | +| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. | +| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. | +| Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. | +| Review | Measures the median time taken to review the merge request that has closing issue pattern, between its creation and until it's merged. | +| Staging | Measures the median time between merging the merge request with closing issue pattern until the very first deployment to production. It's tracked by the [environment] set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a production environment, this is not tracked. | +| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. | + +--- + +How this works, behind the scenes: + +1. Issues and merge requests are grouped together in pairs, such that for each + `<issue, merge request>` pair, the merge request has the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) + for the corresponding issue. All other issues and merge requests are **not** + considered. +1. Then the `<issue, merge request>` pairs are filtered out by last XX days (specified + by the UI - default is 90 days). So it prohibits these pairs from being considered. +1. For the remaining `<issue, merge request>` pairs, we check the information that + we need for the stages, like issue creation date, merge request merge time, + etc. + +To sum up, anything that doesn't follow [GitLab flow] will not be tracked and the +Cycle Analytics dashboard will not present any data for: + +- merge requests that do not close an issue. +- issues not labeled with a label present in the Issue Board or for issues not assigned a milestone. +- staging and production stages, if the project has no `production` or `production/*` + environment. + +## Example workflow + +Below is a simple fictional workflow of a single cycle that happens in a +single day passing through all seven stages. Note that if a stage does not have +a start and a stop mark, it is not measured and hence not calculated in the median +time. It is assumed that milestones are created and CI for testing and setting +environments is configured. + +1. Issue is created at 09:00 (start of **Issue** stage). +1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of + **Plan** stage). +1. Start working on the issue, create a branch locally and make one commit at + 12:00. +1. Make a second commit to the branch which mentions the issue number at 12.30 + (stop of **Plan** stage / start of **Code** stage). +1. Push branch and create a merge request that contains the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) + in its description at 14:00 (stop of **Code** stage / start of **Test** and + **Review** stages). +1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and + takes 5min (stop of **Test** stage). +1. Review merge request, ensure that everything is OK and merge the merge + request at 19:00. (stop of **Review** stage / start of **Staging** stage). +1. Now that the merge request is merged, a deployment to the `production` + environment starts and finishes at 19:30 (stop of **Staging** stage). +1. The cycle completes and the sum of the median times of the previous stages + is recorded to the **Production** stage. That is the time between creating an + issue and deploying its relevant merge request to production. + +From the above example you can conclude the time it took each stage to complete +as long as their total time: + +- **Issue**: 2h (11:00 - 09:00) +- **Plan**: 1h (12:00 - 11:00) +- **Code**: 2h (14:00 - 12:00) +- **Test**: 5min +- **Review**: 5h (19:00 - 14:00) +- **Staging**: 30min (19:30 - 19:00) +- **Production**: Since this stage measures the sum of median time off all + previous stages, we cannot calculate it if we don't know the status of the + stages before. In case this is the very first cycle that is run in the project, + then the **Production** time is 10h 30min (19:30 - 09:00) + +A few notes: + +- In the above example we demonstrated that it doesn't matter if your first + commit doesn't mention the issue number, you can do this later in any commit + of the branch you are working on. +- You can see that the **Test** stage is not calculated to the overall time of + the cycle since it is included in the **Review** process (every MR should be + tested). +- The example above was just **one cycle** of the seven stages. Add multiple + cycles, calculate their median time and the result is what the dashboard of + Cycle Analytics is showing. + +## Permissions + +The current permissions on the Project Cycle Analytics dashboard are: + +- Public projects - anyone can access +- Internal projects - any authenticated user can access +- Private projects - any member Guest and above can access + +You can [read more about permissions][permissions] in general. + +NOTE: **Note:** +As of GitLab 12.2, the project-level page is deprecated. You should access +project-level Cycle Analytics from **Analytics > Cycle Analytics** in the top +navigation bar. We will ensure that the same project-level functionality is available +to CE users in the new analytics space. + +For Cycle Analytics functionality introduced in GitLab 12.2 and later: + +- Users must have Reporter access or above. +- Features are available only on + [Premium or Silver tiers](https://about.gitlab.com/pricing/) and above. + +## More resources + +Learn more about Cycle Analytics in the following resources: + +- [Cycle Analytics feature page](https://about.gitlab.com/features/cycle-analytics/) +- [Cycle Analytics feature preview](https://about.gitlab.com/2016/09/16/feature-preview-introducing-cycle-analytics/) +- [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/) + +[ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986 +[ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975 +[environment]: ../../ci/yaml/README.md#environment +[GitLab flow]: ../../workflow/gitlab_flow.md +[idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab +[permissions]: ../permissions.md +[yml]: ../../ci/yaml/README.md diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md new file mode 100644 index 00000000000..ec719c0b4a1 --- /dev/null +++ b/doc/user/analytics/index.md @@ -0,0 +1,22 @@ +# Analytics workspace + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) in GitLab 12.2 (enabled using `analytics` feature flag). + +The Analytics workspace will make it possible to aggregate analytics across +GitLab, so that users can view information across multiple projects and groups +in one place. + +To access the centralized analytics workspace: + +1. Ensure it's enabled. Requires a GitLab administrator to enable it with the `analytics` feature + flag. +1. Once enabled, click on **Analytics** from the top navigation bar. + +## Available analytics + +From the centralized analytics workspace, the following analytics are available: + +- [Cycle Analytics](cycle_analytics.md). + +NOTE: **Note:** +Project-level Cycle Analytics are still available at a project's **Project > Cycle Analytics**. diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 83ea0ea3386..fcd683ca2db 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -28,7 +28,7 @@ GitLab can scan and report any vulnerabilities found in your project. | [Dependency List](dependency_list/index.md) **(ULTIMATE)** | View your project's dependencies and their known vulnerabilities. | | [Dependency Scanning](dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | | [Dynamic Application Security Testing (DAST)](dast/index.md) **(ULTIMATE)** | Analyze running web applications for known vulnerabilities. | -| [License Management](license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | +| [License Compliance](license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | | [Security Dashboard](security_dashboard/index.md) **(ULTIMATE)** | View vulnerabilities in all your projects and groups. | | [Static Application Security Testing (SAST)](sast/index.md) **(ULTIMATE)** | Analyze source code for known vulnerabilities. | diff --git a/doc/user/application_security/license_management/index.md b/doc/user/application_security/license_management/index.md index c324848c703..912f2f0e209 100644 --- a/doc/user/application_security/license_management/index.md +++ b/doc/user/application_security/license_management/index.md @@ -2,7 +2,7 @@ type: reference, howto --- -# License Management **(ULTIMATE)** +# License Compliance **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5483) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. @@ -10,18 +10,18 @@ in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. ## Overview If you are using [GitLab CI/CD](../../../ci/README.md), you can search your project dependencies for their licenses -using License Management. +using License Compliance. -You can take advantage of License Management by either [including the job](#configuration) +You can take advantage of License Compliance by either [including the job](#configuration) in your existing `.gitlab-ci.yml` file or by implicitly using -[Auto License Management](../../../topics/autodevops/index.md#auto-license-management-ultimate) +[Auto License Compliance](../../../topics/autodevops/index.md#auto-license-compliance-ultimate) that is provided by [Auto DevOps](../../../topics/autodevops/index.md). -GitLab checks the License Management report, compares the licenses between the +GitLab checks the License Compliance report, compares the licenses between the source and target branches, and shows the information right on the merge request. Blacklisted licenses will be clearly visible with an `x` red icon next to them as well as new licenses which need a decision from you. In addition, you can -[manually approve or blacklist](#project-policies-for-license-management) +[manually approve or blacklist](#project-policies-for-license-compliance) licenses in your project's settings. NOTE: **Note:** @@ -31,7 +31,7 @@ will be displayed in the merge request area. That is the case when you add the Consecutive merge requests will have something to compare to and the license management report will be shown properly. - + If you are a project or group Maintainer, you can click on a license to be given the choice to approve it or blacklist it. @@ -66,12 +66,12 @@ The following languages and package managers are supported. ## Requirements -To run a License Management scanning job, you need GitLab Runner with the +To run a License Compliance scanning job, you need GitLab Runner with the [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). ## Configuration -For GitLab 11.9 and later, to enable License Management, you must +For GitLab 11.9 and later, to enable License Compliance, you must [include](../../../ci/yaml/README.md#includetemplate) the [`License-Management.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml) that's provided as a part of your GitLab installation. @@ -89,14 +89,14 @@ The included template will create a `license_management` job in your CI/CD pipel and scan your dependencies to find their licenses. The results will be saved as a -[License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) +[License Compliance report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) that you can later download and analyze. Due to implementation limitations, we -always take the latest License Management artifact available. Behind the scenes, the -[GitLab License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +always take the latest License Compliance artifact available. Behind the scenes, the +[GitLab License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/license-management) is used to detect the languages/frameworks and in turn analyzes the licenses. -The License Management settings can be changed through environment variables by using the -[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. These variables are documented in the [License Management documentation](https://gitlab.com/gitlab-org/security-products/license-management#settings). +The License Compliance settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. These variables are documented in the [License Compliance documentation](https://gitlab.com/gitlab-org/security-products/license-management#settings). ### Installing custom dependencies @@ -143,7 +143,7 @@ license_management: ### Configuring Maven projects -The License Management tool provides a `MAVEN_CLI_OPTS` environment variable which can hold +The License Compliance tool provides a `MAVEN_CLI_OPTS` environment variable which can hold the command line arguments to pass to the `mvn install` command which is executed under the hood. Feel free to use it for the customization of Maven execution. For example: @@ -169,7 +169,7 @@ If you still need to run tests during `mvn install`, add `-DskipTests=false` to > [Introduced](https://gitlab.com/gitlab-org/security-products/license-management/merge_requests/36) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. -License Management uses Python 2.7 and pip 10.0 by default. +License Compliance uses Python 2.7 and pip 10.0 by default. If your project requires Python 3, you can switch to Python 3.5 and pip 19.1 by setting the `LM_PYTHON_VERSION` environment variable to `3`. @@ -182,7 +182,7 @@ license_management: LM_PYTHON_VERSION: 3 ``` -## Project policies for License Management +## Project policies for License Compliance > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5940) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. @@ -196,10 +196,10 @@ To approve or blacklist a license: 1. Either use the **Manage licenses** button in the merge request widget, or navigate to the project's **Settings > CI/CD** and expand the - **License Management** section. + **License Compliance** section. 1. Click the **Add a license** button. -  +  1. In the **License name** dropdown, either: - Select one of the available licenses. You can search for licenses in the field @@ -211,17 +211,17 @@ To approve or blacklist a license: To modify an existing license: -1. In the **License Management** list, click the **Approved/Declined** dropdown to change it to the desired status. +1. In the **License Compliance** list, click the **Approved/Declined** dropdown to change it to the desired status. -  +  Searching for Licenses: 1. Use the **Search** box to search for a specific license. -  +  -## License Management report under pipelines +## License Compliance report under pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5491) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2. @@ -230,7 +230,7 @@ From your project's left sidebar, navigate to **CI/CD > Pipelines** and click on pipeline ID that has a `license_management` job to see the Licenses tab with the listed licenses (if any). - + <!-- ## Troubleshooting diff --git a/doc/user/discussions/img/make_suggestion.png b/doc/user/discussions/img/make_suggestion.png Binary files differindex 20acc1417da..a24e29770aa 100644 --- a/doc/user/discussions/img/make_suggestion.png +++ b/doc/user/discussions/img/make_suggestion.png diff --git a/doc/user/discussions/img/suggestion.png b/doc/user/discussions/img/suggestion.png Binary files differindex 68a67e6ae5e..f7962305a15 100644 --- a/doc/user/discussions/img/suggestion.png +++ b/doc/user/discussions/img/suggestion.png diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index c9fbd7effa0..af37cc896ad 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -43,13 +43,15 @@ Host gitlab.com Below are the settings for [GitLab Pages]. -| Setting | GitLab.com | Default | -| ----------------------- | ---------------- | ------------- | -| Domain name | `gitlab.io` | - | -| IP address | `35.185.44.232` | - | -| Custom domains support | yes | no | -| TLS certificates support| yes | no | +| Setting | GitLab.com | Default | +| --------------------------- | ---------------- | ------------- | +| Domain name | `gitlab.io` | - | +| IP address | `35.185.44.232` | - | +| Custom domains support | yes | no | +| TLS certificates support | yes | no | +| Maximum size (uncompressed) | 1G | 100M | +NOTE: **Note:** The maximum size of your Pages site is regulated by the artifacts maximum size which is part of [GitLab CI/CD](#gitlab-cicd). @@ -59,7 +61,7 @@ Below are the current settings regarding [GitLab CI/CD](../../ci/README.md). | Setting | GitLab.com | Default | | ----------- | ----------------- | ------------- | -| Artifacts maximum size | 1G | 100M | +| Artifacts maximum size (uncompressed) | 1G | 100M | | Artifacts [expiry time](../../ci/yaml/README.md#artifactsexpire_in) | kept forever | deleted after 30 days unless otherwise specified | ## Repository size limit @@ -112,57 +114,6 @@ Below are the shared Runners settings. The full contents of our `config.toml` are: -**DigitalOcean** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - [runners.machine] - IdleCount = 20 - IdleTime = 1800 - OffPeakPeriods = ["* * * * * sat,sun *"] - OffPeakTimezone = "UTC" - OffPeakIdleCount = 5 - OffPeakIdleTime = 1800 - MaxBuilds = 1 - MachineName = "srm-%s" - MachineDriver = "digitalocean" - MachineOptions = [ - "digitalocean-image=X", - "digitalocean-ssh-user=core", - "digitalocean-region=nyc1", - "digitalocean-size=s-2vcpu-2gb", - "digitalocean-private-networking", - "digitalocean-tags=shared_runners,gitlab_com", - "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR", - "digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN", - ] - [runners.cache] - Type = "s3" - BucketName = "runner" - Insecure = true - Shared = true - ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER" - AccessKey = "ACCESS_KEY" - SecretKey = "ACCESS_SECRET_KEY" -``` - **Google Cloud Platform** ```toml @@ -178,20 +129,25 @@ sentry_dsn = "X" token = "SHARED_RUNNER_TOKEN" executor = "docker+machine" environment = [ - "DOCKER_DRIVER=overlay2" + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" ] limit = X [runners.docker] image = "ruby:2.5" privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] [runners.machine] - IdleCount = 20 - IdleTime = 1800 + IdleCount = 50 + IdleTime = 3600 OffPeakPeriods = ["* * * * * sat,sun *"] OffPeakTimezone = "UTC" - OffPeakIdleCount = 5 - OffPeakIdleTime = 1800 - MaxBuilds = 1 + OffPeakIdleCount = 15 + OffPeakIdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. MachineName = "srm-%s" MachineDriver = "google" MachineOptions = [ @@ -202,17 +158,18 @@ sentry_dsn = "X" "google-tags=gitlab-com,srm", "google-use-internal-ip", "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/issues/3214#note_82892928 "google-machine-image=PROJECT/global/images/IMAGE", - "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR" + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 ] [runners.cache] - Type = "s3" - BucketName = "runner" - Insecure = true + Type = "gcs" Shared = true - ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER" - AccessKey = "ACCESS_KEY" - SecretKey = "ACCESS_SECRET_KEY" + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" ``` ## Sidekiq diff --git a/doc/user/group/index.md b/doc/user/group/index.md index d1d4f3740b0..864f1a397d5 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -342,7 +342,7 @@ underlying projects, issues, etc, by IP address. This can help ensure that particular content doesn't leave the premises, while not blocking off access to the entire instance. -Add whitelisted IP subnet using CIDR notation to the group settings and anyone +Add one or more whitelisted IP subnets using CIDR notation in comma separated format to the group settings and anyone coming from a different IP address won't be able to access the restricted content. diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index e3f657af564..bd50367681e 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -39,6 +39,25 @@ However, users will not be prompted to log via SSO on each visit. GitLab will ch We intend to add a similar SSO requirement for [Git and API activity](https://gitlab.com/gitlab-org/gitlab-ee/issues/9152) in the future. +#### Group-managed accounts + +[Introduced in GitLab 12.1](https://gitlab.com/groups/gitlab-org/-/epics/709). + +When SSO is being enforced, groups can enable an additional level of protection by enforcing the creation of dedicated user accounts to access the group. + +Without group-managed accounts, users can link their SAML identity with any existing user on the instance. With group-managed accounts enabled, users are required to create a new, dedicated user linked to the group. The notification email address associated with the user is locked to the email address received from the configured identity provider. + +When this option is enabled: + +- All existing and new users in the group will be required to log in via the SSO URL associated with the group. +- On successfully authenticating, GitLab will prompt the user to create a new, dedicated account using the email address received from the configured identity provider. +- After the group managed account has been created, group activity will require the use of this user account. + +Since use of the group managed account requires the use of SSO, users of group managed accounts will lose access to these accounts when they are no longer able to authenticate with the connected identity provider. In the case of an offboarded employee who has been removed from your identity provider: + +- The user will be unable to access the group (their credentials will no longer work on the identity provider when prompted to SSO). +- Contributions in the group (e.g. issues, merge requests) will remain intact. + ### NameID GitLab.com uses the SAML NameID to identify users. The NameID element: diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index f8bef8b8a6a..5d136ad62da 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -59,15 +59,14 @@ Once [Single sign-on](index.md) has been configured, we can: ### Azure -First, double check the [Single sign-on](index.md) configuration for your group and ensure that **Name identifier value** (NameID) points to `user.objectid` or another unique identifier. This will match the `extern_uid` used on GitLab. +The SAML application that was created during [Single sign-on](index.md) setup now needs to be set up for SCIM. - +1. Check the configuration for your GitLab SAML app and ensure that **Name identifier value** (NameID) points to `user.objectid` or another unique identifier. This will match the `extern_uid` used on GitLab. -#### Set up admin credentials +  -Next, configure your GitLab application in Azure by following the -[Provisioning users and groups to applications that support SCIM](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim) -section in Azure's SCIM setup documentation. +1. Set up automatic provisioning and administrative credentials by following the + [Provisioning users and groups to applications that support SCIM](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim) section in Azure's SCIM setup documentation. During this configuration, note the following: @@ -97,6 +96,7 @@ You can then test the connection by clicking on **Test Connection**. If the conn NOTE: **Note:** If you used a unique identifier **other than** `objectId`, be sure to map it instead to both `id` and `externalId`. 1. Below the mapping list click on **Show advanced options > Edit attribute list for AppName**. + 1. Leave the `id` as the primary and only required field. NOTE: **Note:** @@ -129,8 +129,7 @@ When testing the connection, you may encounter an error: **You appear to have en When checking the Audit Logs for the Provisioning, you can sometimes see the error `Namespace can't be blank, Name can't be blank, and User can't be blank.` -This is likely caused because not all required fields (such as first name and -last name) are present for all users being mapped. +This is likely caused because not all required fields (such as first name and last name) are present for all users being mapped. As a workaround, try an alternate mapping: diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 6cfc8b6429b..17bbed2945d 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -57,12 +57,6 @@ render incorrectly: - milk ``` -1. Chocolate - - dark - - milk - ---- - Simply add a space to each nested item to align the `-` with the first character of the top list item (`C` in this case): @@ -265,8 +259,7 @@ this font installed by default. ### Front matter -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) - in GitLab 11.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) in GitLab 11.6. Front matter is metadata included at the beginning of a markdown document, preceding its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/), @@ -866,18 +859,6 @@ or underscores ___ ``` -Three or more hyphens, - ---- - -asterisks, - -*** - -or underscores - -___ - ### Images Examples: diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 4fd7c5abf78..80d1bf992ec 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -45,7 +45,7 @@ The following table depicts the various user permission levels in a project. | Leave comments | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View Insights charts **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View approved/blacklisted licenses **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View license management reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View License Compliance reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View Security reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View Dependency list **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View [Design Management](project/issues/design_management.md) pages **(PREMIUM)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | @@ -162,7 +162,7 @@ to learn more. ### Cycle Analytics permissions Find the current permissions on the Cycle Analytics dashboard on -the [documentation on Cycle Analytics permissions](project/cycle_analytics.md#permissions). +the [documentation on Cycle Analytics permissions](analytics/cycle_analytics.md#permissions). ### Issue Board permissions @@ -237,13 +237,16 @@ To learn more, read through the documentation on ## Guest User -Create a user and assign to a project with a role as `Guest` user, this user -will be considered as guest user by GitLab and will not take up the license. -There is no specific `Guest` role for newly created users. If this user will -be assigned a higher role to any of the projects and groups then this user will -take a license seat. If a user creates a project this user becomes a maintainer, -therefore, takes up a license seat as well, in order to prevent this you have -to go and edit user profile and mark the user as External. +When a user is given `Guest` permissions on a project and/or group, and holds no +higher permission level on any other project or group on the instance, the user +is considered a guest user by GitLab and will not consume a license seat. +There is no other specific "guest" designation for newly created users. + +If the user is assigned a higher role on any projects or groups, the user will +take a license seat. If a user creates a project, the user becomes a `Maintainer` +on the project, resulting in the use of a license seat. To prevent a guest user +from creating projects, you can edit the user profile to mark the user as +[External](#external-users-permissions). ## External users permissions diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index e2b1a20a605..f7ba921aa7d 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -48,6 +48,7 @@ To enable 2FA: - [andOTP](https://github.com/andOTP/andOTP): feature rich open source app for Android which supports PGP encrypted backups. - [FreeOTP](https://freeotp.github.io/): open source app for Android. - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en): proprietary app for iOS and Android. + - [SailOTP](https://openrepos.net/content/seiichiro0185/sailotp): open source app for SailFish OS. 1. In the application, add a new entry in one of two ways: - Scan the code presented in GitLab with your device's camera to add the entry automatically. diff --git a/doc/user/project/cycle_analytics.md b/doc/user/project/cycle_analytics.md index 424bee6e9f1..87577c9ec88 100644 --- a/doc/user/project/cycle_analytics.md +++ b/doc/user/project/cycle_analytics.md @@ -1,181 +1,5 @@ -# Cycle Analytics - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) at a group level in [GitLab Premium and Silver](https://about.gitlab.com/pricing/) 12.2 (enabled by feature flag `analytics`). - -Cycle Analytics measures the time spent to go from an [idea to production] - also known -as cycle time - for each of your projects. Cycle Analytics displays the median time for an idea to -reach production, along with the time typically spent in each DevOps stage along the way. - -Cycle Analytics is useful in order to quickly determine the velocity of a given -project. It points to bottlenecks in the development process, enabling management -to uncover, triage, and root-cause slowdowns in the software development life cycle. - -Cycle Analytics is tightly coupled with the [GitLab flow] and -calculates a separate median for each stage. - -## Overview - -Cycle Analytics are available at a: - -- Group level from the top navigation bar **Analytics > Cycle Analytics**. **(PREMIUM)** - - In the future, multiple groups will be selectable which will effectively make this an - instance-level feature. - -- Project level from a project's **Project > Cycle Analytics**. - -  - -There are seven stages that are tracked as part of the Cycle Analytics calculations. - -- **Issue** (Tracker) - - Time to schedule an issue (by milestone or by adding it to an issue board) -- **Plan** (Board) - - Time to first commit -- **Code** (IDE) - - Time to create a merge request -- **Test** (CI) - - Time it takes GitLab CI/CD to test your code -- **Review** (Merge Request/MR) - - Time spent on code review -- **Staging** (Continuous Deployment) - - Time between merging and deploying to production -- **Production** (Total) - - Total lifecycle time; i.e. the velocity of the project or team - -## How the data is measured - -Cycle Analytics records cycle time and data based on the project issues with the -exception of the staging and production stages, where only data deployed to -production are measured. - -Specifically, if your CI is not set up and you have not defined a `production` -or `production/*` [environment], then you will not have any data for those stages. - -Below you can see in more detail what the various stages of Cycle Analytics mean. - -| **Stage** | **Description** | -| --------- | --------------- | -| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list][board] created for it. | -| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. | -| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern] to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. | -| Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. | -| Review | Measures the median time taken to review the merge request that has closing issue pattern, between its creation and until it's merged. | -| Staging | Measures the median time between merging the merge request with closing issue pattern until the very first deployment to production. It's tracked by the [environment] set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a production environment, this is not tracked. | -| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. | - +--- +redirect_to: '../analytics/cycle_analytics.md' --- -Here's a little explanation of how this works behind the scenes: - -1. Issues and merge requests are grouped together in pairs, such that for each - `<issue, merge request>` pair, the merge request has the [issue closing pattern] - for the corresponding issue. All other issues and merge requests are **not** - considered. -1. Then the `<issue, merge request>` pairs are filtered out by last XX days (specified - by the UI - default is 90 days). So it prohibits these pairs from being considered. -1. For the remaining `<issue, merge request>` pairs, we check the information that - we need for the stages, like issue creation date, merge request merge time, - etc. - -To sum up, anything that doesn't follow the [GitLab flow] won't be tracked at all. -So, the Cycle Analytics dashboard won't present any data: - -- For merge requests that do not close an issue. -- For issues not labeled with a label present in the Issue Board or for issues not assigned a milestone. -- For staging and production stages, if the project has no `production` or `production/*` - environment. - -## Example workflow - -Below is a simple fictional workflow of a single cycle that happens in a -single day passing through all seven stages. Note that if a stage does not have -a start and a stop mark, it is not measured and hence not calculated in the median -time. It is assumed that milestones are created and CI for testing and setting -environments is configured. - -1. Issue is created at 09:00 (start of **Issue** stage). -1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of - **Plan** stage). -1. Start working on the issue, create a branch locally and make one commit at - 12:00. -1. Make a second commit to the branch which mentions the issue number at 12.30 - (stop of **Plan** stage / start of **Code** stage). -1. Push branch and create a merge request that contains the [issue closing pattern] - in its description at 14:00 (stop of **Code** stage / start of **Test** and - **Review** stages). -1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and - takes 5min (stop of **Test** stage). -1. Review merge request, ensure that everything is OK and merge the merge - request at 19:00. (stop of **Review** stage / start of **Staging** stage). -1. Now that the merge request is merged, a deployment to the `production` - environment starts and finishes at 19:30 (stop of **Staging** stage). -1. The cycle completes and the sum of the median times of the previous stages - is recorded to the **Production** stage. That is the time between creating an - issue and deploying its relevant merge request to production. - -From the above example you can conclude the time it took each stage to complete -as long as their total time: - -- **Issue**: 2h (11:00 - 09:00) -- **Plan**: 1h (12:00 - 11:00) -- **Code**: 2h (14:00 - 12:00) -- **Test**: 5min -- **Review**: 5h (19:00 - 14:00) -- **Staging**: 30min (19:30 - 19:00) -- **Production**: Since this stage measures the sum of median time off all - previous stages, we cannot calculate it if we don't know the status of the - stages before. In case this is the very first cycle that is run in the project, - then the **Production** time is 10h 30min (19:30 - 09:00) - -A few notes: - -- In the above example we demonstrated that it doesn't matter if your first - commit doesn't mention the issue number, you can do this later in any commit - of the branch you are working on. -- You can see that the **Test** stage is not calculated to the overall time of - the cycle since it is included in the **Review** process (every MR should be - tested). -- The example above was just **one cycle** of the seven stages. Add multiple - cycles, calculate their median time and the result is what the dashboard of - Cycle Analytics is showing. - -## Permissions - -The current permissions on the Project Cycle Analytics dashboard are: - -- Public projects - anyone can access -- Internal projects - any authenticated user can access -- Private projects - any member Guest and above can access - -You can [read more about permissions][permissions] in general. - -NOTE: **Note:** -As of GitLab 12.2, the project-level page is deprecated. You should access -project-level Cycle Analytics from **Analytics > Cycle Analytics** in the top -navigation bar. We will ensure that the same project-level functionality is available -to CE users in the new analytics space. - -For Cycle Analytics functionality introduced in GitLab 12.2 and later: - -- Users must have Reporter access or above. -- Features are available only on - [Premium or Silver tiers](https://about.gitlab.com/pricing/) and above. - -## More resources - -Learn more about Cycle Analytics in the following resources: - -- [Cycle Analytics feature page](https://about.gitlab.com/features/cycle-analytics/) -- [Cycle Analytics feature preview](https://about.gitlab.com/2016/09/16/feature-preview-introducing-cycle-analytics/) -- [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/) - -[board]: issue_board.md#creating-a-new-list -[ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986 -[ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975 -[environment]: ../../ci/yaml/README.md#environment -[GitLab flow]: ../../workflow/gitlab_flow.md -[idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab -[issue closing pattern]: issues/managing_issues.md#closing-issues-automatically -[permissions]: ../permissions.md -[yml]: ../../ci/yaml/README.md +This document was moved to [another location](../analytics/cycle_analytics.md) diff --git a/doc/user/project/img/cycle_analytics_landing_page.png b/doc/user/project/img/cycle_analytics_landing_page.png Binary files differdeleted file mode 100644 index c0c07e84a82..00000000000 --- a/doc/user/project/img/cycle_analytics_landing_page.png +++ /dev/null diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index f48a158e2d3..2f87f257754 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -1,21 +1,21 @@ # Project importing from GitLab.com to your private GitLab instance -You can import your existing GitLab.com projects to your GitLab instance. But keep in mind that it is possible only if -GitLab.com integration is enabled on your GitLab instance. +You can import your existing GitLab.com projects to your GitLab instance, but keep in +mind that it is possible only if GitLab.com integration is enabled on your GitLab instance. [Read more about GitLab.com integration for self-managed GitLab instances](../../../integration/gitlab.md). To get to the importer page you need to go to "New project" page. >**Note:** -If you are interested in importing Wiki and Merge Request data to your new -instance, you'll need to follow the instructions for [project export](../settings/import_export.md) +If you are interested in importing Wiki and Merge Request data to your new instance, +you'll need to follow the instructions for [exporting a project](../settings/import_export.md#exporting-a-project-and-its-data) - + -Click on the "Import projects from GitLab.com" link and you will be redirected to GitLab.com +Go to the **Import Projects** tab, then click on **GitLab.com**, and you will be redirected to GitLab.com for permission to access your projects. After accepting, you'll be automatically redirected to the importer.  -To import a project, you can simple click "Import". The importer will import your repository and issues. +To import a project, click "Import". The importer will import your repository and issues. Once the importer is done, a new GitLab project will be created with your imported data. diff --git a/doc/user/project/import/img/gitlab_new_project_page.png b/doc/user/project/import/img/gitlab_new_project_page.png Binary files differdeleted file mode 100644 index c673724f436..00000000000 --- a/doc/user/project/import/img/gitlab_new_project_page.png +++ /dev/null diff --git a/doc/user/project/import/img/gitlab_new_project_page_v12_2.png b/doc/user/project/import/img/gitlab_new_project_page_v12_2.png Binary files differnew file mode 100644 index 00000000000..e79c27f32c0 --- /dev/null +++ b/doc/user/project/import/img/gitlab_new_project_page_v12_2.png diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 30ff0e9ff07..932e7fd10b2 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -98,7 +98,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Maven packages](packages/maven_repository.md): your private Maven repository in GitLab. **(PREMIUM)** - [NPM packages](packages/npm_registry.md): your private NPM package registry in GitLab. **(PREMIUM)** - [Code owners](code_owners.md): specify code owners for certain files **(STARTER)** -- [License Management](../application_security/license_management/index.md): approve and blacklist licenses for projects. **(ULTIMATE)** +- [License Compliance](../application_security/license_management/index.md): approve and blacklist licenses for projects. **(ULTIMATE)** - [Dependency List](../application_security/dependency_list/index.md): view project dependencies. **(ULTIMATE)** ### Project integrations diff --git a/doc/user/project/integrations/img/download_as_csv.png b/doc/user/project/integrations/img/download_as_csv.png Binary files differnew file mode 100644 index 00000000000..0ed5ab8db89 --- /dev/null +++ b/doc/user/project/integrations/img/download_as_csv.png diff --git a/doc/user/project/integrations/img/generate_link_to_chart.png b/doc/user/project/integrations/img/generate_link_to_chart.png Binary files differnew file mode 100644 index 00000000000..03e018969b1 --- /dev/null +++ b/doc/user/project/integrations/img/generate_link_to_chart.png diff --git a/doc/user/project/integrations/img/jira_service_page.png b/doc/user/project/integrations/img/jira_service_page.png Binary files differdeleted file mode 100644 index 76fd5f4641c..00000000000 --- a/doc/user/project/integrations/img/jira_service_page.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_service_page_v12_2.png b/doc/user/project/integrations/img/jira_service_page_v12_2.png Binary files differnew file mode 100644 index 00000000000..ba7dad9b438 --- /dev/null +++ b/doc/user/project/integrations/img/jira_service_page_v12_2.png diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index ca990ee6c32..61f6f6c9412 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -93,7 +93,7 @@ even if the status you are changing to is the same. After saving the configuration, your GitLab project will be able to interact with all Jira projects in your Jira instance and you'll see the Jira link on the GitLab project pages that takes you to the appropriate Jira project. - + ## Jira issues diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index aa7db97c413..9fc0ade809e 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -269,6 +269,12 @@ Note the following properties:  +### Downloading data as CSV + +Data from Prometheus charts on the metrics dashboard can be downloaded as CSV. + + + ### Setting up alerts for Prometheus metrics **(ULTIMATE)** #### Managed Prometheus instances @@ -354,7 +360,7 @@ Prometheus server.  -## Embedding metric charts within Gitlab Flavored Markdown +## Embedding metric charts within GitLab Flavored Markdown > [Introduced][ce-29691] in GitLab 12.2. > Requires [Kubernetes](prometheus_library/kubernetes.md) metrics. @@ -363,6 +369,10 @@ It is possible to display metrics charts within [GitLab Flavored Markdown](../.. To display a metric chart, include a link of the form `https://<root_url>/<project>/environments/<environment_id>/metrics`. +A single chart may also be embedded. You can generate a link to the chart via the dropdown located on the right side of the chart: + + + The following requirements must be met for the metric to unfurl: - The `<environment_id>` must correspond to a real environment. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index bffbcb544e3..1324a90e00b 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -35,9 +35,19 @@ to be enabled: ## Limitations -- Files uploaded must have a file extension of either `png`, `jpg`, `jpeg`, `gif`, `bmp`, `tiff` or `ico`. The [`svg` extension is not yet supported](https://gitlab.com/gitlab-org/gitlab-ee/issues/12771). +- Files uploaded must have a file extension of either `png`, `jpg`, `jpeg`, `gif`, `bmp`, `tiff` or `ico`. + The [`svg` extension is not yet supported](https://gitlab.com/gitlab-org/gitlab-ee/issues/12771). +- Design uploads are limited to 10 files at a time. - [Designs cannot yet be deleted](https://gitlab.com/gitlab-org/gitlab-ee/issues/11089). -- Design Management is [not yet supported in the project export](https://gitlab.com/gitlab-org/gitlab-ee/issues/11090). +- Design Management is + [not yet supported in the project export](https://gitlab.com/gitlab-org/gitlab-ee/issues/11090). +- Design Management data + [isn't deleted when a project is destroyed](https://gitlab.com/gitlab-org/gitlab-ee/issues/13429) yet. +- Design Management data [won't be moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/13426) + when an issue is moved, nor [deleted](https://gitlab.com/gitlab-org/gitlab-ee/issues/13427) + when an issue is deleted. +- Design Management + [isn't supported by Geo](https://gitlab.com/groups/gitlab-org/-/epics/1633) yet. ## The Design Management page diff --git a/doc/user/project/members/img/access_requests_management.png b/doc/user/project/members/img/access_requests_management.png Binary files differindex 8996d9564d7..9a1c9621e41 100644 --- a/doc/user/project/members/img/access_requests_management.png +++ b/doc/user/project/members/img/access_requests_management.png diff --git a/doc/user/project/members/img/request_access_button.png b/doc/user/project/members/img/request_access_button.png Binary files differindex e8b490b91b8..e693f9a9ac2 100644 --- a/doc/user/project/members/img/request_access_button.png +++ b/doc/user/project/members/img/request_access_button.png diff --git a/doc/user/project/members/img/withdraw_access_request_button.png b/doc/user/project/members/img/withdraw_access_request_button.png Binary files differindex 6a3172dfcdb..e5a8fe0b356 100644 --- a/doc/user/project/members/img/withdraw_access_request_button.png +++ b/doc/user/project/members/img/withdraw_access_request_button.png diff --git a/doc/user/project/merge_requests/fast_forward_merge.md b/doc/user/project/merge_requests/fast_forward_merge.md index 4f3c68090e4..2d59e535ce1 100644 --- a/doc/user/project/merge_requests/fast_forward_merge.md +++ b/doc/user/project/merge_requests/fast_forward_merge.md @@ -15,7 +15,7 @@ to accept merge requests without creating merge commits. When the fast-forward merge ([`--ff-only`](https://git-scm.com/docs/git-merge#git-merge---ff-only)) setting is enabled, no merge commits will be created and all merges are fast-forwarded, -which means that merging is only allowed if the branch could be fast-forwarded. +which means that merging is only allowed if the branch can be fast-forwarded. When a fast-forward merge is not possible, the user is given the option to rebase. @@ -28,9 +28,15 @@ When a fast-forward merge is not possible, the user is given the option to rebas Now, when you visit the merge request page, you will be able to accept it **only if a fast-forward merge is possible**. + + +If a fast-forward merge is not possible but a conflict free rebase is possible, +a rebase button will be offered. +  -If the target branch is ahead of the source branch, you need to rebase the +If the target branch is ahead of the source branch and a conflict free rebase is +not possible, you need to rebase the source branch locally before you will be able to do a fast-forward merge.  diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png Binary files differdeleted file mode 100644 index 6a09412533f..00000000000 --- a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png Binary files differnew file mode 100644 index 00000000000..32b9a3b9ce4 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png diff --git a/doc/user/project/merge_requests/img/cross-project-dependencies-edit-inaccessible.png b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_inaccessible_v12_2.png Binary files differindex 2dc02634fd8..2dc02634fd8 100644 --- a/doc/user/project/merge_requests/img/cross-project-dependencies-edit-inaccessible.png +++ b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_inaccessible_v12_2.png diff --git a/doc/user/project/merge_requests/img/cross-project-dependencies-edit.png b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_v12_2.png Binary files differindex 362e7e0ead2..362e7e0ead2 100644 --- a/doc/user/project/merge_requests/img/cross-project-dependencies-edit.png +++ b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_v12_2.png diff --git a/doc/user/project/merge_requests/img/cross-project-dependencies-view.png b/doc/user/project/merge_requests/img/cross_project_dependencies_view_v12_2.png Binary files differindex e00231c839b..e00231c839b 100644 --- a/doc/user/project/merge_requests/img/cross-project-dependencies-view.png +++ b/doc/user/project/merge_requests/img/cross_project_dependencies_view_v12_2.png diff --git a/doc/user/project/merge_requests/img/ff_merge_mr.png b/doc/user/project/merge_requests/img/ff_merge_mr.png Binary files differnew file mode 100644 index 00000000000..241cc990343 --- /dev/null +++ b/doc/user/project/merge_requests/img/ff_merge_mr.png diff --git a/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png b/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png Binary files differnew file mode 100644 index 00000000000..ee94dbdea5c --- /dev/null +++ b/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 8a82b163481..04db54872d3 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -41,7 +41,7 @@ With **[GitLab Enterprise Edition][ee]**, you can also: - View the deployment process across projects with [Multi-Project Pipelines](../../../ci/multi_project_pipelines.md) **(PREMIUM)** - Request [approvals](merge_request_approvals.md) from your managers **(STARTER)** - Analyze the impact of your changes with [Code Quality reports](code_quality.md) **(STARTER)** -- Manage the licenses of your dependencies with [License Management](../../application_security/license_management/index.md) **(ULTIMATE)** +- Manage the licenses of your dependencies with [License Compliance](../../application_security/license_management/index.md) **(ULTIMATE)** - Analyze your source code for vulnerabilities with [Static Application Security Testing](../../application_security/sast/index.md) **(ULTIMATE)** - Analyze your running web applications for vulnerabilities with [Dynamic Application Security Testing](../../application_security/dast/index.md) **(ULTIMATE)** - Analyze your dependencies for vulnerabilities with [Dependency Scanning](../../application_security/dependency_scanning/index.md) **(ULTIMATE)** @@ -57,7 +57,7 @@ A. Consider you are a software developer working in a team: 1. You gather feedback from your team 1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) **(STARTER)** 1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD -1. You avoid using dependencies whose license is not compatible with your project with [License Management reports](license_management.md) **(ULTIMATE)** +1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](license_management.md) **(ULTIMATE)** 1. You request the [approval](#merge-request-approvals-starter) from your manager 1. Your manager pushes a commit with their final review, [approves the merge request](merge_request_approvals.md), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD @@ -497,6 +497,15 @@ list.  +### Incrementally expand merge request diffs + +By default, the diff shows only the parts of a file which are changed. +To view more unchanged lines above or below a change click on the +**Expand up** or **Expand down** icons. You can also click on **Show all lines** +to expand the entire file. + + + ## Ignore whitespace changes in Merge Request diff view If you click the **Hide whitespace changes** button, you can see the diff diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index 5161b25de99..a0432414bcf 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -83,7 +83,7 @@ request approval rules: 1. Give the approval rule a name that describes the set of approvers selected. 1. Click **Add approvers** to submit the new rule. -  +  ## Multiple approval rules **(PREMIUM)** diff --git a/doc/user/project/merge_requests/merge_request_dependencies.md b/doc/user/project/merge_requests/merge_request_dependencies.md index e046b3466c4..b30e24b2386 100644 --- a/doc/user/project/merge_requests/merge_request_dependencies.md +++ b/doc/user/project/merge_requests/merge_request_dependencies.md @@ -2,9 +2,9 @@ type: reference, concepts --- -# Cross-project merge request dependencies **(PREMIUM)** +# Cross-project Merge Request dependencies **(PREMIUM)** -> Introduced in GitLab Premium 12.2 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9688) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. Cross-project merge request dependencies allows a required order of merging between merge requests in different projects to be expressed. If a @@ -24,11 +24,11 @@ merge requests in the same project cannot depend on each other. ## Use cases - Ensure changes to a library are merged before changes to a project that - imports the library + imports the library. - Prevent a documentation-only merge request from being merged before the merge request - implementing the feature to be documented + implementing the feature to be documented. - Require an merge request updating a permissions matrix to be merged before merging an - merge request from someone who hasn't yet been granted permissions + merge request from someone who hasn't yet been granted permissions. It is common for a single logical change to span several merge requests, spread out across multiple projects, and the order in which they are merged can be @@ -60,33 +60,33 @@ new merge request in `awesome-project` (or by editing it, if it already exists). The dependency needs to be configured on the **dependent** merge request. There is a "Cross-project dependencies" section in the form: - + Anyone who can edit a merge request can change the list of dependencies. New dependencies can be added by reference, or by URL. To remove a dependency, -press the "X" by its reference. +press the **X** by its reference. As dependencies are specified across projects, it's possible that someone else has added a dependency for a merge request in a project you don't have access to. These are shown as a simple count: - + -If necessary, you can remove all the dependencies like this by pressing the "X", -just as you would for a single, visible dependency. +If necessary, you can remove all the dependencies like this by pressing the +**X**, just as you would for a single, visible dependency. -Once you're finished, press the "Save changes" button to submit the request, or -"Cancel" to return without making any changes. +Once you're finished, press the **Save changes** button to submit the request, +or **Cancel** to return without making any changes. The list of configured dependencies, and the status of each one, is shown in the merge request widget: - + -Until all dependencies have, themselves, been merged, the "Merge" +Until all dependencies have, themselves, been merged, the **Merge** button will be disabled for the dependent merge request. In -particular, note that **closed** merge request still prevent their +particular, note that **closed merge requests** still prevent their dependents from being merged - it is impossible to automatically determine whether the dependency expressed by a closed merge request has been satisfied in some other way or not. diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index 03ae24242e3..8606d92f20c 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -28,13 +28,13 @@ The reasons to do it like that are: With the new behavior, any job that is triggered by the user, is also marked with their read permissions. When a user does a `git push` or changes files through the web UI, a new pipeline will be usually created. This pipeline will be marked -as created be the pusher (local push or via the UI) and any job created in this +as created by the pusher (local push or via the UI) and any job created in this pipeline will have the read permissions of the pusher but not write permissions. This allows us to make it really easy to evaluate the access for all projects that have [Git submodules][gitsub] or are using container images that the pusher -would have access too. **The permission is granted only for time that job is -running. The access is revoked after the job is finished.** +would have access too. **The permission is granted only for the time that the job +is running. The access is revoked after the job is finished.** ## Types of users @@ -74,7 +74,7 @@ We try to make sure that this token doesn't leak by: 1. Securing all API endpoints to not expose the job token. 1. Masking the job token from job logs. -1. Allowing to use the job token **only** when job is running. +1. Granting permissions to the job token **only** when the job is running. However, this brings a question about the Runners security. To make sure that this token doesn't leak, you should also make sure that you configure @@ -86,12 +86,6 @@ your Runners in the most possible secure way, by avoiding the following: By using an insecure GitLab Runner configuration, you allow the rogue developers to steal the tokens of other jobs. -## Pipeline triggers - -Since 9.0 [pipeline triggers][triggers] do support the new permission model. -The new triggers do impersonate their associated user including their access -to projects and their project permissions. - ## Before GitLab 8.12 In versions before GitLab 8.12, all CI jobs would use the CI Runner's token @@ -203,7 +197,7 @@ Container Registries for private projects. > **Notes:** > > - GitLab Runner versions prior to 1.8 don't incorporate the introduced changes -> for permissions. This makes the `image:` directive to not work with private +> for permissions. This makes the `image:` directive not work with private > projects automatically and it needs to be configured manually on Runner's host > with a predefined account (for example administrator's personal account with > access token created explicitly for this purpose). This issue is resolved with @@ -227,11 +221,22 @@ test: - docker run $CI_REGISTRY/group/other-project:latest ``` +### Pipeline triggers + +Since 9.0 [pipeline triggers][triggers] do support the new permission model. +The new triggers do impersonate their associated user including their access +to projects and their project permissions. + +### API + +GitLab API cannot be used via `CI_JOB_TOKEN` but there is a [proposal](https://gitlab.com/gitlab-org/gitlab-ce/issues/29566) +to support it. + [job permissions]: ../permissions.md#job-permissions [comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302 [gitsub]: ../../ci/git_submodules.md [https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols -[triggers]: ../../ci/triggers/README.md +[triggers]: ../../ci/triggers/README.md#ci-job-token [update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update [workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse [jobenv]: ../../ci/variables/README.md#predefined-environment-variables diff --git a/doc/user/project/operations/feature_flags.md b/doc/user/project/operations/feature_flags.md index 19ccde6e16a..75b0623e6b0 100644 --- a/doc/user/project/operations/feature_flags.md +++ b/doc/user/project/operations/feature_flags.md @@ -112,6 +112,19 @@ If this strategy is selected, then the Unleash client **must** be given a user i **Percent rollout (logged in users)** is implemented using the Unleash [gradualRolloutUserId](https://unleash.github.io/docs/activation_strategy#gradualrolloutuserid) activation strategy. +## Target Users + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8240) in GitLab 12.2. + +A feature flag may be enabled for a list of target users. + + + +CAUTION: **Caution:** +The Unleash client **must** be given a user id for the feature to be enabled for target users. See the [Ruby example](#ruby-application-example) below. + +**Target users** is implemented using the Unleash [userWithId](https://unleash.github.io/docs/activation_strategy#userwithid) activation strategy. + ## Integrating with your application In order to use Feature Flags, you need to first @@ -207,7 +220,7 @@ func main() { Here's an example of how to integrate the feature flags in a Ruby application. -The Unleash client is given a user id for use with a **Percent rollout (logged in users)** rollout strategy. +The Unleash client is given a user id for use with a **Percent rollout (logged in users)** rollout strategy or a list of **Target Users**. ```ruby #!/usr/bin/env ruby diff --git a/doc/user/project/operations/img/target_users_v12_2.png b/doc/user/project/operations/img/target_users_v12_2.png Binary files differnew file mode 100644 index 00000000000..c88d2b7be97 --- /dev/null +++ b/doc/user/project/operations/img/target_users_v12_2.png diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 6758adf2b43..647250bd02a 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -40,18 +40,20 @@ discussions, and descriptions: | `/label ~label1 ~label2` | Add label(s). Label names can also start without ~ but mixed syntax is not supported. | ✓ | ✓ | | `/unlabel ~label1 ~label2` | Remove all or specific label(s)| ✓ | ✓ | | `/relabel ~label1 ~label2` | Replace existing label(s) with those specified | ✓ | ✓ | -| `/copy_metadata <#issue | !merge_request>` | Copy labels and milestone from other issue or merge request in the project | ✓ | ✓ | +| `/copy_metadata <#issue>` | Copy labels and milestone from another issue in the project | ✓ | ✓ | +| `/copy_metadata <!merge_request>` | Copy labels and milestone from another merge request in the project | ✓ | ✓ | | `/estimate <1w 3d 2h 14m>` | Set time estimate | ✓ | ✓ | | `/remove_estimate` | Remove time estimate | ✓ | ✓ | -| `/spend <time(1h 30m | -1h 5m)> <date(YYYY-MM-DD)>` | Add or subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ | +| `/spend <time(1h 30m)> <date(YYYY-MM-DD)>` | Add spent time; optionally, specify the date that time was spent on | ✓ | ✓ | +| `/spend <time(-1h 5m)> <date(YYYY-MM-DD)>` | Subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ | | `/remove_time_spent` | Remove time spent | ✓ | ✓ | | `/lock` | Lock the thread | ✓ | ✓ | | `/unlock` | Unlock the thread | ✓ | ✓ | -| `/due <in 2 days | this Friday | December 31st>`| Set due date | ✓ | | +| `/due <date>` | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | ✓ | | | `/remove_due_date` | Remove due date | ✓ | | -| `/weight <0 | 1 | 2 | ...>` | Set weight **(STARTER)** | ✓ | | +| `/weight <value>` | Set weight. Valid options for `<value>` include `0`, `1`, `2`, etc. **(STARTER)** | ✓ | | | `/clear_weight` | Clears weight **(STARTER)** | ✓ | | -| `/epic <&epic | group&epic | Epic URL>` | Add to epic **(ULTIMATE)** | ✓ | | +| `/epic <epic>` | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. **(ULTIMATE)** | ✓ | | | `/remove_epic` | Removes from epic **(ULTIMATE)** | ✓ | | | `/promote` | Promote issue to epic **(ULTIMATE)** | ✓ | | | `/confidential` | Make confidential | ✓ | | @@ -110,9 +112,9 @@ The following quick actions are applicable for epics threads and description: | `/label ~label1 ~label2` | Add label(s) | | `/unlabel ~label1 ~label2` | Remove all or specific label(s) | | `/relabel ~label1 ~label2` | Replace existing label(s) with those specified | -| `/child_epic <&epic | group&epic | Epic URL>` | Adds child epic to epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) | -| `/remove_child_epic <&epic | group&epic | Epic URL>` | Removes child epic from epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) | -| `/parent_epic <&epic | group&epic | Epic URL>` | Sets parent epic to epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) | +| `/child_epic <epic>` | Adds child epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([Introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) **(ULTIMATE)**| +| `/remove_child_epic <epic>` | Removes child epic from `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([Introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) **(ULTIMATE)** | +| `/parent_epic <epic>` | Sets parent epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) **(ULTIMATE)** | | `/remove_parent_epic` | Removes parent epic from epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) | <!-- ## Troubleshooting diff --git a/doc/user/project/repository/img/repository_languages.png b/doc/user/project/repository/img/repository_languages.png Binary files differdeleted file mode 100644 index 5977ad7faae..00000000000 --- a/doc/user/project/repository/img/repository_languages.png +++ /dev/null diff --git a/doc/user/project/repository/img/repository_languages_v12_2.gif b/doc/user/project/repository/img/repository_languages_v12_2.gif Binary files differnew file mode 100644 index 00000000000..d0a0e57c919 --- /dev/null +++ b/doc/user/project/repository/img/repository_languages_v12_2.gif diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 84d63d29929..bd966185c94 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -182,7 +182,7 @@ were used and display this on the projects pages. If this information is missing be added after updating the default branch on the project. This process can take up to 5 minutes. - + Not all files are detected, among others; documentation, vendored code, and most markup languages are excluded. This behaviour can be diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index e3c6cd6d6ff..d1bab47de25 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -28,11 +28,12 @@ NOTE: **Note:** Requires Developer [permissions](../../permissions.md). Create a new page by clicking the **New page** button that can be found -in all wiki pages. You will be asked to fill in the page name from which GitLab -will create the path to the page. You can specify a full path for the new file -and any missing directories will be created automatically. +in all wiki pages. - +You will be asked to fill in a title for your new wiki page. Wiki titles +also determine the path to the wiki page. You can specify a full path +(using "`/`" for subdirectories) for the new title and any missing +directories will be created automatically. Once you enter the page name, it's time to fill in its content. GitLab wikis support Markdown, RDoc and AsciiDoc. For Markdown based pages, all the diff --git a/doc/workflow/git_annex.md b/doc/workflow/git_annex.md index 9543859f86f..cdc38a5b7eb 100644 --- a/doc/workflow/git_annex.md +++ b/doc/workflow/git_annex.md @@ -200,7 +200,7 @@ searching for your distribution. Although there is no general guide for `git-annex` errors, there are a few tips on how to go around the warnings. -### git-annex-shell: Not a git-annex or gcrypt repository. +### git-annex-shell: Not a git-annex or gcrypt repository This warning can appear on the initial `git annex sync --content` and is caused by differences in `git-annex-shell`. You can read more about it diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md index 3160b0c4deb..7e880b3009d 100644 --- a/doc/workflow/lfs/lfs_administration.md +++ b/doc/workflow/lfs/lfs_administration.md @@ -46,8 +46,7 @@ In `config/gitlab.yml`: ## Storing LFS objects in remote object storage -> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core -in 10.7. +> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7. It is possible to store LFS objects in remote object storage which allows you to offload local hard disk R/W operations, and free up disk space significantly. @@ -91,7 +90,7 @@ Here is a configuration example with S3. | `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` | | `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | -| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with AWS v4 signatures (https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -107,7 +106,9 @@ Here is a configuration example with GCS. | `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | | `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` | -_NOTE: The service account must have permission to access the bucket. [See more](https://cloud.google.com/storage/docs/authentication)_ +NOTE: **Note:** +The service account must have permission to access the bucket. +[See more](https://cloud.google.com/storage/docs/authentication) Here is a configuration example with Rackspace Cloud Files. @@ -119,7 +120,13 @@ Here is a configuration example with Rackspace Cloud Files. | `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | | `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | -_NOTES: Regardless of whether the container has public access enabled or disabled, Fog will use the TempURL method to grant access to LFS objects. If you see errors in logs referencing instantiating storage with a temp-url-key, ensure that you have set they key properly on the Rackspace API and in gitlab.rb. You can verify the value of the key Rackspace has set by sending a GET request with token header to the service access endpoint URL and comparing the output of the returned headers._ +NOTE: **Note:** +Regardless of whether the container has public access enabled or disabled, Fog will +use the TempURL method to grant access to LFS objects. If you see errors in logs referencing +instantiating storage with a temp-url-key, ensure that you have set they key properly +on the Rackspace API and in gitlab.rb. You can verify the value of the key Rackspace +has set by sending a GET request with token header to the service access endpoint URL +and comparing the output of the returned headers. ### Manual uploading to an object storage @@ -167,13 +174,13 @@ On Omnibus installations, the settings are prefixed by `lfs_object_store_`: 1. Save the file and [reconfigure GitLab]s for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: - ```bash - gitlab-rake gitlab:lfs:migrate - ``` + ```bash + gitlab-rake gitlab:lfs:migrate + ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless - `gitlab_rails['lfs_object_store_background_upload']` is set to false. + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless + `gitlab_rails['lfs_object_store_background_upload']` is set to false. ### S3 for installations from source @@ -203,13 +210,13 @@ For source installations the settings are nested under `lfs:` and then 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: - ```bash - sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production - ``` + ```bash + sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production + ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless `background_upload` is set to - false. + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless `background_upload` is set to + false. ## Storage statistics diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index 264372a512d..f5593927cc2 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -35,9 +35,10 @@ Documentation for GitLab instance administrators is under [LFS administration do - Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the URL to Git config manually (see [troubleshooting](#troubleshooting)) ->**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication - still goes over HTTP, but now the SSH client passes the correct credentials - to the Git LFS client, so no action is required by the user. +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. ## Using Git LFS @@ -61,12 +62,13 @@ git commit -am "Added Debian iso" # commit the file meta data git push origin master # sync the git repo and large file to the GitLab server ``` -> **Note**: Make sure that `.gitattributes` is tracked by git. Otherwise Git -> LFS will not be working properly for people cloning the project. -> -> ```bash -> git add .gitattributes -> ``` +NOTE: **Note:** +**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git +LFS will not be working properly for people cloning the project. + +```bash +git add .gitattributes +``` Cloning the repository works the same as before. Git automatically detects the LFS-tracked files and clones them via HTTP. If you performed the git clone @@ -216,9 +218,10 @@ git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs" ### Credentials are always required when pushing an object ->**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication - still goes over HTTP, but now the SSH client passes the correct credentials - to the Git LFS client, so no action is required by the user. +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing the LFS object on every push for every object, user HTTPS credentials are required. diff --git a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md index 75673d23799..5ceeb3253f1 100644 --- a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md +++ b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md @@ -76,7 +76,7 @@ Here you'll find a guide on Since Annex files are stored as objects with symlinks and cannot be directly modified, we need to first remove those symlinks. ->**Note:** +NOTE: **Note:** Make sure the you read about the [`direct` mode][annex-direct] as it contains useful information that may fit in your use case. Note that `annex direct` is deprecated in Git Annex version 6, so you may need to upgrade your repository diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index ccb8844aea3..5abae929355 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -53,7 +53,7 @@ These settings can be configured on group page under the name of the group. It w The group owner can disable email notifications for a group, which also includes it's subgroups and projects. If this is the case, you will not receive any corresponding notifications, -and the notification button will be disabled with an explanatory tooltip. +and the notification button will be disabled with an explanatory tooltip. ### Project Settings @@ -66,7 +66,7 @@ These settings can be configured on project page under the name of the project. The project owner (or it's group owner) can disable email notifications for the project. If this is the case, you will not receive any corresponding notifications, and the notification -button will be disabled with an explanatory tooltip. +button will be disabled with an explanatory tooltip. ## Notification events diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md index 0b8e7d851b0..753518d0424 100644 --- a/doc/workflow/repository_mirroring.md +++ b/doc/workflow/repository_mirroring.md @@ -37,8 +37,7 @@ The following are some possible use cases for repository mirroring: ## Pushing to a remote repository **(CORE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in GitLab Enterprise -> Edition 8.7. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in GitLab Enterprise Edition 8.7. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. For an existing project, you can set up push mirroring as follows: @@ -67,8 +66,7 @@ section. ### Push only protected branches **(CORE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. You can choose to only push your protected branches from GitLab to your remote repository. @@ -98,8 +96,7 @@ The repository will push soon. To force a push, click the appropriate button. ## Pulling from a remote repository **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. -> [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab-ee/issues/10871) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab-ee/issues/10871) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.11. NOTE: **Note:** This feature [is available for free](https://gitlab.com/gitlab-org/gitlab-ee/issues/10361) to GitLab.com users until September 22nd, 2019. @@ -157,8 +154,7 @@ Repository mirrors are updated as Sidekiq becomes available to process them. If ### SSH authentication -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551) for Pull mirroring in [GitLab Starter](https://about.gitlab.com/pricing/) 9.5. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22982) for Push mirroring in [GitLab Core](https://about.gitlab.com/pricing/) 11.6 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551) for Pull mirroring in [GitLab Starter](https://about.gitlab.com/pricing/) 9.5. [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22982) for Push mirroring in [GitLab Core](https://about.gitlab.com/pricing/) 11.6 SSH authentication is mutual: @@ -245,8 +241,7 @@ key to keep the mirror running. ### Overwrite diverged branches **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.6. You can choose to always update your local branches with remote versions, even if they have diverged from the remote. @@ -258,8 +253,7 @@ To use this option, check the **Overwrite diverged branches** box when creating ### Only mirror protected branches **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. You can choose to pull mirror only the protected branches from your remote repository to GitLab. Non-protected branches are not mirrored and can diverge. @@ -268,8 +262,7 @@ To use this option, check the **Only mirror protected branches** box when creati ### Hard failure **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.2. Once the mirroring process is unsuccessfully retried 14 times in a row, it will get marked as hard failed. This will become visible in either the: @@ -282,8 +275,7 @@ project mirroring again by [Forcing an update](#forcing-an-update-core). ### Trigger update using API **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453) in -[GitLab Starter](https://about.gitlab.com/pricing/) 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. Pull mirroring uses polling to detect new branches and commits added upstream, often minutes afterwards. If you notify GitLab by [API](../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), @@ -325,9 +317,10 @@ protected branches. ### Preventing conflicts using a `pre-receive` hook -> **Warning:** The solution proposed will negatively impact the performance of -> Git push operations because they will be proxied to the upstream Git -> repository. +CAUTION: **Warning:** +The solution proposed will negatively impact the performance of +Git push operations because they will be proxied to the upstream Git +repository. A server-side `pre-receive` hook can be used to prevent the race condition described above by only accepting the push after first pushing the commit to diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 0432c406f31..211d242e2d3 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -35,8 +35,8 @@ A To Do displays on your To-Do List when: - You are `@mentioned` in a comment on a commit - A job in the CI pipeline running for your merge request failed, but this job is not allowed to fail -- An open merge request becomes unmergeable due to conflict, and you are either: - - The author +- An open merge request becomes unmergeable due to conflict, and you are either: + - The author - Have set it to automatically merge once the pipeline succeeds To-do triggers are not affected by [GitLab Notification Email settings](notifications.md). |
