diff options
| author | Filipa Lacerda <filipa@gitlab.com> | 2019-07-03 22:39:10 +0100 |
|---|---|---|
| committer | Filipa Lacerda <filipa@gitlab.com> | 2019-07-03 22:39:10 +0100 |
| commit | 50be7237f41b0ac44b9aaf8b73c57993548d4c35 (patch) | |
| tree | ecfeeae58829dadbd90de4f834c730d1d8c55e74 /doc | |
| parent | 35331c435196ea1155eb15161f3f9a481a01501d (diff) | |
| parent | 2ad75a4f96c4d377e18788966e7eefee4d78b6d2 (diff) | |
| download | gitlab-ce-update-todo-in-ui.tar.gz | |
Merge branch 'master' into update-todo-in-uiupdate-todo-in-ui
* master: (435 commits)
Change occurrence of Sidekiq::Testing.inline!
Fix order-dependent spec failure in appearance_spec.rb
Put a failed example from appearance_spec in quarantine
Cache PerformanceBar.allowed_user_ids list locally and in Redis
Add Grafana to Admin > Monitoring menu when enabled
Add changelog entry
Add salesforce logo
Move error_tracking_frontend specs to Jest
Only save Peek session in Redis when Peek is enabled
Migrate markdown header_spec.js to Jest
Fix golint command in Go guide doc to be recursive
Move images to their own dirs
Gitlab -> GitLab
Re-align CE and EE API docs
Rename Release groups in issue_workflow.md
Update api docs to finish aligning EE and CE docs
Update locale.pot
Update TODO: allow_collaboration column renaming
Show upcoming status for releases
Rebased and squashed commits
...
Diffstat (limited to 'doc')
208 files changed, 6300 insertions, 3442 deletions
diff --git a/doc/README.md b/doc/README.md index 3863e17c268..489c8117b9d 100644 --- a/doc/README.md +++ b/doc/README.md @@ -54,7 +54,7 @@ GitLab provides solutions for [all the stages of the DevOps lifecycle](https://a  GitLab is like a top-of-the-line kitchen for making software. As the executive -chef, you decide what software you want serve. Using your recipe, GitLab handles +chef, you decide what software you want to serve. Using your recipe, GitLab handles all the prep work, cooking, and delivery, so you can turn around orders faster than ever. @@ -209,7 +209,7 @@ The following documentation relates to the DevOps **Create** stage: | [GitLab API](api/README.md) | Integrate GitLab via a simple and powerful API. | | [GitLab Integration](integration/README.md) | Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. | | [GitLab Webhooks](user/project/integrations/webhooks.md) | Let GitLab notify you when new code has been pushed to your project. | -| [JIRA Development Panel](integration/jira_development_panel.md) **[PREMIUM]** | See GitLab information in the JIRA Development Panel. | +| [Jira Development Panel](integration/jira_development_panel.md) **[PREMIUM]** | See GitLab information in the Jira Development Panel. | | [Project Services](user/project/integrations/project_services.md) | Integrate a project with external services, such as CI and chat. | | [Trello Power-Up](integration/trello_power_up.md) | Integrate with GitLab's Trello Power-Up. | diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index 760af0cfd1a..c668f19ca7d 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -13,7 +13,7 @@ The steps below cover: ## Configuring Google LDAP client -1. Navigate to https://admin.google.com and sign in as a GSuite domain administrator. +1. Navigate to <https://admin.google.com> and sign in as a GSuite domain administrator. 1. Go to **Apps > LDAP > Add Client**. diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 15f093bb62d..b45966fa920 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -185,7 +185,7 @@ group, as opposed to the full DN. ## Global group memberships lock -"Lock memberships to LDAP synchronization" setting allows instance administrators +"Lock memberships to LDAP synchronization" setting allows instance administrators to lock down user abilities to invite new members to a group. When enabled following happens: 1. Only administrator can manage memberships of any group including access levels. @@ -198,14 +198,14 @@ to lock down user abilities to invite new members to a group. When enabled follo NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. You can manually configure LDAP user sync times by setting the following configuration values. The example below shows how to set LDAP user -sync to run once every 12 hours at the top of the hour. +sync to run once every 12 hours at the top of the hour. **Omnibus installations** @@ -233,7 +233,7 @@ sync to run once every 12 hours at the top of the hour. NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a group sync process every hour, on the hour. @@ -245,8 +245,8 @@ for installations with a large number of LDAP users. Please review the your installation compares before proceeding. You can manually configure LDAP group sync times by setting the -following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. **Omnibus installations** diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 54279897e04..79ac7fe0352 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -1,6 +1,4 @@ -[//]: # (Do *NOT* modify this file in EE documentation. All changes in this) -[//]: # (file should happen in CE, too. If the change is EE-specific, put) -[//]: # (it in `ldap-ee.md`.) +<!-- If the change is EE-specific, put it in `ldap-ee.md`, NOT here. --> # LDAP @@ -396,21 +394,34 @@ omniauth-ldap. ### Escaping special characters -If the `user_filter` DN contains special characters. For example, a comma: +The `user_filter` DN can contain special characters. For example: -``` -OU=GitLab, Inc,DC=gitlab,DC=com -``` +- A comma: -This character needs to be escaped as documented in [RFC 4515](https://tools.ietf.org/search/rfc4515). + ``` + OU=GitLab, Inc,DC=gitlab,DC=com + ``` -Due to the way the string is parsed, the special character needs to be converted -to hex and `\\5C\\` (`5C` = `\` in hex) added before it. -As an example the above DN would look like +- Open and close brackets: -``` -OU=GitLab\\5C\\2C Inc,DC=gitlab,DC=com -``` + ``` + OU=Gitlab (Inc),DC=gitlab,DC=com + ``` + + These characters must be escaped as documented in + [RFC 4515](https://tools.ietf.org/search/rfc4515). + +- Escape commas with `\2C`. For example: + + ``` + OU=GitLab\2C Inc,DC=gitlab,DC=com + ``` + +- Escape open and close brackets with `\28` and `\29`, respectively. For example: + + ``` + OU=Gitlab \28Inc\29,DC=gitlab,DC=com + ``` ## Enabling LDAP sign-in for existing GitLab users diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 00422ec347c..6e48add6930 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -144,20 +144,20 @@ for more details: If you're having trouble, here are some tips: 1. Ensure `discovery` is set to `true`. Setting it to `false` requires -specifying all the URLs and keys required to make OpenID work. + specifying all the URLs and keys required to make OpenID work. 1. Check your system clock to ensure the time is synchronized properly. 1. As mentioned in [the -documentation](https://github.com/m0n9oose/omniauth_openid_connect), -make sure `issuer` corresponds to the base URL of the Discovery URL. For -example, `https://accounts.google.com` is used for the URL -`https://accounts.google.com/.well-known/openid-configuration`. + documentation](https://github.com/m0n9oose/omniauth_openid_connect), + make sure `issuer` corresponds to the base URL of the Discovery URL. For + example, `https://accounts.google.com` is used for the URL + `https://accounts.google.com/.well-known/openid-configuration`. 1. The OpenID Connect client uses HTTP Basic Authentication to send the -OAuth2 access token. For example, if you are seeing 401 errors upon -retrieving the `userinfo` endpoint, you may want to check your OpenID -Web server configuration. For example, for -[oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you -may need to [add a configuration parameter to -Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). + OAuth2 access token. For example, if you are seeing 401 errors upon + retrieving the `userinfo` endpoint, you may want to check your OpenID + Web server configuration. For example, for + [oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you + may need to [add a configuration parameter to + Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 4d55f2357c1..2e4b4efa0ac 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -689,6 +689,20 @@ You can add a configuration option for backwards compatibility. 1. Restart the registry for the changes to take affect. +### Docker connection error + +A Docker connection error can occur when there are special characters in either the group, +project or branch name. Special characters can include: + +* Leading underscore +* Trailing hyphen/dash +* Double hyphen/dash + +To get around this, you can [change the group path](../user/group/index.md#changing-a-groups-path), +[change the project path](../user/project/settings/index.md#renaming-a-repository) or change the +branch name. Another option is to create a [push rule](../push_rules/push_rules.html) to prevent +this at the instance level. + [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 7f3be402b84..98404ff2a10 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -40,16 +40,16 @@ For example, say you have a primary (`db1.gitlab.com`) and two secondaries, `db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 load balancers, one for every host. For example: -* `primary.gitlab.com` forwards to `db1.gitlab.com` -* `secondary1.gitlab.com` forwards to `db2.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db1.gitlab.com` +- `secondary1.gitlab.com` forwards to `db2.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` Now let's say that a failover happens and db2 becomes the new primary. This means forwarding should now happen as follows: -* `primary.gitlab.com` forwards to `db2.gitlab.com` -* `secondary1.gitlab.com` forwards to `db1.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db2.gitlab.com` +- `secondary1.gitlab.com` forwards to `db1.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` GitLab does not take care of this for you, so you will need to do so yourself. @@ -209,9 +209,9 @@ without it immediately leading to errors being presented to the users. The load balancer logs various messages, such as: -* When a host is marked as offline -* When a host comes back online -* When all secondaries are offline +- When a host is marked as offline +- When a host comes back online +- When all secondaries are offline Each log message contains the tag `[DB-LB]` to make searching/filtering of such log entries easier. For example: diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index d4c8c2d3624..e19cd9bbfec 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -143,11 +143,13 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch 1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:  -1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. +1. Navigate to the project's repository directory on both **primary** and **secondary** nodes + (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` + is customized, check the directory layout on your server to be sure. - ```sh - cd /var/opt/gitlab/git-data/repositories - ``` + ```sh + cd /var/opt/gitlab/git-data/repositories + ``` 1. Run the following command on the **primary** node, redirecting the output to a file: diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index f4d31a98080..9a981b49349 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -21,20 +21,20 @@ To bring the former **primary** node up to date: 1. SSH into the former **primary** node that has fallen behind. 1. Make sure all the services are up: - ```sh - sudo gitlab-ctl start - ``` - - > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], - > you need to undo those steps now. For Debian/Ubuntu you just need to run - > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install - > the GitLab instance from scratch and set it up as a **secondary** node by - > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. - > - > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - > for this node during disaster recovery procedure you may need to [block - > all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic) - > during this procedure. + ```sh + sudo gitlab-ctl start + ``` + + NOTE: **Note:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], + you need to undo those steps now. For Debian/Ubuntu you just need to run + `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + the GitLab instance from scratch and set it up as a **secondary** node by + following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. + + NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) + for this node during disaster recovery procedure you may need to [block + all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) + during this procedure. 1. [Setup database replication][database-replication]. Note that in this case, **primary** node refers to the current **primary** node, and **secondary** node refers to the diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 71dc797f281..86182b84062 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -39,48 +39,50 @@ must disable the **primary** node. 1. SSH into the **primary** node to stop and disable GitLab, if possible: - ```sh - sudo gitlab-ctl stop - ``` + ```sh + sudo gitlab-ctl stop + ``` - Prevent GitLab from starting up again if the server unexpectedly reboots: + Prevent GitLab from starting up again if the server unexpectedly reboots: - ```sh - sudo systemctl disable gitlab-runsvdir - ``` + ```sh + sudo systemctl disable gitlab-runsvdir + ``` - > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being - > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). - > It may be safest to uninstall the GitLab package completely: + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). + It may be safest to uninstall the GitLab package completely: - ```sh - yum remove gitlab-ee - ``` + ```sh + yum remove gitlab-ee + ``` - > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu - > or any other distro based on the Upstart init system, you can prevent GitLab - > from starting if the machine reboots by doing the following: + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distro based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots by doing the following: - ```sh - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```sh + initctl stop gitlab-runsvvdir + echo 'manual' > /etc/init/gitlab-runsvdir.override + initctl reload-configuration + ``` 1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (e.g., point the primary DNS record to the **secondary** - node in order to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. - -1. If you plan to - [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (e.g., point the primary DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node @@ -94,26 +96,26 @@ the **secondary** to the **primary**. 1. SSH in to your **secondary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` 1. Promote the **secondary** node to the **primary** node. Execute: - ```sh - gitlab-ctl promote-to-primary-node - ``` + ```sh + gitlab-ctl promote-to-primary-node + ``` 1. Verify you can connect to the newly promoted **primary** node using the URL used previously for the **secondary** node. @@ -129,31 +131,31 @@ do this manually. 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: - ```bash - sudo gitlab-pg-ctl promote - ``` + ```bash + sudo gitlab-pg-ctl promote + ``` 1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: - ```bash - sudo gitlab-rake geo:set_secondary_as_primary - ``` + ```bash + sudo gitlab-rake geo:set_secondary_as_primary + ``` 1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. @@ -167,37 +169,37 @@ secondary domain, like changing Git remotes and API URLs. 1. SSH into the **secondary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Update the primary domain's DNS record. After updating the primary domain's DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the **secondary** node to reflect the new URL: - ```ruby - # Change the existing external_url configuration - external_url 'https://<new_external_url>' - ``` + ```ruby + # Change the existing external_url configuration + external_url 'https://<new_external_url>' + ``` - NOTE: **Note** - Changing `external_url` won't prevent access via the old secondary URL, as - long as the secondary DNS records are still intact. + NOTE: **Note** + Changing `external_url` won't prevent access via the old secondary URL, as + long as the secondary DNS records are still intact. 1. Reconfigure the **secondary** node for the change to take effect: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Execute the command below to update the newly promoted **primary** node URL: - ```sh - gitlab-rake geo:update_primary_node_url - ``` + ```sh + gitlab-rake geo:update_primary_node_url + ``` - This command will use the changed `external_url` configuration defined - in `/etc/gitlab/gitlab.rb`. + This command will use the changed `external_url` configuration defined + in `/etc/gitlab/gitlab.rb`. 1. Verify you can connect to the newly promoted **primary** using its URL. If you updated the DNS records for the primary domain, these changes may @@ -231,62 +233,61 @@ and after that you also need two extra steps. 1. SSH into the new **primary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` - ```ruby - ## Enable a Geo Primary role (if you haven't yet) - roles ['geo_primary_role'] + ```ruby + ## Enable a Geo Primary role (if you haven't yet) + roles ['geo_primary_role'] - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have - postgresql['max_replication_slots'] = 1 + # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + postgresql['max_replication_slots'] = 1 - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` - ``` - - (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) + (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied. - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` - Restart PostgreSQL for its changes to take effect: + Restart PostgreSQL for its changes to take effect: - ```sh - gitlab-ctl restart postgresql - ``` + ```sh + gitlab-ctl restart postgresql + ``` 1. Re-enable migrations now that PostgreSQL is restarted and listening on the private address. - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - ```ruby - gitlab_rails['auto_migrate'] = true - ``` + ```ruby + gitlab_rails['auto_migrate'] = true + ``` - Save the file and reconfigure GitLab: + Save the file and reconfigure GitLab: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` ### Step 2. Initiate the replication process diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index b8071b5993f..c1a95157f8d 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -143,26 +143,26 @@ access to the **primary** node during the maintenance window. all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and the **secondary** node's IP. - For instance, you might run the following commands on the server(s) making up your **primary** node: + For instance, you might run the following commands on the server(s) making up your **primary** node: - ```sh - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT + ```sh + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` - From this point, users will be unable to view their data or make changes on the - **primary** node. They will also be unable to log in to the **secondary** node. - However, existing sessions will work for the remainder of the maintenance period, and - public data will be accessible throughout. + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. @@ -187,10 +187,11 @@ access to the **primary** node during the maintenance window. before it is completed will cause the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: - - All replication meters to each 100% replicated, 0% failures. - - All verification meters reach 100% verified, 0% failures. - - Database replication lag is 0ms. - - The Geo log cursor is up to date (0 events behind). + + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 3d4f69d3abe..dd5e09c0dd7 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -16,11 +16,10 @@ The basic steps of configuring a **secondary** node are to: You are encouraged to first read through all the steps before executing them in your testing/production environment. -> **Notes:** -> - **Do not** setup any custom authentication for the **secondary** nodes. This will be - handled by the **primary** node. -> - Any change that requires access to the **Admin Area** needs to be done in the - **primary** node because the **secondary** node is a read-only replica. +NOTE: **Note:** +**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. +Any change that requires access to the **Admin Area** needs to be done in the +**primary** node because the **secondary** node is a read-only replica. ### Step 1. Manually replicate secret GitLab values @@ -31,47 +30,47 @@ they must be manually replicated to the **secondary** node. 1. SSH into the **primary** node, and execute the command below: - ```sh - sudo cat /etc/gitlab/gitlab-secrets.json - ``` + ```sh + sudo cat /etc/gitlab/gitlab-secrets.json + ``` - This will display the secrets that need to be replicated, in JSON format. + This will display the secrets that need to be replicated, in JSON format. 1. SSH into the **secondary** node and login as the `root` user: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Make a backup of any existing secrets: - ```sh - mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` - ``` + ```sh + mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` + ``` 1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or copy-and-paste the file contents between nodes: - ```sh - sudo editor /etc/gitlab/gitlab-secrets.json + ```sh + sudo editor /etc/gitlab/gitlab-secrets.json - # paste the output of the `cat` command you ran on the primary - # save and exit - ``` + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` 1. Ensure the file permissions are correct: - ```sh - chown root:root /etc/gitlab/gitlab-secrets.json - chmod 0600 /etc/gitlab/gitlab-secrets.json - ``` + ```sh + chown root:root /etc/gitlab/gitlab-secrets.json + chmod 0600 /etc/gitlab/gitlab-secrets.json + ``` 1. Reconfigure the **secondary** node for the change to take effect: - ```sh - gitlab-ctl reconfigure - gitlab-ctl restart - ``` + ```sh + gitlab-ctl reconfigure + gitlab-ctl restart + ``` ### Step 2. Manually replicate the **primary** node's SSH host keys @@ -89,80 +88,80 @@ keys must be manually replicated to the **secondary** node. 1. SSH into the **secondary** node and login as the `root` user: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Make a backup of any existing SSH host keys: - ```sh - find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; - ``` + ```sh + find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + ``` 1. Copy OpenSSH host keys from the **primary** node: - If you can access your **primary** node using the **root** user: + If you can access your **primary** node using the **root** user: - ```sh - # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server - scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh - ``` + ```sh + # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh + ``` - If you only have access through a user with **sudo** privileges: + If you only have access through a user with **sudo** privileges: - ```sh - # Run this from your primary node: - sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + ```sh + # Run this from your primary node: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* - # Run this from your secondary node: - scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . - tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh - ``` + # Run this from your secondary node: + scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + ``` 1. On your **secondary** node, ensure the file permissions are correct: - ```sh - chown root:root /etc/ssh/ssh_host_*_key* - chmod 0600 /etc/ssh/ssh_host_*_key* - ``` + ```sh + chown root:root /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key* + ``` 1. To verify key fingerprint matches, execute the following command on both nodes: - ```sh - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done - ``` + ```sh + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ``` - You should get an output similar to this one and they should be identical on both nodes: + You should get an output similar to this one and they should be identical on both nodes: - ```sh - 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) - 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) - 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) - 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) - ``` + ```sh + 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) + 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) + 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) + 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) + ``` 1. Verify that you have the correct public keys for the existing private keys: - ```sh - # This will print the fingerprint for private keys: - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ```sh + # This will print the fingerprint for private keys: + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done - # This will print the fingerprint for public keys: - for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done - ``` + # This will print the fingerprint for public keys: + for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + ``` - NOTE: **Note**: - The output for private keys and public keys command should generate the same fingerprint. + NOTE: **Note:** + The output for private keys and public keys command should generate the same fingerprint. 1. Restart sshd on your **secondary** node: - ```sh - # Debian or Ubuntu installations - sudo service ssh reload + ```sh + # Debian or Ubuntu installations + sudo service ssh reload - # CentOS installations - sudo service sshd reload - ``` + # CentOS installations + sudo service sshd reload + ``` ### Step 3. Add the **secondary** node @@ -176,22 +175,22 @@ keys must be manually replicated to the **secondary** node. 1. Click the **Add node** button. 1. SSH into your GitLab **secondary** server and restart the services: - ```sh - gitlab-ctl restart - ``` + ```sh + gitlab-ctl restart + ``` - Check if there are any common issue with your Geo setup by running: + Check if there are any common issue with your Geo setup by running: - ```sh - gitlab-rake gitlab:geo:check - ``` + ```sh + gitlab-rake gitlab:geo:check + ``` 1. SSH into your **primary** server and login as root to verify the **secondary** node is reachable or there are any common issue with your Geo setup: - ```sh - gitlab-rake gitlab:geo:check - ``` + ```sh + gitlab-rake gitlab:geo:check + ``` Once added to the admin panel and restarted, the **secondary** node will automatically start replicating missing data from the **primary** node in a process known as **backfill**. @@ -250,9 +249,8 @@ The two most obvious issues that can become apparent in the dashboard are: 1. Database replication not working well. 1. Instance to instance notification not working. In that case, it can be something of the following: - - You are using a custom certificate or custom CA (see the - [troubleshooting document]). - - The instance is firewalled (check your firewall rules). + - You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)). + - The instance is firewalled (check your firewall rules). Please note that disabling a **secondary** node will stop the synchronization process. @@ -304,5 +302,4 @@ See the [troubleshooting document](troubleshooting.md). [gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 [gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 [omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html -[troubleshooting document]: troubleshooting.md [using-geo]: using_a_geo_server.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 6658c37752a..021ed2d9f3c 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -52,186 +52,188 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. SSH into your GitLab **primary** server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Execute the command below to define the node as **primary** node: - ```sh - gitlab-ctl set-geo-primary-node - ``` + ```sh + gitlab-ctl set-geo-primary-node + ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. 1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: - Generate a MD5 hash of the desired password: + Generate a MD5 hash of the desired password: - ```sh - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` - Edit `/etc/gitlab/gitlab.rb`: + Edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` 1. Omnibus GitLab already has a [replication user] called `gitlab_replicator`. You must set the password for this user manually. You will be prompted to enter a password: - ```sh - gitlab-ctl set-replication-password - ``` + ```sh + gitlab-ctl set-replication-password + ``` - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. - If you are using an external database not managed by Omnibus GitLab, you need - to create the replicator user and define a password to it manually: + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually: - ```sql - --- Create a new user 'replicator' - CREATE USER gitlab_replicator; + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; - --- Set/change a password and grants replication privilege - ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; - ``` + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; + ``` 1. Configure PostgreSQL to listen on network interfaces: - For security reasons, PostgreSQL does not listen on any network interfaces - by default. However, Geo requires the **secondary** node to be able to - connect to the **primary** node's database. For this reason, we need the address of - each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). - - If you are using a cloud provider, you can lookup the addresses for each - Geo node through your cloud provider's management console. - - To lookup the address of a Geo node, SSH in to the Geo node and execute: - - ```sh - ## - ## Private address - ## - ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' - - ## - ## Public address - ## - echo "External address: $(curl --silent ipinfo.io/ip)" - ``` - - In most cases, the following addresses will be used to configure GitLab - Geo: - - | Configuration | Address | - |:----------------------------------------|:------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | - - If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC) you can use the **secondary** node's private - address (corresponds to "internal address" for Google Cloud Platform) for - `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. - - The `listen_address` option opens PostgreSQL up to network connections - with the interface corresponding to the given address. See [the PostgreSQL - documentation][pg-docs-runtime-conn] for more details. - - Depending on your network configuration, the suggested addresses may not - be correct. If your **primary** node and **secondary** nodes connect over a local - area network, or a virtual network connecting availability zones like - [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) - you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Primary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_primary_role'] - - ## - ## Primary address - ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node - ## - postgresql['listen_address'] = '<primary_node_ip>' - - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - - ## - ## Replication settings - ## - set this to be the number of Geo secondary nodes you have - ## - postgresql['max_replication_slots'] = 1 - # postgresql['max_wal_senders'] = 10 - # postgresql['wal_keep_segments'] = 10 - - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false - ``` + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. + + NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```sh + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **secondary** node's private + address (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections + with the interface corresponding to the given address. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '<primary_node_ip>' + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` 1. Optional: If you want to add another **secondary** node, the relevant setting would look like: - ```ruby - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] - ``` + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + ``` - You may also want to edit the `wal_keep_segments` and `max_wal_senders` to - match your database replication requirements. Consult the [PostgreSQL - - Replication documentation][pg-docs-runtime-replication] - for more information. + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the [PostgreSQL - + Replication documentation][pg-docs-runtime-replication] + for more information. 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` - Restart PostgreSQL for its changes to take effect: + Restart PostgreSQL for its changes to take effect: - ```sh - gitlab-ctl restart postgresql - ``` + ```sh + gitlab-ctl restart postgresql + ``` 1. Re-enable migrations now that PostgreSQL is restarted and listening on the private address. - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - ```ruby - gitlab_rails['auto_migrate'] = true - ``` + ```ruby + gitlab_rails['auto_migrate'] = true + ``` - Save the file and reconfigure GitLab: + Save the file and reconfigure GitLab: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Now that the PostgreSQL server is set up to accept remote connections, run `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port @@ -241,143 +243,143 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o will be used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL - `server.crt` file on the **primary** node by running this command: + `server.crt` file on the **primary** node by running this command: - ```sh - cat ~gitlab-psql/data/server.crt - ``` + ```sh + cat ~gitlab-psql/data/server.crt + ``` - Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive - data. + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. ### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: - ``` - sudo -i - ``` + ``` + sudo -i + ``` 1. Stop application server and Sidekiq - ``` - gitlab-ctl stop unicorn - gitlab-ctl stop sidekiq - ``` + ``` + gitlab-ctl stop unicorn + gitlab-ctl stop sidekiq + ``` - NOTE: **Note**: - This step is important so we don't try to execute anything before the node is fully configured. + NOTE: **Note:** + This step is important so we don't try to execute anything before the node is fully configured. 1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server: - ```sh - gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] - ``` + ```sh + gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + ``` - NOTE: **Note**: - If this step fails, you may be using the wrong IP address, or a firewall may - be preventing access to the server. Check the IP address, paying close - attention to the difference between public and private addresses and ensure - that, if a firewall is present, the **secondary** node is permitted to connect to the - **primary** node on port 5432. + NOTE: **Note:** + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. 1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: - ``` - editor server.crt - ``` + ``` + editor server.crt + ``` 1. Set up PostgreSQL TLS verification on the **secondary** node: - Install the `server.crt` file: + Install the `server.crt` file: - ```sh - install \ - -D \ - -o gitlab-psql \ - -g gitlab-psql \ - -m 0400 \ - -T server.crt ~gitlab-psql/.postgresql/root.crt - ``` + ```sh + install \ + -D \ + -o gitlab-psql \ + -g gitlab-psql \ + -m 0400 \ + -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` - PostgreSQL will now only recognize that exact certificate when verifying TLS - connections. The certificate can only be replicated by someone with access - to the private key, which is **only** present on the **primary** node. + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. 1. Test that the `gitlab-psql` user can connect to the **primary** node's database (the default Omnibus database name is gitlabhq_production): - ```sh - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - --list \ - -U gitlab_replicator \ - -d "dbname=gitlabhq_production sslmode=verify-ca" \ - -W \ - -h <primary_node_ip> - ``` + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + --list \ + -U gitlab_replicator \ + -d "dbname=gitlabhq_production sslmode=verify-ca" \ + -W \ + -h <primary_node_ip> + ``` - When prompted enter the password you set in the first step for the - `gitlab_replicator` user. If all worked correctly, you should see - the list of **primary** node's databases. + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. - A failure to connect here indicates that the TLS configuration is incorrect. - Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node - match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. 1. Configure PostgreSQL to enable FDW support: - This step is similar to how we configured the **primary** instance. - We need to enable this, to enable FDW support, even if using a single node. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## Enable FDW support for the Geo Tracking Database (improves performance) - ## - geo_secondary['db_fdw'] = true - ``` - - For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + This step is similar to how we configured the **primary** instance. + We need to enable this, to enable FDW support, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## Enable FDW support for the Geo Tracking Database (improves performance) + ## + geo_secondary['db_fdw'] = true + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Restart PostgreSQL for the IP change to take effect and reconfigure again: - ```sh - gitlab-ctl restart postgresql - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl restart postgresql + gitlab-ctl reconfigure + ``` - This last reconfigure will provision the FDW configuration and enable it. + This last reconfigure will provision the FDW configuration and enable it. ### Step 3. Initiate the replication process @@ -394,9 +396,9 @@ data before running `pg_basebackup`. 1. SSH into your GitLab **secondary** server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Choose a database-friendly name to use for your **secondary** node to use as the replication slot name. For example, if your domain is @@ -404,38 +406,40 @@ data before running `pg_basebackup`. name as shown in the commands below. 1. Execute the command below to start a backup/restore and begin the replication - CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. - - ```sh - gitlab-ctl replicate-geo-database \ - --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> - ``` - - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` - user in the first step. - - This command also takes a number of additional options. You can use `--help` - to list them all, but here are a couple of tips: - - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need - to increase the timeout, e.g., `--backup-timeout=3600` if you expect the - initial replication to take under an hour. - - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether - (e.g., you know the network path is secure, or you are using a site-to-site - VPN). This is **not** safe over the public Internet! - - You can read more details about each `sslmode` in the - [PostgreSQL documentation][pg-docs-ssl]; - the instructions above are carefully written to ensure protection against - both passive eavesdroppers and active "man-in-the-middle" attackers. - - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the - replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to - add `--force` to the command line. - - When not in a production machine you can disable backup step if you - really sure this is what you want by adding `--skip-backup` + + CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```sh + gitlab-ctl replicate-geo-database \ + --slot-name=<secondary_node_name> \ + --host=<primary_node_ip> + ``` + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` The replication process is now complete. @@ -452,42 +456,42 @@ it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] work: 1. On the **primary** Geo database, enter the PostgreSQL on the console as an - admin user. If you are using an Omnibus-managed database, log onto the **primary** - node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): + admin user. If you are using an Omnibus-managed database, log onto the **primary** + node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): - ```sh - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/postgresql gitlabhq_production - ``` + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` 1. Then create the read-only user: - ```sql - -- NOTE: Use the password defined earlier - CREATE USER gitlab_geo_fdw WITH password 'mypassword'; - GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; - GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; - -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" - -- automatically. - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; - ``` + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` 1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: - ``` - geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' - ``` + ``` + geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' + ``` 1. Save the file and reconfigure GitLab for the changes to be applied: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` ## Troubleshooting diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 177ca68613e..452e4f490a6 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -4,7 +4,7 @@ This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or manually installed and configured PostgreSQL instances. -NOTE: **Note**: +NOTE: **Note:** We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. @@ -13,17 +13,17 @@ developed and tested. We aim to be compatible with most external 1. SSH into a GitLab **primary** application server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Execute the command below to define the node as **primary** node: - ```sh - gitlab-ctl set-geo-primary-node - ``` + ```sh + gitlab-ctl set-geo-primary-node + ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. ### Configure the external database to be replicated @@ -101,26 +101,27 @@ To configure the connection to the external read-replica database and enable Log 1. SSH into a GitLab **secondary** application server and login as root: - ```bash - sudo -i - ``` + ```bash + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` and add the following - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = '<your_password_here>' - # note this is shared between both databases, - # make sure you define the same password in both - gitlab_rails['db_password'] = '<your_password_here>' + gitlab_rails['db_username'] = 'gitlab' + gitlab_rails['db_host'] = '<database_read_replica_host>' + ``` - gitlab_rails['db_username'] = 'gitlab' - gitlab_rails['db_host'] = '<database_read_replica_host>' - ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) ### Configure the tracking database @@ -147,73 +148,72 @@ the tracking database on port 5432. 1. SSH into a GitLab **secondary** server and login as root: - ```bash - sudo -i - ``` + ```bash + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for - the machine with the PostgreSQL instance: + the machine with the PostgreSQL instance: - ```ruby - geo_secondary['db_username'] = 'gitlab_geo' - geo_secondary['db_password'] = '<your_password_here>' + ```ruby + geo_secondary['db_username'] = 'gitlab_geo' + geo_secondary['db_password'] = '<your_password_here>' - geo_secondary['db_host'] = '<tracking_database_host>' - geo_secondary['db_port'] = <tracking_database_port> # change to the correct port - geo_secondary['db_fdw'] = true # enable FDW - geo_postgresql['enable'] = false # don't use internal managed instance - ``` + geo_secondary['db_host'] = '<tracking_database_host>' + geo_secondary['db_port'] = <tracking_database_port> # change to the correct port + geo_secondary['db_fdw'] = true # enable FDW + geo_postgresql['enable'] = false # don't use internal managed instance + ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) 1. Run the tracking database migrations: - ```bash - gitlab-rake geo:db:create - gitlab-rake geo:db:migrate - ``` - -1. Configure the - [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) - connection and credentials: - - Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection - params to match your environment. Execute it to set up the FDW connection. - - ```bash - #!/bin/bash - - # Secondary Database connection params: - DB_HOST="<public_ip_or_vpc_private_ip>" - DB_NAME="gitlabhq_production" - DB_USER="gitlab" - DB_PASS="<your_password_here>" - DB_PORT="5432" - - # Tracking Database connection params: - GEO_DB_HOST="<public_ip_or_vpc_private_ip>" - GEO_DB_NAME="gitlabhq_geo_production" - GEO_DB_USER="gitlab_geo" - GEO_DB_PORT="5432" - - query_exec () { - gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" - } - - query_exec "CREATE EXTENSION postgres_fdw;" - query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" - query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" - query_exec "CREATE SCHEMA gitlab_secondary;" - query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" - ``` - - NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, - but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using - `psql` for AWS RDS. + ```bash + gitlab-rake geo:db:create + gitlab-rake geo:db:migrate + ``` + +1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```bash + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="<public_ip_or_vpc_private_ip>" + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PASS="<your_password_here>" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="<public_ip_or_vpc_private_ip>" + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using + `psql` for AWS RDS. 1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) 1. Populate the FDW tables: - ```bash - gitlab-rake geo:db:refresh_foreign_tables - ``` + ```bash + gitlab-rake geo:db:refresh_foreign_tables + ``` diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index dd1af0dbf9c..c527248bc72 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -6,8 +6,8 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin ## How does Geo know which projects to sync? -On each **secondary** node, there is a read-only replicated copy of the GitLab database. -A **secondary** node also has a tracking database where it stores which projects have been synced. +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. @@ -15,19 +15,19 @@ At the start, this tracking database is empty, so Geo will start trying to updat For each project to sync: 1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. -If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. + If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. 1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. -When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when -the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. -Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. -If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. -So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. ## Can I use Geo in a disaster recovery situation? diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 921a3ef1c7a..61e18df2480 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -50,17 +50,17 @@ The following steps enable a GitLab cluster to serve as the **primary** node. 1. Edit `/etc/gitlab/gitlab.rb` and add the following: - ```ruby - ## - ## Enable the Geo primary role - ## - roles ['geo_primary_role'] - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - ``` + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -79,9 +79,9 @@ The **primary** database will require modification later, as part of A **secondary** cluster is similar to any other GitLab HA cluster, with two major differences: -* The main PostgreSQL database is a read-only replica of the **primary** node's +- The main PostgreSQL database is a read-only replica of the **primary** node's PostgreSQL database. -* There is also a single PostgreSQL database for the **secondary** cluster, +- There is also a single PostgreSQL database for the **secondary** cluster, called the "tracking database", which tracks the synchronization state of various resources. @@ -93,9 +93,9 @@ from the normal HA setup. Configure the following services, again using the non-Geo high availability documentation: -* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high +- [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high availability. -* [NFS](../../high_availability/nfs.md) which will store data that is +- [NFS](../../high_availability/nfs.md) which will store data that is synchronized from the **primary** node. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node @@ -107,36 +107,36 @@ Configure the [**secondary** database](database.md) as a read-only replica of the **primary** database. Use the following as a guide. 1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the PostgreSQL role - ## - roles ['postgres_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - ``` + following: + + ```ruby + ## + ## Configure the PostgreSQL role + ## + roles ['postgres_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -151,47 +151,47 @@ only a single machine, rather than as a PostgreSQL cluster. Configure the tracking database. 1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: - - ```ruby - ## - ## Enable the Geo secondary tracking database - ## - geo_postgresql['enable'] = true - geo_postgresql['listen_address'] = '<ip_address_of_this_host>' - geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' - - ## - ## Configure FDW connection to the replica database - ## - geo_secondary['db_fdw'] = true - geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' - geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] - gitlab_rails['db_host'] = '<replica_database_ip>' - - # Prevent reconfigure from attempting to run migrations on the replica DB - gitlab_rails['auto_migrate'] = false - - ## - ## Disable all other services that aren't needed, since we don't have a role - ## that does this. - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_monitor['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - sidekiq['enable'] = false - unicorn['enable'] = false - ``` + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '<ip_address_of_this_host>' + geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' + geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] + gitlab_rails['db_host'] = '<replica_database_ip>' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -211,50 +211,50 @@ following modifications: 1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** cluster, and add the following: - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role', 'application_role'] - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '<geo_tracking_db_host>' - geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '<replica_database_host>' - gitlab_rails['db_password'] = '<replica_database_password>' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '<redis_host>' - gitlab_rails['redis_password'] = '<redis_password>' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` NOTE: **Note:** If you had set up PostgreSQL cluster using the omnibus package and you had set @@ -270,15 +270,15 @@ After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the chang On the secondary the following GitLab frontend services will be enabled: -* geo-logcursor -* gitlab-pages -* gitlab-workhorse -* logrotate -* nginx -* registry -* remote-syslog -* sidekiq -* unicorn +- geo-logcursor +- gitlab-pages +- gitlab-workhorse +- logrotate +- nginx +- registry +- remote-syslog +- sidekiq +- unicorn Verify these services by running `sudo gitlab-ctl status` on the frontend application servers. diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 54377f7ae4e..8e1d1cb46ba 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -80,8 +80,8 @@ In this diagram: - If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. - A **secondary** node performs different type of synchronizations against the **primary** node, using a special authorization protected by JWT: - - Repositories are cloned/updated via Git over HTTPS. - - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. From the perspective of a user performing Git operations: @@ -107,8 +107,8 @@ The following are required to run Geo: - An operating system that supports OpenSSH 6.9+ (needed for [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) The following operating systems are known to ship with a current version of OpenSSH: - - [CentOS](https://www.centos.org) 7.4+ - - [Ubuntu](https://www.ubuntu.com) 16.04+ + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://www.ubuntu.com) 16.04+ - PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index b190fe7d42d..6bdaad8f783 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -10,41 +10,42 @@ Once removed from the Geo admin page, you must stop and uninstall the **secondar 1. On the **secondary** node, stop GitLab: - ```bash - sudo gitlab-ctl stop - ``` + ```bash + sudo gitlab-ctl stop + ``` + 1. On the **secondary** node, uninstall GitLab: - ```bash - # Stop gitlab and remove its supervision process - sudo gitlab-ctl uninstall + ```bash + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall - # Debian/Ubuntu - sudo dpkg --remove gitlab-ee + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee - # Redhat/Centos - sudo rpm --erase gitlab-ee - ``` + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: 1. On the **primary** node, start a PostgreSQL console session: - ```bash - sudo gitlab-psql - ``` + ```bash + sudo gitlab-psql + ``` - NOTE: **Note:** - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + NOTE: **Note:** + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. - ```sql - SELECT * FROM pg_replication_slots; - ``` + ```sql + SELECT * FROM pg_replication_slots; + ``` 1. Remove the replication slot for the **secondary** node: - ```sql - SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 846afd8f5f4..c7c78407084 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -184,17 +184,17 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou 1. Start a PostgreSQL console session: - ```sh - sudo gitlab-psql gitlabhq_production - ``` + ```sh + sudo gitlab-psql gitlabhq_production + ``` - > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: - ```sql - SELECT * FROM pg_replication_slots; - ``` + ```sql + SELECT * FROM pg_replication_slots; + ``` Slots where `active` is `f` are not active. @@ -204,9 +204,9 @@ Slots where `active` is `f` are not active. - If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the PostgreSQL console session: - ```sql - SELECT pg_drop_replication_slot('<name_of_extra_slot>'); - ``` + ```sql + SELECT pg_drop_replication_slot('<name_of_extra_slot>'); + ``` ### Very large repositories never successfully synchronize on the **secondary** node @@ -237,82 +237,82 @@ to start again from scratch, there are a few steps that can help you: 1. Stop Sidekiq and the Geo LogCursor - It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and - wait until the current jobs to finish processing. + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. - You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** - when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. - ```sh - gitlab-ctl status sidekiq - # run: sidekiq: (pid 10180) <- this is the PID you will use - kill -TSTP 10180 # change to the correct PID + ```sh + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID - gitlab-ctl stop sidekiq - gitlab-ctl stop geo-logcursor - ``` + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` - You can watch sidekiq logs to know when sidekiq jobs processing have finished: + You can watch sidekiq logs to know when sidekiq jobs processing have finished: - ```sh - gitlab-ctl tail sidekiq - ``` + ```sh + gitlab-ctl tail sidekiq + ``` 1. Rename repository storage folders and create new ones - ```sh - mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old - mkdir -p /var/opt/gitlab/git-data/repositories - chown git:git /var/opt/gitlab/git-data/repositories - ``` + ```sh + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` - TIP: **Tip** - You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future - as soon as you confirmed that you don't need it anymore, to save disk space. + TIP: **Tip** + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. 1. _(Optional)_ Rename other data folders and create new ones - CAUTION: **Caution**: - You may still have files on the **secondary** node that have been removed from **primary** node but - removal have not been reflected. If you skip this step, they will never be removed - from this Geo node. + CAUTION: **Caution**: + You may still have files on the **secondary** node that have been removed from **primary** node but + removal have not been reflected. If you skip this step, they will never be removed + from this Geo node. - Any uploaded content like file attachments, avatars or LFS objects are stored in a - subfolder in one of the two paths below: + Any uploaded content like file attachments, avatars or LFS objects are stored in a + subfolder in one of the two paths below: - 1. /var/opt/gitlab/gitlab-rails/shared - 1. /var/opt/gitlab/gitlab-rails/uploads + - /var/opt/gitlab/gitlab-rails/shared + - /var/opt/gitlab/gitlab-rails/uploads - To rename all of them: + To rename all of them: - ```sh - gitlab-ctl stop + ```sh + gitlab-ctl stop - mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old - mkdir -p /var/opt/gitlab/gitlab-rails/shared + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared - mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old - mkdir -p /var/opt/gitlab/gitlab-rails/uploads - ``` + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + ``` - Reconfigure in order to recreate the folders and make sure permissions and ownership - are correctly + Reconfigure in order to recreate the folders and make sure permissions and ownership + are correctly - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Reset the Tracking Database - ```sh - gitlab-rake geo:db:reset - ``` + ```sh + gitlab-rake geo:db:reset + ``` 1. Restart previously stopped services - ```sh - gitlab-ctl start - ``` + ```sh + gitlab-ctl start + ``` ## Fixing Foreign Data Wrapper errors @@ -331,7 +331,7 @@ There are a few key points to remember: 1. The FDW settings are configured on the Geo **tracking** database. 1. The configured foreign server enables a login to the Geo -**secondary**, read-only database. + **secondary**, read-only database. By default, the Geo secondary and tracking database are running on the same host on different ports. That is, 5432 and 5431 respectively. @@ -345,108 +345,106 @@ To check the configuration: 1. Enter the database console: - ```sh - gitlab-geo-psql - ``` + ```sh + gitlab-geo-psql + ``` 1. Check whether any tables are present. If everything is working, you -should see something like this: - - ```sql - gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; - foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n - ame - -------------------------+----------------------+-------------------------------------------------+-------------------------+----------------- - ---- - gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary - <snip> - ``` - - However, if the query returns with `0 rows`, then continue onto the next steps. + should see something like this: + + ```sql + gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; + foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name + -------------------------+----------------------+-------------------------------------------------+-------------------------+--------------------- + gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary + <snip> + ``` + + However, if the query returns with `0 rows`, then continue onto the next steps. 1. Check that the foreign server mapping is correct via `\des+`. The results should look something like this: - ```sql - gitlabhq_geo_production=# \des+ - List of foreign servers - -[ RECORD 1 ]--------+------------------------------------------------------------ - Name | gitlab_secondary - Owner | gitlab-psql - Foreign-data wrapper | postgres_fdw - Access privileges | "gitlab-psql"=U/"gitlab-psql" + - | gitlab_geo=U/"gitlab-psql" - Type | - Version | - FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') - Description | - ``` - - NOTE: **Note:** Pay particular attention to the host and port under - FDW options. That configuration should point to the Geo secondary - database. - - If you need to experiment with changing the host or password, the - following queries demonstrate how: - - ```sql - ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); - ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); - ``` - - If you change the host and/or port, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: - - - `gitlab_rails['db_host']` - - `gitlab_rails['db_port']` + ```sql + gitlabhq_geo_production=# \des+ + List of foreign servers + -[ RECORD 1 ]--------+------------------------------------------------------------ + Name | gitlab_secondary + Owner | gitlab-psql + Foreign-data wrapper | postgres_fdw + Access privileges | "gitlab-psql"=U/"gitlab-psql" + + | gitlab_geo=U/"gitlab-psql" + Type | + Version | + FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') + Description | + ``` + + NOTE: **Note:** Pay particular attention to the host and port under + FDW options. That configuration should point to the Geo secondary + database. + + If you need to experiment with changing the host or password, the + following queries demonstrate how: + + ```sql + ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); + ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); + ``` + + If you change the host and/or port, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_host']` + - `gitlab_rails['db_port']` 1. Check that the user mapping is configured properly via `\deu+`: - ```sql - gitlabhq_geo_production=# \deu+ - List of user mappings - Server | User name | FDW Options - ------------------+------------+-------------------------------------------------------------------------------- - gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') - (1 row) - ``` + ```sql + gitlabhq_geo_production=# \deu+ + List of user mappings + Server | User name | FDW Options + ------------------+------------+-------------------------------------------------------------------------------- + gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') + (1 row) + ``` - Make sure the password is correct. You can test that logins work by running `psql`: + Make sure the password is correct. You can test that logins work by running `psql`: - ```sh - # Connect to the tracking database as the `gitlab_geo` user - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` + ```sh + # Connect to the tracking database as the `gitlab_geo` user + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` - If you need to correct the password, the following query shows how: + If you need to correct the password, the following query shows how: - ```sql - ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); - ``` + ```sql + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); + ``` - If you change the user or password, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: + If you change the user or password, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: - - `gitlab_rails['db_username']` - - `gitlab_rails['db_password']` + - `gitlab_rails['db_username']` + - `gitlab_rails['db_password']` - If you are using [PgBouncer in front of the secondary - database](database.md#pgbouncer-support-optional), be sure to update - the following settings: + If you are using [PgBouncer in front of the secondary + database](database.md#pgbouncer-support-optional), be sure to update + the following settings: - - `geo_postgresql['fdw_external_user']` - - `geo_postgresql['fdw_external_password']` + - `geo_postgresql['fdw_external_user']` + - `geo_postgresql['fdw_external_password']` #### Manual reload of FDW schema @@ -456,34 +454,34 @@ reload of the FDW schema. To manually reload the FDW schema: 1. On the node running the Geo tracking database, enter the PostgreSQL console via the `gitlab_geo` user: - ```sh - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` + ```sh + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` - Be sure to adjust the port and hostname for your configuration. You - may be asked to enter a password. + Be sure to adjust the port and hostname for your configuration. You + may be asked to enter a password. 1. Reload the schema via: - ```sql - DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; - CREATE SCHEMA gitlab_secondary; - GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; - IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; - ``` + ```sql + DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; + CREATE SCHEMA gitlab_secondary; + GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; + IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; + ``` 1. Test that queries work: - ```sql - SELECT * from information_schema.foreign_tables; - SELECT * FROM gitlab_secondary.projects limit 1; - ``` + ```sql + SELECT * from information_schema.foreign_tables; + SELECT * FROM gitlab_secondary.projects limit 1; + ``` [database-start-replication]: database.md#step-3-initiate-the-replication-process [database-pg-replication]: database.md#postgresql-replication @@ -504,6 +502,15 @@ To resolve this, run the following command: sudo gitlab-rake geo:db:refresh_foreign_tables ``` +## Expired artifacts + +If you notice for some reason there are more artifacts on the Geo +secondary node than on the Geo primary node, you can use the rake task +to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). + +On a Geo **secondary** node, this command will also clean up all Geo +registry record related to the orphan files on disk. + ## Fixing common errors This section documents common errors reported in the Admin UI and how to fix them. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 933a75c47d8..d56a59f4967 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -30,70 +30,70 @@ We now require this change as we use this password to enable the Foreign Data Wr the Geo Tracking Database. We are also improving security by disabling the use of **trust** authentication method. -1. **[primary]** Login to your **primary** node and run: +1. **(primary)** Login to your **primary** node and run: - ```sh - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` - Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: + Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this - ``` + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this + ``` -1. **[primary]** Reconfigure and restart: +1. **(primary)** Reconfigure and restart: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` -1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: +1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' - # Enable Foreign Data Wrapper - geo_secondary['db_fdw'] = true + # Enable Foreign Data Wrapper + geo_secondary['db_fdw'] = true - # Secondary address in CIDR format, for example '5.6.7.8/32' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - ``` + # Secondary address in CIDR format, for example '5.6.7.8/32' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + ``` - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this - ``` + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this + ``` -1. **[secondary]** Reconfigure and restart: +1. **(secondary)** Reconfigure and restart: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` ## Upgrading to GitLab 10.5 @@ -169,11 +169,11 @@ After you've verified that HTTP/HTTPS replication is working, you should remove the now-unused SSH keys from your secondaries, as they may cause problems if the **secondary** node if ever promoted to a **primary** node: -1. **[secondary]** Login to **all** your **secondary** nodes and run: +1. **(secondary)** Login to **all** your **secondary** nodes and run: - ```ruby - sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub - ``` + ```ruby + sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub + ``` ### Hashed Storage @@ -236,12 +236,12 @@ instructions below. When in doubt, it does not hurt to do a resync. The easiest way to do this in Omnibus is the following: - 1. Make sure you have Omnibus GitLab on the **primary** server. - 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. - 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. - 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. - 1. Install GitLab on the **secondary** server. - 1. Re-run the [database replication process][database-replication]. +1. Make sure you have Omnibus GitLab on the **primary** server. +1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. +1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. +1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. +1. Install GitLab on the **secondary** server. +1. Re-run the [database replication process][database-replication]. ## Special update notes for 9.0.x @@ -260,157 +260,154 @@ Make sure to follow the steps in the exact order as they appear below and pay extra attention in what node (either **primary** or **secondary**) you execute them! Each step is prepended with the relevant node for better clarity: -1. **[secondary]** Login to **all** your **secondary** nodes and stop all services: +1. **(secondary)** Login to **all** your **secondary** nodes and stop all services: - ```ruby - sudo gitlab-ctl stop - ``` + ```ruby + sudo gitlab-ctl stop + ``` -1. **[secondary]** Make a backup of the `recovery.conf` file on **all** +1. **(secondary)** Make a backup of the `recovery.conf` file on **all** **secondary** nodes to preserve PostgreSQL's credentials: - ```sh - sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ - ``` + ```sh + sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ + ``` -1. **[primary]** Update the **primary** node to GitLab 9.0 following the +1. **(primary)** Update the **primary** node to GitLab 9.0 following the [regular update docs][update]. At the end of the update, the **primary** node will be running with PostgreSQL 9.6. -1. **[primary]** To prevent a de-synchronization of the repository replication, +1. **(primary)** To prevent a de-synchronization of the repository replication, stop all services except `postgresql` as we will use it to re-initialize the **secondary** node's database: - ```sh - sudo gitlab-ctl stop - sudo gitlab-ctl start postgresql - ``` + ```sh + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` -1. **[secondary]** Run the following steps on each of the **secondary** nodes: +1. **(secondary)** Run the following steps on each of the **secondary** nodes: - 1. **[secondary]** Stop all services: + 1. **(secondary)** Stop all services: - ```sh - sudo gitlab-ctl stop - ``` + ```sh + sudo gitlab-ctl stop + ``` - 1. **[secondary]** Prevent running database migrations: + 1. **(secondary)** Prevent running database migrations: - ```sh - sudo touch /etc/gitlab/skip-auto-migrations - ``` + ```sh + sudo touch /etc/gitlab/skip-auto-migrations + ``` - 1. **[secondary]** Move the old database to another directory: + 1. **(secondary)** Move the old database to another directory: - ```sh - sudo mv /var/opt/gitlab/postgresql{,.bak} - ``` + ```sh + sudo mv /var/opt/gitlab/postgresql{,.bak} + ``` - 1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update]. - At the end of the update, the node will be running with PostgreSQL 9.6. + 1. **(secondary)** Update to GitLab 9.0 following the [regular update docs][update]. + At the end of the update, the node will be running with PostgreSQL 9.6. - 1. **[secondary]** Make sure all services are up: + 1. **(secondary)** Make sure all services are up: - ```sh - sudo gitlab-ctl start - ``` + ```sh + sudo gitlab-ctl start + ``` - 1. **[secondary]** Reconfigure GitLab: + 1. **(secondary)** Reconfigure GitLab: - ```sh - sudo gitlab-ctl reconfigure - ``` + ```sh + sudo gitlab-ctl reconfigure + ``` - 1. **[secondary]** Run the PostgreSQL upgrade command: + 1. **(secondary)** Run the PostgreSQL upgrade command: - ```sh - sudo gitlab-ctl pg-upgrade - ``` + ```sh + sudo gitlab-ctl pg-upgrade + ``` - 1. **[secondary]** See the stored credentials for the database that you will - need to re-initialize the replication: + 1. **(secondary)** See the stored credentials for the database that you will + need to re-initialize the replication: - ```sh - sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf - ``` + ```sh + sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf + ``` - 1. **[secondary]** Create the `replica.sh` script as described in the - [database configuration document][database-source-replication]. + 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the + embedded paths if necessary: - 1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the - embedded paths if necessary: + ``` + #!/bin/bash - ``` - #!/bin/bash + PORT="5432" + USER="gitlab_replicator" + echo --------------------------------------------------------------- + echo WARNING: Make sure this script is run from the secondary server + echo --------------------------------------------------------------- + echo + echo Enter the IP or FQDN of the primary PostgreSQL server + read HOST + echo Enter the password for $USER@$HOST + read -s PASSWORD + echo Enter the required sslmode + read SSLMODE - PORT="5432" - USER="gitlab_replicator" - echo --------------------------------------------------------------- - echo WARNING: Make sure this script is run from the secondary server - echo --------------------------------------------------------------- - echo - echo Enter the IP or FQDN of the primary PostgreSQL server - read HOST - echo Enter the password for $USER@$HOST - read -s PASSWORD - echo Enter the required sslmode - read SSLMODE + echo Stopping PostgreSQL and all GitLab services + sudo service gitlab stop + sudo service postgresql stop - echo Stopping PostgreSQL and all GitLab services - sudo service gitlab stop - sudo service postgresql stop + echo Backing up postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ - echo Backing up postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ + echo Cleaning up old cluster directory + sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data - echo Cleaning up old cluster directory - sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data + echo Starting base backup as the replicator user + echo Enter the password for $USER@$HOST + sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P - echo Starting base backup as the replicator user - echo Enter the password for $USER@$HOST - sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P + echo Writing recovery.conf file + sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ + standby_mode = 'on' + primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' + _EOF1_ + " - echo Writing recovery.conf file - sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ - standby_mode = 'on' - primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' - _EOF1_ - " + echo Restoring postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ - echo Restoring postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ + echo Starting PostgreSQL + sudo service postgresql start + ``` - echo Starting PostgreSQL - sudo service postgresql start - ``` + 1. **(secondary)** Run the recovery script using the credentials from the + previous step: - 1. **[secondary]** Run the recovery script using the credentials from the - previous step: + ```sh + sudo bash /tmp/replica.sh + ``` - ```sh - sudo bash /tmp/replica.sh - ``` + 1. **(secondary)** Reconfigure GitLab: - 1. **[secondary]** Reconfigure GitLab: + ```sh + sudo gitlab-ctl reconfigure + ``` - ```sh - sudo gitlab-ctl reconfigure - ``` + 1. **(secondary)** Start all services: - 1. **[secondary]** Start all services: + ```sh + sudo gitlab-ctl start + ``` - ```sh - sudo gitlab-ctl start - ``` + 1. **(secondary)** Repeat the steps for the remaining **secondary** nodes. - 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes. - -1. **[primary]** After all **secondary** nodes are updated, start all services in +1. **(primary)** After all **secondary** nodes are updated, start all services in **primary** node: - ```sh - sudo gitlab-ctl start - ``` + ```sh + sudo gitlab-ctl start + ``` ## Check status after updating @@ -419,9 +416,9 @@ everything is working correctly: 1. Run the Geo raketask on all nodes, everything should be green: - ```sh - sudo gitlab-rake gitlab:geo:check - ``` + ```sh + sudo gitlab-rake gitlab:geo:check + ``` 1. Check the **primary** node's Geo dashboard for any errors. 1. Test the data replication by pushing code to the **primary** node and see if it @@ -435,9 +432,9 @@ and it is required since 10.0. 1. Run database migrations on tracking database: - ```sh - sudo gitlab-rake geo:db:migrate - ``` + ```sh + sudo gitlab-rake geo:db:migrate + ``` 1. Repeat this step for each **secondary** node. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index da8f1ee1529..a3cbc4272f0 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -48,7 +48,7 @@ used by Omnibus and the GitLab source installation guide. Starting with GitLab 11.4, Gitaly is able to serve all Git requests without needed a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the -[Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still be leveraged for redudancy on block level of the Git data. But only has to be mounted on the Gitaly server. @@ -267,7 +267,7 @@ repository from your GitLab server over HTTP. Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url -scheme in the `gitaly_address` of the corresponding storage entry in the gitlab configuration. +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. The admin needs to bring their own certificate as we do not provide that automatically. The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 0c4f926c579..e81d2741082 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -171,12 +171,12 @@ are a work-in-progress representation of the work so far. Quality will be certifying this environment in FY20-Q2. The specifications may be adjusted prior to certification based on performance testing. -- 3 PostgreSQL - 4 CPU, 8GB RAM +- 3 PostgreSQL - 4 CPU, 8GB RAM per node - 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM -- 3 Consul/Sentinel - 2 CPU, 2GB RAM -- 4 Sidekiq - 4 CPU, 8GB RAM -- 5 GitLab application nodes - 20 CPU, 64GB RAM +- 2 Redis - 2 CPU, 8GB RAM per node +- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node +- 4 Sidekiq - 4 CPU, 8GB RAM per node +- 5 GitLab application nodes - 20 CPU, 64GB RAM per node - 1 Gitaly - 20 CPU, 64GB RAM - 1 Monitoring node - 4 CPU, 8GB RAM diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 4db53353218..20bbfdb2603 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -83,7 +83,7 @@ deploy the bundled PostgreSQL. plain text password. These will be necessary when configuring the GitLab application servers later. 1. [Enable monitoring](#enable-monitoring) - + Advanced configuration options are supported and can be added if needed. @@ -204,9 +204,9 @@ Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - If you are using a different username, you will have to specify it. We -will refer to it with `CONSUL_USERNAME`, + will refer to it with `CONSUL_USERNAME`, - There will be a database user created with read only access to the repmgr -database + database - Passwords will be stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 90e5f71d835..b7eaa4ce105 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -2,13 +2,13 @@ Gitaly does not yet support full high availability. However, Gitaly is quite stable and is in use on GitLab.com. Scaled and highly available GitLab environments -should consider using Gitaly on a separate node. +should consider using Gitaly on a separate node. -See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to -track plans and progress toward high availability support. +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. This document is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) -environments and [High Availability Architecture](README.md#high-availability-architecture-examples). +environments and [High Availability Architecture](README.md#high-availability-architecture-examples). ## Running Gitaly on its own server @@ -24,23 +24,25 @@ Continue configuration of other components by going back to: > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. - 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - ``` + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } - 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 0e655e49922..3045be616a6 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -138,6 +138,8 @@ need some extra configuration. If you enable Monitoring, it must be enabled on **all** GitLab servers. +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: ```ruby @@ -158,12 +160,11 @@ If you enable Monitoring, it must be enabled on **all** GitLab servers. sidekiq['listen_address'] = "0.0.0.0" unicorn['listen'] = '0.0.0.0' - # Add the monitoring node's IP address to the monitoring whitelist and allow it to scrape the NGINX metrics - # Replace placeholder - # monitoring.gitlab.example.com - # with the addresses gathered for the monitoring node - gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com'] - nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com'] + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with + # the address and/or subnets gathered from the monitoring node(s). + gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] ``` 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index d16bf7dc0f0..ef415dde10a 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -16,6 +16,8 @@ Omnibus: package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 762179cf756..053dae25823 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -62,6 +62,33 @@ See our [HA documentation for PostgreSQL](database.md) for information on runnin 1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + + If you enable Monitoring, it must be enabled on **all** pgbouncer servers. + + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' + ``` + + 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ### Interacting with pgbouncer #### Administrative console diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index f61a8834af3..874525dd836 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -22,10 +22,10 @@ environments including [Basic Scaling](README.md#basic-scaling) and ### Provide your own Redis instance **[CORE ONLY]** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. ### Standalone Redis using GitLab Omnibus **[CORE ONLY]** @@ -62,11 +62,11 @@ Omnibus: pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false - + redis['bind'] = '0.0.0.0' redis['port'] = '6379' redis['password'] = 'SECRET_PASSWORD_HERE' - + gitlab_rails['auto_migrate'] = false ``` @@ -74,7 +74,7 @@ Omnibus: 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when configuring the GitLab application servers later. -1. [Enable Monitoring](#enable-monitoring) +1. [Enable Monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -91,10 +91,10 @@ environments including [Horizontal](README.md#horizontal), ### Provide your own Redis instance **[CORE ONLY]** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. ### High Availability with GitLab Omnibus **[PREMIUM ONLY]** @@ -368,7 +368,7 @@ The prerequisites for a HA Redis setup are the following: ```ruby # Specify server role as 'redis_master_role' roles ['redis_master_role'] - + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -382,7 +382,6 @@ The prerequisites for a HA Redis setup are the following: # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'redis-password-goes-here' ``` - 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following @@ -394,9 +393,9 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. ### Step 2. Configuring the slave Redis instances @@ -412,7 +411,7 @@ The prerequisites for a HA Redis setup are the following: ```ruby # Specify server role as 'redis_slave_role' roles ['redis_slave_role'] - + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -443,9 +442,9 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> `roles ['redis_sentinel_role', 'redis_slave_role']`. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. --- @@ -754,28 +753,30 @@ gitlab_rails['redis_sentinels'] = [ > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. - If you enable Monitoring, it must be enabled on **all** Redis servers. +If you enable Monitoring, it must be enabled on **all** Redis servers. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` - 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` - 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. ## Advanced configuration diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 022c23d02ce..9921ffd8ea0 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -125,7 +125,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/integrations_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/integrations_json.log` for installations from source. -It contains information about [integrations](../user/project/integrations/project_services.md) activities such as JIRA, Asana and Irker services. It uses JSON format like the example below: +It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below: ``` json {"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"} diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 187fb2f73a1..51b0d78681d 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -103,6 +103,21 @@ repository for more information on this process. [grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards +## Integration with GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/61005) in GitLab 12.1. + +If you have set up Grafana, you can enable a link to access it easily from the sidebar: + +1. Go to the admin area under **Settings > Metrics and profiling** + and expand "Metrics - Grafana". +1. Check the "Enable access to Grafana" checkbox. +1. If Grafana is enabled through Omnibus GitLab and on the same server, + leave "Grafana URL" unchanged. In any other case, enter the full URL + path of the Grafana instance. +1. Click **Save changes**. +1. The new link will be available in the admin area under **Monitoring > Metrics Dashboard**. + --- Read more on: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 84b71ae6f1c..2d9e3f7f18b 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -108,7 +108,7 @@ Some basic Ruby runtime metrics are available: [GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat -## Puma Metrics **[EXPERIMENTAL]** +## Puma Metrics **(EXPERIMENTAL)** When Puma is used instead of Unicorn, following metrics are available: diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 286b99aceb5..7297507f599 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,70 +1,132 @@ # Extra Sidekiq processes **[STARTER ONLY]** -GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +NOTE: **Note:** +The information in this page applies only to Omnibus GitLab. + +GitLab Starter allows one to start an extra set of Sidekiq processes besides the default one. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. -## Starting extra processes via Omnibus GitLab +## Available Sidekiq queues -To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true` -setting `/etc/gitlab/gitlab.rb`: +For a list of the existing Sidekiq queues, check the following files: -```ruby -sidekiq_cluster['enable'] = true -``` +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/workers/all_queues.yml) -You will then specify how many additional processes to create via `sidekiq-cluster` -as well as which queues for them to handle. This is done via the -`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain -which queues to process. Each item in the array will equate to one additional -sidekiq process. +Each entry in the above files represents a queue on which extra Sidekiq processes +can be started. -As an example, to make additional sidekiq processes that process the -`elastic_indexer` and `mailers` queues, you would apply the following: +## Starting extra processes -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer", - "mailers" -] -``` +To start extra Sidekiq processes, you must enable `sidekiq-cluster`: -To have an additional sidekiq process handle multiple queues, you simply put a -comma after the first queue name and then put the next queue name: +1. Edit `/etc/gitlab/gitlab.rb` and add: -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer,elastic_commit_indexer", - "mailers" -] -``` + ```ruby + sidekiq_cluster['enable'] = true + ``` -Keep in mind, all changes must be followed by reconfiguring your GitLab -application via `sudo gitlab-ctl reconfigure`. +1. You will then need to specify how many additional processes to create via `sidekiq-cluster` + and which queue they should handle via the `sidekiq_cluster['queue_groups']` + array setting. Each item in the array equates to one additional Sidekiq + process, and values in each item determine the queues it works on. -### Monitoring + For example, the following setting adds additional Sidekiq processes to two + queues, one to `elastic_indexer` and one to `mailers`: -Once the Sidekiq processes are added, you can visit the "Background Jobs" + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer", + "mailers" + ] + ``` + + To have an additional Sidekiq process handle multiple queues, add multiple + queue names to its item delimited by commas. For example: + + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer, elastic_commit_indexer", + "mailers" + ] + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +Once the extra Sidekiq processes are added, you can visit the "Background Jobs" section under the admin area in GitLab (`/admin/background_jobs`). - + -### All queues with exceptions +## Negating settings -To have the additional sidekiq processes work on every queue EXCEPT the ones +To have the additional Sidekiq processes work on every queue **except** the ones you list: +1. After you follow the steps for [starting extra processes](#starting-extra-processes), + edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['negate'] = true + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +## Ignore all GitHub import queues + +When [importing from GitHub](../../user/project/import/github.md), Sidekiq might +use all of its resources to perform those operations. To set up a separate +`sidekiq-cluster` process to ignore all GitHub import-related queues: + 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby + sidekiq_cluster['enable'] = true sidekiq_cluster['negate'] = true + sidekiq_cluster['queue_groups'] = [ + "github_import_advance_stage", + "github_importer:github_import_import_diff_note", + "github_importer:github_import_import_issue", + "github_importer:github_import_import_note", + "github_importer:github_import_import_lfs_object", + "github_importer:github_import_import_pull_request", + "github_importer:github_import_refresh_import_jid", + "github_importer:github_import_stage_finish_import", + "github_importer:github_import_stage_import_base_data", + "github_importer:github_import_stage_import_issues_and_diff_notes", + "github_importer:github_import_stage_import_notes", + "github_importer:github_import_stage_import_lfs_objects", + "github_importer:github_import_stage_import_pull_requests", + "github_importer:github_import_stage_import_repository" + ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: + ```sh + sudo gitlab-ctl reconfigure + ``` -### Limiting concurrency +## Number of threads + +Each process defined under `sidekiq_cluster` starts with a +number of threads that equals the number of queues, plus one spare thread. +For example, a process that handles the `process_commit` and `post_receive` +queues will use three threads in total. + +## Limiting concurrency + +To limit the concurrency of the Sidekiq processes: 1. Edit `/etc/gitlab/gitlab.rb` and add: @@ -72,11 +134,22 @@ you list: sidekiq_cluster['concurrency'] = 25 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: -Keep in mind, this normally would not exceed the number of CPU cores available. + ```sh + sudo gitlab-ctl reconfigure + ``` -### Modifying the check interval +For each queue group, the concurrency factor will be set to `min(number of queues, N)`. +Setting the value to 0 will disable the limit. Keep in mind this normally would +not exceed the number of CPU cores available. + +Each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) +for more details. + +## Modifying the check interval To modify the check interval for the additional Sidekiq processes: @@ -90,9 +163,14 @@ To modify the check interval for the additional Sidekiq processes: This tells the additional processes how often to check for enqueued jobs. -## Starting extra processes via command line +## Troubleshooting using the CLI -Starting extra Sidekiq processes can be done using the command +CAUTION: **Warning:** +It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. +If you experience a problem, you should contact GitLab support. Use the command +line at your own risk. + +For debugging purposes, you can start extra Sidekiq processes by using the command `/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster`. This command takes arguments using the following syntax: @@ -111,29 +189,29 @@ see the relevant section in the [Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). For example, say you want to start 2 extra processes: one to process the -"process_commit" queue, and one to process the "post_receive" queue. This can be +`process_commit` queue, and one to process the `post_receive` queue. This can be done as follows: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit post_receive ``` -If you instead want to start one process processing both queues you'd use the +If you instead want to start one process processing both queues, you'd use the following syntax: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive ``` -If you want to have one Sidekiq process process the "process_commit" and -"post_receive" queues, and one process to process the "gitlab_shell" queue, +If you want to have one Sidekiq process dealing with the `process_commit` and +`post_receive` queues, and one process to process the `gitlab_shell` queue, you'd use the following: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive gitlab_shell ``` -### Monitoring +### Monitoring the `sidekiq-cluster` command The `sidekiq-cluster` command will not terminate once it has started the desired amount of Sidekiq processes. Instead, the process will continue running and @@ -172,24 +250,24 @@ command and not the PID(s) of the started Sidekiq processes. The Rails environment can be set by passing the `--environment` flag to the `sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The -default value is "development". +default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`. -### All queues with exceptions +### Using negation You're able to run all queues in `sidekiq_queues.yml` file on a single or multiple processes with exceptions using the `--negate` flag. For example, say you want to run a single process for all queues, -except "process_commit" and "post_receive". You can do so by executing: +except `process_commit` and `post_receive`: ```bash -sidekiq-cluster process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive --negate ``` -For multiple processes of all queues (except "process_commit" and "post_receive"): +For multiple processes of all queues (except `process_commit` and `post_receive`): ```bash -sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate ``` ### Limiting concurrency @@ -201,18 +279,3 @@ the `-m N` option. For example, this would cap the maximum number of threads to ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive -m 1 ``` - -For each queue group, the concurrency factor will be set to min(number of -queues, N). Setting the value to 0 will disable the limit. - -Note that each thread requires a Redis connection, so adding threads may -increase Redis latency and potentially cause client timeouts. See the [Sidekiq -documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) -for more details. - -## Number of threads - -Each process started using `sidekiq-cluster` (whether it be via command line or -via the gitlab.rb file) starts with a number of threads that equals the number -of queues, plus one spare thread. For example, a process that handles the -"process_commit" and "post_receive" queues will use 3 threads in total. diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md index 9b0ac23b41c..a677a9c9a33 100644 --- a/doc/api/group_boards.md +++ b/doc/api/group_boards.md @@ -27,7 +27,69 @@ Example response: [ { "id": 1, - "group_id": 5, + "name:": "group issue board", + "group": { + "id": 5, + "name": "Documentcloud", + "web_url": "http://example.com/groups/documentcloud" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +] +``` + +Users on GitLab [Premium, Silver, or higher](https://about.gitlab.com/pricing/) will see +different parameters, due to the ability to have multiple group boards. Refer to the table +above to see what enpoint(s) belong to each tier. + +Example response: + +```json +[ + { + "id": 1, + "name:": "group issue board", + "group": { + "id": 5, + "name": "Documentcloud", + "web_url": "http://example.com/groups/documentcloud" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, "lists" : [ { "id" : 1, @@ -83,7 +145,16 @@ Example response: ```json { "id": 1, - "group_id": 5, + "name:": "group issue board", + "group": { + "id": 5, + "name": "Documentcloud", + "web_url": "http://example.com/groups/documentcloud" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, "lists" : [ { "id" : 1, @@ -116,6 +187,206 @@ Example response: } ``` +Users on GitLab [Premium, Silver, or higher](https://about.gitlab.com/pricing/) will see +different parameters, due to the ability to have multiple group boards: + +Example response: + +```json + { + "id": 1, + "name:": "group issue board", + "group": { + "id": 5, + "name": "Documentcloud", + "web_url": "http://example.com/groups/documentcloud" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +``` + +## Create a Group Issue Board **[PREMIUM]** + +Creates a Group Issue Board. + +``` +POST /groups/:id/boards +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the new board | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards?name=newboard +``` + +Example response: + +```json + { + "id": 1, + "name": "newboard", + "group": { + "id": 5, + "name": "Documentcloud", + "web_url": "http://example.com/groups/documentcloud" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +``` + +## Update a Group Issue Board **[PREMIUM]** + +> [Introduced][ee-5954] in GitLab 11.1. + +Updates a Group Issue Board. + +``` +PUT /groups/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `name` | string | no | The new name of the board | +| `assignee_id` | integer | no | The assignee the board should be scoped to | +| `milestone_id` | integer | no | The milestone the board should be scoped to | +| `labels` | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` | integer | no | The weight range from 0 to 9, to which the board should be scoped to | + + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards/1?name=new_name&milestone_id=44&assignee_id=1&labels=GroupLabel&weight=4 +``` + +Example response: + +```json + { + "id": 1, + "project": null, + "lists": [], + "name": "new_name", + "group": { + "id": 5, + "name": "Documentcloud", + "web_url": "http://example.com/groups/documentcloud" + }, + "milestone": { + "id": 44, + "iid": 1, + "group_id": 5, + "title": "Group Milestone", + "description": "Group Milestone Desc", + "state": "active", + "created_at": "2018-07-03T07:15:19.271Z", + "updated_at": "2018-07-03T07:15:19.271Z", + "due_date": null, + "start_date": null, + "web_url": "http://example.com/groups/documentcloud/-/milestones/1" + }, + "assignee": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://example.com/root" + }, + "labels": [{ + "id": 11, + "name": "GroupLabel", + "color": "#428BCA", + "description": "" + }], + "weight": 4 + } +``` + +## Delete a Group Issue Board **[PREMIUM]** + +Deletes a Group Issue Board. + +``` +DELETE /groups/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards/1 +``` + ## List board lists Get a list of the board's lists. @@ -282,3 +553,5 @@ DELETE /groups/:id/boards/:board_id/lists/:list_id ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards/1/lists/1 ``` + +[ee-5954]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5954 diff --git a/doc/api/group_milestones.md b/doc/api/group_milestones.md index 260eb09cc38..e780a60a416 100644 --- a/doc/api/group_milestones.md +++ b/doc/api/group_milestones.md @@ -136,3 +136,18 @@ Parameters: - `milestone_id` (required) - The ID of a group milestone [ce-12819]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12819 + +## Get all burndown chart events for a single milestone **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4737) in GitLab 12.1 + +Get all burndown chart events for a single milestone. + +``` +GET /groups/:id/milestones/:milestone_id/burndown_events +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of a group milestone diff --git a/doc/api/groups.md b/doc/api/groups.md index 20789a1d4a4..80a2fb8e4d9 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -70,8 +70,8 @@ GET /groups?statistics=true "repository_size" : 33, "wiki_size" : 100, "lfs_objects_size" : 123, - "job_artifacts_size" : 57 - + "job_artifacts_size" : 57, + "packages_size": 0 } } ] @@ -375,6 +375,16 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `shared_runners_minutes_limit` and `extra_shared_runners_minutes_limit` parameters: + +Additional response parameters: **[STARTER]** + +```json + "shared_runners_minutes_limit": 133, + "extra_shared_runners_minutes_limit": 133, +``` + When adding the parameter `with_projects=false`, projects will not be returned. ```bash @@ -412,13 +422,15 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `name` | string | yes | The name of the group | -| `path` | string | yes | The path of the group | -| `description` | string | no | The group's description | -| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | -| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group | -| `request_access_enabled` | boolean | no | Allow users to request member access. | -| `parent_id` | integer | no | The parent group id for creating nested group. | +| `name` | string | yes | The name of the group | +| `path` | string | yes | The path of the group | +| `description` | string | no | The group's description | +| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | +| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `parent_id` | integer | no | The parent group ID for creating nested group. | +| `shared_runners_minutes_limit` | integer | no | **[STARTER ONLY]** Pipeline minutes quota for this group. | +| `extra_shared_runners_minutes_limit` | integer | no | **[STARTER ONLY]** Extra pipeline minutes quota for this group. | ## Transfer project to group @@ -445,14 +457,18 @@ PUT /groups/:id | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer | yes | The ID of the group | -| `name` | string | no | The name of the group | -| `path` | string | no | The path of the group | -| `description` | string | no | The description of the group | -| `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | -| `lfs_enabled` (optional) | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group | -| `request_access_enabled` | boolean | no | Allow users to request member access. | -| `file_template_project_id` | integer | no | **(Premium)** The ID of a project to load custom file templates from | +| `id` | integer | yes | The ID of the group | +| `name` | string | no | The name of the group | +| `path` | string | no | The path of the group | +| `description` | string | no | The description of the group | +| `membership_lock` | boolean | no | **[STARTER]** Prevent adding new members to project membership within this group | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group | +| `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | +| `lfs_enabled` (optional) | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `file_template_project_id` | integer | no | **[PREMIUM]** The ID of a project to load custom file templates from | +| `shared_runners_minutes_limit` | integer | no | **[STARTER ONLY]** Pipeline minutes quota for this group | +| `extra_shared_runners_minutes_limit` | integer | no | **[STARTER ONLY]** Extra pipeline minutes quota for this group | ```bash curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5?name=Experimental" @@ -520,7 +536,7 @@ Example response: ## Remove group -Removes group with all projects inside. +Removes group with all projects inside. Only available to group owners and administrators. ``` DELETE /groups/:id @@ -552,10 +568,62 @@ GET /groups?search=foobar ] ``` +## Sync group with LDAP **[CORE ONLY]** + +Syncs the group with its linked LDAP group. Only available to group owners and administrators. + +``` +POST /groups/:id/ldap_sync +``` + +Parameters: + +- `id` (required) - The ID or path of a user group + ## Group members Please consult the [Group Members](members.md) documentation. +### Add LDAP group link **[CORE ONLY]** + +Adds an LDAP group link. + +``` +POST /groups/:id/ldap_group_links +``` + +Parameters: + +- `id` (required) - The ID of a group +- `cn` (required) - The CN of a LDAP group +- `group_access` (required) - Minimum access level for members of the LDAP group +- `provider` (required) - LDAP provider for the LDAP group + +### Delete LDAP group link **[CORE ONLY]** + +Deletes an LDAP group link. + +``` +DELETE /groups/:id/ldap_group_links/:cn +``` + +Parameters: + +- `id` (required) - The ID of a group +- `cn` (required) - The CN of a LDAP group + +Deletes a LDAP group link for a specific LDAP provider + +``` +DELETE /groups/:id/ldap_group_links/:provider/:cn +``` + +Parameters: + +- `id` (required) - The ID of a group +- `cn` (required) - The CN of a LDAP group +- `provider` (required) - Name of a LDAP provider + ## Namespaces in groups By default, groups only get 20 namespaces at a time because the API results are paginated. diff --git a/doc/api/issues.md b/doc/api/issues.md index 0d96cfa1b21..b29626525da 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -3,7 +3,7 @@ Every API call to issues must be authenticated. If a user is not a member of a project and the project is private, a `GET` -request on that project will result to a `404` status code. +request on that project will result in a `404` status code. ## Issues pagination @@ -39,7 +39,7 @@ GET /issues?confidential=true | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` | string | no | Return `all` issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | -| `with_labels_details`| Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | +| `with_labels_details` | Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13004] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | | `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | @@ -47,6 +47,7 @@ GET /issues?confidential=true | `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | | `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Simillar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `weight` **[STARTER]** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | @@ -144,6 +145,20 @@ Example response: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +[ + { + "state" : "opened", + "description" : "Ratione dolores corrupti mollitia soluta quia.", + "weight": null, + ... + } +] +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -174,7 +189,7 @@ GET /groups/:id/issues?confidential=true | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | -| `with_labels_details`| Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | +| `with_labels_details` | Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | | `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13004] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | @@ -183,6 +198,7 @@ GET /groups/:id/issues?confidential=true | `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | | `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Simillar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `weight` **[STARTER]** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search group issues against their `title` and `description` | @@ -278,6 +294,20 @@ Example response: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +[ + { + "project_id" : 4, + "description" : "Omnis vero earum sunt corporis dolor et placeat.", + "weight": null, + ... + } +] +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -309,7 +339,7 @@ GET /projects/:id/issues?confidential=true | `iids[]` | Array[integer] | no | Return only the milestone having the given `iid` | | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | -| `with_labels_details`| Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | +| `with_labels_details` | Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13004] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | | `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | @@ -317,6 +347,7 @@ GET /projects/:id/issues?confidential=true | `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | | `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Simillar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `weight` **[STARTER]** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search project issues against their `title` and `description` | @@ -420,6 +451,20 @@ Example response: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +[ + { + "project_id" : 4, + "description" : "Omnis vero earum sunt corporis dolor et placeat.", + "weight": null, + ... + } +] +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -520,6 +565,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +{ + "project_id" : 4, + "description" : "Omnis vero earum sunt corporis dolor et placeat.", + "weight": null, + ... +} +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -536,16 +593,17 @@ POST /projects/:id/issues |-------------------------------------------|----------------|----------|--------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `iid` | integer/string | no | The internal ID of the project's issue (requires admin or project owner rights) | -| `title` | string | yes | The title of an issue | -| `description` | string | no | The description of an issue | -| `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | -| `assignee_ids` | Array[integer] | no | The ID of the users to assign issue | -| `milestone_id` | integer | no | The global ID of a milestone to assign issue | -| `labels` | string | no | Comma-separated label names for an issue | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. `2016-03-11T03:45:40Z` (requires admin or project/group owner rights) | -| `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, e.g. `2016-03-11` | -| `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This will fill the issue with a default description and mark all discussions as resolved. When passing a description or title, these values will take precedence over the default values.| -| `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This will fill in the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | +| `title` | string | yes | The title of an issue | +| `description` | string | no | The description of an issue | +| `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | +| `assignee_ids` | Array[integer] | no | The ID of a user to assign issue | +| `milestone_id` | integer | no | The global ID of a milestone to assign issue | +| `labels` | string | no | Comma-separated label names for an issue | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. `2016-03-11T03:45:40Z` (requires admin or project/group owner rights) | +| `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, e.g. `2016-03-11` | +| `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This will fill the issue with a default description and mark all discussions as resolved. When passing a description or title, these values will take precedence over the default values.| +| `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This will fill in the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | +| `weight` **[STARTER]** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/issues?title=Issues%20with%20auth&labels=bug @@ -607,6 +665,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +{ + "project_id" : 4, + "description" : null, + "weight": null, + ... +} +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -622,17 +692,18 @@ PUT /projects/:id/issues/:issue_iid | Attribute | Type | Required | Description | |----------------|---------|----------|------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `issue_iid` | integer | yes | The internal ID of a project's issue | | `title` | string | no | The title of an issue | | `description` | string | no | The description of an issue | | `confidential` | boolean | no | Updates an issue to be confidential | -| `assignee_ids` | Array[integer] | no | The ID of the user(s) to assign the issue to. Set to `0` or provide an empty value to unassign all assignees. | +| `assignee_ids` | Array[integer] | no | The ID of the user(s) to assign the issue to. Set to `0` or provide an empty value to unassign all assignees. | | `milestone_id` | integer | no | The global ID of a milestone to assign the issue to. Set to `0` or provide an empty value to unassign a milestone.| | `labels` | string | no | Comma-separated label names for an issue. Set to an empty string to unassign all labels. | | `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it | | `updated_at` | string | no | Date time string, ISO 8601 formatted, e.g. `2016-03-11T03:45:40Z` (requires admin or project owner rights) | | `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, e.g. `2016-03-11` | +| `weight` **[STARTER]** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. 0 | | `discussion_locked` | boolean | no | Flag indicating if the issue's discussion is locked. If the discussion is locked only project members can add or edit comments. | ```bash @@ -702,6 +773,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +{ + "project_id" : 4, + "description" : null, + "weight": null, + ... +} +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -812,6 +895,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +{ + "project_id": 5, + "description": "Repellat voluptas quibusdam voluptatem exercitationem.", + "weight": null, + ... +} +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. @@ -901,6 +996,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `weight` parameter: + +```json +{ + "project_id": 5, + "description": "Repellat voluptas quibusdam voluptatem exercitationem.", + "weight": null, + ... +} +``` + **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 72973b69117..223bfed91a9 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -14,7 +14,7 @@ GET /projects/:id/jobs | `scope` | string **or** array of strings | no | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | ```sh -curl --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/1/jobs?scope[]=pending&scope[]=running' +curl --globoff --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/1/jobs?scope[]=pending&scope[]=running' ``` Example of response diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index ddac81328b9..49aaac06b46 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -222,7 +222,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/approvals "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } } ], @@ -314,7 +314,7 @@ PUT /projects/:id/merge_requests/:merge_request_iid/approvers "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } } ], @@ -387,7 +387,7 @@ does not match, the response code will be `409`. "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } }, { @@ -397,7 +397,7 @@ does not match, the response code will be `409`. "id": 2, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/cf7ad14b34162a76d593e3affca2adca?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/ryley" + "web_url": "http://localhost:3000/ryley" } } ], diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 7b58aa3100e..69db9f97a35 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -29,27 +29,28 @@ GET /merge_requests?search=foo&in=title Parameters: -| Attribute | Type | Required | Description | -| ------------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | -| `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged` | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | -| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | -| `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | -| `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | -| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | -| `created_after` | datetime | no | Return merge requests created on or after the given time | -| `created_before` | datetime | no | Return merge requests created on or before the given time | -| `updated_after` | datetime | no | Return merge requests updated on or after the given time | -| `updated_before` | datetime | no | Return merge requests updated on or before the given time | -| `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead. | -| `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me` | -| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | -| `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | -| `source_branch` | string | no | Return merge requests with the given source branch | -| `target_branch` | string | no | Return merge requests with the given target branch | -| `search` | string | no | Search merge requests against their `title` and `description` | -| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | -| `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* WIP merge requests, `no` to return *non* WIP merge requests | +| Attribute | Type | Required | Description | +| ------------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | +| `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged` | +| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | +| `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | +| `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | +| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | +| `created_after` | datetime | no | Return merge requests created on or after the given time | +| `created_before` | datetime | no | Return merge requests created on or before the given time | +| `updated_after` | datetime | no | Return merge requests updated on or after the given time | +| `updated_before` | datetime | no | Return merge requests updated on or before the given time | +| `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead. | +| `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me` | +| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | +| `approver_ids` **[STARTER]** | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `source_branch` | string | no | Return merge requests with the given source branch | +| `target_branch` | string | no | Return merge requests with the given target branch | +| `search` | string | no | Search merge requests against their `title` and `description` | +| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | +| `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* WIP merge requests, `no` to return *non* WIP merge requests | ```json [ @@ -147,6 +148,20 @@ Parameters: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +[ + { + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... + } +] +``` + ## List project merge requests Get all merge requests for this project. @@ -191,6 +206,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13060] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced][ce-13060] in GitLab 9.5)_ | +| `approver_ids` **[STARTER]** | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -293,6 +309,20 @@ Parameters: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +[ + { + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... + } +] +``` + ## List group merge requests Get all merge requests for this group and its subgroups. @@ -328,6 +358,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced][ce-13060] in GitLab 9.5)_ | +| `approver_ids` **[STARTER]** | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -427,6 +458,20 @@ Parameters: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +[ + { + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... + } +] +``` + ## Get single MR Shows information about a single merge request. @@ -565,6 +610,18 @@ Parameters: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + ## Get single MR participants Get a list of merge request participants. @@ -767,6 +824,7 @@ Parameters: ## Create MR Creates a new merge request. + ``` POST /projects/:id/merge_requests ``` @@ -788,6 +846,15 @@ POST /projects/:id/merge_requests | `allow_maintainer_to_push` | boolean | no | Deprecated, see allow_collaboration | | `squash` | boolean | no | Squash commits into a single commit when merging | +If `approvals_before_merge` **[STARTER]** is not provided, it inherits the value from the +target project. If it is provided, then the following conditions must hold in +order for it to take effect: + +1. The target project's `approvals_before_merge` must be greater than zero. (A + value of zero disables approvals for that project.) +2. The provided value of `approvals_before_merge` must be greater than the + target project's `approvals_before_merge`. + ```json { "id": 1, @@ -893,6 +960,18 @@ POST /projects/:id/merge_requests } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + ## Update MR Updates an existing merge request. You can change the target branch, title, or even close the MR. @@ -1034,6 +1113,18 @@ Must include at least one non-required attribute from above. } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + ## Delete a merge request Only for admins and project owners. Soft deletes the merge request in question. @@ -1191,7 +1282,19 @@ Parameters: } ``` -## Returns the up to date merge-ref HEAD commit +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + +## Merge to default merge ref path Merge the changes between the merge request source and target branches into `refs/merge-requests/:iid/merge` ref, of the target project repository, if possible. This ref will have the state the target branch would have if @@ -1349,6 +1452,18 @@ Parameters: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + ## Rebase a merge request Automatically rebase the `source_branch` of the merge request against its @@ -1473,7 +1588,7 @@ Example response when the GitLab issue tracker is used: ] ``` -Example response when an external issue tracker (e.g. JIRA) is used: +Example response when an external issue tracker (e.g. Jira) is used: ```json [ @@ -1616,6 +1731,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + ## Unsubscribe from a merge request Unsubscribes the authenticated user from a merge request to not receive @@ -1749,6 +1876,18 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 1, + "title": "test1", + "approvals_before_merge": null + ... +} +``` + ## Create a todo Manually creates a todo for the current user on a merge request. @@ -1970,6 +2109,7 @@ Example response: }] } ``` + ## Set a time estimate for a merge request Sets an estimated time of work for this merge request. @@ -2114,3 +2254,7 @@ Example response: [ce-14016]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14016 [ce-15454]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15454 [ce-18935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18935 + +## Approvals **[STARTER]** + +For approvals, please see [Merge Request Approvals](merge_request_approvals.md) diff --git a/doc/api/milestones.md b/doc/api/milestones.md index 90d8c033cd6..e745d0c2e6c 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -147,3 +147,18 @@ Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `milestone_id` (required) - The ID of a project milestone + +## Get all burndown chart events for a single milestone **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4737) in GitLab 12.1 + +Gets all burndown chart events for a single milestone. + +``` +GET /projects/:id/milestones/:milestone_id/burndown_events +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of a project milestone diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index b66a3198ffb..b354e2b9ab2 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -54,7 +54,21 @@ Example response: ] ``` -**Note**: `members_count_with_descendants` are presented only for group maintainers/owners. +Users on GitLab.com [Bronze or higher](https://about.gitlab.com/pricing/#gitlab-com) may also see +the `plan` parameter associated with a namespace: + +```json +[ + { + "id": 1, + "name": "user1", + "plan": "bronze", + ... + } +] +``` + +**Note**: Only group maintainers/owners are presented with `members_count_with_descendants`, as well as `plan` **[BRONZE ONLY]**. ## Search for namespace diff --git a/doc/api/notification_settings.md b/doc/api/notification_settings.md index e21e73c7dac..0342622f384 100644 --- a/doc/api/notification_settings.md +++ b/doc/api/notification_settings.md @@ -1,6 +1,6 @@ # Notification settings API ->**Note:** This feature was [introduced][ce-5632] in GitLab 8.12. +>**Note:** This feature was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5632) in GitLab 8.12. **Valid notification levels** @@ -17,22 +17,22 @@ custom If the `custom` level is used, specific email events can be controlled. Available events are returned by `NotificationSetting.email_events`. Currently, these events are recognized: -``` -new_note -new_issue -reopen_issue -close_issue -reassign_issue -issue_due -new_merge_request -push_to_merge_request -reopen_merge_request -close_merge_request -reassign_merge_request -merge_merge_request -failed_pipeline -success_pipeline -``` +- `new_note` +- `new_issue` +- `reopen_issue` +- `close_issue` +- `reassign_issue` +- `issue_due` +- `new_merge_request` +- `push_to_merge_request` +- `reopen_merge_request` +- `close_merge_request` +- `reassign_merge_request` +- `merge_merge_request` +- `failed_pipeline` +- `success_pipeline` +- `new_epic` **[ULTIMATE]** + ## Global notification settings @@ -85,6 +85,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab. | `merge_merge_request` | boolean | no | Enable/disable this notification | | `failed_pipeline` | boolean | no | Enable/disable this notification | | `success_pipeline` | boolean | no | Enable/disable this notification | +| `new_epic` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6626) in 11.3) **[ULTIMATE]** | Example response: @@ -153,6 +154,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab. | `merge_merge_request` | boolean | no | Enable/disable this notification | | `failed_pipeline` | boolean | no | Enable/disable this notification | | `success_pipeline` | boolean | no | Enable/disable this notification | +| `new_epic` | boolean | no | Enable/disable this notification ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6626) in 11.3) **[ULTIMATE]** | Example responses: @@ -182,4 +184,17 @@ Example responses: } ``` -[ce-5632]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5632 +Users on GitLab [Ultimate or Gold](https://about.gitlab.com/pricing/) will also see +the `new_epic` parameter: + +```json +{ + "level": "custom", + "events": { + "new_note": true, + "new_issue": false, + "new_epic": false, + ... + } +} +``` diff --git a/doc/api/project_aliases.md b/doc/api/project_aliases.md new file mode 100644 index 00000000000..76343b4cd82 --- /dev/null +++ b/doc/api/project_aliases.md @@ -0,0 +1,111 @@ +# Project Aliases API **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3264) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.1. + +All methods require administrator authorization. + +## List all project aliases + +Get a list of all project aliases: + +``` +GET /project_aliases +``` + +``` +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases" +``` + +Example response: + +```json +[ + { + "id": 1, + "project_id": 1, + "name": "gitlab-ce" + }, + { + "id": 2, + "project_id": 2, + "name": "gitlab-ee" + } +] +``` + +## Get project alias' details + +Get details of a project alias: + +``` +GET /project_aliases/:name +``` + +| Attribute | Type | Required | Description | +|-----------|--------|----------|-----------------------| +| `name` | string | yes | The name of the alias | + +``` +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases/gitlab-ee" +``` + +Example response: + +```json +{ + "id": 1, + "project_id": 1, + "name": "gitlab-ee" +} +``` + +## Create a project alias + +Add a new alias for a project. Responds with a 201 when successful, +400 when there are validation errors (e.g. alias already exists): + +``` +POST /project_aliases +``` + +| Attribute | Type | Required | Description | +|--------------|----------------|----------|----------------------------------------| +| `project_id` | integer/string | yes | The ID or path of the project. | +| `name` | string | yes | The name of the alias. Must be unique. | + +``` +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=1" --form "name=gitlab-ee" +``` + +or + +``` +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=gitlab-org/gitlab-ee" --form "name=gitlab-ee" +``` + +Example response: + +```json +{ + "id": 1, + "project_id": 1, + "name": "gitlab-ee" +} +``` + +## Delete a project alias + +Removes a project aliases. Responds with a 204 when project alias +exists, 404 when it doesn't: + +``` +DELETE /project_aliases/:name +``` + +| Attribute | Type | Required | Description | +|-----------|--------|----------|-----------------------| +| `name` | string | yes | The name of the alias | + +``` +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases/gitlab-ee" +``` diff --git a/doc/api/project_snippets.md b/doc/api/project_snippets.md index 0ccb0517e08..2b2e40fb276 100644 --- a/doc/api/project_snippets.md +++ b/doc/api/project_snippets.md @@ -13,6 +13,12 @@ Constants for snippet visibility levels are: | `internal` | The snippet is visible for any logged in user | | `public` | The snippet can be accessed without any authentication | +NOTE: **Note:** +From July 2019, the `Internal` visibility setting is disabled for new projects, groups, +and snippets on GitLab.com. Existing projects, groups, and snippets using the `Internal` +visibility setting keep this setting. You can read more about the change in the +[relevant issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/12388). + ## List snippets Get a list of project snippets. diff --git a/doc/api/projects.md b/doc/api/projects.md index 1d58e390d9e..b8ccf25581e 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -14,7 +14,7 @@ Values for the project visibility level are: The project can be cloned by any logged in user. - `public`: - The project can be cloned without any authentication. + The project can be accessed without any authentication. ## Project merge method @@ -41,23 +41,23 @@ GET /projects | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `archived` | boolean | no | Limit by archived status | -| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | -| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | -| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | -| `search` | string | no | Return list of projects matching the search criteria | -| `simple` | boolean | no | Return only limited fields for each project. This is a no-op without authentication as then _only_ simple fields are returned. | -| `owned` | boolean | no | Limit by projects explicitly owned by the current user | -| `membership` | boolean | no | Limit by projects that the current user is a member of | -| `starred` | boolean | no | Limit by projects starred by the current user | -| `statistics` | boolean | no | Include project statistics | -| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | -| `with_issues_enabled` | boolean | no | Limit by enabled issues feature | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of projects matching the search criteria | +| `simple` | boolean | no | Return only limited fields for each project. This is a no-op without authentication as then _only_ simple fields are returned. | +| `owned` | boolean | no | Limit by projects explicitly owned by the current user | +| `membership` | boolean | no | Limit by projects that the current user is a member of | +| `starred` | boolean | no | Limit by projects starred by the current user | +| `statistics` | boolean | no | Include project statistics | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | +| `with_issues_enabled` | boolean | no | Limit by enabled issues feature | | `with_merge_requests_enabled` | boolean | no | Limit by enabled merge requests feature | -| `with_programming_language` | string | no | Limit by projects which use the given programming language | -| `wiki_checksum_failed` | boolean | no | Limit projects where the wiki checksum calculation has failed _([Introduced][ee-6137] in [GitLab Premium][eep] 11.2)_ | -| `repository_checksum_failed` | boolean | no | Limit projects where the repository checksum calculation has failed _([Introduced][ee-6137] in [GitLab Premium][eep] 11.2)_ | -| `min_access_level` | integer | no | Limit by current user minimal [access level](members.md) | +| `with_programming_language` | string | no | Limit by projects which use the given programming language | +| `wiki_checksum_failed` | boolean | no | **[PREMIUM]** Limit projects where the wiki checksum calculation has failed *([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6137) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2)* | +| `repository_checksum_failed` | boolean | no | **[PREMIUM]** Limit projects where the repository checksum calculation has failed *([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6137) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2)* | +| `min_access_level` | integer | no | Limit by current user minimal [access level](members.md) | When `simple=true` or the user is unauthenticated this returns something like: @@ -156,7 +156,8 @@ When the user is authenticated and `simple` is not set this returns something li "repository_size": 1038090, "wiki_size" : 0, "lfs_objects_size": 0, - "job_artifacts_size": 0 + "job_artifacts_size": 0, + "packages_size": 0 }, "_links": { "self": "http://example.com/api/v4/projects", @@ -239,7 +240,8 @@ When the user is authenticated and `simple` is not set this returns something li "repository_size": 2066080, "wiki_size" : 0, "lfs_objects_size": 0, - "job_artifacts_size": 0 + "job_artifacts_size": 0, + "packages_size": 0 }, "_links": { "self": "http://example.com/api/v4/projects", @@ -254,6 +256,20 @@ When the user is authenticated and `simple` is not set this returns something li ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +[ + { + "id": 4, + "description": null, + "approvals_before_merge": 0, + ... + } +] +``` + You can filter by [custom attributes](custom_attributes.md) with: ``` @@ -349,7 +365,8 @@ GET /users/:user_id/projects "repository_size": 1038090, "wiki_size" : 0, "lfs_objects_size": 0, - "job_artifacts_size": 0 + "job_artifacts_size": 0, + "packages_size": 0 }, "_links": { "self": "http://example.com/api/v4/projects", @@ -432,7 +449,8 @@ GET /users/:user_id/projects "repository_size": 2066080, "wiki_size" : 0, "lfs_objects_size": 0, - "job_artifacts_size": 0 + "job_artifacts_size": 0, + "packages_size": 0 }, "_links": { "self": "http://example.com/api/v4/projects", @@ -548,6 +566,7 @@ GET /projects/:id "group_access_level": 10 } ], + "repository_storage": "default", "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "printing_merge_requests_link_enabled": true, @@ -559,7 +578,8 @@ GET /projects/:id "repository_size": 1038090, "wiki_size" : 0, "lfs_objects_size": 0, - "job_artifacts_size": 0 + "job_artifacts_size": 0, + "packages_size": 0 }, "_links": { "self": "http://example.com/api/v4/projects", @@ -573,6 +593,18 @@ GET /projects/:id } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `approvals_before_merge` parameter: + +```json +{ + "id": 3, + "description": null, + "approvals_before_merge": 0, + ... +} +``` + **Note**: The `web_url` and `avatar_url` attributes on `namespace` were [introduced][ce-27427] in GitLab 11.11. If the project is a fork, and you provide a valid token to authenticate, the @@ -674,6 +706,7 @@ POST /projects | `name` | string | yes if path is not provided | The name of the new project. Equals path if not provided. | | `path` | string | yes if name is not provided | Repository name for new project. Generated based on name if not provided (generated lowercased with dashes). | | `namespace_id` | integer | no | Namespace for the new project (defaults to the current user's namespace) | +| `default_branch` | string | no | `master` by default | | `description` | string | no | Short project description | | `issues_enabled` | boolean | no | Enable issues for this project | | `merge_requests_enabled` | boolean | no | Enable merge requests for this project | @@ -695,8 +728,16 @@ POST /projects | `avatar` | mixed | no | Image file for avatar of the project | | `printing_merge_request_link_enabled` | boolean | no | Show link to create/view merge request when pushing from the command line | | `ci_config_path` | string | no | The path to CI config file | +| `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | +| `approvals_before_merge` | integer | no | **[STARTER]** How many approvers should approve merge requests by default | +| `mirror` | boolean | no | **[STARTER]** Enables pull mirroring in a project | +| `mirror_trigger_builds` | boolean | no | **[STARTER]** Pull mirroring triggers builds | | `initialize_with_readme` | boolean | no | `false` by default | +>**Note**: If your HTTP repository is not publicly accessible, +add authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git` +where `password` is a public access key with the `api` scope enabled. + ## Create project for user Creates a new project owned by the specified user. Available only for admins. @@ -710,7 +751,6 @@ POST /projects/user/:user_id | `user_id` | integer | yes | The user ID of the project owner | | `name` | string | yes | The name of the new project | | `path` | string | no | Custom repository name for new project. By default generated based on name | -| `default_branch` | string | no | `master` by default | | `namespace_id` | integer | no | Namespace for the new project (defaults to the current user's namespace) | | `description` | string | no | Short project description | | `issues_enabled` | boolean | no | Enable issues for this project | @@ -733,6 +773,15 @@ POST /projects/user/:user_id | `avatar` | mixed | no | Image file for avatar of the project | | `printing_merge_request_link_enabled` | boolean | no | Show link to create/view merge request when pushing from the command line | | `ci_config_path` | string | no | The path to CI config file | +| `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | +| `approvals_before_merge` | integer | no | **[STARTER]** How many approvers should approve merge requests by default | +| `external_authorization_classification_label` | string | no | **[CORE ONLY]** The classification label for the project | +| `mirror` | boolean | no | **[STARTER]** Enables pull mirroring in a project | +| `mirror_trigger_builds` | boolean | no | **[STARTER]** Pull mirroring triggers builds | + +>**Note**: If your HTTP repository is not publicly accessible, +add authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git` +where `password` is a public access key with the `api` scope enabled. ## Edit project @@ -769,6 +818,19 @@ PUT /projects/:id | `avatar` | mixed | no | Image file for avatar of the project | | `ci_config_path` | string | no | The path to CI config file | | `ci_default_git_depth` | integer | no | Default number of revisions for [shallow cloning](../user/project/pipelines/settings.md#git-shallow-clone) | +| `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | +| `approvals_before_merge` | integer | no | **[STARTER]** How many approvers should approve merge request by default | +| `external_authorization_classification_label` | string | no | **[CORE ONLY]** The classification label for the project | +| `mirror` | boolean | no | **[STARTER]** Enables pull mirroring in a project | +| `mirror_user_id` | integer | no | **[STARTER]** User responsible for all the activity surrounding a pull mirror event | +| `mirror_trigger_builds` | boolean | no | **[STARTER]** Pull mirroring triggers builds | +| `only_mirror_protected_branches` | boolean | no | **[STARTER]** Only mirror protected branches | +| `mirror_overwrites_diverged_branches` | boolean | no | **[STARTER]** Pull mirror overwrites diverged branches | +| `packages_enabled` | boolean | no | **[PREMIUM ONLY]** Enable or disable packages repository feature | + +>**Note**: If your HTTP repository is not publicly accessible, +add authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git` +where `password` is a public access key with the `api` scope enabled. ## Fork project @@ -1548,9 +1610,104 @@ POST /projects/:id/housekeeping | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | + +## Push Rules **[STARTER]** + +### Get project push rules + +Get the push rules of a project. + +``` +GET /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | + +```json +{ + "id": 1, + "project_id": 3, + "commit_message_regex": "Fixes \d+\..*", + "branch_name_regex": "", + "deny_delete_tag": false, + "created_at": "2012-10-12T17:04:47Z", + "member_check": false, + "prevent_secrets": false, + "author_email_regex": "", + "file_name_regex": "", + "max_file_size": 5, + "commit_committer_check": false +} +``` + +The following attributes are restricted to certain plans, and will not appear if +you do not have access to those features: + +* `commit_committer_check` only available on **[PREMIUM]** + +### Add project push rule + +Adds a push rule to a specified project. + +``` +POST /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| -------------------------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `deny_delete_tag` **[STARTER]** | boolean | no | Deny deleting a tag | +| `member_check` **[STARTER]** | boolean | no | Restrict commits by author (email) to existing GitLab users | +| `prevent_secrets` **[STARTER]** | boolean | no | GitLab will reject any files that are likely to contain secrets | +| `commit_message_regex` **[STARTER]** | string | no | All commit messages must match this, e.g. `Fixed \d+\..*` | +| `branch_name_regex` **[STARTER]** | string | no | All branch names must match this, e.g. `(feature|hotfix)\/*` | +| `author_email_regex` **[STARTER]** | string | no | All commit author emails must match this, e.g. `@my-company.com$` | +| `file_name_regex` **[STARTER]** | string | no | All commited filenames must **not** match this, e.g. `(jar|exe)$` | +| `max_file_size` **[STARTER]** | integer | no | Maximum file size (MB) | +| `commit_committer_check` **[PREMIUM]** | boolean | no | Users can only push commits to this repository that were committed with one of their own verified emails. | + +### Edit project push rule + +Edits a push rule for a specified project. + +``` +PUT /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| -------------------------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `deny_delete_tag` **[STARTER]** | boolean | no | Deny deleting a tag | +| `member_check` **[STARTER]** | boolean | no | Restrict commits by author (email) to existing GitLab users | +| `prevent_secrets` **[STARTER]** | boolean | no | GitLab will reject any files that are likely to contain secrets | +| `commit_message_regex` **[STARTER]** | string | no | All commit messages must match this, e.g. `Fixed \d+\..*` | +| `branch_name_regex` **[STARTER]** | string | no | All branch names must match this, e.g. `(feature|hotfix)\/*` | +| `author_email_regex` **[STARTER]** | string | no | All commit author emails must match this, e.g. `@my-company.com$` | +| `file_name_regex` **[STARTER]** | string | no | All commited filenames must **not** match this, e.g. `(jar|exe)$` | +| `max_file_size` **[STARTER]** | integer | no | Maximum file size (MB) | +| `commit_committer_check` **[PREMIUM]** | boolean | no | Users can only push commits to this repository that were committed with one of their own verified emails. | + +### Delete project push rule + +> Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 9.0. + +Removes a push rule from a project. This is an idempotent method and can be called multiple times. +Either the push rule is available or not. + +``` +DELETE /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | -### Transfer a project to a new namespace +## Transfer a project to a new namespace + +> Introduced in GitLab 11.1. ``` PUT /projects/:id/transfer @@ -1572,6 +1729,22 @@ Read more in the [Project import/export](project_import_export.md) documentation Read more in the [Project members](members.md) documentation. +## Start the pull mirroring process for a Project **[STARTER]** + +> Introduced in [GitLab Starter](https://about.gitlab.com/pricing) 10.3. + +``` +POST /projects/:id/mirror/pull +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/mirror/pull +``` + ## Project badges Read more in the [Project Badges](project_badges.md) documentation. @@ -1602,6 +1775,4 @@ GET /projects/:id/snapshot | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `wiki` | boolean | no | Whether to download the wiki, rather than project, repository | -[eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" -[ee-6137]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6137 [ce-27427]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27427 diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index a261bb75be5..6e41584afef 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -10,6 +10,7 @@ The access levels are defined in the `ProtectedRefAccess.allowed_access_levels` 0 => No access 30 => Developer access 40 => Maintainer access +60 => Admin access ``` ## List protected branches @@ -51,6 +52,36 @@ Example response: ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `user_id` and `group_id` parameters: + +Example response: + +```json +[ + { + "name": "master", + "push_access_levels": [ + { + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "merge_access_levels": [ + { + "access_level": null, + "user_id": null, + "group_id": 1234, + "access_level_description": "Example Merge Group" + } + ] + }, + ... +] +``` + ## Get a single protected branch or wildcard protected branch Gets a single protected branch or wildcard protected branch. @@ -88,6 +119,33 @@ Example response: } ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `user_id` and `group_id` parameters: + +Example response: + +```json +{ + "name": "master", + "push_access_levels": [ + { + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ], + "merge_access_levels": [ + { + "access_level": null, + "user_id": null, + "group_id": 1234, + "access_level_description": "Example Merge Group" + } + ] +} +``` + ## Protect repository branches Protects a single repository branch or several project repository @@ -98,15 +156,47 @@ POST /projects/:id/protected_branches ``` ```bash -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30' +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30&unprotect_access_level=40' ``` | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the branch or wildcard | -| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, maintainer access level) | -| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, maintainer access level) | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the branch or wildcard | +| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, maintainer access level) | +| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, maintainer access level) | +| `unprotect_access_level` | string | no | Access levels allowed to unprotect (defaults: `40`, maintainer access level) | +| `allowed_to_push` | array | no | **[STARTER]** Array of access levels allowed to push, with each described by a hash | +| `allowed_to_merge` | array | no | **[STARTER]** Array of access levels allowed to merge, with each described by a hash | +| `allowed_to_unprotect` | array | no | **[STARTER]**Array of access levels allowed to unprotect, with each described by a hash | + +Example response: + +```json +{ + "name": "*-stable", + "push_access_levels": [ + { + "access_level": 30, + "access_level_description": "Developers + Maintainers" + } + ], + "merge_access_levels": [ + { + "access_level": 30, + "access_level_description": "Developers + Maintainers" + ], + "unprotect_access_levels": [ + { + "access_level": 40, + "access_level_description": "Maintainers" + } + ] +} +``` + +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `user_id` and `group_id` parameters: Example response: @@ -116,13 +206,65 @@ Example response: "push_access_levels": [ { "access_level": 30, + "user_id": null, + "group_id": null, "access_level_description": "Developers + Maintainers" } ], "merge_access_levels": [ { "access_level": 30, + "user_id": null, + "group_id": null, "access_level_description": "Developers + Maintainers" + ], + "unprotect_access_levels": [ + { + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ] +} +``` + +### Example with user / group level access **[STARTER]** + +Elements in the `allowed_to_push` / `allowed_to_merge` / `allowed_to_unprotect` array should take the +form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). These access levels allow [more granular control over protected branch access](../user/project/protected_branches.md#restricting-push-and-merge-access-to-certain-users-starter) and were [added to the API in ][ee-3516] in GitLab 10.3 EE. + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&allowed_to_push%5B%5D%5Buser_id%5D=1' +``` + +Example response: + +```json +{ + "name":"*-stable", + "push_access_levels": [ + { + "access_level":null, + "user_id":1, + "group_id":null, + "access_level_description":"Administrator" + } + ], + "merge_access_levels": [ + { + "access_level":40, + "user_id":null, + "group_id":null, + "access_level_description":"Maintainers" + } + ], + "unprotect_access_levels": [ + { + "access_level":40, + "user_id":null, + "group_id":null, + "access_level_description":"Maintainers" } ] } @@ -144,3 +286,5 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" 'https://git | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the branch | + +[ee-3516]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3516 "ProtectedBranches API handles per user/group granularity" diff --git a/doc/api/search.md b/doc/api/search.md index da81c8321c9..abb77ae05dc 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -285,7 +285,7 @@ Example response: ### Scope: wiki_blobs **[STARTER]** -This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. ```bash curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=wiki_blobs&search=bye @@ -346,6 +346,7 @@ Example response: This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. Filters are available for this scope: + - filename - path - extension @@ -679,6 +680,7 @@ Example response: This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. Filters are available for this scope: + - filename - path - extension diff --git a/doc/api/services.md b/doc/api/services.md index 042fee4a21a..c811d0e84ca 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -291,7 +291,6 @@ Parameters: | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `tag_push_events` | boolean | false | Enable notifications for tag push events | - ### Delete Drone CI service Delete Drone CI service for a project. @@ -551,21 +550,21 @@ Get Irker (IRC gateway) service settings for a project. GET /projects/:id/services/irker ``` -## JIRA +## Jira -JIRA issue tracker. +Jira issue tracker. -### Get JIRA service settings +### Get Jira service settings -Get JIRA service settings for a project. +Get Jira service settings for a project. ``` GET /projects/:id/services/jira ``` -### Create/Edit JIRA service +### Create/Edit Jira service -Set JIRA service for a project. +Set Jira service for a project. > Starting with GitLab 8.14, `api_url`, `issues_url`, `new_issue_url` and > `project_url` are replaced by `url`. If you are using an @@ -579,18 +578,18 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `url` | string | yes | The URL to the JIRA project which is being linked to this GitLab project. For example, `https://jira.example.com`. | -| `api_url` | string | no | The base URL to the JIRA instance API. Web URL value will be used if not set. For example, `https://jira-api.example.com`. | -| `username` | string | yes | The username of the user created to be used with GitLab/JIRA. | -| `password` | string | yes | The password of the user created to be used with GitLab/JIRA. | +| `url` | string | yes | The URL to the Jira project which is being linked to this GitLab project. For example, `https://jira.example.com`. | +| `api_url` | string | no | The base URL to the Jira instance API. Web URL value will be used if not set. For example, `https://jira-api.example.com`. | +| `username` | string | yes | The username of the user created to be used with GitLab/Jira. | +| `password` | string | yes | The password of the user created to be used with GitLab/Jira. | | `active` | boolean | no | Activates or deactivates the service. Defaults to false (deactivated). | -| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the JIRA workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | +| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the Jira workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | | `commit_events` | boolean | false | Enable notifications for commit events | | `merge_requests_events` | boolean | false | Enable notifications for merge request events | -### Delete JIRA service +### Delete Jira service -Remove all previously JIRA settings from a project. +Remove all previously Jira settings from a project. ``` DELETE /projects/:id/services/jira @@ -744,7 +743,7 @@ Parameters: | --------- | ---- | -------- | ----------- | | `username` | string | yes | The username of a Packagist account | | `token` | string | yes | API token to the Packagist server | -| `server` | boolean | no | URL of the Packagist server. Leave blank for default: https://packagist.org | +| `server` | boolean | no | URL of the Packagist server. Leave blank for default: <https://packagist.org> | | `push_events` | boolean | false | Enable notifications for push events | | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `tag_push_events` | boolean | false | Enable notifications for tag push events | @@ -1161,7 +1160,7 @@ PUT /projects/:id/services/jenkins Parameters: -- `jenkins_url` (**required**) - Jenkins URL like http://jenkins.example.com +- `jenkins_url` (**required**) - Jenkins URL like `http://jenkins.example.com` - `project_name` (**required**) - The URL-friendly project name. Example: my_project_name - `username` (optional) - A user with access to the Jenkins server, if applicable - `password` (optional) - The password of the user @@ -1196,7 +1195,7 @@ PUT /projects/:id/services/jenkins-deprecated Parameters: -- `project_url` (**required**) - Jenkins project URL like http://jenkins.example.com/job/my-project/ +- `project_url` (**required**) - Jenkins project URL like `http://jenkins.example.com/job/my-project/` - `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin - `pass_unstable` (optional) - Unstable builds will be treated as passing diff --git a/doc/api/settings.md b/doc/api/settings.md index c2a1f7feefd..876a5a75590 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -67,6 +67,14 @@ Example response: } ``` +Users on GitLab [Premium or Ultimate](https://about.gitlab.com/pricing/) may also see +the `file_template_project_id` or the `geo_node_allowed_ips` parameters: **[PREMIUM ONLY]** + +```json + "file_template_project_id": 1, + "geo_node_allowed_ips": "0.0.0.0/0, ::/0" +``` + ## Change application settings Use an API call to modify GitLab instance @@ -121,10 +129,33 @@ Example response: "performance_bar_allowed_group_id": 42, "instance_statistics_visibility_private": false, "user_show_add_ssh_key_message": true, - "local_markdown_version": 0 + "file_template_project_id": 1, + "local_markdown_version": 0, + "geo_node_allowed_ips": "0.0.0.0/0, ::/0" } ``` +Users on GitLab [Premium or Ultimate](https://about.gitlab.com/pricing/) may also see +these parameters: + +- `external_authorization_service_enabled` +- `external_authorization_service_url` +- `external_authorization_service_default_label` +- `external_authorization_service_timeout` +- `file_template_project_id` +- `geo_node_allowed_ips` + +Example responses: **[PREMIUM ONLY]** + +```json + "external_authorization_service_enabled": true, + "external_authorization_service_url": "https://authorize.me", + "external_authorization_service_default_label": "default", + "external_authorization_service_timeout": 0.5, + "file_template_project_id": 1, + "geo_node_allowed_ips": "0.0.0.0/0, ::/0" +``` + ## List of settings that can be accessed via API calls In general, all settings are optional. Certain settings though, if enabled, will @@ -138,10 +169,12 @@ are listed in the descriptions of the relevant settings. | `after_sign_up_text` | string | no | Text shown to the user after signing up | | `akismet_api_key` | string | required by: `akismet_enabled` | API key for akismet spam protection. | | `akismet_enabled` | boolean | no | (**If enabled, requires:** `akismet_api_key`) Enable or disable akismet spam protection. | +| `allow_group_owners_to_manage_ldap` | boolean | no | **[Premium]** Set to `true` to allow group owners to manage LDAP | | `allow_local_requests_from_hooks_and_services` | boolean | no | Allow requests to the local network from hooks and services. | | `authorized_keys_enabled` | boolean | no | By default, we write to the `authorized_keys` file to support Git over SSH without additional configuration. GitLab can be optimized to authenticate SSH keys via the database file. Only disable this if you have configured your OpenSSH server to use the AuthorizedKeysCommand. | | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It will automatically build, test, and deploy applications based on a predefined CI/CD configuration. | +| `check_namespace_plan` | boolean | no | **[Premium]** Enabling this will make only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `clientside_sentry_dsn` | string | required by: `clientside_sentry_enabled` | Clientside Sentry Data Source Name. | | `clientside_sentry_enabled` | boolean | no | (**If enabled, requires:** `clientside_sentry_dsn`) Enable Sentry error reporting for the client side. | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | @@ -158,10 +191,31 @@ are listed in the descriptions of the relevant settings. | `dsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded DSA key. Default is `0` (no restriction). `-1` disables DSA keys. | | `ecdsa_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ECDSA key. Default is `0` (no restriction). `-1` disables ECDSA keys. | | `ed25519_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ED25519 key. Default is `0` (no restriction). `-1` disables ED25519 keys. | +| `elasticsearch_aws` | boolean | no | **[Premium]** Enable the use of AWS hosted Elasticsearch | +| `elasticsearch_aws_access_key` | string | no | **[Premium]** AWS IAM access key | +| `elasticsearch_aws_region` | string | no | **[Premium]** The AWS region the elasticsearch domain is configured | +| `elasticsearch_aws_secret_access_key` | string | no | **[Premium]** AWS IAM secret access key | +| `elasticsearch_experimental_indexer` | boolean | no | **[Premium]** Use the experimental elasticsearch indexer. More info: <https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer> | +| `elasticsearch_indexing` | boolean | no | **[Premium]** Enable Elasticsearch indexing | +| `elasticsearch_search` | boolean | no | **[Premium]** Enable Elasticsearch search | +| `elasticsearch_url` | string | no | **[Premium]** The url to use for connecting to Elasticsearch. Use a comma-separated list to support cluster (e.g., `http://localhost:9200, http://localhost:9201"`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `elasticsearch_limit_indexing` | boolean | no | **[Premium]** Limit Elasticsearch to index certain namespaces and projects | +| `elasticsearch_project_ids` | array of integers | no | **[Premium]** The projects to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | +| `elasticsearch_namespace_ids` | array of integers | no | **[Premium]** The namespaces to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. | +| `email_additional_text` | string | no | **[Premium]** Additional text added to the bottom of every email for legal/auditing/compliance reasons | | `email_author_in_body` | boolean | no | Some email servers do not support overriding the email sender name. Enable this option to include the name of the author of the issue, merge request or comment in the email body instead. | | `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | | `enforce_terms` | boolean | no | (**If enabled, requires:** `terms`) Enforce application ToS to all users. | -| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | +| `external_auth_client_cert` | string | no | **[Premium]** (**If enabled, requires:** `external_auth_client_key`) The certificate to use to authenticate with the external authorization service | +| `external_auth_client_key` | string | required by: `external_auth_client_cert` | **[Premium]** Private key for the certificate when authentication is required for the external authorization service, this is encrypted when stored | +| `external_auth_client_key_pass` | string | no | **[Premium]** Passphrase to use for the private key when authenticating with the external service this is encrypted when stored | +| `external_authorization_service_enabled` | boolean | no | **[Premium]** (**If enabled, requires:** `external_authorization_service_default_label`, `external_authorization_service_timeout` and `external_authorization_service_url` ) Enable using an external authorization service for accessing projects | +| `external_authorization_service_default_label` | string | required by: `external_authorization_service_enabled` | **[Premium]** The default classification label to use when requesting authorization and no classification label has been specified on the project | +| `external_authorization_service_timeout` | float | required by: `external_authorization_service_enabled` | **[Premium]** The timeout after which an authorization request is aborted, in seconds. When a request times out, access is denied to the user. (min: 0.001, max: 10, step: 0.001) | +| `external_authorization_service_url` | string | required by: `external_authorization_service_enabled` | **[Premium]** URL to which authorization requests will be directed | +| `file_template_project_id` | integer | no | **[Premium]** The ID of a project to load custom file templates from | +| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | +| `geo_status_timeout` | integer | no | **[Premium]** The amount of seconds after which a request to get a secondary node status will time out. | | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | @@ -170,6 +224,7 @@ are listed in the descriptions of the relevant settings. | `help_page_hide_commercial_content` | boolean | no | Hide marketing-related entries from help. | | `help_page_support_url` | string | no | Alternate support URL for help page. | | `help_page_text` | string | no | Custom text displayed on the help page. | +| `help_text` | string | no | **[Premium]** GitLab server administrator information | | `hide_third_party_offers` | boolean | no | Do not display offers from third parties within GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | @@ -192,6 +247,9 @@ are listed in the descriptions of the relevant settings. | `metrics_sample_interval` | integer | required by: `metrics_enabled` | The sampling interval in seconds. | | `metrics_timeout` | integer | required by: `metrics_enabled` | The amount of seconds after which InfluxDB will time out. | | `mirror_available` | boolean | no | Allow mirrors to be set up for projects. If disabled, only admins will be able to set up mirrors in projects. | +| `mirror_capacity_threshold` | integer | no | **[Premium]** Minimum capacity to be available before scheduling more mirrors preemptively | +| `mirror_max_capacity` | integer | no | **[Premium]** Maximum number of mirrors that can be synchronizing at the same time. | +| `mirror_max_delay` | integer | no | **[Premium]** Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | | `pages_domain_verification_enabled` | boolean | no | Require users to prove ownership of custom domains. Domain verification is an essential security measure for public GitLab sites. Users are required to demonstrate they control a domain before it is enabled. | | `password_authentication_enabled_for_git` | boolean | no | Enable authentication for Git over HTTP(S) via a GitLab account password. Default is `true`. | | `password_authentication_enabled_for_web` | boolean | no | Enable authentication for the web interface via a GitLab account password. Default is `true`. | @@ -203,23 +261,28 @@ are listed in the descriptions of the relevant settings. | `polling_interval_multiplier` | decimal | no | Interval multiplier used by endpoints that perform polling. Set to `0` to disable polling. | | `project_export_enabled` | boolean | no | Enable project export. | | `prometheus_metrics_enabled` | boolean | no | Enable prometheus metrics. | +| `pseudonymizer_enabled` | boolean | no | **[Premium]** When enabled, GitLab will run a background job that will produce pseudonymized CSVs of the GitLab database that will be uploaded to your configured object storage directory. | `recaptcha_enabled` | boolean | no | (**If enabled, requires:** `recaptcha_private_key` and `recaptcha_site_key`) Enable recaptcha. | | `recaptcha_private_key` | string | required by: `recaptcha_enabled` | Private key for recaptcha. | | `recaptcha_site_key` | string | required by: `recaptcha_enabled` | Site key for recaptcha. | | `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | +| `repository_size_limit` | integer | no | **[Premium]** Size limit per repository (MB) | | `repository_storages` | array of strings | no | A list of names of enabled storage paths, taken from `gitlab.yml`. New projects will be created in one of these stores, chosen at random. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | | `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-admin users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | | `rsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded RSA key. Default is `0` (no restriction). `-1` disables RSA keys. | | `send_user_confirmation_email` | boolean | no | Send confirmation email on sign-up. | -| `sentry_dsn` | string | required by: `sentry_enabled` | Sentry Data Source Name. | -| `sentry_enabled` | boolean | no | (**If enabled, requires:** `sentry_dsn`) Sentry is an error reporting and logging tool which is currently not shipped with GitLab, available at <https://sentry.io>. | | `session_expire_delay` | integer | no | Session duration in minutes. GitLab restart is required to apply changes | -| `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text`) Enable shared runners for new projects. | +| `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text` and `shared_runners_minutes`) Enable shared runners for new projects. | +| `shared_runners_minutes` | integer | required by: `shared_runners_enabled` | **[Premium]** Set the maximum number of pipeline minutes that a group can use on shared Runners per month. | | `shared_runners_text` | string | required by: `shared_runners_enabled` | Shared runners text. | | `sign_in_text` | string | no | Text on the login page. | | `signin_enabled` | string | no | (Deprecated: Use `password_authentication_enabled_for_web` instead) Flag indicating if password authentication is enabled for the web interface. | | `signup_enabled` | boolean | no | Enable registration. Default is `true`. | +| `slack_app_enabled` | boolean | no | **[Premium]** (**If enabled, requires:** `slack_app_id`, `slack_app_secret` and `slack_app_secret`) Enable Slack app. | +| `slack_app_id` | string | required by: `slack_app_enabled` | **[Premium]** The app id of the Slack-app. | +| `slack_app_secret` | string | required by: `slack_app_enabled` | **[Premium]** The app secret of the Slack-app. | +| `slack_app_verification_token` | string | required by: `slack_app_enabled` | **[Premium]** The verification token of the Slack-app. | | `terminal_max_session_time` | integer | no | Maximum time for web terminal websocket connection (in seconds). Set to `0` for unlimited time. | | `terms` | text | required by: `enforce_terms` | (**Required by:** `enforce_terms`) Markdown content for the ToS. | | `throttle_authenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_api_period_in_seconds` and `throttle_authenticated_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (e.g. from crawlers or abusive bots). | @@ -231,6 +294,7 @@ are listed in the descriptions of the relevant settings. | `throttle_unauthenticated_enabled` | boolean | no | (**If enabled, requires:** `throttle_unauthenticated_period_in_seconds` and `throttle_unauthenticated_requests_per_period`) Enable unauthenticated request rate limit. Helps reduce request volume (e.g. from crawlers or abusive bots). | | `throttle_unauthenticated_period_in_seconds` | integer | required by: `throttle_unauthenticated_enabled` | Rate limit period in seconds. | | `throttle_unauthenticated_requests_per_period` | integer | required by: `throttle_unauthenticated_enabled` | Max requests per period per IP. | +| `time_tracking_limit_to_hours` | boolean | no | Limit display of time tracking units to hours. Default is `false`. | | `two_factor_grace_period` | integer | required by: `require_two_factor_authentication` | Amount of time (in hours) that users are allowed to skip forced configuration of two-factor authentication. | | `unique_ips_limit_enabled` | boolean | no | (**If enabled, requires:** `unique_ips_limit_per_user` and `unique_ips_limit_time_window`) Limit sign in from multiple ips. | | `unique_ips_limit_per_user` | integer | required by: `unique_ips_limit_enabled` | Maximum number of ips per user. | @@ -241,3 +305,4 @@ are listed in the descriptions of the relevant settings. | `user_show_add_ssh_key_message` | boolean | no | When set to `false` disable the "You won't be able to pull or push project code via SSH" warning shown to users with no uploaded SSH key. | | `version_check_enabled` | boolean | no | Let GitLab inform you when an update is available. | | `local_markdown_version` | integer | no | Increase this value when any cached markdown should be invalidated. | +| `geo_node_allowed_ips` | string | yes | **[Premium]** Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | diff --git a/doc/api/users.md b/doc/api/users.md index 4bc0335ae33..6be097e6364 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -147,6 +147,24 @@ GET /users ] ``` +Users on GitLab [Silver or higher](https://about.gitlab.com/pricing/) will also see +the `group_saml` provider option: + +```json +[ + { + "id": 1, + ... + "identities": [ + {"provider": "github", "extern_uid": "2435223452345"}, + {"provider": "bitbucket", "extern_uid": "john.smith"}, + {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"}, + {"provider": "group_saml", "extern_uid": "123789", "saml_provider_id": 10} + ], + ... + } +] + You can lookup users by external UID and provider: ``` @@ -223,6 +241,8 @@ Parameters: - `id` (required) - The ID of a user +Example Responses: + ```json { "id": 1, @@ -258,9 +278,27 @@ Parameters: "can_create_project": true, "two_factor_enabled": true, "external": false, - "private_profile": false, - "shared_runners_minutes_limit": 133 + "private_profile": false +} +``` + +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see +the `shared_runners_minutes_limit` and `extra_shared_runners_minutes_limit` parameters. +Users on GitLab Silver will also see the `group_saml` option: + +```json +{ + "id": 1, + "username": "john_smith", + "shared_runners_minutes_limit": 133, "extra_shared_runners_minutes_limit": 133 + "identities": [ + {"provider": "github", "extern_uid": "2435223452345"}, + {"provider": "bitbucket", "extern_uid": "john.smith"}, + {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"}, + {"provider": "group_saml", "extern_uid": "123789", "saml_provider_id": 10} + ], + ... } ``` @@ -272,7 +310,14 @@ GET /users/:id?with_custom_attributes=true ## User creation -Creates a new user. Note only administrators can create new users. Either `password` or `reset_password` should be specified (`reset_password` takes priority). If `reset_password` is `false`, then `password` is required. +Creates a new user. Note only administrators can create new +users. Either `password`, `reset_password`, or `force_random_password` +must be specified. If `reset_password` and `force_random_password` are +both `false`, then `password` is required. + +Note that `force_random_password` and `reset_password` take priority +over `password`. In addition, `reset_password` and +`force_random_password` can be used together. ``` POST /users @@ -293,7 +338,6 @@ Parameters: - `projects_limit` (optional) - Number of projects user can create - `extern_uid` (optional) - External UID - `provider` (optional) - External provider name -- `group_id_for_saml` (optional) - ID of group where SAML has been configured - `bio` (optional) - User's biography - `location` (optional) - User's location - `public_email` (optional) - The public email of the user @@ -303,8 +347,8 @@ Parameters: - `external` (optional) - Flags the user as external - true or false(default) - `avatar` (optional) - Image file for user's avatar - `private_profile` (optional) - User's profile is private - true or false -- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user -- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user +- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **[STARTER]** +- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **[STARTER]** ## User modification @@ -340,6 +384,8 @@ Parameters: - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user - `avatar` (optional) - Image file for user's avatar - `private_profile` (optional) - User's profile is private - true or false +- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **[STARTER]** +- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **[STARTER]** On password update, user will be forced to change it upon next login. Note, at the moment this method does only return a `404` error, @@ -1262,4 +1308,4 @@ Example response: Please note that `last_activity_at` is deprecated, please use `last_activity_on`. -[gemojione-index]: https://github.com/jonathanwiesel/gemojione/blob/master/config/index.json
\ No newline at end of file +[gemojione-index]: https://github.com/jonathanwiesel/gemojione/blob/master/config/index.json diff --git a/doc/ci/README.md b/doc/ci/README.md index 1743c38eb46..d851a56ee0e 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -63,7 +63,7 @@ Once you're familiar with how GitLab CI/CD works, see the for all the attributes you can set and use. NOTE: **Note:** -GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/admin_area/settings/continuous_integration.md#extra-shared-runners-pipeline-minutes-quota). +GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/admin_area/settings/continuous_integration.md#extra-shared-runners-pipeline-minutes-quota-free-only). ## Configuration @@ -81,6 +81,7 @@ GitLab CI/CD supports numerous configuration options: | [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| | [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | | [Pipelines triggers](triggers/README.md) | Trigger pipelines through the API. | +| [Pipelines for Merge Requests](merge_request_pipelines/index.md) | Design a pipeline structure for running a pipeline in merge requests. | | [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | | [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own GitLab Runners to execute your scripts. | | [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repos. | diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index b4c4bea6447..efdcaf5a6f5 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -205,7 +205,14 @@ An example project using this approach can be found here: <https://gitlab.com/gi ### Use Docker socket binding -The third approach is to bind-mount `/var/run/docker.sock` into the container so that docker is available in the context of that image. +The third approach is to bind-mount `/var/run/docker.sock` into the +container so that Docker is available in the context of that image. + +NOTE: **Note:** +If you bind the Docker socket [when using GitLab Runner 11.11 or +newer](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1261), +you can no longer use `docker:dind` as a service because volume bindings +are done to the services as well, making these incompatible. In order to do that, follow the steps: diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index e4dc289dbdb..e012f4f8595 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -463,8 +463,6 @@ that runner. > support for using private registries, which required manual configuration > of credentials on runner's host. We recommend to upgrade your Runner to > at least version **1.8** if you want to use private registries. -> - If the repository is private you need to authenticate your GitLab Runner in the -> registry. Learn more about how [GitLab Runner works in this case][runner-priv-reg]. To access private container registries, the GitLab Runner process can use: @@ -489,6 +487,20 @@ it's provided as an environment variable. This is because GitLab Runnner uses ** runtime. ### Using statically-defined credentials +There are two approaches that you can take in order to access a +private registry. Both require setting the environment variable +`DOCKER_AUTH_LOGIN` with appropriate authentication info. + +1. Per-job: To configure one job to access a private registry, add + `DOCKER_AUTH_LOGIN` as a job variable. +1. Per-runner: To configure a Runner so all its jobs can access a + private registry, add `DOCKER_AUTH_LOGIN` to the environment in the + Runner's configuration. + +See below for examples of each. + +#### Determining your `DOCKER_AUTH_LOGIN` data + As an example, let's assume that you want to use the `registry.example.com:5000/private/image:latest` image which is private and requires you to login into a private container registry. @@ -500,30 +512,41 @@ Let's also assume that these are the login credentials: | username | `my_username` | | password | `my_password` | -To configure access for `registry.example.com:5000`, follow these steps: +There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: + +- **First way -** Do a `docker login` on your local machine: + + ```bash + docker login registry.example.com:5000 --username my_username --password my_password + ``` -1. Find what the value of `DOCKER_AUTH_CONFIG` should be. There are two ways to - accomplish this: - - **First way -** Do a `docker login` on your local machine: + Then copy the content of `~/.docker/config.json`. - ```bash - docker login registry.example.com:5000 --username my_username --password my_password - ``` + If you don't need access to the registry from your computer, you + can do a `docker logout`: + + ```bash + docker logout registry.example.com:5000 + ``` + +- **Second way -** In some setups, it's possible that Docker client +will use the available system keystore to store the result of `docker +login`. In that case, it's impossible to read `~/.docker/config.json`, +so you will need to prepare the required base64-encoded version of +`${username}:${password}` manually. Open a terminal and execute the +following command: + + ```bash + echo -n "my_username:my_password" | base64 - Then copy the content of `~/.docker/config.json`. - - **Second way -** In some setups, it's possible that Docker client will use - the available system keystore to store the result of `docker login`. In - that case, it's impossible to read `~/.docker/config.json`, so you will - need to prepare the required base64-encoded version of - `${username}:${password}` manually. Open a terminal and execute the - following command: + # Example output to copy + bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= + ``` - ```bash - echo -n "my_username:my_password" | base64 +#### Configuring a job - # Example output to copy - bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= - ``` +To configure a single job with access for `registry.example.com:5000`, +follow these steps: 1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: @@ -538,14 +561,6 @@ To configure access for `registry.example.com:5000`, follow these steps: } ``` -1. Optionally,if you followed the first way of finding the `DOCKER_AUTH_CONFIG` - value, do a `docker logout` on your computer if you don't need access to the - registry from it: - - ```bash - docker logout registry.example.com:5000 - ``` - 1. You can now use any private image from `registry.example.com:5000` defined in `image` and/or `services` in your `.gitlab-ci.yml` file: @@ -566,6 +581,37 @@ for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. Specifying only `registry.example.com` will not work. +### Configuring a Runner + +If you have many pipelines that access the same registry, it'll +probably be better to setup registry access at the runner level. This +allows pipeline authors to have access to a private registry just by +running a job on the appropriate runner. It also makes registry +changes and credential rotations much simpler. + +Of course this means that any job on that runner can access the +registry with the same privilege, even across projects. If you need to +control access to the registry, you'll need to be sure to control +access to the runner. + +To add `DOCKER_AUTH_CONFIG` to a Runner: + +1. Modify the Runner's `config.toml` file as follows: + + ```toml + [[runners]] + environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] + ``` + +1. Restart the Runner service. + +NOTE: **Note:** The double quotes included in the `DOCKER_AUTH_CONFIG` +data must be escaped with backslashes. This prevents them from being +interpreted as TOML. + +NOTE: **Note:** The `environment` option is a list. So your Runner may +have existing entries and you should add this to the list, not replace +it. ### Using Credentials Store @@ -574,7 +620,7 @@ Specifying only `registry.example.com` will not work. To configure credentials store, follow these steps: 1. To use a credentials store, you need an external helper program to interact with a specific keychain or external store. -Make sure helper program is available in GitLab Runner `$PATH`. + Make sure helper program is available in GitLab Runner `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - Create a @@ -741,7 +787,6 @@ creation. [tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/ [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ -[runner-priv-reg]: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry [entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint [cmd]: https://docs.docker.com/engine/reference/builder/#cmd [register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index 0e33a1ba060..bd899240307 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -47,10 +47,10 @@ deploy: In the above configuration: - The `before_script` installs [SBT](http://www.scala-sbt.org/) and -displays the version that is being used. + displays the version that is being used. - The `test` stage executes SBT to compile and test the project. - - [sbt-scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT -plugin to measure test coverage. + - [sbt-scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT + plugin to measure test coverage. - The `deploy` stage automatically deploys the project to Heroku using dpl. You can use other versions of Scala and SBT by defining them in diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index 5adb7ebd30d..e70ae0bd154 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,14 +1,9 @@ --- -type: reference +type: reference, index +last_update: 2019-07-03 --- -# Pipelines for merge requests - -NOTE: **Note**: -As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 -or higher due to the [recent refspecs -changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). -Anything lower will cause the pipeline to fail. +# Pipelines for Merge Requests > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) in GitLab 11.6. @@ -23,6 +18,16 @@ for when you are running a pipeline in a merge request. This could be either adding or removing steps in the pipeline, to make sure that your pipelines are as efficient as possible. +## Requirements and limitations + +Pipelines for merge requests have the following requirements and limitations: + +- As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 + or higher due to the + [recent refspecs changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). +- Pipelines for merge requests are incompatible with + [CI/CD for external repositories](../ci_cd_for_external_repos/index.md). + ## Configuring pipelines for merge requests To configure pipelines for merge requests, add the `only: merge_requests` parameter to @@ -73,117 +78,11 @@ when a merge request was created or updated. For example: ## Pipelines for Merged Results **[PREMIUM]** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. -> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222), but [can be enabled manually](#enabling-pipelines-for-merged-results). - -It's possible for your source and target branches to diverge, which can result -in the scenario that source branch's pipeline was green, the target's pipeline was green, -but the combined output fails. - -By having your merge request pipeline automatically -create a new ref that contains the merge result of the source and target branch -(then running a pipeline on that ref), we can better test that the combined result -is also valid. - -GitLab can run pipelines for merge requests -on this merged result. That is, where the source and target branches are combined into a -new ref and a pipeline for this ref validates the result prior to merging. - - - -There are some cases where creating a combined ref is not possible or not wanted. -For example, a source branch that has conflicts with the target branch -or a merge request that is still in WIP status. In this case, the merge request pipeline falls back to a "detached" state -and runs on the source branch ref as if it was a regular pipeline. - -The detached state serves to warn you that you are working in a situation -subjected to merge problems, and helps to highlight that you should -get out of WIP status or resolve merge conflicts as soon as possible. - -### Enabling Pipelines for Merged Results - -To enable pipelines on merged results at the project level: - -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. -1. Click **Save changes** button. - - - -CAUTION: **Warning:** -Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](#configuring-pipelines-for-merge-requests), -otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. - -### Pipelines for Merged Result's limitations - -- This feature requires [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- This feature requires [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. -- Forking/cross-repo workflows are not currently supported. To follow progress, see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). -- This feature is not available for [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). - -## Merge Trains **[PREMIUM]** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. -> This feature is disabled by default, but [can be enabled manually](#enabling-merge-trains). +Read the [documentation on Pipelines for Merged Results](pipelines_for_merged_results/index.md). -[Pipelines for merged results](#pipelines-for-merged-results-premium) introduces -running a build on the result of the merged code prior to merging, as a way to keep master green. -There's a scenario, however, for teams with a high number of changes in the target branch (typically master) where in many or even all cases, -by the time the merged code is validated another commit has made it to master, invalidating the merged result. -You'd need some kind of queuing, cancellation or retry mechanism for these scenarios -in order to ensure an orderly flow of changes into the target branch. +### Merge Trains **[PREMIUM]** -Each MR that joins a merge train joins as the last item in the train, -just as it works in the current state. However, instead of queuing and waiting, -each item takes the completed state of the previous (pending) merge ref, adds its own changes, -and starts the pipeline immediately in parallel under the assumption that everything is going to pass. -In this way, if all the pipelines in the train merge successfully, no pipeline time is wasted either queuing or retrying. -If the button is subsequently pressed in a different MR, instead of creating a new pipeline for the target branch, -it creates a new pipeline targeting the merge result of the previous MR plus the target branch. -Pipelines invalidated through failures are immediately canceled and requeued. - -CAUTION: **Caution:** -At the moment, each merge train can generate a merge ref and run a pipeline **one at a time**. We plan to make the pipelines for merged results [run in parallel](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222) in a future release. - -### Enabling Merge Trains - -To enable merge trains at the project level: - -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Allow merge trains**. -1. Click **Save changes** button. - - - -CAUTION: **Warning:** -This feature requires [Pipelines for merged results](#pipelines-for-merged-results-premium) to be **configured properly**. - -### How to add a merge request to a merge train - -To add a merge request to a merge train: - -1. Click "Start/Add merge train" button in a merge request - - - -### How to remove a merge request from a merge train - -1. Click "Remove from merge train" button in the merge request widget. - - - -### Tips: Start/Add to merge train when pipeline succeeds - -You can add a merge request to a merge train only when the latest pipeline in the -merge request finished. While the pipeline is running or pending, you cannot add -the merge request to a train because the current change of the merge request may -be broken thus it could affect the following merge requests. - -In this case, you can schedule to add the merge request to a merge train **when the latest -pipeline succeeds**. You can see the following button instead of the regular "Start/Add merge train" -button while the latest pipeline is running. - - +Read the [documentation on Merge Trains](pipelines_for_merged_results/merge_trains/index.md). ## Excluding certain jobs @@ -269,3 +168,15 @@ to integrate your job with [GitLab Merge Request API](../../api/merge_requests.m You can find the list of available variables in [the reference sheet](../variables/predefined_variables.md). The variable names begin with the `CI_MERGE_REQUEST_` prefix. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png Binary files differindex 58d5581f628..58d5581f628 100644 --- a/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png Binary files differindex 0a84e61d284..0a84e61d284 100644 --- a/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md new file mode 100644 index 00000000000..3c5088089fa --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -0,0 +1,78 @@ +--- +type: reference +last_update: 2019-07-03 +--- + +# Pipelines for Merged Results **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. +> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222), but [can be enabled manually](#enabling-pipelines-for-merged-results). + +It's possible for your source and target branches to diverge, which can result +in the scenario that source branch's pipeline was green, the target's pipeline was green, +but the combined output fails. + +By having your merge request pipeline automatically +create a new ref that contains the merge result of the source and target branch +(then running a pipeline on that ref), we can better test that the combined result +is also valid. + +GitLab can run pipelines for merge requests +on this merged result. That is, where the source and target branches are combined into a +new ref and a pipeline for this ref validates the result prior to merging. + + + +There are some cases where creating a combined ref is not possible or not wanted. +For example, a source branch that has conflicts with the target branch +or a merge request that is still in WIP status. In this case, the merge request pipeline falls back to a "detached" state +and runs on the source branch ref as if it was a regular pipeline. + +The detached state serves to warn you that you are working in a situation +subjected to merge problems, and helps to highlight that you should +get out of WIP status or resolve merge conflicts as soon as possible. + +## Requirements and limitations + +Pipelines for merged results require: + +- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. +- [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. + +In addition, pipelines for merged results have the following limitations: + +- Forking/cross-repo workflows are not currently supported. To follow progress, + see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). +- This feature is not available for + [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. + To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). + +## Enabling Pipelines for Merged Results + +To enable pipelines on merged results at the project level: + +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. +1. Click **Save changes** button. + + + +CAUTION: **Warning:** +Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](../index.md#configuring-pipelines-for-merge-requests), +otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. + +## Merge Trains **[PREMIUM]** + +Read the [documentation on Merge Trains](merge_trains/index.md). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/merge_request_pipelines/img/merge_train_cancel.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png Binary files differindex 1561fdcc7a5..1561fdcc7a5 100644 --- a/doc/ci/merge_request_pipelines/img/merge_train_cancel.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png diff --git a/doc/ci/merge_request_pipelines/img/merge_train_config.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png Binary files differindex fb0af43556e..fb0af43556e 100644 --- a/doc/ci/merge_request_pipelines/img/merge_train_config.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png diff --git a/doc/ci/merge_request_pipelines/img/merge_train_start.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png Binary files differindex f20108157d2..f20108157d2 100644 --- a/doc/ci/merge_request_pipelines/img/merge_train_start.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png diff --git a/doc/ci/merge_request_pipelines/img/merge_train_start_when_pipeline_succeeds.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png Binary files differindex 62c2f2f5ff5..62c2f2f5ff5 100644 --- a/doc/ci/merge_request_pipelines/img/merge_train_start_when_pipeline_succeeds.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md new file mode 100644 index 00000000000..c5ff6f9ebed --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -0,0 +1,88 @@ +--- +type: reference +last_update: 2019-07-03 +--- + +# Merge Trains **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. +> This feature is disabled by default, but [can be enabled manually](#enabling-merge-trains). + +[Pipelines for merged results](../index.md#pipelines-for-merged-results-premium) introduces +running a build on the result of the merged code prior to merging, as a way to keep master green. + +There's a scenario, however, for teams with a high number of changes in the target branch (typically master) where in many or even all cases, +by the time the merged code is validated another commit has made it to master, invalidating the merged result. +You'd need some kind of queuing, cancellation or retry mechanism for these scenarios +in order to ensure an orderly flow of changes into the target branch. + +Each MR that joins a merge train joins as the last item in the train, +just as it works in the current state. However, instead of queuing and waiting, +each item takes the completed state of the previous (pending) merge ref, adds its own changes, +and starts the pipeline immediately in parallel under the assumption that everything is going to pass. + +In this way, if all the pipelines in the train merge successfully, no pipeline time is wasted either queuing or retrying. +If the button is subsequently pressed in a different MR, instead of creating a new pipeline for the target branch, +it creates a new pipeline targeting the merge result of the previous MR plus the target branch. +Pipelines invalidated through failures are immediately canceled and requeued. + +## Requirements and limitations + +Merge trains have the following requirements and limitations: + +- This feature requires that + [pipelines for merged results](../index.md#pipelines-for-merged-results-premium) are + **configured properly**. +- Each merge train can generate a merge ref and run a pipeline **one at a time**. + We plan to make the pipelines for merged results + [run in parallel](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222) in a + future release. + +## Enabling Merge Trains + +To enable merge trains at the project level: + +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Allow merge trains**. +1. Click **Save changes** button. + + + +## How to add a merge request to a merge train + +To add a merge request to a merge train: + +1. Click "Start/Add merge train" button in a merge request + + + +## How to remove a merge request from a merge train + +1. Click "Remove from merge train" button in the merge request widget. + + + +## Start/Add to merge train when pipeline succeeds + +You can add a merge request to a merge train only when the latest pipeline in the +merge request finished. While the pipeline is running or pending, you cannot add +the merge request to a train because the current change of the merge request may +be broken thus it could affect the following merge requests. + +In this case, you can schedule to add the merge request to a merge train **when the latest +pipeline succeeds**. You can see the following button instead of the regular "Start/Add merge train" +button while the latest pipeline is running. + + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index c8c92002be2..1b50273eca2 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -339,7 +339,7 @@ Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. 1. Inputing variable types, keys, and values in the **Variables** section. -Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. + Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. Once you set them, they will be available for all subsequent pipelines. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 4655eec51de..e911e97d3c8 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -30,7 +30,7 @@ future GitLab releases.** | `CI_BUILDS_DIR` | all | 11.10 | Top-level directory where builds are executed. | | `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | -| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. Only populated when there is a merge request associated with the pipeline. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | | `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | | `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built | @@ -78,7 +78,7 @@ future GitLab releases.** | `CI_PIPELINE_ID` | 8.10 | all | The unique id of the current pipeline that GitLab CI uses internally | | `CI_PIPELINE_IID` | 11.0 | all | The unique id of the current pipeline scoped to project | | `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | -| `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered] | +| `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered](../triggers/README.md) | | `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | | `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | | `CI_PROJECT_ID` | all | all | The unique id of the current project that GitLab CI uses internally | diff --git a/doc/customization/system_header_and_footer_messages.md b/doc/customization/system_header_and_footer_messages.md index 9d6931c730d..7eee79abc77 100644 --- a/doc/customization/system_header_and_footer_messages.md +++ b/doc/customization/system_header_and_footer_messages.md @@ -1,6 +1,7 @@ # Adding a system message to every page -> [Introduced][ee-1283] in [GitLab Premium][eep] 10.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5023) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.7. +> [Added](https://gitlab.com/gitlab-org/gitlab-ce/issues/55057) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. Navigate to the **Admin** area and go to the **Appearance** page. diff --git a/doc/development/README.md b/doc/development/README.md index af3207671e6..5df6ec5fd56 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -108,6 +108,7 @@ description: 'Learn how to contribute to GitLab.' - [Database Debugging and Troubleshooting](database_debugging.md) - [Query Count Limits](query_count_limits.md) - [Database helper modules](database_helpers.md) +- [Code comments](code_comments.md) ## Integration guides diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index f7f36901fe1..aeddad14995 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -198,9 +198,9 @@ abilities as in the Rails app. If the: - Currently authenticated user fails the authorization, the authorized -resource will be returned as `null`. + resource will be returned as `null`. - Resource is part of a collection, the collection will be filtered to -exclude the objects that the user's authorization checks failed against. + exclude the objects that the user's authorization checks failed against. TIP: **Tip:** Try to load only what the currently authenticated user is allowed to @@ -447,7 +447,7 @@ want to validate the abilities for. Alternatively, we can add a `find_object` method that will load the object on the mutation. This would allow you to use the -`authorized_find!` and `authorized_find!` helper methods. +`authorized_find!` helper method. When a user is not allowed to perform the action, or an object is not found, we should raise a @@ -496,4 +496,4 @@ it 'returns a successful response' do expect(response).to have_gitlab_http_status(:success) expect(graphql_mutation_response(:merge_request_set_wip)['errors']).to be_empty end -```
\ No newline at end of file +``` diff --git a/doc/development/architecture.md b/doc/development/architecture.md index d14a760f972..ee6d00331e3 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -40,7 +40,7 @@ A complete architecture diagram is available in our  -<!-- +<!-- To update this diagram, GitLab team members can edit this source file: https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit. --> @@ -93,8 +93,8 @@ graph TB Prometheus --> Alertmanager Migrations --> PostgreSQL Runner -- TCP 443 --> NGINX - Unicorn -- TCP 9200 --> ElasticSearch - Sidekiq -- TCP 9200 --> ElasticSearch + Unicorn -- TCP 9200 --> Elasticsearch + Sidekiq -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 80, 443 --> Sentry Unicorn -- TCP 80, 443 --> Sentry Sidekiq -- UDP 6831 --> Jaeger @@ -116,10 +116,10 @@ graph TB ### Component legend -* ✅ - Installed by default -* ⚙ - Requires additional configuration, or GitLab Managed Apps -* ⤓ - Manual installation required -* ❌ - Not supported or no instructions available +- ✅ - Installed by default +- ⚙ - Requires additional configuration, or GitLab Managed Apps +- ⤓ - Manual installation required +- ❌ - Not supported or no instructions available Component statuses are linked to configuration documentation for each component. @@ -158,7 +158,7 @@ Component statuses are linked to configuration documentation for each component. | [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | [⤓][gitlab-yml] | [⤓][ldap-gdk] | CE & EE | | [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | | [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | -| [ElasticSearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | [⤓][elasticsearch-source] | [⤓][elasticsearch-gdk] | EE Only | +| [Elasticsearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | [⤓][elasticsearch-source] | [⤓][elasticsearch-gdk] | EE Only | | [Sentry integration](#sentry) | Error tracking for deployed apps | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | CE & EE | | [Jaeger integration](#jaeger) | Distributed tracing for deployed apps | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | EE Only | | [GitLab Managed Apps](#gitlab-managed-apps) | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | CE & EE | @@ -304,7 +304,7 @@ GitLab is comprised of a large number of services that all log. We started bundl - Configuration: [Omnibus][mattermost-omnibus], [Charts][mattermost-charts] - Layer: Core Service (Processor) -Mattermost is an open source, private cloud, Slack-alternative from https://mattermost.com. +Mattermost is an open source, private cloud, Slack-alternative from <https://mattermost.com>. #### MinIO @@ -321,7 +321,7 @@ MinIO is an object storage server released under Apache License v2.0. It is comp - Layer: Core Service (Processor) - Process: `nginx` -Nginx as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. +Nginx as an ingress port for all HTTP requests and routes them to the appropriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. #### Node Exporter diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index c63ec53414c..a7b402c3fb0 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -1,12 +1,13 @@ # Chatops on GitLab.com -Chatops on GitLab.com allows GitLabbers to run various automation tasks on GitLab.com using Slack. +ChatOps on GitLab.com allows GitLab team members to run various automation tasks on GitLab.com using Slack. ## Requesting access -GitLabbers may need access to Chatops on GitLab.com for administration tasks such as: +GitLab team-members may need access to Chatops on GitLab.com for administration +tasks such as: -- Configuring feature flags on staging. +- Configuring feature flags. - Running `EXPLAIN` queries against the GitLab.com production replica. To request access to Chatops on GitLab.com: @@ -18,4 +19,4 @@ To request access to Chatops on GitLab.com: - [Chatops Usage](https://docs.gitlab.com/ee/ci/chatops/README.html) - [Understanding EXPLAIN plans](understanding_explain_plans.md) - - [Feature Groups](feature_flags.md#feature-groups) + - [Feature Groups](feature_flags/development.md#feature-groups) diff --git a/doc/development/code_comments.md b/doc/development/code_comments.md new file mode 100644 index 00000000000..36962eb46d4 --- /dev/null +++ b/doc/development/code_comments.md @@ -0,0 +1,14 @@ +# Code comments + +Whenever you add comment to the code that is expected to be addressed at any time +in future, please create a technical debt issue for it. Then put a link to it +to the code comment you've created. This will allow other developers to quickly +check if a comment is still relevant and what needs to be done to address it. + +Examples: + +```rb +# Deprecated scope until code_owner column has been migrated to rule_type. +# To be removed with https://gitlab.com/gitlab-org/gitlab-ee/issues/11834. +scope :code_owner, -> { where(code_owner: true).or(where(rule_type: :code_owner)) } +``` diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 29e2aa1a581..6123f9f845a 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -60,6 +60,8 @@ from teams other than your own. 1. If your merge request includes backend changes [^1], it must be **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_backend)**. + 1. If your merge request includes database migrations or changes to expensive queries [^2], it must be + **approved by a [database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_database)**. 1. If your merge request includes frontend changes [^1], it must be **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_frontend)**. 1. If your merge request includes UX changes [^1], it must be @@ -377,3 +379,4 @@ Largely based on the [thoughtbot code review guide]. [team]: https://about.gitlab.com/team/ [build handbook]: https://about.gitlab.com/handbook/build/handbook/build#how-to-work-with-build [^1]: Please note that specs other than JavaScript specs are considered backend code. +[^2]: We encourage you to seek guidance from a database maintainer if your merge request is potentially introducing expensive queries. It is most efficient to comment on the line of code in question with the SQL queries so they can give their advice. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 3d36a7bf3b1..f4fa0caeb01 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -6,10 +6,13 @@ scheduling into milestones. Labelling is a task for everyone. Most issues will have labels for at least one of the following: -- Type: ~feature, ~bug, ~customer, etc. -- Subject: ~wiki, ~"Container Registry", ~ldap, ~api, ~frontend, etc. -- Team: ~Plan, ~Manage, ~Quality, etc. -- Stage: ~"devops:plan", ~"devops:create", etc. +- Type: ~feature, ~bug, ~backstage, etc. +- Subject: ~wiki, ~"Container Registry", ~ldap, ~api, etc. +- Team: ~Documentation, ~Delivery, etc. +- Stage: ~"devops::plan", ~"devops::create", etc. +- Group: ~"group::source code" ~"group::knowledge" ~"group::editor", etc. +- Department: ~UX, ~Quality +- Specialization: ~frontend, ~backend - Release Scoping: ~Deliverable, ~Stretch, ~"Next Patch Release" - Priority: ~P1, ~P2, ~P3, ~P4 - Severity: ~S1, ~S2, ~S3, ~S4 @@ -27,8 +30,7 @@ labels, you can _always_ add the team and type, and often also the subject. Type labels are very important. They define what kind of issue this is. Every issue should have one or more. -Examples of type labels are ~feature, ~bug, ~customer, ~security, -and ~direction. +Examples of type labels are ~feature, ~bug, ~backstage and ~security A number of type labels have a priority assigned to them, which automatically makes them float to the top, depending on their importance. @@ -43,6 +45,10 @@ The descriptions on the [labels page][labels-page] explain what falls under each Subject labels are labels that define what area or feature of GitLab this issue hits. They are not always necessary, but very convenient. +Subject labels are now used to infer and apply relevant group and devops stage +labels. Please apply them whenever possible to facilitate accurate matching. +Please refer to [this merge request][inferred-labels] for more information. + Examples of subject labels are ~wiki, ~ldap, ~api, ~issues, ~"merge requests", ~labels, and ~"Container Registry". @@ -52,31 +58,37 @@ issue is labeled with a subject label corresponding to your expertise. Subject labels are always all-lowercase. -## Team labels +## Team labels + +**Important**: Most of the team labels will be soon deprecated in favor of [Group labels](#group-labels). Team labels specify what team is responsible for this issue. Assigning a team label makes sure issues get the attention of the appropriate people. -The current team labels are: +The team labels planned for deprecation are: - ~Configure - ~Create - ~Defend - ~Distribution -- ~Documentation +- ~Ecosystem - ~Geo - ~Gitaly - ~Growth - ~Manage +- ~Memory - ~Monitor - ~Plan -- ~Quality - ~Release - ~Secure -- ~UX - ~Verify +The following team labels are **true** teams per our [organization structure](https://about.gitlab.com/company/team/structure/#organizational-structure) which will remain post deprecation. + +- ~Delivery +- ~Documentation + The descriptions on the [labels page][labels-page] explain what falls under the responsibility of each team. @@ -86,6 +98,7 @@ indicate if an issue needs backend work, frontend work, or both. Team labels are always capitalized so that they show up as the first label for any issue. + ## Stage labels Stage labels specify which [DevOps stage][devops-stages] the issue belongs to. @@ -126,10 +139,44 @@ The Stage labels are used to generate the [direction pages][direction-pages] aut Group labels specify which [groups][structure-groups] the issue belongs to. -Examples include: - -- ~"group::control" -- ~"group::editor" +The current group labels are: + +* ~"group::access" +* ~"group::measure" +* ~"group::source code" +* ~"group::knowledge" +* ~"group::editor" +* ~"group::gitaly" +* ~"group::gitter" +* ~"group::team planning" +* ~"group::enterprise planning" +* ~"group::certify" +* ~"group::ci and runner" +* ~"group::testing" +* ~"group::package" +* ~"group::progressive delivery" +* ~"group::release management" +* ~"group::autodevops and kubernetes" +* ~"group::serverless and paas" +* ~"group::apm" +* ~"group::health" +* ~"group::static analysis" +* ~"group::dynamic analysis" +* ~"group::software composition analysis" +* ~"group::runtime application security" +* ~"group::threat management" +* ~"group::application infrastructure security" +* ~"group::activation" +* ~"group::adoption" +* ~"group::upsell" +* ~"group::retention" +* ~"group::fulfillment" +* ~"group::telemetry" +* ~"group::distribution" +* ~"group::geo" +* ~"group::memory" +* ~"group::ecosystem" + These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) and thus are mutually exclusive. @@ -141,6 +188,20 @@ can be applied to a single issue. You can find the groups listed in the [structure-groups]: https://about.gitlab.com/company/team/structure/#groups [product-categories]: https://about.gitlab.com/handbook/product/categories/ +## Department labels + +The current department labels are: + +* ~UX +* ~Quality + +## Specialization labels + +These labels narrow the [specialization](https://about.gitlab.com/company/team/structure/#specialist) on a unit of work. + +* ~frontend +* ~backend + ## Release Scoping labels Release Scoping labels help us clearly communicate expectations of the work for the @@ -186,20 +247,20 @@ There can be multiple facets of the impact. The below is a guideline. If a bug seems to fall between two severity labels, assign it to the higher-severity label. -* Example(s) of ~S1 - * Data corruption/loss. - * Security breach. - * Unable to create an issue or merge request. - * Unable to add a comment or discussion to the issue or merge request. -* Example(s) of ~S2 - * Cannot submit changes through the web IDE but the commandline works. - * A status widget on the merge request page is not working but information can be seen in the test pipeline page. -* Example(s) of ~S3 - * Can create merge requests only from the Merge Requests list view, not from an Issue page. - * Status is not updated in real time and needs a page refresh. -* Example(s) of ~S4 - * Label colors are incorrect. - * UI elements are not fully aligned. +- Example(s) of ~S1 + - Data corruption/loss. + - Security breach. + - Unable to create an issue or merge request. + - Unable to add a comment or discussion to the issue or merge request. +- Example(s) of ~S2 + - Cannot submit changes through the web IDE but the commandline works. + - A status widget on the merge request page is not working but information can be seen in the test pipeline page. +- Example(s) of ~S3 + - Can create merge requests only from the Merge Requests list view, not from an Issue page. + - Status is not updated in real time and needs a page refresh. +- Example(s) of ~S4 + - Label colors are incorrect. + - UI elements are not fully aligned. ## Label for community contributors @@ -442,3 +503,4 @@ A recent example of this was the issue for [labels-page]: https://gitlab.com/gitlab-org/gitlab-ce/labels [ce-tracker]: https://gitlab.com/gitlab-org/gitlab-ce/issues [ee-tracker]: https://gitlab.com/gitlab-org/gitlab-ee/issues +[inferred-labels]: https://gitlab.com/gitlab-org/quality/triage-ops/merge_requests/155 diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 8a4aa5dfa7f..6064f59ed10 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -126,16 +126,16 @@ When writing commit messages, please follow the guidelines below: - The commit subject must contain at least 3 words. - The commit subject should ideally contain up to 50 characters, -and must not be longer than 72 characters. + and must not be longer than 72 characters. - The commit subject must start with a capital letter. - The commit subject must not end with a period. - The commit subject and body must be separated by a blank line. - The commit body must not contain more than 72 characters per line. - Commits that change 30 or more lines across at least 3 files must -describe these changes in the commit body. + describe these changes in the commit body. - The commit subject or body must not contain Emojis. - Use issues and merge requests' full URLs instead of short references, -as they are displayed as plain text outside of GitLab. + as they are displayed as plain text outside of GitLab. - The merge request must not contain more than 10 commit messages. If the guidelines are not met, the MR will not pass the diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index f319d00d7fe..87e61a7476f 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -31,8 +31,8 @@ This is also the style used by linting tools such as [Return to Contributing documentation](index.md) -[rss-source]: https://github.com/bbatsov/ruby-style-guide/blob/master/README.md#source-code-layout -[rss-naming]: https://github.com/bbatsov/ruby-style-guide/blob/master/README.md#naming +[rss-source]: https://github.com/rubocop-hq/ruby-style-guide/blob/master/README.adoc#source-code-layout +[rss-naming]: https://github.com/rubocop-hq/ruby-style-guide/blob/master/README.adoc#naming-conventions [doc-guidelines]: ../documentation/index.md "Documentation guidelines" [js-styleguide]: ../fe_guide/style_guide_js.md "JavaScript styleguide" [scss-styleguide]: ../fe_guide/style_guide_scss.md "SCSS styleguide" diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 68d33a9d8e0..de2c5b43411 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -85,3 +85,21 @@ eric 37709 0.0 0.0 2518640 7524 s006 S Wed11AM 0:00.79 s $ kill 87304 $ kill 37709 ``` + +### db:migrate `database version is too old to be migrated` error + +Users receive this error when `db:migrate` detects that the current schema version +is older than the `MIN_SCHEMA_VERSION` defined in the `Gitlab::Database` library +module. + +Over time we cleanup/combine old migrations in the codebase, so it is not always +possible to migrate GitLab from every previous version. + +In some cases you may want to bypass this check. For example, if you were on a version +of GitLab schema later than the `MIN_SCHEMA_VERSION`, and then rolled back the +to an older migration, from before. In this case, in order to migrate forward again, +you should set the `SKIP_SCHEMA_VERSION_CHECK` environment variable. + +```sh +bundle exec rake db:migrate SKIP_SCHEMA_VERSION_CHECK=true +``` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index c7fa40af930..418e58b22d5 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -76,10 +76,10 @@ After a given documentation path is aligned across CE and EE, all merge requests affecting that path must be submitted to CE, regardless of the content it has. This means that: -* For **EE-only docs changes**, you only have to submit a CE MR. -* For **EE-only features** that touch both the code and the docs, you have to submit -an EE MR containing all changes, and a CE MR containing only the docs changes -and without a changelog entry. +- For **EE-only docs changes**, you only have to submit a CE MR. +- For **EE-only features** that touch both the code and the docs, you have to submit + an EE MR containing all changes, and a CE MR containing only the docs changes + and without a changelog entry. This might seem like a duplicate effort, but it's only for the short term. A list of the already aligned docs can be found in diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index a441ede6e42..20eeebf444f 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -4,8 +4,9 @@ description: "Learn how GitLab docs' global navigation works and how to add new # Global navigation -> [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/362) -in November 2018 for GitLab 11.6. +> - [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/362) +in GitLab 11.6. +> - [Updated](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/482) in GitLab 12.1. The global nav adds to the left sidebar the ability to navigate and explore the contents of GitLab's documentation. @@ -217,13 +218,13 @@ and the following syntax rules. - Always use relative paths against the home of CE and EE. Examples: - For `https://docs.gitlab.com/ee/README.html`, the relative URL is `README.html`. - For `https://docs.gitlab.com/ee/user/project/cycle_analytics.html`, the relative - URL is `user/project/cycle_analytics.html` + URL is `user/project/cycle_analytics.html`. - For `README.html` files, add the complete path `path/to/README.html`. - For `index.html` files, use the clean (canonical) URL: `path/to/`. - For EE-only docs, use the same relative path, but add the attribute `ee_only: true` below - the `doc_url` or `category_url`, as explained above. This will guarantee that when - the user is looking at the CE docs, it will link to the EE docs. It also displays - an "info" icon on the CE nav to make the user aware that it's a different link. + the `doc_url` or `category_url`, as explained above. This displays + an "info" icon on the nav to make the user aware that the feature is + EE-only. DANGER: **Important!** All links present on the data file must end in `.html`, not `.md`. Do not @@ -293,7 +294,7 @@ point to `/ee/` docs. On the other hand, if the user is looking at `/ce/` docs, all the links in the CE nav should link internally to `/ce/` -files, except for [`ee-only` docs](#ee-only-docs). +files. ```html <% if dir != 'ce' %> @@ -314,21 +315,12 @@ categories (`cat[:category_url]`), and docs (`doc[:doc_url]`) URLs. #### `ee-only` docs -If the user is looking at the CE nav, a given doc is present only -in `/ee/`, it's tagged in the data file by `ee-only`, linking it -directly to `/ee/`. +Docs for features present only in GitLab EE are tagged +in the data file by `ee-only` and an icon is displayed on the nav +link indicating that the `ee-only` feature is not available in CE. -```html -<% if dir == 'ce' && cat[:ee_only] %> - <a href="/ee/<%= cat[:category_url] %>">...</a> -<% end %> -``` - -To make it clear that it it's a different link, an icon is displayed -on the nav link indicating that the `ee-only` doc is not available in CE. - -The `ee-only` attribute is available for `categories` (`<% if dir == 'ce' && cat[:ee_only] %>`) -and `docs` (`<% if dir == 'ce' && doc[:ee_only] %>`), but not for `sections`. +The `ee-only` attribute is available for `categories` (`<% if cat[:ee_only] %>`) +and `docs` (`<% if doc[:ee_only] %>`), but not for `sections`. ### CSS classes diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index ee3a9caf9a0..6dd12b5efa7 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -11,8 +11,40 @@ and deploy it to <https://docs.gitlab.com>. While the source of the documentation content is stored in GitLab's respective product repositories, the source that is used to build the documentation site _from that content_ -is located at <https://gitlab.com/gitlab-com/gitlab-docs>. See the README there for -detailed information. +is located at <https://gitlab.com/gitlab-com/gitlab-docs>. + +The following diagram illustrates the relationship between the repositories +from where content is sourced, the `gitlab-docs` project, and the published output. + +```mermaid + graph LR + A[gitlab-ce/doc] + B[gitlab-ee/doc] + C[gitlab-runner/docs] + D[omnibus-gitlab/doc] + E[charts/doc] + F[gitlab-docs] + A --> F + B --> F + C --> F + D --> F + E --> F + F -- Build pipeline --> G + G[docs.gitlab.com] + H[/ce/] + I[/ee/] + J[/runner/] + K[/omnibus/] + L[/charts/] + G --> H + G --> I + G --> J + G --> K + G --> L +``` + +See the [README there](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md) +for detailed information. ## Assets @@ -22,9 +54,9 @@ the GitLab Documentation website. ### Libraries -- [Bootstrap 3.3 components](https://getbootstrap.com/docs/3.3/components/) -- [Bootstrap 3.3 JS](https://getbootstrap.com/docs/3.3/javascript/) -- [jQuery](https://jquery.com/) 3.2.1 +- [Bootstrap 4.3.1 components](https://getbootstrap.com/docs/4.3/components/) +- [Bootstrap 4.3.1 JS](https://getbootstrap.com/docs/4.3/getting-started/javascript/) +- [jQuery](https://jquery.com/) 3.3.1 - [Clipboard JS](https://clipboardjs.com/) - [Font Awesome 4.7.0](https://fontawesome.com/v4.7.0/icons/) diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index b8506a72666..6bfedcb1047 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -76,8 +76,8 @@ and cross-link between any related content. We employ a **docs-first methodology** to help ensure that the docs remain a complete and trusted resource, and to make communicating about the use of GitLab more efficient. -* If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. -* When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request to add this information to the docs. You can then share the MR in order to communicate this information. +- If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. +- When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request to add this information to the docs. You can then share the MR in order to communicate this information. New information that would be useful toward the future usage or troubleshooting of GitLab should not be written directly in a forum or other messaging system, but added to a docs MR and then referenced, as described above. Note that among any other doc changes, you can always add a Troubleshooting section to a doc if none exists, or un-comment and use the placeholder Troubleshooting section included as part of our [doc template](structure.md#template-for-new-docs), if present. @@ -100,6 +100,13 @@ use regular Markdown markup, following the rules in the linked style guide. Note that Kramdown-specific markup (e.g., `{:.class}`) will not render properly on GitLab instances under [`/help`](index.md#gitlab-help). +Hard-coded HTML is valid, although it's discouraged to be used while we have `/help`. HTML is permitted as long as: + +- There's no equivalent markup in markdown. +- Advanced tables are necessary. +- Special styling is required. +- Reviewed and approved by a technical writer. + ## Structure ### Organize by topic, not by type @@ -143,7 +150,8 @@ The table below shows what kind of documentation goes where. a proper naming would be `import_projects_from_github.md`. The same rule applies to images. 1. For image files, do not exceed 100KB. -1. We do not yet support embedded videos. Please link out. +1. Do not upload video files to the product repositories. + [Link or embed videos](#videos) instead. 1. There are four main directories, `user`, `administration`, `api` and `development`. 1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, `profile/`, `dashboard/` and `admin_area/`. @@ -165,13 +173,13 @@ The table below shows what kind of documentation goes where. `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. General user- and admin- related documentation, should be placed accordingly. 1. The directories `/workflow/`, `/university/`, and `/articles/` have -been **deprecated** and the majority their docs have been moved to their correct location -in small iterations. + been **deprecated** and the majority their docs have been moved to their correct location + in small iterations. If you are unsure where a document or a content addition should live, this should not stop you from authoring and contributing. You can use your best judgment and then ask the reviewer of your MR to confirm your decision, and/or ask a technical writer -at any stage in the process. The techncial writing team will review all documentation +at any stage in the process. The technical writing team will review all documentation changes, regardless, and can move content if there is a better place for it. ### Avoid duplication @@ -207,6 +215,7 @@ Do not include the same information in multiple places. [Link to a SSOT instead. ## Text +- [Write in markdown](#markdown). - Splitting long lines (preferably up to 100 characters) can make it easier to provide feedback on small chunks of text. - Insert an empty line for new paragraphs. - Use sentence case for titles, headings, labels, menu items, and buttons. @@ -409,11 +418,20 @@ To indicate the steps of navigation through the UI: ## Images - Place images in a separate directory named `img/` in the same directory where - the `.md` document that you're working on is located. Always prepend their - names with the name of the document that they will be included in. For - example, if there is a document called `twitter.md`, then a valid image name - could be `twitter_login_screen.png`. -- Images should have a specific, non-generic name that will differentiate and describe them properly. + the `.md` document that you're working on is located. +- Images should have a specific, non-generic name that will + differentiate and describe them properly. +- Always add to the end of the file name the GitLab release version + number corresponding to the release milestone the image was added to, + or corresponding to the release the screenshot was taken from, using the + format `image_name_vX_Y.png`. + ([Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/61027) in GitLab 12.1.) +- For example, for a screenshot taken from the pipelines page of + GitLab 11.1, a valid name is `pipelines_v11_1.png`. If you're + adding an illustration that does not include parts of the UI, + add the release number corresponding to the release the image + was added to. Example, for an MR added to 11.1's milestone, + a valid name for an illustration is `devops_diagram_v11_1.png`. - Keep all file names in lower case. - Consider using PNG images instead of JPEG. - Compress all images with <https://tinypng.com/> or similar tool. @@ -421,12 +439,12 @@ To indicate the steps of navigation through the UI: - Images should be used (only when necessary) to _illustrate_ the description of a process, not to _replace_ it. - Max image size: 100KB (gifs included). -- The GitLab docs do not support videos yet. +- See also how to link and embed [videos](#videos) to illustrate the docs. Inside the document: - The Markdown way of using an image inside a document is: - `` + `` - Always use a proper description for what the image is about. That way, when a browser fails to show the image, this text will be used as an alternative description. @@ -446,6 +464,85 @@ directly to an HTML `img` tag: <img src="path/to/image.jpg" alt="Alt text (required)" class="image-noshadow"> ``` +## Videos + +Adding GitLab's existing YouTube video tutorials to the documentation is +highly encouraged, unless the video is outdated. Videos should not +replace documentation, but complement or illustrate it. If content in a video is +fundamental to a feature and its key use cases, but this is not adequately covered in the documentation, +add this detail to the documentation text or create an issue to review the video and do so. + +Do not upload videos to the product repositories. [Link](#link-to-video) or [embed](#embed-videos) them instead. + +### Link to video + +To link out to a video, include a YouTube icon so that readers can +quickly and easily scan the page for videos before reading: + +```md +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Video Title](link-to-video). +``` + +You can link any up-to-date video that is useful to the GitLab user. + +### Embed videos + +> [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/472) in GitLab 12.1. + +GitLab docs (docs.gitlab.com) support embedded videos. + +You can only embed videos from +[GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). +For videos from other sources, [link](#link-to-video) them instead. + +In most cases, it is better to [link to video](#link-to-video) instead, +because an embed takes up a lot of space on the page and can be distracting +to readers. + +To embed a video, follow the instructions below and make sure +you have your MR reviewed and approved by a technical writer. + +1. Copy the code below and paste it into your markdown file. + Leave a blank line above and below it. Do NOT edit the code + (don't remove or add any spaces, etc). +1. On YouTube, visit the video URL you want to display. Copy + the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) + and replace the video title and link in the line under `<div class="video-fallback">`. +1. On YouTube, click **Share**, then **Embed**. +1. Copy the `<iframe>` source (`src`) **URL only** + (`https://www.youtube.com/embed/VIDEO-ID`), + and paste it, replacing the content of the `src` field in the + `iframe` tag. + +```html +leave a blank line here +<div class="video-fallback"> + See the video: [Video title](https://www.youtube.com/watch?v=MqL6BMOySIQ). +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> +leave a blank line here +``` + +This is how it renders on docs.gitlab.com: + +<div class="video-fallback"> + See the video: [What is GitLab](https://www.youtube.com/watch?v=enMumwvLAug). +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + +> Notes: +> +> - The `figure` tag is required for semantic SEO and the `video_container` +class is necessary to make sure the video is responsive and displays +nicely on different mobile devices. +> - The `<div class="video-fallback">` is a fallback necessary for GitLab's +`/help`, as GitLab's markdown processor does not support iframes. It's hidden on the docs site but will be displayed on GitLab's `/help`. + ## Code blocks - Always wrap code added to a sentence in inline code blocks (``` ` ```). @@ -480,7 +577,7 @@ directly to an HTML `img` tag: ## Alert boxes -Whenever you want to call the attention to a particular sentence, +Whenever you need to call special attention to particular sentences, use the following markup for highlighting. _Note that the alert boxes only work for one paragraph only. Multiple paragraphs, @@ -488,6 +585,23 @@ lists, headers, etc will not render correctly. For multiple lines, use blockquot ### Note +Notes catch the eye of most readers, and therefore should be used very sparingly. +In most cases, content considered for a note should be included: + +- As just another sentence in the previous paragraph or the most-relevant paragraph. +- As its own standalone paragraph. +- As content under a new subheading that introduces the topic, making it more visible/findable. + +#### When to use + +Use a note when there is a reason that most or all readers who browse the +section should see the content. That is, if missed, it’s likely to cause +major trouble for a minority of users or significant trouble for a majority +of users. + +Weigh the costs of distracting users to whom the content is not relevant against +the cost of users missing the content if it were not expressed as a note. + ```md NOTE: **Note:** This is something to note. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index cca52706ddc..6f4a36d4066 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -909,11 +909,12 @@ import bundle from 'ee_else_ce/protected_branches/protected_branches_bundle.js'; See the frontend guide [performance section](fe_guide/performance.md) for information on managing page-specific javascript within EE. - ## Vue code in `assets/javascript` + ### script tag #### Child Component only used in EE + To separate Vue template differences we should [async import the components](https://vuejs.org/v2/guide/components-dynamic-async.html#Async-Components). Doing this allows for us to load the correct component in EE whilst in CE @@ -937,10 +938,12 @@ export default { ``` #### For JS code that is EE only, like props, computed properties, methods, etc, we will keep the current approach - - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. + +- Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin ##### Example: + ```javascript import mixin from 'ee_else_ce/path/mixin'; @@ -955,18 +958,20 @@ import mixin from 'ee_else_ce/path/mixin'; - You can see an MR with an example [here](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9762) #### `template` tag -* **EE Child components** + +- **EE Child components** - Since we are using the async loading to check which component to load, we'd still use the component's name, check [this example](#child-component-only-used-in-ee). -* **EE extra HTML** +- **EE extra HTML** - For the templates that have extra HTML in EE we should move it into a new component and use the `ee_else_ce` dynamic import ### Non Vue Files + For regular JS files, the approach is similar. 1. We will keep using the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) helper, this means that EE only code should be inside the `ee/` folder. - 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. - 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: + 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. + 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: ##### Example: @@ -996,6 +1001,7 @@ to isolate such ruleset from rest of CE rules (along with adding comment describ to avoid conflicts during CE to EE merge. #### Bad + ```scss .section-body { .section-title { @@ -1011,6 +1017,7 @@ to avoid conflicts during CE to EE merge. ``` #### Good + ```scss .section-body { .section-title { @@ -1029,7 +1036,14 @@ to avoid conflicts during CE to EE merge. ### Backporting changes from EE to CE -When working in EE-specific features, you might have to tweak a few files that are not EE-specific. Here is a workflow to make sure those changes end up backported safely into CE too. +Until the work completed to merge the ce and ee codebases, which is tracked on [epic &802](https://gitlab.com/groups/gitlab-org/-/epics/802), there exists times in which some changes for EE require specific changes to the CE +code base. Examples of backports include the following: + +* Features intended or originally built for EE that are later decided to move to CE +* Sometimes some code in CE may impact the EE feature + +Here is a workflow to make sure those changes end up backported safely into CE too. + (This approach does not refer to changes introduced via [csslab](https://gitlab.com/gitlab-org/csslab/).) 1. **Make your changes in the EE branch.** If possible, keep a separated commit (to be squashed) to help backporting and review. diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index c8c70fa7216..603a756ff56 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -2,14 +2,14 @@ This area is to maintain a compendium of useful information when working with elasticsearch. -Information on how to enable ElasticSearch and perform the initial indexing is kept in ../integration/elasticsearch.md#enabling-elasticsearch +Information on how to enable Elasticsearch and perform the initial indexing is kept in ../integration/elasticsearch.md#enabling-elasticsearch ## Deep Dive -In June 2019, Mario de la Ossa hosted a [Deep Dive] on GitLab's [ElasticSearch integration] to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. +In June 2019, Mario de la Ossa hosted a [Deep Dive] on GitLab's [Elasticsearch integration] to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. [Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 -[ElasticSearch integration]: ../integration/elasticsearch.md +[Elasticsearch integration]: ../integration/elasticsearch.md [recording on YouTube]: https://www.youtube.com/watch?v=vrvl-tN2EaA [Google Slides]: https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit [PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf @@ -57,27 +57,32 @@ Additionally, if you need large repos or multiple forks for testing, please cons ## How does it work? -The ElasticSearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). +The Elasticsearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs). Search queries are generated by the concerns found in [ee/app/models/concerns/elastic](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! ## Existing Analyzers/Tokenizers/Filters -These are all defined in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb + +These are all defined in <https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb> ### Analyzers + #### `path_analyzer` + Used when indexing blobs' paths. Uses the `path_tokenizer` and the `lowercase` and `asciifolding` filters. Please see the `path_tokenizer` explanation below for an example. #### `sha_analyzer` + Used in blobs and commits. Uses the `sha_tokenizer` and the `lowercase` and `asciifolding` filters. Please see the `sha_tokenizer` explanation later below for an example. #### `code_analyzer` + Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: `code`, `edgeNGram_filter`, `lowercase`, and `asciifolding` The `whitespace` tokenizer was selected in order to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` in order to be properly searched. @@ -85,15 +90,19 @@ The `whitespace` tokenizer was selected in order to have more control over how t Please see the `code` filter for an explanation on how tokens are split. #### `code_search_analyzer` + Not directly used for indexing, but rather used to transform a search input. Uses the `whitespace` tokenizer and the `lowercase` and `asciifolding` filters. ### Tokenizers + #### `sha_tokenizer` + This is a custom tokenizer that uses the [`edgeNGram` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenizer.html) to allow SHAs to be searcheable by any sub-set of it (minimum of 5 chars). -example: +Example: `240c29dc7e` becomes: + - `240c2` - `240c29` - `240c29d` @@ -102,21 +111,26 @@ example: - `240c29dc7e` #### `path_tokenizer` + This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` in order to allow searches to find paths no matter how much or how little of the path is given as input. -example: +Example: `'/some/path/application.js'` becomes: + - `'/some/path/application.js'` - `'some/path/application.js'` - `'path/application.js'` - `'application.js'` ### Filters + #### `code` -Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. + +Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. Patterns: + - `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCased and lowedCameCased strings as separate tokens - `"(\\d+)"`: extracts digits - `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCased strings recursively. Ex: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` @@ -126,6 +140,7 @@ Patterns: - `'\/?([^\/]+)(?=\/|\b)'`: separate path terms `like/this/one` #### `edgeNGram_filter` + Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenfilter.html) to allow inputs with only parts of a token to find the token. For example it would turn `glasses` into permutations starting with `gl` and ending with `glasses`, which would allow a search for "`glass`" to find the original token `glasses` ## Gotchas @@ -140,13 +155,13 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/ You might get an error such as ``` -[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] - flood stage disk watermark [95%] exceeded on - [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], +[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] + flood stage disk watermark [95%] exceeded on + [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], all indices on this node will be marked read-only ``` -This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. +This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. In addition, the `read_only_allow_delete` setting will be set to `true`. It will block indexing, `forcemerge`, etc @@ -158,19 +173,19 @@ Add this to your `elasticsearch.yml` file: ``` # turn off the disk allocator -cluster.routing.allocation.disk.threshold_enabled: false +cluster.routing.allocation.disk.threshold_enabled: false ``` _or_ ``` # set your own limits -cluster.routing.allocation.disk.threshold_enabled: true +cluster.routing.allocation.disk.threshold_enabled: true cluster.routing.allocation.disk.watermark.flood_stage: 5gb # ES 6.x only -cluster.routing.allocation.disk.watermark.low: 15gb +cluster.routing.allocation.disk.watermark.low: 15gb cluster.routing.allocation.disk.watermark.high: 10gb ``` -Restart ElasticSearch, and the `read_only_allow_delete` will clear on it's own. +Restart Elasticsearch, and the `read_only_allow_delete` will clear on it's own. _from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator) and [6.x](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html)_ diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index df32242a522..64c793cfd64 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -5,8 +5,16 @@ [Chrome Accessibility Developer Tools][chrome-accessibility-developer-tools] are useful for testing for potential accessibility problems in GitLab. -Accessibility best-practices and more in-depth information is available on -[the Audit Rules page][audit-rules] for the Chrome Accessibility Developer Tools. +The [axe][axe-website] browser extension (available for [Firefox][axe-firefox-extension] and [Chrome][axe-chrome-extension]) is +also a handy tool for running audits and getting feedback on markup, CSS and even potentially problematic color usages. + +Accessibility best-practices and more in-depth information are available on +[the Audit Rules page][audit-rules] for the Chrome Accessibility Developer Tools. The "[awesome a11y][awesome-a11y]" list is also a +useful compilation of accessibility-related material. [chrome-accessibility-developer-tools]: https://github.com/GoogleChrome/accessibility-developer-tools [audit-rules]: https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules +[axe-website]: https://www.deque.com/axe/ +[axe-firefox-extension]: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/ +[axe-chrome-extension]: https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd +[awesome-a11y]: https://github.com/brunopulis/awesome-a11y diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index b25dce65ffe..5220c9eeea3 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -6,6 +6,7 @@ easy to maintain, and performant for the end-user. ## Rules ### Utility Classes + As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950) led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes. @@ -31,12 +32,12 @@ New utility classes should be added to [`utilities.scss`](https://gitlab.com/git #### When should I create component classes? -We recommend a "utility-first" approach. +We recommend a "utility-first" approach. 1. Start with utility classes. 2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it. -This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). +This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). Examples of component classes that were created using "utility-first" include: @@ -45,8 +46,8 @@ Examples of component classes that were created using "utility-first" include: Inspiration: -- https://tailwindcss.com/docs/utility-first -- https://tailwindcss.com/docs/extracting-components +- <https://tailwindcss.com/docs/utility-first> +- <https://tailwindcss.com/docs/extracting-components> ### Naming diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 8c6a73c6824..6c7572352ec 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -49,7 +49,9 @@ provided as a prop to the main component. Be sure to read about [page-specific JavaScript][page_specific_javascript]. ### Bootstrapping Gotchas + #### Providing data from HAML to JavaScript + While mounting a Vue application may be a need to provide data from Rails to JavaScript. To do that, provide the data through `data` attributes in the HTML element and query them while mounting the application. @@ -83,7 +85,8 @@ document.addEventListener('DOMContentLoaded', () => new Vue({ ``` #### Accessing the `gl` object -When we need to query the `gl` object for data that won't change during the application's life cyle, we should do it in the same place where we query the DOM. + +When we need to query the `gl` object for data that won't change during the application's life cycle, we should do it in the same place where we query the DOM. By following this practice, we can avoid the need to mock the `gl` object, which will make tests easier. It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component: @@ -118,12 +121,13 @@ You can read more about components in Vue.js site, [Component System][component- ### A folder for the Store #### Vuex + Check this [page](vuex.md) for more details. ### Mixing Vue and jQuery - Mixing Vue and jQuery is not recommended. -- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it][https://vuejs.org/v2/examples/select2.html]. +- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it](https://vuejs.org/v2/examples/select2.html). - It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. - It is not recommended to add new jQuery events for Vue to interact with jQuery. @@ -212,6 +216,7 @@ describe('Todos App', () => { ``` ### `mountComponent` helper + There is a helper in `spec/javascripts/helpers/vue_mount_component_helper.js` that allows you to mount a component with the given props: ```javascript @@ -225,10 +230,12 @@ const vm = mountComponent(Component, data); ``` ### Test the component's output + The main return value of a Vue component is the rendered output. In order to test the component we need to test the rendered output. [Vue][vue-test] guide's to unit test show us exactly that: ## Vue.js Expert Role + One should apply to be a Vue.js expert by opening an MR when the Merge Request's they create and review show: - Deep understanding of Vue and Vuex reactivy diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 13f0c5cc33e..6bad91d6287 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -1,127 +1 @@ -# Manage feature flags - -Starting from GitLab 9.3 we support feature flags for features in GitLab via -[Flipper](https://github.com/jnunemaker/flipper/). You should use the `Feature` -class (defined in `lib/feature.rb`) in your code to get, set and list feature -flags. - -During runtime you can set the values for the gates via the -[features API](../api/features.md) (accessible to admins only). - -## Feature groups - -Starting from GitLab 9.4 we support feature groups via -[Flipper groups](https://github.com/jnunemaker/flipper/blob/v0.10.2/docs/Gates.md#2-group). - -Feature groups must be defined statically in `lib/feature.rb` (in the -`.register_feature_groups` method), but their implementation can obviously be -dynamic (querying the DB etc.). - -Once defined in `lib/feature.rb`, you will be able to activate a -feature for a given feature group via the [`feature_group` param of the features API](../api/features.md#set-or-create-a-feature) - -For GitLab.com, [team members have access to feature flags through Chatops](chatops_on_gitlabcom.md). Only -percentage gates are supported at this time. Setting a feature to be used 50% of -the time, you should execute `/chatops run feature set my_feature_flag 50`. - -## Feature flags for user applications - -This document only covers feature flags used in the development of GitLab -itself. Feature flags in deployed user applications can be found at -[Feature Flags](../user/project/operations/feature_flags.md) - -## Developing with feature flags - -In general, it's better to have a group- or user-based gate, and you should prefer -it over the use of percentage gates. This would make debugging easier, as you -filter for example logs and errors based on actors too. Furthermore, this allows -for enabling for the `gitlab-org` group first, while the rest of the users -aren't impacted. - -```ruby -# Good -Feature.enabled?(:feature_flag, project) - -# Avoid, if possible -Feature.enabled?(:feature_flag) -``` - -To use feature gates based on actors, the model needs to respond to -`flipper_id`. For example, to enable for the Foo model: - -```ruby -class Foo < ActiveRecord::Base - include FeatureGate -end -``` - -Features that are developed and are intended to be merged behind a feature flag -should not include a changelog entry. The entry should be added in the merge -request removing the feature flags. - -In the rare case that you need the feature flag to be on automatically, use -`default_enabled: true` when checking: - -```ruby -Feature.enabled?(:feature_flag, project, default_enabled: true) -``` - -For more information about rolling out changes using feature flags, refer to the -[Rolling out changes using feature flags](rolling_out_changes_using_feature_flags.md) -guide. - -### Frontend - -For frontend code you can use the method `push_frontend_feature_flag`, which is -available to all controllers that inherit from `ApplicationController`. Using -this method you can expose the state of a feature flag as follows: - -```ruby -before_action do - push_frontend_feature_flag(:vim_bindings) -end - -def index - # ... -end - -def edit - # ... -end -``` - -You can then check for the state of the feature flag in JavaScript as follows: - -```javascript -if ( gon.features.vimBindings ) { - // ... -} -``` - -The name of the feature flag in JavaScript will always be camelCased, meaning -that checking for `gon.features.vim_bindings` would not work. - -### Specs - -In the test environment `Feature.enabled?` is stubbed to always respond to `true`, -so we make sure behavior under feature flag doesn't go untested in some non-specific -contexts. - -Whenever a feature flag is present, make sure to test _both_ states of the -feature flag. - -See the -[testing guide](testing_guide/best_practices.md#feature-flags-in-tests) -for information and examples on how to stub feature flags in tests. - -## Enabling a feature flag (in development) - -In the rails console (`rails c`), enter the following command to enable your feature flag - -```ruby -Feature.enable(:feature_flag_name) -``` - -## Enabling a feature flag (in production) - -Check how to [roll out changes using feature flags](rolling_out_changes_using_feature_flags.md). +This document was moved to [another location](feature_flags/index.md). diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md new file mode 100644 index 00000000000..c67467b7c11 --- /dev/null +++ b/doc/development/feature_flags/controls.md @@ -0,0 +1,123 @@ +# Access for enabling a feature flag in production + +In order to be able to turn on/off features behind feature flags in any of the +GitLab Inc. provided environments such as staging and production, you need to +have access to the chatops bot. Chatops bot is currently running on the ops instance, +which is different from GitLab.com or dev.gitlab.org. + +Follow the Chatops document to [request access](https://docs.gitlab.com/ee/development/chatops_on_gitlabcom.html#requesting-access). + +Once you are added to the project test if your access propagated, +run: + +``` +/chatops run feature --help +``` + +## Rolling out changes + +When the changes are deployed to the environments it is time to start +rolling out the feature to our users. The exact procedure of rolling out a +change is unspecified, as this can vary from change to change. However, in +general we recommend rolling out changes incrementally, instead of enabling them +for everybody right away. We also recommend you to _not_ enable a feature +_before_ the code is being deployed. +This allows you to separate rolling out a feature from a deploy, making it +easier to measure the impact of both separately. + +GitLab's feature library (using +[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature +Flags process](process.md) guide) supports rolling out changes to a percentage of +users. This in turn can be controlled using [GitLab chatops](../../ci/chatops/README.md). + +For an up to date list of feature flag commands please see [the source +code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). +Note that all the examples in that file must be preceded by +`/chatops run`. + +If you get an error "Whoops! This action is not allowed. This incident +will be reported." that means your Slack account is not allowed to +change feature flags or you do not [have access](#access-for-enabling-a-feature-flag-in-production). + +### Enabling feature for staging and dev.gitlab.org + +As a first step in a feature rollout, you should enable the feature on <https://staging.gitlab.com> +and <https://dev.gitlab.org>. + +For example, to enable a feature for 25% of all users, run the following in +Slack: + +``` +/chatops run feature set new_navigation_bar 25 --dev +/chatops run feature set new_navigation_bar 25 --staging +``` + +These two environments have different scopes. +`dev.gitlab.org` is a production CE environment that has internal GitLab Inc. +traffic and is used for some development and other related work. +`staging.gitlab.com` has a smaller subset of GitLab.com database and repositories +and does not have regular traffic. Staging is an EE instance and can give you +a (very) rough estimate of how your feature will look/behave on GitLab.com. +Both of these instances are connected to Sentry so make sure you check the projects +there for any exceptions while testing your feature after enabling the feature flag. + +Once you are confident enough that these environments are in a good state with your +feature enabled, you can roll out the change to GitLab.com. + +## Enabling feature for GitLab.com + +Similar to above, to enable a feature for 25% of all users, run the following in +Slack: + +``` +/chatops run feature set new_navigation_bar 25 +``` + +This will enable the feature for GitLab.com, with `new_navigation_bar` being the +name of the feature. + +If you are not certain what percentages to use, simply use the following steps: + +1. 25% +1. 50% +1. 75% +1. 100% + +Between every step you'll want to wait a little while and monitor the +appropriate graphs on <https://dashboards.gitlab.net>. The exact time to wait +may differ. For some features a few minutes is enough, while for others you may +want to wait several hours or even days. This is entirely up to you, just make +sure it is clearly communicated to your team, and the Production team if you +anticipate any potential problems. + +Feature gates can also be actor based, for example a feature could first be +enabled for only the `gitlab-ce` project. The project is passed by supplying a +`--project` flag: + +``` +/chatops run feature set --project=gitlab-org/gitlab-ce some_feature true +``` + +For groups the `--group` flag is available: + +``` +/chatops run feature set --group=gitlab-org some_feature true +``` + +## Cleaning up + +Once the change is deemed stable, submit a new merge request to remove the +feature flag. This ensures the change is available to all users and self-hosted +instances. Make sure to add the ~"feature flag" label to this merge request so +release managers are aware the changes are hidden behind a feature flag. If the +merge request has to be picked into a stable branch, make sure to also add the +appropriate "Pick into X" label (e.g. "Pick into XX.X"). +See [the process document](https://docs.gitlab.com/ee/development/feature_flags/process.html#including-a-feature-behind-feature-flag-in-the-final-release) for further details. + +When a feature gate has been removed from the code base, the value still exists +in the database. +This can be removed through ChatOps: + +``` +/chatops run feature delete some_feature +``` diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md new file mode 100644 index 00000000000..238052529d9 --- /dev/null +++ b/doc/development/feature_flags/development.md @@ -0,0 +1,131 @@ +# Developing with feature flags + +In general, it's better to have a group- or user-based gate, and you should prefer +it over the use of percentage gates. This would make debugging easier, as you +filter for example logs and errors based on actors too. Furthermore, this allows +for enabling for the `gitlab-org` or `gitlab-com` group first, while the rest of +the users aren't impacted. + +```ruby +# Good +Feature.enabled?(:feature_flag, project) + +# Avoid, if possible +Feature.enabled?(:feature_flag) +``` + +To use feature gates based on actors, the model needs to respond to +`flipper_id`. For example, to enable for the Foo model: + +```ruby +class Foo < ActiveRecord::Base + include FeatureGate +end +``` + +Features that are developed and are intended to be merged behind a feature flag +should not include a changelog entry. The entry should be added in the merge +request removing the feature flags. + +In the rare case that you need the feature flag to be on automatically, use +`default_enabled: true` when checking: + +```ruby +Feature.enabled?(:feature_flag, project, default_enabled: true) +``` + +The [`Project#feature_available?`][project-fa], +[`Namespace#feature_available?`][namespace-fa] (EE), and +[`License.feature_available?`][license-fa] (EE) methods all implicitly check for +a feature flag by the same name as the provided argument. + +For example if a feature is license-gated, there's no need to add an additional +explicit feature flag check since the flag will be checked as part of the +`License.feature_available?` call. Similarly, there's no need to "clean up" a +feature flag once the feature has reached general availability. + +You'd still want to use an explicit `Feature.enabled?` check if your new feature +isn't gated by a License or Plan. + +[project-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/app/models/project_feature.rb#L63-68 +[namespace-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/ee/namespace.rb#L71-85 +[license-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/license.rb#L293-300 + +An important side-effect of the implicit feature flags mentioned above is that +unless the feature is explicitly disabled or limited to a percentage of users, +the feature flag check will default to `true`. + +As an example, if you were to ship the backend half of a feature behind a flag, +you'd want to explicitly disable that flag until the frontend half is also ready +to be shipped. [You can do this via Chatops](https://docs.gitlab.com/ee/development/feature_flags/controls.html): + +``` +/chatops run feature set some_feature 0 +``` + +Note that you can do this at any time, even before the merge request using the +flag has been merged! + +## Feature groups + +Starting from GitLab 9.4 we support feature groups via +[Flipper groups](https://github.com/jnunemaker/flipper/blob/v0.10.2/docs/Gates.md#2-group). + +Feature groups must be defined statically in `lib/feature.rb` (in the +`.register_feature_groups` method), but their implementation can obviously be +dynamic (querying the DB etc.). + +Once defined in `lib/feature.rb`, you will be able to activate a +feature for a given feature group via the [`feature_group` param of the features API](../../api/features.md#set-or-create-a-feature) + +### Frontend + +For frontend code you can use the method `push_frontend_feature_flag`, which is +available to all controllers that inherit from `ApplicationController`. Using +this method you can expose the state of a feature flag as follows: + +```ruby +before_action do + push_frontend_feature_flag(:vim_bindings) +end + +def index + # ... +end + +def edit + # ... +end +``` + +You can then check for the state of the feature flag in JavaScript as follows: + +```javascript +if ( gon.features.vimBindings ) { + // ... +} +``` + +The name of the feature flag in JavaScript will always be camelCased, meaning +that checking for `gon.features.vim_bindings` would not work. + +### Specs + +In the test environment `Feature.enabled?` is stubbed to always respond to `true`, +so we make sure behavior under feature flag doesn't go untested in some non-specific +contexts. + +Whenever a feature flag is present, make sure to test _both_ states of the +feature flag. + +See the +[testing guide](../testing_guide/best_practices.md#feature-flags-in-tests) +for information and examples on how to stub feature flags in tests. + +### Enabling a feature flag (in development) + +In the rails console (`rails c`), enter the following command to enable your feature flag + +```ruby +Feature.enable(:feature_flag_name) +``` diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md new file mode 100644 index 00000000000..56872f8c075 --- /dev/null +++ b/doc/development/feature_flags/index.md @@ -0,0 +1,12 @@ +# Feature flags in development of GitLab + +Feature flags can be used to gradually roll out changes, be +it a new feature, or a performance improvement. By using feature flags, we can +comfortably measure the impact of our changes, while still being able to easily +disable those changes, without having to revert an entire release. + +Before using feature flags for GitLab's development, read through the following: + +- [Process for using features flags](process.md). +- [Developing with feature flags documentation](development.md). +- [Controlling feature flags documentation](controls.md). diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md new file mode 100644 index 00000000000..28d6080ce87 --- /dev/null +++ b/doc/development/feature_flags/process.md @@ -0,0 +1,131 @@ +# Feature flags process + +## Feature flags for user applications + +This document only covers feature flags used in the development of GitLab +itself. Feature flags in deployed user applications can be found at +[Feature Flags feature documentation](../../user/project/operations/feature_flags.md). + +## Feature flags in GitLab development + +The following highlights should be considered when deciding if feature flags +should be leveraged: + +- By default, the feature flags should be **off**. +- Feature flags should remain in the codebase for as short period as possible + to reduce the need for feature flag accounting. +- The person operating with feature flags is responsible for clearly communicating + the status of a feature behind the feature flag with responsible stakeholders. +- Merge requests that make changes hidden behind a feature flag, or remove an + existing feature flag because a feature is deemed stable must have the + ~"feature flag" label assigned. + +One might be tempted to think that feature flags will delay the release of a +feature by at least one month (= one release). This is not the case. A feature +flag does not have to stick around for a specific amount of time +(e.g. at least one release), instead they should stick around until the feature +is deemed stable. Stable means it works on GitLab.com without causing any +problems, such as outages. + +### When to use feature flags + +Starting with GitLab 11.4, developers are required to use feature flags for +non-trivial changes. Such changes include: + +- New features (e.g. a new merge request widget, epics, etc). +- Complex performance improvements that may require additional testing in + production, such as rewriting complex queries. +- Invasive changes to the user interface, such as a new navigation bar or the + removal of a sidebar. +- Adding support for importing projects from a third-party service. + +In all cases, those working on the changes can best decide if a feature flag is +necessary. For example, changing the color of a button doesn't need a feature +flag, while changing the navigation bar definitely needs one. In case you are +uncertain if a feature flag is necessary, simply ask about this in the merge +request, and those reviewing the changes will likely provide you with an answer. + +When using a feature flag for UI elements, make sure to _also_ use a feature +flag for the underlying backend code, if there is any. This ensures there is +absolutely no way to use the feature until it is enabled. + +### Including a feature behind feature flag in the final release + +In order to build a final release and present the feature for self-hosted +users, the feature flag should be at least defaulted to **on**. If the feature +is deemed stable and there is confidence that removing the feature flag is safe, +consider removing the feature flag altogether. Take into consideration that such +action can make the feature available on GitLab.com shortly after the change to +the feature flag is merged. + +Changing the default state or removing the feature flag has to be done before +the 22nd of the month, _at least_ 2 working days before, in order for the change +to be included in the final self-managed release. + +In addition to this, the feature behind feature flag should: + +- Run in all GitLab.com environments for a sufficient period of time. This time + period depends on the feature behind the feature flag, but as a general rule of + thumb 2-4 working days should be sufficient to gather enough feedback. +- The feature should be exposed to all users within the GitLab.com plan during + the above mentioned period of time. Exposing the feature to a smaller percentage + or only a group of users might not expose a sufficient amount of information to aid in + making a decision on feature stability. + +While rare, release managers may decide to reject picking or revert a change in +a stable branch, even when feature flags are used. This might be necessary if +the changes are deemed problematic, too invasive, or there simply isn't enough +time to properly measure how the changes behave on GitLab.com. + +### The cost of feature flags + +When reading the above, one might be tempted to think this procedure is going to +add a lot of work. Fortunately, this is not the case, and we'll show why. For +this example we'll specify the cost of the work to do as a number, ranging from +0 to infinity. The greater the number, the more expensive the work is. The cost +does _not_ translate to time, it's just a way of measuring complexity of one +change relative to another. + +Let's say we are building a new feature, and we have determined that the cost of +this is 10. We have also determined that the cost of adding a feature flag check +in a variety of places is 1. If we do not use feature flags, and our feature +works as intended, our total cost is 10. This however is the best case scenario. +Optimizing for the best case scenario is guaranteed to lead to trouble, whereas +optimizing for the worst case scenario is almost always better. + +To illustrate this, let's say our feature causes an outage, and there's no +immediate way to resolve it. This means we'd have to take the following steps to +resolve the outage: + +1. Revert the release. +1. Perform any cleanups that might be necessary, depending on the changes that + were made. +1. Revert the commit, ensuring the "master" branch remains stable. This is + especially necessary if solving the problem can take days or even weeks. +1. Pick the revert commit into the appropriate stable branches, ensuring we + don't block any future releases until the problem is resolved. + +As history has shown, these steps are time consuming, complex, often involve +many developers, and worst of all: our users will have a bad experience using +GitLab.com until the problem is resolved. + +Now let's say that all of this has an associated cost of 10. This means that in +the worst case scenario, which we should optimize for, our total cost is now 20. + +If we had used a feature flag, things would have been very different. We don't +need to revert a release, and because feature flags are disabled by default we +don't need to revert and pick any Git commits. In fact, all we have to do is +disable the feature, and in the worst case, perform cleanup. Let's say that +the cost of this is 2. In this case, our best case cost is 11: 10 to build the +feature, and 1 to add the feature flag. The worst case cost is now 13: 10 to +build the feature, 1 to add the feature flag, and 2 to disable and clean up. + +Here we can see that in the best case scenario the work necessary is only a tiny +bit more compared to not using a feature flag. Meanwhile, the process of +reverting our changes has been made significantly and reliably cheaper. + +In other words, feature flags do not slow down the development process. Instead, +they speed up the process as managing incidents now becomes _much_ easier. Once +continuous deployments are easier to perform, the time to iterate on a feature +is reduced even further, as you no longer need to wait weeks before your changes +are available on GitLab.com. diff --git a/doc/development/geo.md b/doc/development/geo.md index 6e59fab34c7..a10f13b069f 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -14,10 +14,10 @@ Geo handles replication for different components: - [Database](#database-replication): includes the entire application, except cache and jobs. - [Git repositories](#repository-replication): includes both projects and wikis. - [Uploaded blobs](#uploads-replication): includes anything from images attached on issues -to raw logs and assets from CI. + to raw logs and assets from CI. With the exception of the Database replication, on a *secondary* node, everything is coordinated -by the [Geo Log Cursor](#geo-log-cursor). +by the [Geo Log Cursor](#geo-log-cursor). ### Geo Log Cursor daemon @@ -31,8 +31,8 @@ picks the event up and schedules a `Geo::ProjectSyncWorker` job which will use the `Geo::RepositorySyncService` and `Geo::WikiSyncService` classes to update the repository and the wiki respectively. -The Geo Log Cursor daemon can operate in High Availability mode automatically. -The daemon will try to acquire a lock from time to time and once acquired, it +The Geo Log Cursor daemon can operate in High Availability mode automatically. +The daemon will try to acquire a lock from time to time and once acquired, it will behave as the *active* daemon. Any additional running daemons on the same node, will be in standby @@ -164,20 +164,20 @@ The Git Push Proxy exists as a functionality built inside the `gitlab-shell` com It is active on a **secondary** node only. It allows the user that has cloned a repository from the secondary node to push to the same URL. -Git `push` requests directed to a **secondary** node will be sent over to the **primary** node, +Git `push` requests directed to a **secondary** node will be sent over to the **primary** node, while `pull` requests will continue to be served by the **secondary** node for maximum efficiency. HTTPS and SSH requests are handled differently: - With HTTPS, we will give the user a `HTTP 302 Redirect` pointing to the project on the **primary** node. -The git client is wise enough to understand that status code and process the redirection. + The git client is wise enough to understand that status code and process the redirection. - With SSH, because there is no equivalent way to perform a redirect, we have to proxy the request. -This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request -to the HTTP protocol, and then proxying it to the **primary** node. + This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request + to the HTTP protocol, and then proxying it to the **primary** node. -The [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) daemon knows when to proxy based on the response -from `/api/v4/allowed`. A special `HTTP 300` status code is returned and we execute a "custom action", -specified in the response body. The response contains additional data that allows the proxied `push` operation +The [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) daemon knows when to proxy based on the response +from `/api/v4/allowed`. A special `HTTP 300` status code is returned and we execute a "custom action", +specified in the response body. The response contains additional data that allows the proxied `push` operation to happen on the **primary** node. ## Using the Tracking Database @@ -229,17 +229,17 @@ named `gitlab_secondary`. This configuration exists within the database's user context only. To access the `gitlab_secondary`, GitLab needs to use the same database user that had previously been configured. -The Geo Tracking Database accesses the readonly database replica via FDW as a regular user, -limited by its own restrictions. The credentials are configured as a -`USER MAPPING` associated with the `SERVER` mapped previously +The Geo Tracking Database accesses the readonly database replica via FDW as a regular user, +limited by its own restrictions. The credentials are configured as a +`USER MAPPING` associated with the `SERVER` mapped previously (`gitlab_secondary`). FDW configuration and credentials definition are managed automatically by the -Omnibus GitLab `gitlab-ctl reconfigure` command. +Omnibus GitLab `gitlab-ctl reconfigure` command. #### Refeshing the Foreign Tables -Whenever a new Geo node is configured or the database schema changes on the +Whenever a new Geo node is configured or the database schema changes on the **primary** node, you must refresh the foreign tables on the **secondary** node by running the following: @@ -279,11 +279,11 @@ on the Tracking Database: SELECT project_registry.* FROM project_registry JOIN gitlab_secondary.projects - ON (project_registry.project_id = gitlab_secondary.projects.id + ON (project_registry.project_id = gitlab_secondary.projects.id AND gitlab_secondary.projects.archived IS FALSE) ``` -At the ActiveRecord level, we have additional Models that represent the +At the ActiveRecord level, we have additional Models that represent the foreign tables. They must be mapped in a slightly different way, and they are read-only. Check the existing FDW models in `ee/app/models/geo/fdw` for reference. diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index b512d7611d3..c103a4527ff 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -113,7 +113,7 @@ are as follows: (`pool.source_project`) > TODO Fix invalid SQL data for pools created prior to GitLab 11.11 -> https://gitlab.com/gitlab-org/gitaly/issues/1653. +> <https://gitlab.com/gitlab-org/gitaly/issues/1653>. ### Assumptions @@ -157,7 +157,7 @@ are as follows: repository. > TODO should forks of forks be deduplicated? -> https://gitlab.com/gitlab-org/gitaly/issues/1532 +> <https://gitlab.com/gitlab-org/gitaly/issues/1532> ### Consequences @@ -215,4 +215,4 @@ the secondary, at which stage Git objects will get deduplicated. > TODO How do we handle the edge case where at the time the Geo > secondary tries to create the pool repository, the source project does -> not exist? https://gitlab.com/gitlab-org/gitaly/issues/1533 +> not exist? <https://gitlab.com/gitlab-org/gitaly/issues/1533> diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index a0585fed2fc..5552d5d37b4 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -88,12 +88,12 @@ Until GitLab has eliminated most of these inefficiencies or the use of NFS is discontinued for Git data, Rugged implementations of some of the most commonly-used RPCs can be enabled via feature flags: -* `rugged_find_commit` -* `rugged_get_tree_entries` -* `rugged_tree_entry` -* `rugged_commit_is_ancestor` -* `rugged_commit_tree_entry` -* `rugged_list_commits_by_oid` +- `rugged_find_commit` +- `rugged_get_tree_entries` +- `rugged_tree_entry` +- `rugged_commit_is_ancestor` +- `rugged_commit_tree_entry` +- `rugged_list_commits_by_oid` A convenience Rake task can be used to enable or disable these flags all together. To enable: diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 4dad8815fcb..f961a2fc837 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -82,7 +82,7 @@ go lint: image: golang:1.11 script: - go get -u golang.org/x/lint/golint - - golint -set_exit_status + - golint -set_exit_status $(go list ./... | grep -v "vendor/") ``` Once [recursive includes](https://gitlab.com/gitlab-org/gitlab-ce/issues/56836) diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index ce310672dad..17462887162 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -135,7 +135,7 @@ There is also and alternative method to [translate messages from validation erro ### Interpolation Placeholders in translated text should match the code style of the respective source file. -For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. +For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make sure to [avoid splitting sentences when adding links](#avoid-splitting-sentences-when-adding-links). - In Ruby/HAML: @@ -267,20 +267,41 @@ should be externalized as follows: This also applies when using links in between translated sentences, otherwise these texts are not translatable in certain languages. -Instead of: +- In Ruby/HAML, instead of: + + ```haml + - zones_link = link_to(s_('ClusterIntegration|zones'), 'https://cloud.google.com/compute/docs/regions-zones/regions-zones', target: '_blank', rel: 'noopener noreferrer') + = s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } + ``` + + Set the link starting and ending HTML fragments as variables like so: + + ```haml + - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' + - zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } + = s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } + ``` -```haml -- zones_link = link_to(s_('ClusterIntegration|zones'), 'https://cloud.google.com/compute/docs/regions-zones/regions-zones', target: '_blank', rel: 'noopener noreferrer') -= s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } -``` +- In JavaScript, instead of: -Set the link starting and ending HTML fragments as variables like so: + ```js + {{ + sprintf(s__("ClusterIntegration|Learn more about %{link}"), { + link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' + }) + }} + ``` -```haml -- zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' -- zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } -= s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } -``` + Set the link starting and ending HTML fragments as variables like so: + + ```js + {{ + sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { + linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">' + linkEnd: '</a>', + }) + }} + ``` The reasoning behind this is that in some languages words change depending on context. For example in Japanese は is added to the subject of a sentence and を to the object. This is impossible to translate correctly if we extract individual words from the sentence. diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index 4564f678ec0..879b54bd93c 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -10,16 +10,16 @@ yarn clean ## Creating feature flags in development -The process for creating a feature flag is the same as [enabling a feature flag in development](../feature_flags.md#enabling-a-feature-flag-in-development). +The process for creating a feature flag is the same as [enabling a feature flag in development](../feature_flags/development.md#enabling-a-feature-flag-in-development). Your feature flag can now be: -- [made available to the frontend](../feature_flags.md#frontend) via the `gon` -- queried in [tests](../feature_flags.md#specs) -- queried in HAML templates and ruby files via the `Feature.enabled?(:my_shiny_new_feature_flag)` method +- [Made available to the frontend](../feature_flags/development.md#frontend) via the `gon` +- Queried in [tests](../feature_flags/development.md#specs) +- Queried in HAML templates and ruby files via the `Feature.enabled?(:my_shiny_new_feature_flag)` method ### More on feature flags - [Deleting a feature flag](../../api/features.md#delete-a-feature) -- [Manage feature flags](../feature_flags.md) +- [Manage feature flags](../feature_flags/process.md) - [Feature flags API](../../api/features.md) diff --git a/doc/development/performance.md b/doc/development/performance.md index 0e21d45f57c..c034f4a344b 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -424,7 +424,7 @@ might find using these gems more convenient: ### Examples You may find some useful examples in this snippet: -https://gitlab.com/gitlab-org/gitlab-ce/snippets/33946 +<https://gitlab.com/gitlab-org/gitlab-ce/snippets/33946> [#15607]: https://gitlab.com/gitlab-org/gitlab-ce/issues/15607 [yorickpeterse]: https://gitlab.com/yorickpeterse diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index de5e1323e6a..17d8428b0c0 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -73,7 +73,7 @@ Similar to source browsing, is [Documentation browsing](https://github.com/pry/p ### Command history -With <kdb>Ctrl+R</kbd> you can search your [command history](https://github.com/pry/pry/wiki/History). +With **Ctrl+R** you can search your [command history](https://github.com/pry/pry/wiki/History). ## Stepping diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 01cedf734fb..59da02ed6fd 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -127,6 +127,42 @@ Everything in `lib/api`. Everything that resides in `app/services`. +#### ServiceResponse + +Service classes usually have an `execute` method, which can return a +`ServiceResponse`. You can use `ServiceResponse.success` and +`ServiceResponse.error` to return a response in `execute` method. + +In a successful case: + +``` ruby +response = ServiceResponse.success(message: 'Branch was deleted') + +response.success? # => true +response.error? # => false +response.status # => :success +response.message # => 'Branch was deleted' +``` + +In a failed case: + +``` ruby +response = ServiceResponse.error(message: 'Unsupported operation') + +response.success? # => false +response.error? # => true +response.status # => :error +response.message # => 'Unsupported operation' +``` + +An additional payload can also be attached: + +``` ruby +response = ServiceResponse.success(payload: { issue: issue }) + +response.payload[:issue] # => issue +``` + ### Finders Everything in `app/finders`, typically used for retrieving data from a database. diff --git a/doc/development/rolling_out_changes_using_feature_flags.md b/doc/development/rolling_out_changes_using_feature_flags.md index 84028b1b342..6bad91d6287 100644 --- a/doc/development/rolling_out_changes_using_feature_flags.md +++ b/doc/development/rolling_out_changes_using_feature_flags.md @@ -1,225 +1 @@ -# Rolling out changes using feature flags - -[Feature flags](feature_flags.md) can be used to gradually roll out changes, be -it a new feature, or a performance improvement. By using feature flags, we can -comfortably measure the impact of our changes, while still being able to easily -disable those changes, without having to revert an entire release. - -## When to use feature flags - -Starting with GitLab 11.4, developers are required to use feature flags for -non-trivial changes. Such changes include: - -- New features (e.g. a new merge request widget, epics, etc). -- Complex performance improvements that may require additional testing in - production, such as rewriting complex queries. -- Invasive changes to the user interface, such as a new navigation bar or the - removal of a sidebar. -- Adding support for importing projects from a third-party service. - -In all cases, those working on the changes can best decide if a feature flag is -necessary. For example, changing the color of a button doesn't need a feature -flag, while changing the navigation bar definitely needs one. In case you are -uncertain if a feature flag is necessary, simply ask about this in the merge -request, and those reviewing the changes will likely provide you with an answer. - -When using a feature flag for UI elements, make sure to _also_ use a feature -flag for the underlying backend code, if there is any. This ensures there is -absolutely no way to use the feature until it is enabled. - -## The cost of feature flags - -When reading the above, one might be tempted to think this procedure is going to -add a lot of work. Fortunately, this is not the case, and we'll show why. For -this example we'll specify the cost of the work to do as a number, ranging from -0 to infinity. The greater the number, the more expensive the work is. The cost -does _not_ translate to time, it's just a way of measuring complexity of one -change relative to another. - -Let's say we are building a new feature, and we have determined that the cost of -this is 10. We have also determined that the cost of adding a feature flag check -in a variety of places is 1. If we do not use feature flags, and our feature -works as intended, our total cost is 10. This however is the best case scenario. -Optimising for the best case scenario is guaranteed to lead to trouble, whereas -optimising for the worst case scenario is almost always better. - -To illustrate this, let's say our feature causes an outage, and there's no -immediate way to resolve it. This means we'd have to take the following steps to -resolve the outage: - -1. Revert the release. -1. Perform any cleanups that might be necessary, depending on the changes that - were made. -1. Revert the commit, ensuring the "master" branch remains stable. This is - especially necessary if solving the problem can take days or even weeks. -1. Pick the revert commit into the appropriate stable branches, ensuring we - don't block any future releases until the problem is resolved. - -As history has shown, these steps are time consuming, complex, often involve -many developers, and worst of all: our users will have a bad experience using -GitLab.com until the problem is resolved. - -Now let's say that all of this has an associated cost of 10. This means that in -the worst case scenario, which we should optimise for, our total cost is now 20. - -If we had used a feature flag, things would have been very different. We don't -need to revert a release, and because feature flags are disabled by default we -don't need to revert and pick any Git commits. In fact, all we have to do is -disable the feature, and in the worst case, perform cleanup. Let's say that -the cost of this is 2. In this case, our best case cost is 11: 10 to build the -feature, and 1 to add the feature flag. The worst case cost is now 13: 10 to -build the feature, 1 to add the feature flag, and 2 to disable and clean up. - -Here we can see that in the best case scenario the work necessary is only a tiny -bit more compared to not using a feature flag. Meanwhile, the process of -reverting our changes has been made significantly and reliably cheaper. - -In other words, feature flags do not slow down the development process. Instead, -they speed up the process as managing incidents now becomes _much_ easier. Once -continuous deployments are easier to perform, the time to iterate on a feature -is reduced even further, as you no longer need to wait weeks before your changes -are available on GitLab.com. - -## Rolling out changes - -The procedure of using feature flags is straightforward, and similar to not -using them. You add the necessary tests (make sure to test both the on and off -states of your feature flag(s)), make sure they all pass, have the code -reviewed, etc. You then submit your merge request, and add the ~"feature flag" -label. This label is used to signal to release managers that your changes are -hidden behind a feature flag and that it is safe to pick the MR into a stable -branch, without the need for an exception request. - -When the changes are deployed it is time to start rolling out the feature to our -users. The exact procedure of rolling out a change is unspecified, as this can -vary from change to change. However, in general we recommend rolling out changes -incrementally, instead of enabling them for everybody right away. We also -recommend you to _not_ enable a feature _before_ the code is being deployed. -This allows you to separate rolling out a feature from a deploy, making it -easier to measure the impact of both separately. - -GitLab's feature library (using -[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags](feature_flags.md) guide) supports rolling out changes to a percentage of -users. This in turn can be controlled using [GitLab -chatops](../ci/chatops/README.md). - -For an up to date list of feature flag commands please see [the source -code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). -Note that all the examples in that file must be preceded by -`/chatops run`. - -If you get an error "Whoops! This action is not allowed. This incident -will be reported." that means your Slack account is not allowed to -change feature flags. To test if you are allowed to do anything at all, -run: - -``` -/chatops run feature --help -``` - -For example, to enable a feature for 25% of all users, run the following in -Slack: - -``` -/chatops run feature set new_navigation_bar 25 -``` - -This will enable the feature for GitLab.com, with `new_navigation_bar` being the -name of the feature. We can also enable the feature for <https://dev.gitlab.org> -or <https://staging.gitlab.com>: - -``` -/chatops run feature set new_navigation_bar 25 --dev -/chatops run feature set new_navigation_bar 25 --staging -``` - -If you are not certain what percentages to use, simply use the following steps: - -1. 25% -1. 50% -1. 75% -1. 100% - -Between every step you'll want to wait a little while and monitor the -appropriate graphs on <https://dashboards.gitlab.net>. The exact time to wait -may differ. For some features a few minutes is enough, while for others you may -want to wait several hours or even days. This is entirely up to you, just make -sure it is clearly communicated to your team, and the Production team if you -anticipate any potential problems. - -Feature gates can also be actor based, for example a feature could first be -enabled for only the `gitlab-ce` project. The project is passed by supplying a -`--project` flag: - -``` -/chatops run feature set --project=gitlab-org/gitlab-ce some_feature true -``` - -For groups the `--group` flag is available: - -``` -/chatops run feature set --group=gitlab-org some_feature true -``` - -Once a change is deemed stable, submit a new merge request to remove the -feature flag. This ensures the change is available to all users and self-hosted -instances. Make sure to add the ~"feature flag" label to this merge request so -release managers are aware the changes are hidden behind a feature flag. If the -merge request has to be picked into a stable branch (e.g. after the 7th), make -sure to also add the appropriate "Pick into X" label (e.g. "Pick into 11.4"). - -One might be tempted to think this will delay the release of a feature by at -least one month (= one release). This is not the case. A feature flag does not -have to stick around for a specific amount of time (e.g. at least one release), -instead they should stick around until the feature is deemed stable. Stable -means it works on GitLab.com without causing any problems, such as outages. In -most cases this will translate to a feature (with a feature flag) being shipped -in RC1, followed by the feature flag being removed in RC2. This in turn means -the feature will be stable by the time we publish a stable package around the -22nd of the month. - -## Implicit feature flags - -The [`Project#feature_available?`][project-fa], -[`Namespace#feature_available?`][namespace-fa] (EE), and -[`License.feature_available?`][license-fa] (EE) methods all implicitly check for -a feature flag by the same name as the provided argument. - -For example if a feature is license-gated, there's no need to add an additional -explicit feature flag check since the flag will be checked as part of the -`License.feature_available?` call. Similarly, there's no need to "clean up" a -feature flag once the feature has reached general availability. - -You'd still want to use an explicit `Feature.enabled?` check if your new feature -isn't gated by a License or Plan. - -[project-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/app/models/project_feature.rb#L63-68 -[namespace-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/ee/namespace.rb#L71-85 -[license-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/license.rb#L293-300 - -### Undefined feature flags default to "on" - -An important side-effect of the [implicit feature flags](#implicit-feature-flags) -mentioned above is that unless the feature is explicitly disabled or limited to a -percentage of users, the feature flag check will default to `true`. - -As an example, if you were to ship the backend half of a feature behind a flag, -you'd want to explicitly disable that flag until the frontend half is also ready -to be shipped. You can do this via ChatOps: - -``` -/chatops run feature set some_feature 0 -``` - -Note that you can do this at any time, even before the merge request using the -flag has been merged! - -### Cleaning up - -When a feature gate has been removed from the code base, the value still exists -in the database. This can be removed through ChatOps: - -``` -/chatops run feature delete some_feature -``` +This document was moved to [another location](feature_flags/index.md). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 71e3b7740cb..448d9fd01c4 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -327,7 +327,7 @@ However, if a spec makes direct Redis calls, it should mark itself with the `:clean_gitlab_redis_queues` traits as appropriate. Sidekiq jobs are typically not run in specs, but this behaviour can be altered -in each spec through the use of `Sidekiq::Testing.inline!` blocks. Any spec that +in each spec through the use of `perform_enqueued_jobs` blocks. Any spec that causes Sidekiq jobs to be pushed to Redis should use the `:sidekiq` trait, to ensure that they are removed once the spec completes. diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index f7b3ca8bc89..aec0a3ede5a 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -5,8 +5,8 @@ We devised a solution to solve common test automation problems such as the dread Other problems that dynamic element validations solve are... - When we perform an action with the mouse, we expect something to occur. -- When our test is navigating to (or from) a page, we ensure that we are on the page we expect before -test continuation. +- When our test is navigating to (or from) a page, we ensure that we are on the page we expect before + test continuation. ## How it works @@ -19,7 +19,7 @@ We interpret user actions on the page to have some sort of effect. These actions When a page is navigated to, there are elements that will always appear on the page unconditionally. -Dynamic element validation is instituted when using +Dynamic element validation is instituted when using ```ruby Runtime::Browser.visit(:gitlab, Some::Page) @@ -27,7 +27,7 @@ Runtime::Browser.visit(:gitlab, Some::Page) ### Clicks -When we perform a click within our tests, we expect something to occur. That something could be a component to now +When we perform a click within our tests, we expect something to occur. That something could be a component to now appear on the webpage, or the test to navigate away from the page entirely. Dynamic element validation is instituted when using @@ -71,7 +71,7 @@ class MyPage < Page::Base element :another_element, required: true element :conditional_element end - + def open_layer click_element :my_element, Layer::MyLayer end @@ -95,7 +95,7 @@ execute_stuff ``` will invoke GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to -`execute_stuff` +`execute_stuff` ### Clicking diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 73e1fd862c1..05cb03eb4bd 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -82,7 +82,7 @@ module Page end # ... - end + end end end ``` @@ -134,7 +134,7 @@ for each element defined. In our case, `qa-login-field`, `qa-password-field` and `qa-sign-in-button` -**app/views/my/view.html.haml** +**app/views/my/view.html.haml** ```haml = f.text_field :login, class: "form-control top qa-login-field", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required." @@ -146,7 +146,7 @@ Things to note: - The CSS class must be `kebab-cased` (separated with hyphens "`-`") - If the element appears on the page unconditionally, add `required: true` to the element. See -[Dynamic element validation](dynamic_element_validation.md) + [Dynamic element validation](dynamic_element_validation.md) ## Running the test locally diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md index f96c85be1ba..041bdf716b3 100644 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -242,7 +242,7 @@ module QA issue = Resource::Issue.fabricate_via_api! do |issue| issue.title = 'Issue to test the scoped labels' - issue.labels = @initial_label + issue.labels = [@initial_label] end [@new_label_same_scope, @new_label_different_scope].each do |label| @@ -365,6 +365,14 @@ Add the following `attribute :id` and `attribute :labels` right above the [`attr > We add the attributes above the existing attribute to keep them alphabetically organized. +Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: + +```ruby +def initialize + @labels = [] +end +``` + Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. ```ruby @@ -378,7 +386,7 @@ end def api_post_body { - labels: [labels], + labels: labels, title: title } end diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 4c9d1684c00..28ebb6f0f64 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -27,14 +27,30 @@ we need to solve before being able to use Jest for all our needs. ### Differences to Karma - Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). -- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. +- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. +- Jest does not have access to Webpack loaders or aliases. + The aliases used by Jest are defined in its [own config](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/jest.config.js). - All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks). - `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/en/manual-mocks). +- No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. + This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. + Instead you should declare shared variables in the context that they are needed (via `const` / `let`). - The following will cause tests to fail in Jest: - Unmocked requests. - Unhandled Promise rejections. - Calls to `console.warn`, including warnings from libraries like Vue. +### Limitations of jsdom + +As mentioned [above](#differences-to-karma), Jest uses jsdom instead of a browser for running tests. +This comes with a number of limitations, namely: + +- [No scrolling support](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/browser/Window.js#L623-L625) +- [No element sizes or positions](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/living/nodes/Element-impl.js#L334-L371) +- [No layout engine](https://github.com/jsdom/jsdom/issues/1322) in general + +See also the issue for [support running Jest tests in browsers](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). + ### Debugging Jest tests Running `yarn jest-debug` will run Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 63b7b97c32f..ae40d628717 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -257,7 +257,7 @@ find a way to limit it to only us.** ## Other resources -* [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) +- [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) [charts-1068]: https://gitlab.com/charts/gitlab/issues/1068 [gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/44362587 diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index 1cf883679d7..b7e6844f43a 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -29,7 +29,9 @@ git clone PASTE HTTPS OR SSH HERE A clone of the project will be created in your computer. ->**Note:** If you clone your project via a URL that contains special characters, make sure that characters are URL-encoded. +NOTE: **Note:** +If you clone your project via a URL that contains special characters, make sure +that characters are URL-encoded. ### Go into a project directory to work in it @@ -86,12 +88,18 @@ cat README.md ### Remove a file +DANGER: **Danger:** +This will permanently delete the file. + ``` rm NAME-OF-FILE ``` ### Remove a directory and all of its contents +DANGER: **Danger:** +This will permanently delete the directory and all of its contents. + ``` rm -r NAME-OF-DIRECTORY ``` @@ -113,9 +121,13 @@ history You will be asked for an administrator’s password. ``` -sudo +sudo COMMAND ``` +CAUTION: **Caution:** +Be careful of the commands you run with `sudo`. Certain commands may cause +damage to your data and system. + ### Show which directory I am in ``` diff --git a/doc/install/installation.md b/doc/install/installation.md index 4c1021d097f..70e5ab28931 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -560,7 +560,7 @@ NOTE: **Note:** If you want to use HTTPS, see [Using HTTPS](#using-https) for the additional steps. NOTE: **Note:** -Make sure your hostname can be resolved on the machine itself by either a proper DNS record or an additional line in `/etc/hosts` ("127.0.0.1 hostname"). This might be necessary, for example, if you set up GitLab behind a reverse proxy. If the hostname cannot be resolved, the final installation check will fail with "Check GitLab API access: FAILED. code: 401" and pushing commits will be rejected with "[remote rejected] master -> master (hook declined)". +Make sure your hostname can be resolved on the machine itself by either a proper DNS record or an additional line in `/etc/hosts` ("127.0.0.1 hostname"). This might be necessary, for example, if you set up GitLab behind a reverse proxy. If the hostname cannot be resolved, the final installation check will fail with `Check GitLab API access: FAILED. code: 401` and pushing commits will be rejected with `[remote rejected] master -> master (hook declined)`. NOTE: **Note:** GitLab Shell application startup time can be greatly reduced by disabling RubyGems. This can be done in several ways: diff --git a/doc/install/kubernetes/gitlab_chart.md b/doc/install/kubernetes/gitlab_chart.md index 43655767002..d067c341be8 100644 --- a/doc/install/kubernetes/gitlab_chart.md +++ b/doc/install/kubernetes/gitlab_chart.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/ +redirect_to: 'https://docs.gitlab.com/charts/' --- This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index 43655767002..d067c341be8 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/ +redirect_to: 'https://docs.gitlab.com/charts/' --- This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index 08ccf2cf9ad..be58c957166 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/runner/install/kubernetes.html +redirect_to: 'https://docs.gitlab.com/runner/install/kubernetes.html' --- This document was moved to [another location](https://docs.gitlab.com/runner/install/kubernetes.html). diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index 43655767002..d067c341be8 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/ +redirect_to: 'https://docs.gitlab.com/charts/' --- This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/preparation/connect.md b/doc/install/kubernetes/preparation/connect.md index db55e03d3d4..839461c982c 100644 --- a/doc/install/kubernetes/preparation/connect.md +++ b/doc/install/kubernetes/preparation/connect.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/cloud/ +redirect_to: 'https://docs.gitlab.com/charts/installation/cloud/' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/). diff --git a/doc/install/kubernetes/preparation/eks.md b/doc/install/kubernetes/preparation/eks.md index 975d35c11c6..c3f53c2f580 100644 --- a/doc/install/kubernetes/preparation/eks.md +++ b/doc/install/kubernetes/preparation/eks.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/cloud/eks.html +redirect_to: 'https://docs.gitlab.com/charts/installation/cloud/eks.html' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/eks.html). diff --git a/doc/install/kubernetes/preparation/networking.md b/doc/install/kubernetes/preparation/networking.md index 2af16a752dc..7e88bbd3cd1 100644 --- a/doc/install/kubernetes/preparation/networking.md +++ b/doc/install/kubernetes/preparation/networking.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns +redirect_to: 'https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns). diff --git a/doc/install/kubernetes/preparation/rbac.md b/doc/install/kubernetes/preparation/rbac.md index f94e7c24cdc..fc18b91641c 100644 --- a/doc/install/kubernetes/preparation/rbac.md +++ b/doc/install/kubernetes/preparation/rbac.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#rbac +redirect_to: 'https://docs.gitlab.com/charts/installation/deployment.html#rbac' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#rbac). diff --git a/doc/install/kubernetes/preparation/tiller.md b/doc/install/kubernetes/preparation/tiller.md index 66d6c8faece..c1c7910703e 100644 --- a/doc/install/kubernetes/preparation/tiller.md +++ b/doc/install/kubernetes/preparation/tiller.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/tools.html +redirect_to: 'https://docs.gitlab.com/charts/installation/tools.html' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/kubernetes/preparation/tools_installation.md b/doc/install/kubernetes/preparation/tools_installation.md index 66d6c8faece..c1c7910703e 100644 --- a/doc/install/kubernetes/preparation/tools_installation.md +++ b/doc/install/kubernetes/preparation/tools_installation.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/tools.html +redirect_to: 'https://docs.gitlab.com/charts/installation/tools.html' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 18981c43464..45d07ec5d11 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -145,12 +145,12 @@ Login successful. You have access to the following projects and can switch between them with 'oc project <projectname>': - * cockpit - * default (current) - * delete - * openshift - * openshift-infra - * sample +- cockpit +- default (current) +- delete +- openshift +- openshift-infra +- sample Using project "default". ``` diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 92122fca7dd..68c1bcbc801 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -9,7 +9,7 @@ as the hardware requirements that are needed to install and use GitLab. ## Operating Systems -### Supported Unix distributions +### Supported Linux distributions - Ubuntu - Debian @@ -21,7 +21,7 @@ as the hardware requirements that are needed to install and use GitLab. For the installations options, see [the main installation page](README.md). -### Unsupported Unix distributions +### Unsupported Linux distributions and Unix-like operating systems - Arch Linux - Fedora @@ -29,13 +29,13 @@ For the installations options, see [the main installation page](README.md). - Gentoo - macOS -On the above unsupported distributions is still possible to install GitLab yourself. +Installation of GitLab on these operating systems is possible, but not supported. Please see the [installation from source guide](installation.md) and the [installation guides](https://about.gitlab.com/installation/) for more information. -### Non-Unix operating systems such as Windows +### Microsoft Windows -GitLab is developed for Unix operating systems. -It does **not** run on Windows, and we have no plans to support it in the near future. For the latest development status view this [issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46567). +GitLab is developed for Linux-based operating systems. +It does **not** run on Microsoft Windows, and we have no plans to support it in the near future. For the latest development status view this [issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46567). Please consider using a virtual machine to run GitLab. ## Ruby versions diff --git a/doc/integration/README.md b/doc/integration/README.md index f74da97119a..135952a1b08 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -13,10 +13,10 @@ See the documentation below for details on how to configure these services. - [Auth0 OmniAuth](auth0.md) Enable the Auth0 OmniAuth provider - [Bitbucket](bitbucket.md) Import projects from Bitbucket.org and login to your GitLab instance with your Bitbucket.org account - [CAS](cas.md) Configure GitLab to sign in using CAS -- [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. +- [External issue tracker](external-issue-tracker.md) Redmine, Jira, etc. - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages - [Jenkins](jenkins.md) Integrate with the Jenkins CI -- [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Jira](../user/project/integrations/jira.md) Integrate with the Jira issue tracker - [Kerberos](kerberos.md) Integrate with Kerberos - [LDAP](ldap.md) Set up sign in via LDAP - [OAuth2 provider](oauth_provider.md) OAuth2 application creation diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index a2f38a2fcdf..877330b8c44 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -130,7 +130,7 @@ The following Elasticsearch settings are available: | `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). | | `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). | | `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | -| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., `http://host1, https://host2:9200`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | | `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) | | `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. | | `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects). @@ -156,7 +156,7 @@ If no namespaces or projects are selected, no Elasticsearch indexing will take p CAUTION: **Warning**: If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data for filtering to work correctly. To do this run the rake tasks `gitlab:elastic:create_empty_index` and -`gitlab:elastic:clear_index_status` Afterwards, removing a namespace or a projeect from the list will delete the data +`gitlab:elastic:clear_index_status`. Afterwards, removing a namespace or a project from the list will delete the data from the Elasticsearch index as expected. ## Disabling Elasticsearch @@ -167,7 +167,7 @@ To disable the Elasticsearch integration: 1. Find the 'Elasticsearch' section and uncheck 'Search with Elasticsearch enabled' and 'Elasticsearch indexing' 1. Click **Save** for the changes to take effect -1. [Optional] Delete the existing index by running the command `sudo gitlab-rake gitlab:elastic:delete_index` +1. (Optional) Delete the existing index by running the command `sudo gitlab-rake gitlab:elastic:delete_index` ## Adding GitLab's data to the Elasticsearch index @@ -304,7 +304,7 @@ For Elasticsearch 6.x, before proceeding with the force merge, the index should ```bash curl --request PUT localhost:9200/gitlab-production/_settings --data '{ "settings": { - "index.blocks.write": true + "index.blocks.write": true } }' ``` @@ -319,7 +319,7 @@ After this, if your index is in read-only, switch back to read-write: ```bash curl --request PUT localhost:9200/gitlab-production/_settings --data '{ "settings": { - "index.blocks.write": false + "index.blocks.write": false } }' ``` @@ -392,9 +392,9 @@ When performing a search, the GitLab index will use the following scopes: Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index. -In general, we recommend simply letting Elasticseach merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ +In general, we recommend simply letting Elasticsearch merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ -However, some larger installations may wish to tune the merge policy settings: +However, some larger installations may wish to tune the merge policy settings: - Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently. @@ -425,14 +425,14 @@ Here are some common pitfalls and how to overcome them: - **How can I verify my GitLab instance is using Elasticsearch?** The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: - + ```ruby u = User.find_by_username('your-username') s = SearchService.new(u, {:search => 'search_term'}) pp s.search_objects.class.name ``` - - If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. + + If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. - **I updated GitLab and now I can't find anything** @@ -443,23 +443,23 @@ Here are some common pitfalls and how to overcome them: - **I indexed all the repositories but I can't find anything** Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). - + Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. - + If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): - + ```ruby u = User.find_by_username('your-username') s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’}) pp s.search_objects.to_a ``` - + See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data. -- **I indexed all the repositories but then switched elastic search servers and now I can't find anything** +- **I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything** You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. - + - **The indexing process is taking a very long time** The more data present in your GitLab instance, the longer the indexing process takes. diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 5950737b964..e6496ae3a2e 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -25,22 +25,22 @@ and [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF ## Use cases - Suppose you are new to GitLab, and want to keep using Jenkins until you prepare -your projects to build with [GitLab CI/CD](../ci/README.md). You set up the -integration between GitLab and Jenkins, then you migrate to GitLab CI later. While -you organize yourself and your team to onboard GitLab, you keep your pipelines -running with Jenkins, but view the results in your project's repository in GitLab. + your projects to build with [GitLab CI/CD](../ci/README.md). You set up the + integration between GitLab and Jenkins, then you migrate to GitLab CI later. While + you organize yourself and your team to onboard GitLab, you keep your pipelines + running with Jenkins, but view the results in your project's repository in GitLab. - Your team uses [Jenkins Plugins](https://plugins.jenkins.io/) for other proceedings, -therefore, you opt for keep using Jenkins to build your apps. Show the results of your -pipelines directly in GitLab. + therefore, you opt for keep using Jenkins to build your apps. Show the results of your + pipelines directly in GitLab. For a real use case, read the blog post [Continuous integration: From Jenkins to GitLab using Docker](https://about.gitlab.com/2017/07/27/docker-my-precious/). ## Requirements -* [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) -* [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) -* Git clone access for Jenkins from the GitLab repository -* GitLab API access to report build status +- [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +- [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) +- Git clone access for Jenkins from the GitLab repository +- GitLab API access to report build status ## Configure GitLab users @@ -65,7 +65,7 @@ Go to Manage Jenkins -> Configure System and scroll down to the 'GitLab' section Enter the GitLab server URL in the 'GitLab host URL' field and paste the API token copied earlier in the 'API Token' field. -For more information, see GitLab Plugin documentation about +For more information, see GitLab Plugin documentation about [Jenkins-to-GitLab authentication](https://github.com/jenkinsci/gitlab-plugin#jenkins-to-gitlab-authentication)  @@ -76,8 +76,8 @@ Follow the GitLab Plugin documentation about [Jenkins Job Configuration](https:/ NOTE: **Note:** Be sure to include the steps about [Build status configuration](https://github.com/jenkinsci/gitlab-plugin#build-status-configuration). -The 'Publish build status to GitLab' post-build step is required to view -Jenkins build status in GitLab Merge Requests. +The 'Publish build status to GitLab' post-build step is required to view +Jenkins build status in GitLab Merge Requests. ## Configure a GitLab project @@ -114,21 +114,21 @@ and storing build status for Commits and Merge Requests. All steps are implemented using AJAX requests on the merge request page. 1. In order to display the build status in a merge request you must create a project service in GitLab. -2. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. -3. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. -4. The response is parsed to give a response in GitLab (success/failed/pending). +1. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. +1. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. +1. The response is parsed to give a response in GitLab (success/failed/pending). ## Troubleshooting ### Error in merge requests - "Could not connect to the CI server" This integration relies on Jenkins reporting the build status back to GitLab via -the [Commit Status API](../api/commits.md#commit-status). +the [Commit Status API](../api/commits.md#commit-status). The error 'Could not connect to the CI server' usually means that GitLab did not receive a build status update via the API. Either Jenkins was not properly -configured or there was an error reporting the status via the API. +configured or there was an error reporting the status via the API. 1. [Configure the Jenkins server](#configure-the-jenkins-server) for GitLab API access -2. [Configure a Jenkins project](#configure-a-jenkins-project), including the - 'Publish build status to GitLab' post-build action. +1. [Configure a Jenkins project](#configure-a-jenkins-project), including the + 'Publish build status to GitLab' post-build action. diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md index 8001c5dbd83..eae705c9637 100644 --- a/doc/integration/jenkins_deprecated.md +++ b/doc/integration/jenkins_deprecated.md @@ -8,13 +8,13 @@ Please use documentation for the new [Jenkins CI service](jenkins.md). Integration includes: -* Trigger Jenkins build after push to repo -* Show build status on Merge Request page +- Trigger Jenkins build after push to repo +- Show build status on Merge Request page Requirements: -* [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) -* git clone access for Jenkins from GitLab repo (via ssh key) +- [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +- git clone access for Jenkins from GitLab repo (via ssh key) ## Jenkins diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index b1637181855..d04428fdbfe 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -29,6 +29,12 @@ in users. Any logged in user will have [Guest permissions](../user/permissions.md) on the repository. +NOTE: **Note:** +From July 2019, the `Internal` visibility setting is disabled for new projects, groups, +and snippets on GitLab.com. Existing projects, groups, and snippets using the `Internal` +visibility setting keep this setting. You can read more about the change in the +[relevant issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/12388). + ### Private projects Private projects can only be cloned and viewed by project members. diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index b2d626a0a74..2142f5a5f69 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -26,11 +26,11 @@ Every push rule could have its own use case, but let's consider some examples. Let's assume you have the following requirements for your workflow: -- every commit should reference a JIRA issue, for example: `Refactored css. Fixes JIRA-123.` +- every commit should reference a Jira issue, for example: `Refactored css. Fixes JIRA-123.` - users should not be able to remove git tags with `git push` All you need to do is write a simple regular expression that requires the mention -of a JIRA issue in the commit message, like `JIRA\-\d+`. +of a Jira issue in the commit message, like `JIRA\-\d+`. Now when a user tries to push a commit with a message `Bugfix`, their push will be declined. Only pushing commits with messages like `Bugfix according to JIRA-123` diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index c7aa22b11f8..092b4375208 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -18,16 +18,16 @@ installed on your system. If you installed GitLab: -- Using the Omnibus package, you're all set. -- From source, make sure `rsync` is installed: +- Using the Omnibus package, you're all set. +- From source, make sure `rsync` is installed: - ```sh - # Debian/Ubuntu - sudo apt-get install rsync + ```sh + # Debian/Ubuntu + sudo apt-get install rsync - # RHEL/CentOS - sudo yum install rsync - ``` + # RHEL/CentOS + sudo yum install rsync + ``` ### Tar @@ -269,17 +269,17 @@ For Omnibus GitLab packages: 1. Add the following to `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-west-1', - 'aws_access_key_id' => 'AKIAKIAKI', - 'aws_secret_access_key' => 'secret123' - # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key - # 'use_iam_profile' => true - } - gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' - ``` + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' + # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key + # 'use_iam_profile' => true + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` 1. [Reconfigure GitLab] for the changes to take effect @@ -289,16 +289,16 @@ This example can be used for a bucket in Amsterdam (AMS3). 1. Add the following to `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'ams3', - 'aws_access_key_id' => 'AKIAKIAKI', - 'aws_secret_access_key' => 'secret123', - 'endpoint' => 'https://ams3.digitaloceanspaces.com' - } - gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' - ``` + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'ams3', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123', + 'endpoint' => 'https://ams3.digitaloceanspaces.com' + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` 1. [Reconfigure GitLab] for the changes to take effect @@ -321,31 +321,31 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - # snip - upload: - # Fog storage connection settings, see http://fog.io/storage/ . - connection: - provider: AWS - region: eu-west-1 - aws_access_key_id: AKIAKIAKI - aws_secret_access_key: 'secret123' - # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty - # ie. aws_access_key_id: '' - # use_iam_profile: 'true' - # The remote 'directory' to store your backups. For S3, this would be the bucket name. - remote_directory: 'my.s3.bucket' - # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional - # encryption: 'AES256' - # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional - # This should be set to the base64-encoded encryption key for Amazon S3 to use to encrypt or decrypt your data. - # 'encryption' must also be set in order for this to have any effect. - # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. - # encryption_key: '<base64 key>' - # Specifies Amazon S3 storage class to use for backups, this is optional - # storage_class: 'STANDARD' - ``` + ```yaml + backup: + # snip + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: AWS + region: eu-west-1 + aws_access_key_id: AKIAKIAKI + aws_secret_access_key: 'secret123' + # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty + # ie. aws_access_key_id: '' + # use_iam_profile: 'true' + # The remote 'directory' to store your backups. For S3, this would be the bucket name. + remote_directory: 'my.s3.bucket' + # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional + # encryption: 'AES256' + # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional + # This should be set to the base64-encoded encryption key for Amazon S3 to use to encrypt or decrypt your data. + # 'encryption' must also be set in order for this to have any effect. + # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. + # encryption_key: '<base64 key>' + # Specifies Amazon S3 storage class to use for backups, this is optional + # storage_class: 'STANDARD' + ``` 1. [Restart GitLab] for the changes to take effect @@ -417,14 +417,14 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'Google', - 'google_storage_access_key_id' => 'Access Key', - 'google_storage_secret_access_key' => 'Secret' - } - gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' - ``` + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_storage_access_key_id' => 'Access Key', + 'google_storage_secret_access_key' => 'Secret' + } + gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' + ``` 1. [Reconfigure GitLab] for the changes to take effect @@ -434,15 +434,15 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - upload: - connection: - provider: 'Google' - google_storage_access_key_id: 'Access Key' - google_storage_secret_access_key: 'Secret' - remote_directory: 'my.google.bucket' - ``` + ```yaml + backup: + upload: + connection: + provider: 'Google' + google_storage_access_key_id: 'Access Key' + google_storage_secret_access_key: 'Secret' + remote_directory: 'my.google.bucket' + ``` 1. [Restart GitLab] for the changes to take effect @@ -477,16 +477,16 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - :provider => 'Local', - :local_root => '/mnt/backups' - } + ```ruby + gitlab_rails['backup_upload_connection'] = { + :provider => 'Local', + :local_root => '/mnt/backups' + } - # The directory inside the mounted folder to copy backups to - # Use '.' to store them in the root directory - gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' - ``` + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -496,17 +496,17 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - upload: - # Fog storage connection settings, see http://fog.io/storage/ . - connection: - provider: Local - local_root: '/mnt/backups' - # The directory inside the mounted folder to copy backups to - # Use '.' to store them in the root directory - remote_directory: 'gitlab_backups' - ``` + ```yaml + backup: + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: Local + local_root: '/mnt/backups' + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + remote_directory: 'gitlab_backups' + ``` 1. [Restart GitLab] for the changes to take effect. @@ -521,9 +521,9 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable - ``` + ```ruby + gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -533,10 +533,10 @@ For installations from source: 1. Edit `/home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - archive_permissions: 0644 # Makes the backup archives world-readable - ``` + ```yaml + backup: + archive_permissions: 0644 # Makes the backup archives world-readable + ``` 1. [Restart GitLab] for the changes to take effect. @@ -550,10 +550,10 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - ## Limit backup lifetime to 7 days - 604800 seconds - gitlab_rails['backup_keep_time'] = 604800 - ``` + ```ruby + ## Limit backup lifetime to 7 days - 604800 seconds + gitlab_rails['backup_keep_time'] = 604800 + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -586,11 +586,11 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - ## Limit backup lifetime to 7 days - 604800 seconds - keep_time: 604800 - ``` + ```yaml + backup: + ## Limit backup lifetime to 7 days - 604800 seconds + keep_time: 604800 + ``` 1. [Restart GitLab] for the changes to take effect. @@ -840,13 +840,13 @@ columns containing sensitive information. If the key is lost, GitLab will be unable to decrypt those columns. This will break a wide range of functionality, including (but not restricted to): -* [CI/CD variables](../ci/variables/README.md) -* [Kubernetes / GCP integration](../user/project/clusters/index.md) -* [Custom Pages domains](../user/project/pages/getting_started_part_three.md) -* [Project error tracking](../user/project/operations/error_tracking.md) -* [Runner authentication](../ci/runners/README.md) -* [Project mirroring](../workflow/repository_mirroring.md) -* [Web hooks](../user/project/integrations/webhooks.md) +- [CI/CD variables](../ci/variables/README.md) +- [Kubernetes / GCP integration](../user/project/clusters/index.md) +- [Custom Pages domains](../user/project/pages/getting_started_part_three.md) +- [Project error tracking](../user/project/operations/error_tracking.md) +- [Runner authentication](../ci/runners/README.md) +- [Project mirroring](../workflow/repository_mirroring.md) +- [Web hooks](../user/project/integrations/webhooks.md) In cases like CI/CD variables and Runner authentication, you might experience some unexpected behavior such as: @@ -865,72 +865,71 @@ backup beforehand. #### Reset CI/CD variables -1. Enter the DB console: +1. Enter the DB console: - For Omnibus GitLab packages: + For Omnibus GitLab packages: - ```sh - sudo gitlab-rails dbconsole - ``` + ```sh + sudo gitlab-rails dbconsole + ``` - For installations from source: + For installations from source: - ```sh - sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production - ``` + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` -1. Check the `ci_group_variables` and `ci_variables` tables: +1. Check the `ci_group_variables` and `ci_variables` tables: - ```sql - SELECT * FROM public."ci_group_variables"; - SELECT * FROM public."ci_variables"; - ``` + ```sql + SELECT * FROM public."ci_group_variables"; + SELECT * FROM public."ci_variables"; + ``` - Those are the variables that you need to delete. + Those are the variables that you need to delete. -1. Drop the table: +1. Drop the table: - ```sql - DELETE FROM ci_group_variables; - DELETE FROM ci_variables; - ``` + ```sql + DELETE FROM ci_group_variables; + DELETE FROM ci_variables; + ``` 1. You may need to reconfigure or restart GitLab for the changes to take effect. - #### Reset Runner registration tokens -1. Enter the DB console: +1. Enter the DB console: - For Omnibus GitLab packages: + For Omnibus GitLab packages: - ```sh - sudo gitlab-rails dbconsole - ``` + ```sh + sudo gitlab-rails dbconsole + ``` - For installations from source: + For installations from source: - ```sh - sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production - ``` + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` 1. Clear all the tokens for projects, groups, and the whole instance: - CAUTION: **Caution:** - The last UPDATE operation will stop the runners being able to pick up - new jobs. You must register new runners. - - ```sql - -- Clear project tokens - UPDATE projects SET runners_token = null, runners_token_encrypted = null; - -- Clear group tokens - UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; - -- Clear instance tokens - UPDATE application_settings SET runners_registration_token_encrypted = null; - -- Clear runner tokens - UPDATE ci_runners SET token = null, token_encrypted = null; - ``` + CAUTION: **Caution:** + The last UPDATE operation will stop the runners being able to pick up + new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted, GitLab can be brought back into working order, diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index f5c788af578..f880f31c39e 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -92,3 +92,48 @@ I, [2018-08-02T10:26:47.598424 #45087] INFO -- : Looking for orphaned remote up I, [2018-08-02T10:26:47.753131 #45087] INFO -- : Moved to lost and found: @hashed/6b/DSC_6152.JPG -> lost_and_found/@hashed/6b/DSC_6152.JPG I, [2018-08-02T10:26:47.764356 #45087] INFO -- : Moved to lost and found: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg -> lost_and_found/@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg ``` + +## Remove orphan artifact files + +When you notice there are more job artifacts files on disk than there +should be, you can run: + +```shell +gitlab-rake gitlab:cleanup:orphan_job_artifact_files +``` + +This command: + +- Scans through the entire artifacts folder. +- Checks which files still have a record in the database. +- If no database record is found, the file is deleted from disk. + +By default, this task does not delete anything but shows what it can +delete. Run the command with `DRY_RUN=false` if you actually want to +delete the files: + +```shell +gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false +``` + +You can also limit the number of files to delete with `LIMIT`: + +```shell +gitlab-rake gitlab:cleanup:orphan_job_artifact_files LIMIT=100` +``` + +This will only delete up to 100 files from disk. You can use this to +delete a small set for testing purposes. + +If you provide `DEBUG=1`, you'll see the full path of every file that +is detected as being an orphan. + +If `ionice` is installed, the tasks uses it to ensure the command is +not causing too much load on the disk. You can configure the niceness +level with `NICENESS`. Below are the valid levels, but consult +`man 1 ionice` to be sure. + +- `0` or `None` +- `1` or `Realtime` +- `2` or `Best-effort` (default) +- `3` or `Idle` diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index df3dab118b2..2c6ae0749dd 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -2,42 +2,54 @@ ## Add a webhook for **ALL** projects: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" - # source installations - bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" +# source installations +bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production +``` ## Add a webhook for projects in a given **NAMESPACE**: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme - # source installations - bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme +# source installations +bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +``` ## Remove a webhook from **ALL** projects using: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" - # source installations - bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" +# source installations +bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production +``` ## Remove a webhook from projects in a given **NAMESPACE**: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme - # source installations - bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme +# source installations +bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +``` ## List **ALL** webhooks: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:list - # source installations - bundle exec rake gitlab:web_hook:list RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:list +# source installations +bundle exec rake gitlab:web_hook:list RAILS_ENV=production +``` ## List the webhooks from projects in a given **NAMESPACE**: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme - # source installations - bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme +# source installations +bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production +``` diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index fa4b0d1fb09..8695b5d2194 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -53,8 +53,9 @@ For more information on how to use these options check out The following settings can be configured: - `enabled`: By default this is set to `false`. Set this to `true` to enable Rack Attack. -- `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a ruby array. - For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3"]`. +- `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a Ruby array. + CIDR notation is supported in GitLab v12.1 and up. + For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`. - `maxretry`: The maximum amount of times a request can be made in the specified time. - `findtime`: The maximum amount of time that failed requests can count against an IP diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 37051f6b10f..745a2253e84 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -14,7 +14,7 @@ Learn how GitLab helps you in the stages of the DevOps lifecycle by learning mor ### Self-managed: Install GitLab -Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. +Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. ### GitLab.com: Create a user and group @@ -74,11 +74,11 @@ Please note that you need to be a group owner to associate a group to your subsc To see the status of your GitLab.com subscription, you can click on the Billings section of the relevant namespace: -* For individuals, this is located at https://gitlab.com/profile/billings under -in your Settings, -* For groups, this is located under the group's Settings dropdown, under Billing. +- For individuals, this is located at <https://gitlab.com/profile/billings> under + in your Settings, +- For groups, this is located under the group's Settings dropdown, under Billing. -For groups, you can see details of your subscription - including your current +For groups, you can see details of your subscription - including your current plan - in the included table:  @@ -86,11 +86,11 @@ plan - in the included table: | Field | Description | | ------ | ------ | | Seats in subscription | If this is a paid plan, this represents the number of seats you've paid to support in your group. | -| Seats currently in use | The number of active seats currently in use. | -| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | -| Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | -| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | -| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | +| Seats currently in use | The number of active seats currently in use. | +| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | +| Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | +| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | +| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | ### Subscription changes and your data diff --git a/doc/topics/application_development_platform/index.md b/doc/topics/application_development_platform/index.md index e8960a76dd9..8742606479d 100644 --- a/doc/topics/application_development_platform/index.md +++ b/doc/topics/application_development_platform/index.md @@ -1,12 +1,21 @@ # Application Development Platform -The GitLab Application Development Platform refers to the set of GitLab features that can be used by operations teams to -provide a full development environment to internal software development teams. +The GitLab Application Development Platform refers to the set of GitLab features used to create, configure, and manage +a complete software development environment. It provides development, operations, and security teams with a robust feature set aimed at supporting best practices out of the box. ## Overview -The GitLab Application Development Platform aims to reduce and even eliminate the time it takes for an Operations team -to provide a full environment for software developers. It comprises the following high-level elements: +The GitLab Application Development Platform aims to: + +- Reduce and even eliminate the time it takes for an Operations team + to provide a full environment for software developers. +- Get developers up and running fast so they can focus on writing + great applications with a robust development feature set. +- Provide best-of-breed security features so that applications developed + with GitLab are not affected by vulnerabilities that may lead to security + problems and unintended use. + +It is comprised of the following high-level elements: 1. Compute 1. Build, test, and deploy a wide range of applications diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 702245b22a0..bd788cb138c 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -175,8 +175,8 @@ When using Auto DevOps, you may want to deploy different environments to different Kubernetes clusters. This is possible due to the 1:1 connection that [exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters-premium). -In the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) -(used behind the scenes by Auto DevOps), there are currently 3 defined environment names that you need to know: +In the [Auto DevOps template] (used behind the scenes by Auto DevOps), there +are currently 3 defined environment names that you need to know: - `review/` (every environment starting with `review/`) - `staging` @@ -662,25 +662,32 @@ to the desired environment. See [Limiting environment scopes of variables](../.. ### Customizing `.gitlab-ci.yml` -If you want to modify the CI/CD pipeline used by Auto DevOps, you can copy the -[Auto DevOps template] into your project's repo and edit as you see fit. +Everything about Auto DevOps is customizable since the [Auto DevOps template] +is just an example of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) and uses +only features that are available to any `.gitlab-ci.yml`. -Assuming that your project is new or it doesn't have a `.gitlab-ci.yml` file -present: +Auto DevOps is completely customizable because the [Auto DevOps template]: -1. From your project home page, either click on the "Set up CI/CD" button, or click - on the plus button and (`+`), then "New file" -1. Pick `.gitlab-ci.yml` as the template type -1. Select "Auto-DevOps" from the template dropdown -1. Edit the template or add any jobs needed -1. Give an appropriate commit message and hit "Commit changes" +- Is just an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) file. +- Uses only features available to any implementation of `.gitlab-ci.yml`. -TIP: **Tip:** The Auto DevOps template includes useful comments to help you -customize it. For example, if you want deployments to go to a staging environment -instead of directly to a production one, you can enable the `staging` job by -renaming `.staging` to `staging`. Then make sure to uncomment the `when` key of -the `production` job to turn it into a manual action instead of deploying -automatically. +If you want to modify the CI/CD pipeline used by Auto DevOps, you can [`include` +the template](../../ci/yaml/README.md#includetemplate) and customize as +needed. To do this, add a `.gitlab-ci.yml` file to the root of your repository +containing the following: + +```yml +include: + - template: Auto-DevOps.gitlab-ci.yml +``` + +Then add any extra changes you want. Your additions will be merged with the +[Auto DevOps template] using the behaviour described for +[`include`](../../ci/yaml/README.md#include). + +It is also possible to copy and paste the contents of the [Auto DevOps +template] into your project and edit this as needed. You may prefer to do it +that way if you want to specifically remove any part of it. ### Using components of Auto-DevOps @@ -723,7 +730,7 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From Gitlab 11.11, this variable can be used to set a username to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD) | | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From Gitlab 11.11, this variable can be used to set a password to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_USERNAME) | | `REPLICAS` | The number of replicas to deploy; defaults to 1. | -| `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. This takes precedence over `REPLICAS`; defaults to 1. | +| `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | | `CANARY_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md); defaults to 1 | | `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1 | | `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. | diff --git a/doc/university/README.md b/doc/university/README.md index 1d2c0f2068a..9d861460618 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -181,7 +181,7 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres ### 3.9. Integrations -1. [How to Integrate JIRA and Jenkins with GitLab - Video](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) +1. [How to Integrate Jira and Jenkins with GitLab - Video](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) 1. [How to Integrate Jira with GitLab](../user/project/integrations/jira.md) 1. [How to Integrate Jenkins with GitLab](../integration/jenkins.md) 1. [How to Integrate Bamboo with GitLab](../user/project/integrations/bamboo.md) diff --git a/doc/university/process/README.md b/doc/university/process/README.md index fdf6224f7f6..b278e02ccd5 100644 --- a/doc/university/process/README.md +++ b/doc/university/process/README.md @@ -9,11 +9,11 @@ title: University | Process # Suggesting improvements If you would like to teach a class or participate or help in any way please -submit a merge request and assign it to [Job](https://gitlab.com/u/JobV). +submit a merge request and assign it to [Job](https://gitlab.com/JobV). If you have suggestions for additional courses you would like to see, please submit a merge request to add an upcoming class, assign to -[Chad](https://gitlab.com/u/chadmalchow) and /cc [Job](https://gitlab.com/u/JobV). +[Chad](https://gitlab.com/chadmalchow) and /cc [Job](https://gitlab.com/JobV). ## Adding classes @@ -31,4 +31,4 @@ please submit a merge request to add an upcoming class, assign to 1. Please upload any video recordings to our Youtube channel. We prefer them to be public, if needed they can be unlisted but if so they should be linked from this page. -1. Please create a merge request and assign to [Erica](https://gitlab.com/u/Erica). +1. Please create a merge request and assign to [Erica](https://gitlab.com/Erica). diff --git a/doc/university/support/README.md b/doc/university/support/README.md index 35e65b60768..2c6e52acfde 100644 --- a/doc/university/support/README.md +++ b/doc/university/support/README.md @@ -80,7 +80,7 @@ Our integrations add great value to GitLab. User questions often relate to integ - Learn about our Integrations (specially, not only): - [LDAP](../../integration/ldap.md) - - [JIRA](../../project_services/jira.md) + - [Jira](../../project_services/jira.md) - [Jenkins](../../integration/jenkins.md) - [SAML](../../integration/saml.md) diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index 428377adb19..7ae716d2cb3 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -25,14 +25,14 @@ Branch names use the format `major-minor-stable-ee` for Enterprise Edition, and `major-minor-stable` for Community Edition. For example, for 11.8.0 you would use the following branches: -* Enterprise Edition: `11-8-stable-ee` -* Community Edition: `11-8-stable` +- Enterprise Edition: `11-8-stable-ee` +- Community Edition: `11-8-stable` ### 0. Backup Make a backup just in case something goes wrong: -```bash +```sh cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` @@ -42,13 +42,13 @@ privileges to the GitLab user on the database version. ### 1. Stop server -```bash +```sh sudo service gitlab stop ``` ### 2. Get the EE code -```bash +```sh cd /home/git/gitlab sudo -u git -H git remote add -f ee https://gitlab.com/gitlab-org/gitlab-ee.git sudo -u git -H git checkout EE_BRANCH @@ -56,7 +56,7 @@ sudo -u git -H git checkout EE_BRANCH ### 3. Install libs, migrations, etc. -```bash +```sh cd /home/git/gitlab # MySQL installations (note: the line below states '--without postgres') @@ -80,7 +80,7 @@ document linked above and enable the indexer usage in the GitLab admin settings. ### 5. Start application -```bash +```sh sudo service gitlab start sudo service nginx restart ``` @@ -89,13 +89,13 @@ sudo service nginx restart Check if GitLab and its environment are configured correctly: -```bash +```sh sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production ``` To make sure you didn't miss anything run a more thorough check with: -```bash +```sh sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production ``` @@ -105,14 +105,14 @@ If all items are green, then congratulations upgrade complete! ### 1. Revert the code to the previous version -```bash +```sh cd /home/git/gitlab sudo -u git -H git checkout CE_BRANCH ``` ### 2. Restore from the backup -```bash +```sh cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production ``` diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index add5971e01c..d2947ae3371 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -21,7 +21,7 @@ The Admin Area is made up of the following sections: | Section | Description | |:------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Overview](#overview-section) | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administering-projects), [users](#administering-users), [groups](#administering-groups), [jobs](#administering-jobs), [Runners](#administering-runners), and [Gitaly servers](#administering-gitaly-servers). | -| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | +| Monitoring | View GitLab [system information](#system-info), and information on [background jobs](#background-jobs), [logs](#logs), [health checks](monitoring/health_check.md), [requests profiles](#requests-profiles), and [audit logs](#audit-log-premium-only). | | Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | | System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | | Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | @@ -186,7 +186,7 @@ the sort order to *Last Contacted* from the dropdown beside the search field. To search Runners' descriptions: 1. In the **Search or filter results...** field, type the description of the Runner you want to -find. + find. 1. Press Enter. You can also filter Runners by status, type, and tag. To filter: @@ -229,3 +229,66 @@ For each Gitaly server, the following details are listed: | Server version | Gitaly version | | Git version | Version of Git installed on the Gitaly server | | Up to date | Indicates if the Gitaly server version is the latest version available. A green dot indicates the server is up to date. | + +## Monitoring section + +The following topics document the **Monitoring** section of the Admin Area. + +### System Info + +The **System Info** page provides the following statistics: + +| Field | Description | +| :----------- | :---------- | +| CPU | Number of CPU cores available | +| Memory Usage | Memory in use, and total memory available | +| Disk Usage | Disk space in use, and total disk space available | +| Uptime | Approximate uptime of the GitLab instance | + +These statistics are updated only when you navigate to the **System Info** page, or you refresh the page in your browser. + +### Background Jobs + +The **Background Jobs** page displays the Sidekiq dashboard. Sidekiq is used by GitLab to +perform processing in the background. + +The Sidekiq dashboard consists of the following elements: + +- A tab per jobs' status. +- A breakdown of background job statistics. +- A live graph of **Processed** and **Failed** jobs, with a selectable polling interval. +- An historical graph of **Processed** and **Failed** jobs, with a selectable time span. +- Redis statistics, including: + - Version number + - Uptime, measured in days + - Number of connections + - Current memory usage, measured in MB + - Peak memory usage, measured in MB + +### Logs + +The **Logs** page provides access to the following log files: + +| Log file | Contents | +| :---------------------- | :------- | +| `application.log` | GitLab user activity | +| `githost.log` | Failed GitLab interaction with Git repositories | +| `production.log` | Requests received from Unicorn, and the actions taken to serve those requests | +| `sidekiq.log` | Background jobs | +| `repocheck.log` | Repository activity | +| `integrations_json.log` | Activity between GitLab and integrated systems | +| `kubernetes.log` | Kubernetes activity | + +The contents of these log files can be useful when troubleshooting a problem. Access is available to GitLab admins, without requiring direct access to the log files. + +For details of these log files and their contents, see [Log system](../../administration/logs.md). + +The content of each log file is listed in chronological order. To minimize performance issues, a maximum 2000 lines of each log file are shown. + +### Requests Profiles + +The **Requests Profiles** page contains the token required for profiling. For more details, see [Request Profiling](../../administration/monitoring/performance/request_profiling.md). + +### Audit Log **[PREMIUM ONLY]** + +The **Audit Log** page lists changes made within the GitLab server. With this information you can control, analyze, and track every change. diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index f80d4e9d2aa..35e7b6fb541 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -41,42 +41,51 @@ The readiness and liveness probes will provide a report of system health in JSON ```json { - "queues_check" : { - "status" : "ok" + "db_check":{ + "status":"ok" }, - "redis_check" : { - "status" : "ok" + "redis_check":{ + "status":"ok" }, - "shared_state_check" : { - "status" : "ok" + "cache_check":{ + "status":"ok" }, - "db_check" : { - "status" : "ok" + "queues_check":{ + "status":"ok" }, - "cache_check" : { - "status" : "ok" + "shared_state_check":{ + "status":"ok" + }, + "gitaly_check":{ + "status":"ok", + "labels":{ + "shard":"default" + } + } } -} ``` `liveness` probe example output: ```json { - "cache_check" : { - "status" : "ok" + "db_check":{ + "status":"ok" + }, + "redis_check":{ + "status":"ok" }, - "db_check" : { - "status" : "ok" + "cache_check":{ + "status":"ok" }, - "redis_check" : { - "status" : "ok" + "queues_check":{ + "status":"ok" }, - "queues_check" : { - "status" : "ok" + "shared_state_check":{ + "status":"ok" }, - "shared_state_check" : { - "status" : "ok" + "gitaly_check":{ + "status":"ok" } } ``` diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index fde7d1aeaf7..84596ff6a2c 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -94,14 +94,11 @@ a group in the **Usage Quotas** page available to the group page settings list.  +## Extra Shared Runners pipeline minutes quota **[FREE ONLY]** -## Extra Shared Runners pipeline minutes quota - -NOTE: **Note:** -Only available on GitLab.com. - -You can purchase additional CI minutes so your pipelines will not be blocked after you have -used all your CI minutes from your main quota. +If you're using GitLab.com, you can purchase additional CI minutes so your +pipelines will not be blocked after you have used all your CI minutes from your +main quota. In order to purchase additional minutes, you should follow these steps: @@ -110,27 +107,27 @@ In order to purchase additional minutes, you should follow these steps:  1. Locate the subscription card that is linked to your group on GitLab.com, -click on **Buy more CI minutes**, and complete the details about the transaction. + click on **Buy more CI minutes**, and complete the details about the transaction.  1. Once we have processed your payment, the extra CI minutes -will be synced to your Group and you can visualize it from the -**Group > Settings > Pipelines quota** page: + will be synced to your Group and you can visualize it from the + **Group > Settings > Pipelines quota** page:  Be aware that: 1. If you have purchased extra CI minutes before the purchase of a paid plan, -we will calculate a pro-rated charge for your paid plan. That means you may -be charged for less than one year since your subscription was previously -created with the extra CI minutes. + we will calculate a pro-rated charge for your paid plan. That means you may + be charged for less than one year since your subscription was previously + created with the extra CI minutes. 1. Once the extra CI minutes has been assigned to a Group they cannot be transferred -to a different Group. + to a different Group. 1. If you have some minutes used over your default quota, these minutes will -be deducted from your Additional Minutes quota immediately after your purchase of additional -minutes. + be deducted from your Additional Minutes quota immediately after your purchase of additional + minutes. ## What happens when my CI minutes quota run out diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index a5f8d05f662..a1bce5a6c69 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -17,7 +17,7 @@ To enforce acceptance of a Terms of Service and Privacy Policy: 1. Go to **Admin Area > Settings > General**. 1. Expand the **Terms of Service and Privacy Policy** section. 1. Check the **Require all users to accept Terms of Service and Privacy Policy when they access -GitLab.** checkbox. + GitLab.** checkbox. 1. Input the text of the **Terms of Service and Privacy Policy**. Markdown formatting can be used in this input box. 1. Click **Save changes**. 1. When you are presented with the **Terms of Service** statement, click **Accept terms**. diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 4a2fb1d7190..9dfbe326f1d 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -206,6 +206,11 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). + ## Troubleshooting ### docker: Error response from daemon: failed to copy xattrs diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index a722aa88f9d..2283efe3a44 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -259,3 +259,8 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md new file mode 100644 index 00000000000..937ded287e5 --- /dev/null +++ b/doc/user/application_security/dependency_scanning/analyzers.md @@ -0,0 +1,133 @@ +# Dependency Scanning Analyzers **[ULTIMATE]** + +Dependency Scanning relies on underlying third party tools that are wrapped into +what we call "Analyzers". An analyzer is a +[dedicated project](https://gitlab.com/gitlab-org/security-products/analyzers) +that wraps a particular tool to: + +- Expose its detection logic. +- Handle its execution. +- Convert its output to the common format. + +This is achieved by implementing the [common API](https://gitlab.com/gitlab-org/security-products/analyzers/common). + +Dependency Scanning supports the following official analyzers: + +- [`bundler-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit) +- [`gemnasium`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) +- [`gemnasium-maven`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven) +- [`gemnasium-python`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python) +- [`retire.js`](https://gitlab.com/gitlab-org/security-products/analyzers/retire.js) + +The analyzers are published as Docker images that Dependency Scanning will use +to launch dedicated containers for each analysis. + +Dependency Scanning is pre-configured with a set of **default images** that are +maintained by GitLab, but users can also integrate their own **custom images**. + +## Official default analyzers + +Any custom change to the official analyzers can be achieved by using an +[environment variable in your `.gitlab-ci.yml`](index.md#customizing-the-dependency-scanning-settings). + +### Using a custom Docker mirror + +You can switch to a custom Docker registry that provides the official analyzer +images under a different prefix. For instance, the following instructs Dependency +Scanning to pull `my-docker-registry/gl-images/gemnasium` +instead of `registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium`. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_ANALYZER_IMAGE_PREFIX: my-docker-registry/gl-images +``` + +This configuration requires that your custom registry provides images for all +the official analyzers. + +### Selecting specific analyzers + +You can select the official analyzers you want to run. Here's how to enable +`bundler-audit` and `gemnasium` while disabling all the other default ones. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_DEFAULT_ANALYZERS: "bundler-audit,gemnasium" +``` + +`bundler-audit` runs first. When merging the reports, Dependency Scanning will +remove the duplicates and will keep the `bundler-audit` entries. + +### Disabling default analyzers + +Setting `DS_DEFAULT_ANALYZERS` to an empty string will disable all the official +default analyzers. In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_DEFAULT_ANALYZERS: "" +``` + +That's needed when one totally relies on [custom analyzers](#custom-analyzers). + +## Custom analyzers + +You can provide your own analyzers as a comma separated list of Docker images. +Here's how to add `analyzers/nugget` and `analyzers/perl` to the default images. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_ANALYZER_IMAGES: "my-docker-registry/analyzers/nugget,amy-docker-registry/nalyzers/perl" +``` + +The values must be the full path to the container registry images, +like what you would feed to the `docker pull` command. + +NOTE: **Note:** +This configuration doesn't benefit from the integrated detection step. Dependency +Scanning has to fetch and spawn each Docker image to establish whether the +custom analyzer can scan the source code. + +## Analyzers data + +The following table lists the data available for each official analyzer. + +| Property \ Tool | Gemnasium | bundler-audit | Retire.js | +|---------------------------------------|:------------------:|:------------------:|:------------------:| +| Severity | 𐄂 | ✓ | ✓ | +| Title | ✓ | ✓ | ✓ | +| File | ✓ | ⚠ | ✓ | +| Start line | 𐄂 | 𐄂 | 𐄂 | +| End line | 𐄂 | 𐄂 | 𐄂 | +| External ID (e.g., CVE) | ✓ | ✓ | ⚠ | +| URLs | ✓ | ✓ | ✓ | +| Internal doc/explanation | ✓ | 𐄂 | 𐄂 | +| Solution | ✓ | ✓ | 𐄂 | +| Confidence | 𐄂 | 𐄂 | 𐄂 | +| Affected item (e.g. class or package) | ✓ | ✓ | ✓ | +| Source code extract | 𐄂 | 𐄂 | 𐄂 | +| Internal ID | ✓ | 𐄂 | 𐄂 | +| Date | ✓ | 𐄂 | 𐄂 | +| Credits | ✓ | 𐄂 | 𐄂 | + +- ✓ => we have that data +- ⚠ => we have that data but it's partially reliable, or we need to extract that data from unstructured content +- 𐄂 => we don't have that data or it would need to develop specific or inefficient/unreliable logic to obtain it. + +The values provided by these tools are heterogeneous so they are sometimes +normalized into common values (e.g., `severity`, `confidence`, etc). diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index a4e5b19bdc7..9145e034dcb 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -46,17 +46,33 @@ this is enabled by default. The following languages and dependency managers are supported. -| Language (package managers) | Scan tool | -|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| -| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [Retire.js](https://retirejs.github.io/retire.js) | -| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | -| Ruby ([gem](https://rubygems.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [bundler-audit](https://github.com/rubysec/bundler-audit) | -| Java ([Maven](https://maven.apache.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | -| PHP ([Composer](https://getcomposer.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | - -Some scanners require to send a list of project dependencies to GitLab's central -servers to check for vulnerabilities. To learn more about this or to disable it, -refer to the [GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). +| Language (package managers) | Supported | Scan tool(s) | +|----------------------------- | --------- | ------------ | +| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js) | +| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| Ruby ([gem](https://rubygems.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) | +| Java ([Maven](https://maven.apache.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| PHP ([Composer](https://getcomposer.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| Python ([poetry](https://poetry.eustace.io/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7006 "Support Poetry in Dependency Scanning")) | not available | +| Python ([Pipfile](https://docs.pipenv.org/en/latest/basics/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/11756 "Pipfile.lock support for Dependency Scanning"))| not available | +| Go ([Golang](https://golang.org/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7132 "Dependency Scanning for Go")) | not available | + +## Remote checks + +While some tools pull a local database to check vulnerabilities, some others +like Gemnasium require sending data to GitLab central servers to analyze them: + +1. Gemnasium scans the dependencies of your project locally and sends a list of + packages to GitLab central servers. +1. The servers return the list of known vulnerabilities for all versions of + these packages. +1. The client picks up the relevant vulnerabilities by comparing with the versions + of the packages that are used by the project. + +The Gemnasium client does **NOT** send the exact package versions your project relies on. + +You can disable the remote checks by [using](#customizing-the-dependency-scanning-settings) +the `DS_DISABLE_REMOTE_CHECKS` environment variable and setting it to `true`. ## Configuring Dependency Scanning @@ -97,17 +113,10 @@ The report will be saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest Dependency Scanning artifact available. -Some security scanners require to send a list of project dependencies to GitLab -central servers to check for vulnerabilities. To learn more about this or to -disable it, check the -[GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). - #### Customizing the Dependency Scanning settings -The Dependency Scanning settings can be changed through environment variables by using the +The Dependency Scanning settings can be changed through [environment variables](#available-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. -These variables are documented in the -[Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings). For example: @@ -116,7 +125,7 @@ include: template: Dependency-Scanning.gitlab-ci.yml variables: - DEP_SCAN_DISABLE_REMOTE_CHECKS: true + DS_DISABLE_REMOTE_CHECKS: true ``` Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline @@ -137,6 +146,24 @@ dependency_scanning: CI_DEBUG_TRACE: "true" ``` +#### Available variables + +Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings) +using environment variables. + +| Environment variable | Function | +|-------------------------------- |----------| +| `DS_ANALYZER_IMAGES` | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). | +| `DS_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | +| `DS_ANALYZER_IMAGE_TAG` | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). | +| `DS_DEFAULT_ANALYZERS` | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). | +| `DS_DISABLE_REMOTE_CHECKS` | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). | +| `DS_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to `0` to disable). | +| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | +| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | +| `DS_PULL_ANALYZER_IMAGE_TIMEOUT` | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | +| `DS_RUN_ANALYZER_TIMEOUT` | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | + ### Manual job definition for GitLab 11.5 and later For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning` @@ -171,7 +198,7 @@ dependency_scanning: dependency_scanning: gl-dependency-scanning-report.json ``` -You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings) +You can supply many other [settings variables](#available-variables) via `docker run --env` to customize your job execution. ### Manual job definition for GitLab 11.4 and earlier (deprecated) @@ -377,6 +404,11 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). + ## Dependency List > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. @@ -388,6 +420,10 @@ supported by Gemnasium. To see the generated dependency list, navigate to your project's **Project > Dependency List**. +## Versioning and release process + +Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md). + ## Contributing to the vulnerability database You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 679847b76d7..69fa1ec5da6 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -10,7 +10,7 @@ high-level view on projects and groups, and start remediation processes when nee GitLab can scan and report any vulnerabilities found in your project. -| Secure scanning tools | Description | +| Secure scanning tool | Description | |:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------| | [Container Scanning](container_scanning/index.md) **[ULTIMATE]** | Scan Docker containers for known vulnerabilities. | | [Dependency Scanning](dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | @@ -19,6 +19,29 @@ GitLab can scan and report any vulnerabilities found in your project. | [Security Dashboard](security_dashboard/index.md) **[ULTIMATE]** | View vulnerabilities in all your projects and groups. | | [Static Application Security Testing (SAST)](sast/index.md) **[ULTIMATE]** | Analyze source code for known vulnerabilities. | +## Maintenance and update of the vulnerabilities database + +The various scanning tools and the vulnerabilities database are updated regularly. + +| Secure scanning tool | Vulnerabilities database updates | +|:-------------------------------------------------------------|-------------------------------------------| +| [Container Scanning](container_scanning/index.md) | Uses `clair` underneath and the latest `clair-db` version is used for each job run by running the [`latest` docker image tag](https://gitlab.com/gitlab-org/gitlab-ee/blob/438a0a56dc0882f22bdd82e700554525f552d91b/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml#L37). The `clair-db` database [is updated daily according to the author](https://github.com/arminc/clair-local-scan#clair-server-or-local). | +| [Dependency Scanning](dependency_scanning/index.md) | Relies on `bundler-audit` (for Rubygems), `retire.js` (for NPM packages) and `gemnasium` (GitLab's own tool for all libraries). `bundler-audit` and `retire.js` both fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated at least once a week. | +| [Dynamic Application Security Testing (DAST)](dast/index.md) | Updated weekly on Sundays. The underlying tool, `zaproxy`, downloads fresh rules at startup. | +| [Static Application Security Testing (SAST)](sast/index.md) | Relies exclusively on [the tools GitLab is wrapping](sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. | + +You don't have to update GitLab to benefit from the latest vulnerabilities definitions, +but you may have to in the future. + +The security tools are released as Docker images, and the vendored job definitions +to enable them are using the `x-y-stable` image tags that get overridden each time a new +release of the tools is pushed. The Docker images are updated to match the +previous GitLab releases, so they automatically get the latest versions of the +scanning tools without the user having to do anything. + +This workflow comes with some drawbacks and there's a +[plan to change this](https://gitlab.com/gitlab-org/gitlab-ee/issues/9725). + ## Interacting with the vulnerabilities > Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.8. diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index ec3f7fbde76..9074ac3f4a1 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -269,7 +269,7 @@ it highlighted: "url": "https://cwe.mitre.org/data/definitions/330.html" } ] - }, + }, { "category": "sast", "message": "Probable insecure usage of temp file/directory.", @@ -296,7 +296,7 @@ it highlighted: "url": "https://docs.openstack.org/bandit/latest/plugins/b108_hardcoded_tmp_directory.html" } ] - }, + }, ], "remediations": [] } @@ -320,7 +320,7 @@ the report JSON unless stated otherwise. Presence of optional fields depends on | `vulnerabilities[].scanner` | A node that describes the analyzer used to find this vulnerability. | | `vulnerabilities[].scanner.id` | Id of the scanner as a snake_case string. | | `vulnerabilities[].scanner.name` | Name of the scanner, for display purposes. | -| `vulnerabilities[].location` | A node that tells where the vulnerability is located. | +| `vulnerabilities[].location` | A node that tells where the vulnerability is located. | | `vulnerabilities[].location.file` | Path to the file where the vulnerability is located. Optional. | | `vulnerabilities[].location.start_line` | The first line of the code affected by the vulnerability. Optional. | | `vulnerabilities[].location.end_line` | The last line of the code affected by the vulnerability. Optional. | @@ -330,7 +330,7 @@ the report JSON unless stated otherwise. Presence of optional fields depends on | `vulnerabilities[].identifiers[].type` | Type of the identifier. Possible values: common identifier types (among `cve`, `cwe`, `osvdb`, and `usn`) or analyzer-dependent ones (e.g., `bandit_test_id` for [Bandit analyzer](https://wiki.openstack.org/wiki/Security/Projects/Bandit)). | | `vulnerabilities[].identifiers[].name` | Name of the identifier for display purposes. | | `vulnerabilities[].identifiers[].value` | Value of the identifier for matching purposes. | -| `vulnerabilities[].identifiers[].url` | URL to identifier's documentation. Optional. | +| `vulnerabilities[].identifiers[].url` | URL to identifier's documentation. Optional. | ## Secret detection @@ -363,3 +363,8 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index a22b285b114..0ed9bf3f518 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -6,7 +6,7 @@ Consult the [Asciidoctor User Manual](https://asciidoctor.org/docs/user-manual) ## Syntax Here's a brief reference of the most commonly used AsciiDoc syntax. -You can find the full documentation for the AsciiDoc syntax at https://asciidoctor.org/docs. +You can find the full documentation for the AsciiDoc syntax at <https://asciidoctor.org/docs>. ### Paragraphs diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index b520c4fb579..2246ea8ed5a 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -111,7 +111,7 @@ file. [Ingress](https://kubernetes.github.io/ingress-nginx/) can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto -DevOps] or deploy your own web apps. +DevOps](../../topics/autodevops/index.md) or deploy your own web apps. NOTE: **Note:** The @@ -251,6 +251,8 @@ The applications below can be uninstalled. | Application | GitLab version | Notes | | ----------- | -------------- | ----- | +| Ingress | 12.1+ | The associated load balancer and IP will be deleted and cannot be restored. Furthermore, it can only be uninstalled if JupyterHub is not installed. | +| JupyterHub | 12.1+ | All data not committed to GitLab will be deleted and cannot be restored. | | Prometheus | 11.11+ | All data will be deleted and cannot be restored. | To uninstall an application: @@ -287,4 +289,3 @@ To avoid installation errors: kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem diff a.pem b.pem ``` - diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 2fb6cec55fa..886fb6e6f55 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -352,3 +352,13 @@ High Performance TCP/HTTP Load Balancer: [unicorn-worker-killer]: https://rubygems.org/gems/unicorn-worker-killer "unicorn-worker-killer" [4010]: https://gitlab.com/gitlab-com/infrastructure/issues/4010 "Find a good value for maximum timeout for Shared Runners" [4070]: https://gitlab.com/gitlab-com/infrastructure/issues/4070 "Configure per-runner timeout for shared-runners-manager-X on GitLab.com" + +## Other admin area settings + +This area highlights other noteworthy admin area settings on GitLab.com that differ from default settings. This list is not exhaustive. + +NOTE: **Note:** +From July 2019, the `Internal` visibility setting is disabled for new projects, groups, +and snippets on GitLab.com. Existing projects, groups, and snippets using the `Internal` +visibility setting keep this setting. You can read more about the change in the +[relevant issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/12388). diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 26d764fa2cf..8d4ffd93f59 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -138,14 +138,6 @@ The result will then be: - The Staging cluster will be used for the `deploy to staging` job. - The Production cluster will be used for the `deploy to production` job. -## Unavailable features - -The following features are not currently available for group-level clusters: - -1. Terminals (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55487)). -1. Pod logs (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55488)). -1. Deployment boards (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55489)). - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 7e6cb24a51e..a555b7723df 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -21,9 +21,9 @@ page. ## Use cases - Analyze your team's contributions over a period of time, and offer a bonus for the top -contributors. + contributors. - Identify opportunities for improvement with group members who may benefit from additional -support. + support. ## Using Contribution Analytics @@ -54,13 +54,13 @@ Select the desired period from the calendar dropdown. Contributions per group member are also presented in tabular format. Click a column header to sort the table by that column: -* Member name -* Number of pushed events -* Number of opened issues -* Number of closed issues -* Number of opened MRs -* Number of accepted MRs -* Number of total contributions +- Member name +- Number of pushed events +- Number of opened issues +- Number of closed issues +- Number of opened MRs +- Number of accepted MRs +- Number of total contributions  diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 2e4106f55e5..f53c1dd95d7 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -202,8 +202,8 @@ You may also consult the [group permissions table][permissions]. ## Thread - Comments: collaborate on that epic by posting comments in its thread. -These text fields also fully support -[GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). + These text fields also fully support + [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). ## Comment, or start a discussion @@ -216,7 +216,7 @@ Once you wrote your comment, you can either: - You can [award an emoji](../../award_emojis.md) to that epic or its comments. -## Notifications +## Notifications - [Receive notifications](../../../workflow/notifications.md) for epic events. diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 4fde45da6c4..7240b8e118b 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -41,12 +41,13 @@ You can create groups for numerous reasons. To name a couple: - Make it easier to `@mention` all of your team at once in issues and merge requests by creating a group and including the appropriate members. For example, you could create a group for your company members, and create a [subgroup](subgroups/index.md) for each individual team. Let's say you create a group called `company-team`, and you create subgroups in this group for the individual teams `backend-team`, `frontend-team`, and `production-team`. - - When you start a new implementation from an issue, you add a comment: - _"`@company-team`, let's do it! `@company-team/backend-team` you're good to go!"_ - - When your backend team needs help from frontend, they add a comment: - _"`@company-team/frontend-team` could you help us here please?"_ - - When the frontend team completes their implementation, they comment: - _"`@company-team/backend-team`, it's done! Let's ship it `@company-team/production-team`!"_ + +- When you start a new implementation from an issue, you add a comment: + _"`@company-team`, let's do it! `@company-team/backend-team` you're good to go!"_ +- When your backend team needs help from frontend, they add a comment: + _"`@company-team/frontend-team` could you help us here please?"_ +- When the frontend team completes their implementation, they comment: + _"`@company-team/backend-team`, it's done! Let's ship it `@company-team/production-team`!"_ ## Namespaces @@ -217,6 +218,8 @@ Get an overview of the vulnerabilities of all the projects in a group and its su ## Insights **[ULTIMATE]** +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + Configure the Insights that matter for your groups or projects, allowing users to explore data such as: diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 1aba5d0986b..e6ba47939b3 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -4,8 +4,7 @@ type: reference, howto # Insights **[ULTIMATE]** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. -> **Generally Available** (GA) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. Configure the Insights that matter for your groups to explore data such as triage hygiene, issues created/closed per a given period, average time for merge diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 96cc523f4ec..5aef463d782 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -24,27 +24,27 @@ The following identity providers are supported: ## Requirements -- [Group SSO](index.md) needs to be configured. +- [Group SSO](index.md) needs to be configured. - The `scim_group` feature flag must be enabled: Run the following commands in a Rails console: - + ```sh # Omnibus GitLab gitlab-rails console - + # Installation from source cd /home/git/gitlab sudo -u git -H bin/rails console RAILS_ENV=production ``` - + To enable SCIM for a group named `group_name`: - + ```ruby group = Group.find_by_full_path('group_name') Feature.enable(:group_scim, group) ``` - + ### GitLab configuration Once [Single sign-on](index.md) has been configured, we can: @@ -53,7 +53,7 @@ Once [Single sign-on](index.md) has been configured, we can: 1. Click on the **Generate a SCIM token** button. 1. Save the token and URL so they can be used in the next step. - + ## SCIM IdP configuration @@ -63,15 +63,15 @@ In the [Single sign-on](index.md) configuration for the group, make sure that the **Name identifier value** (NameID) points to a unique identifier, such as the `user.objectid`. This will match the `extern_uid` used on GitLab. -The GitLab app in Azure needs to be configured following +The GitLab app in Azure needs to be configured following [Azure's SCIM setup](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#getting-started). Note the following: - The `Tenant URL` and `secret token` are the ones retrieved in the -[previous step](#gitlab-configuration). + [previous step](#gitlab-configuration). - Should there be any problems with the availability of GitLab or similar -errors, the notification email set will get those. + errors, the notification email set will get those. - For mappings, we will only leave `Synchronize Azure Active Directory Users to AppName` enabled. You can then test the connection clicking on `Test Connection`. @@ -79,14 +79,14 @@ You can then test the connection clicking on `Test Connection`. ### Synchronize Azure Active Directory users 1. Click on `Synchronize Azure Active Directory Users to AppName`, to configure -the attribute mapping. + the attribute mapping. 1. Select the unique identifier (in the example `objectId`) as the `id` and `externalId`, -and enable the `Create`, `Update`, and `Delete` actions. + and enable the `Create`, `Update`, and `Delete` actions. 1. Map the `userPricipalName` to `emails[type eq "work"].value` and `mailNickname` to -`userName`. + `userName`. Example configuration: - +  1. Click on **Show advanced options > Edit attribute list for AppName**. @@ -95,11 +95,11 @@ and enable the `Create`, `Update`, and `Delete` actions. NOTE: **Note:** `username` should neither be primary nor required as we don't support that field on GitLab SCIM yet. - +  1. Save all the screens and, in the **Provisioning** step, set -the `Provisioning Status` to `ON`. + the `Provisioning Status` to `ON`. NOTE: **Note:** You can control what is actually synced by selecting the `Scope`. For example, diff --git a/doc/user/img/color_inline_colorchip_render_gfm.png b/doc/user/img/color_inline_colorchip_render_gfm.png Binary files differdeleted file mode 100644 index 6f93dbeeb10..00000000000 --- a/doc/user/img/color_inline_colorchip_render_gfm.png +++ /dev/null diff --git a/doc/user/img/markdown_inline_diffs_tags_rendered.png b/doc/user/img/markdown_inline_diffs_tags_rendered.png Binary files differdeleted file mode 100644 index 4279a20b5a0..00000000000 --- a/doc/user/img/markdown_inline_diffs_tags_rendered.png +++ /dev/null diff --git a/doc/user/img/math_inline_sup_render_gfm.png b/doc/user/img/math_inline_sup_render_gfm.png Binary files differdeleted file mode 100644 index 3ee2abb14df..00000000000 --- a/doc/user/img/math_inline_sup_render_gfm.png +++ /dev/null diff --git a/doc/user/img/task_list_ordered_render_gfm.png b/doc/user/img/task_list_ordered_render_gfm.png Binary files differdeleted file mode 100644 index 98ec791e958..00000000000 --- a/doc/user/img/task_list_ordered_render_gfm.png +++ /dev/null diff --git a/doc/user/index.md b/doc/user/index.md index 1fc4e4c43cf..899026a801f 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -66,7 +66,7 @@ With GitLab Enterprise Edition, you can also: - Leverage continuous delivery method with [Canary Deployments](project/canary_deployments.md). - Scan your code for vulnerabilities and [display them in merge requests](application_security/sast/index.md). -You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, JIRA, and a lot more. +You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, Jira, and a lot more. ## Projects @@ -153,7 +153,7 @@ you have quick access to. You can also gather feedback on them through ## Integrations [Integrate GitLab](../integration/README.md) with your preferred tool, -such as Trello, JIRA, etc. +such as Trello, Jira, etc. ## Webhooks diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 31c8093ced7..eaae9964367 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -1,15 +1,20 @@ # GitLab Markdown -This markdown guide is **valid for GitLab's system markdown entries and files**. -It is not valid for the [GitLab documentation website](https://docs.gitlab.com) -nor [GitLab's main website](https://about.gitlab.com), as they both use -[Kramdown](https://kramdown.gettalong.org) as their markdown engine. -The documentation website uses an extended Kramdown gem, [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown). -Consult the [GitLab Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) for a complete Kramdown reference. +This markdown guide is **valid only for GitLab's internal markdown rendering system for entries and files**. +It is **not** valid for the [GitLab documentation website](https://docs.gitlab.com) +or [GitLab's main website](https://about.gitlab.com), as they both use +[Kramdown](https://kramdown.gettalong.org) as their markdown engine. The documentation +website uses an extended Kramdown gem, [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown). +Consult the [GitLab Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) +for a complete Kramdown reference. + +NOTE: **Note:** We encourage you to view this document as [rendered by GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md). ## GitLab Flavored Markdown (GFM) -GitLab uses "GitLab Flavored Markdown" (GFM). It extends the [CommonMark specification][commonmark-spec] (which is based on standard Markdown) in a few significant ways to add additional useful functionality. It was inspired by [GitHub Flavored Markdown](https://help.github.com/articles/basic-writing-and-formatting-syntax/). +GitLab uses "GitLab Flavored Markdown" (GFM). It extends the [CommonMark specification](https://spec.commonmark.org/current/) +(which is based on standard Markdown) in several ways to add additional useful functionality. +It was inspired by [GitHub Flavored Markdown](https://help.github.com/articles/basic-writing-and-formatting-syntax/). You can use GFM in the following areas: @@ -22,35 +27,29 @@ You can use GFM in the following areas: - Markdown documents inside repositories - Epics **[ULTIMATE]** -You can also use other rich text files in GitLab. You might have to install a -dependency to do so. Please see the [`github-markup` gem readme](https://github.com/gitlabhq/markup#markups) for more information. - -> **Notes:** -> -> We encourage you to view this document as [rendered by GitLab itself](markdown.md). -> -> As of 11.1, GitLab uses the [CommonMark Ruby Library][commonmarker] for Markdown -processing of all new issues, merge requests, comments, and other Markdown content -in the GitLab system. As of 11.3, wiki pages and Markdown files (`.md`) in the -repositories are also processed with CommonMark. As of 11.8, the [Redcarpet -Ruby library][redcarpet] has been removed and all issues/comments, including -those from pre-11.1, are now processed using [CommonMark Ruby -Library][commonmarker]. -> -> The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) -in October 2018. -> -> _Where there are significant differences, we will try to call them out in this document._ +You can also use other rich text files in GitLab. You might have to install a dependency +to do so. Please see the [`gitlab-markup` gem project](https://gitlab.com/gitlab-org/gitlab-markup) +for more information. + +### Transition from Redcarpet to CommonMark -### Transitioning to CommonMark +Since 11.1, GitLab uses the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker) +for Markdown processing of all new issues, merge requests, comments, and other Markdown +content in the GitLab system. Since 11.3, wiki pages and Markdown files (`*.md`) in +repositories are also processed with CommonMark. As of 11.8, the [Redcarpet Ruby library](https://github.com/vmg/redcarpet) +has been removed and all issues and comments, including those from pre-11.1, are now processed +using the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker). -You may have older issues/merge requests or Markdown documents in your -repository that were written using some of the nuances of RedCarpet's version +The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) +in October 2018. + +You may have older issues, merge requests, or Markdown documents in your +repository that were written using some of the nuances of GitLab's RedCarpet version of Markdown. Since CommonMark uses a slightly stricter syntax, these documents -may now display a little strangely since we've transitioned to CommonMark. -Numbered lists with nested lists in particular can be displayed incorrectly. +may now display a little differently since we've transitioned to CommonMark. -It is usually quite easy to fix. In the case of a nested list such as this: +It is usually quite easy to fix. For example, numbered lists with nested lists may +render incorrectly: ```markdown 1. Chocolate @@ -58,7 +57,14 @@ It is usually quite easy to fix. In the case of a nested list such as this: - milk ``` -simply add a space to each nested item: +1. Chocolate + - dark + - milk + +--- + +Simply add a space to each nested item to align the `-` with the first character of +the top list item (`C` in this case): ```markdown 1. Chocolate @@ -66,515 +72,674 @@ simply add a space to each nested item: - milk ``` -In the documentation below, we try to highlight some of the differences. +1. Chocolate + - dark + - milk + +NOTE: **Note:** We will flag any significant differences between Redcarpet and CommonMark + markdown in this document. If you have a large volume of Markdown files, it can be tedious to determine -if they will be displayed correctly or not. You can use the +if they will display correctly or not. You can use the [diff_redcarpet_cmark](https://gitlab.com/digitalmoksha/diff_redcarpet_cmark) -tool (not an officially supported product) to generate a list of files and +tool (not an officially supported product) to generate a list of files, and the differences between how RedCarpet and CommonMark render the files. It can give -you a great idea if anything needs to be changed - many times nothing will need -to changed. +an indication if anything needs to be changed - often nothing will need +to change. + +### GFM extends standard markdown + +GitLab makes full use of the standard (CommonMark) formatting, but also includes additional +functionality useful for GitLab users. + +It makes use of [new markdown features](#new-GFM-markdown-extensions), +not found in standard markdown: + +- [Color "chips" written in HEX, RGB or HSL](#colors) +- [Diagrams and flowcharts using Mermaid](#diagrams-and-flowcharts-using-mermaid) +- [Emoji](#emoji) +- [Front matter](#front-matter) +- [Inline diffs](#inline-diff) +- [Math equations and symbols written in LaTeX](#math) +- [Special GitLab references](#special-gitlab-references) +- [Task Lists](#task-lists) +- [Wiki specific markdown](#wiki-specific-markdown) + +It also has [extended markdown features](#standard-markdown-and-extensions-in-gitlab), without +changing how standard markdown is used: + +| Standard markdown | Extended markdown in GitLab | +| ------------------------------------- | ------------------------- | +| [blockquotes](#blockquotes) | [multiline blockquotes](#multiline-blockquote) | +| [code blocks](#code-spans-and-blocks) | [colored code and syntax highlighting](#colored-code-and-syntax-highlighting) | +| [emphasis](#emphasis) | [multiple underscores in words](#multiple-underscores-in-words-and-mid-word-emphasis) +| [headers](#headers) | [linkable Header IDs](#header-ids-and-links) | +| [images](#images) | [embedded videos](#videos) | +| [linebreaks](#line-breaks) | [more linebreak control](#newlines) | +| [links](#links) | [automatically linking URLs](#url-auto-linking) | + +## New GFM markdown extensions -### Newlines +### Colors -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#newlines +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colors). -GFM honors the markdown specification in how [paragraphs and line breaks are handled][commonmark-spec]. +It is possible to have color written in HEX, RGB or HSL format rendered with a color +indicator. -A paragraph is simply one or more consecutive lines of text, separated by one or -more blank lines. -Line-breaks, or soft returns, are rendered if you end a line with two or more spaces: +Supported formats (named colors are not supported): -<!-- (Do *NOT* remove the two ending whitespaces in the following line.) --> -<!-- (They are needed for the Markdown text to render correctly.) --> - Roses are red [followed by two or more spaces] - Violets are blue +- HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` +- RGB: `` `RGB[A](R, G, B[, A])` `` +- HSL: `` `HSL[A](H, S, L[, A])` `` - Sugar is sweet +Color written inside backticks will be followed by a color "chip": -<!-- (Do *NOT* remove the two ending whitespaces in the following line.) --> -<!-- (They are needed for the Markdown text to render correctly.) --> -Roses are red -Violets are blue +```markdown +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.3)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.3)` +``` -Sugar is sweet +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.3)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.3)` -### Multiple underscores in words +### Diagrams and flowcharts using Mermaid -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiple-underscores-in-words +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15107) in +GitLab 10.3. -It is not reasonable to italicize just _part_ of a word, especially when you're -dealing with code and names that often appear with multiple underscores. -Therefore, GFM ignores multiple underscores in words: +It is possible to generate diagrams and flowcharts from text using [Mermaid](https://mermaidjs.github.io/). +Visit the official page for more details. - perform_complicated_task +In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block: - do_this_and_do_that_and_another_thing +~~~ +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` +~~~ -perform_complicated_task +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` -do_this_and_do_that_and_another_thing +### Emoji -### URL auto-linking +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji). -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#url-auto-linking +```md +Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: -GFM will autolink almost any URL you copy and paste into your text: +:zap: You can use emoji anywhere GFM is supported. :v: - * https://www.google.com - * https://google.com/ - * ftp://ftp.us.debian.org/debian/ - * smb://foo/bar/baz - * irc://irc.freenode.net/gitlab - * http://localhost:3000 +You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People will :heart: you for that. -* https://www.google.com -* https://google.com/ -* ftp://ftp.us.debian.org/debian/ -* <a href="smb://foo/bar/baz">smb://foo/bar/baz</a> -* <a href="irc://irc.freenode.net/gitlab">irc://irc.freenode.net/gitlab</a> -* http://localhost:3000 +If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up one of the supported codes. -### Multiline blockquote +Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: +``` -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiline-blockquote +Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: -On top of standard Markdown [blockquotes](#blockquotes), which require prepending `>` to quoted lines, -GFM supports multiline blockquotes fenced by <code>>>></code>: +<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0"> -``` ->>> -If you paste a message from somewhere else +You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. -that +If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. -spans +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> -multiple lines, +> **Note:** The emoji example above uses hard-coded images for this documentation. The emoji, +when rendered within GitLab, may appear different depending on the OS and browser used. -you can quote that without having to manually prepend `>` to every line! ->>> -``` +Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. -<blockquote dir="auto"> -<p>If you paste a message from somewhere else</p> -<p>that</p> -<p>spans</p> -<p>multiple lines,</p> -<p>you can quote that without having to manually prepend <code>></code> to every line!</p> -</blockquote> +NOTE: **Note:** On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) +to get full native emoji support. Ubuntu 18.04 (like many modern Linux distros) has +this font installed by default. -### Code and syntax highlighting +### Front matter -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#code-and-syntax-highlighting +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) + in GitLab 11.6. -_GitLab uses the [Rouge Ruby library][rouge] for syntax highlighting. For a -list of supported languages visit the Rouge website._ +Front matter is metadata included at the beginning of a markdown document, preceding +its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/), +[Hugo](https://gohugo.io/content-management/front-matter/), and many other applications. -Blocks of code are either fenced by lines with three back-ticks <code>```</code>, -or are indented with four spaces. Only the fenced code blocks support syntax -highlighting: +When you view a Markdown file rendered by GitLab, any front matter is displayed as-is, +in a box at the top of the document, before the rendered HTML content. To view an example, +you can toggle between the source and rendered version of a [GitLab documentation file](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/README.md). -``` -Inline `code` has `back-ticks around` it. -``` +In GitLab, front matter is only used in Markdown files and wiki pages, not the other +places where Markdown formatting is supported. It must be at the very top of the document, +and must be between delimiters, as explained below. -Inline `code` has `back-ticks around` it. +The following delimeters are supported: -Example: +- YAML (`---`): - ```javascript - var s = "JavaScript syntax highlighting"; - alert(s); - ``` + ~~~yaml + --- + title: About Front Matter + example: + language: yaml + --- + ~~~ - ```python - def function(): - #indenting works just fine in the fenced code block - s = "Python syntax highlighting" - print s - ``` +- TOML (`+++`): - ```ruby - require 'redcarpet' - markdown = Redcarpet.new("Hello World!") - puts markdown.to_html - ``` + ~~~toml + +++ + title = "About Front Matter" + [example] + language = "toml" + +++ + ~~~ - ``` - No language indicated, so no syntax highlighting. - s = "There is no highlighting for this." - But let's throw in a <b>tag</b>. - ``` +- JSON (`;;;`): -becomes: + ~~~json + ;;; + { + "title": "About Front Matter" + "example": { + "language": "json" + } + } + ;;; + ~~~ -```javascript -var s = "JavaScript syntax highlighting"; -alert(s); -``` +Other languages are supported by adding a specifier to any of the existing +delimiters. For example: -```python -def function(): - #indenting works just fine in the fenced code block - s = "Python syntax highlighting" - print s +```php +---php +$title = "About Front Matter"; +$example = array( + 'language' => "php", +); +--- ``` -```ruby -require 'redcarpet' -markdown = Redcarpet.new("Hello World!") -puts markdown.to_html -``` +### Inline diff -``` -No language indicated, so no syntax highlighting. -s = "There is no highlighting for this." -But let's throw in a <b>tag</b>. -``` +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-diff). -### Inline diff +With inline diff tags you can display `{+ additions +}` or `[- deletions -]`. + +The wrapping tags can be either curly braces or square brackets: -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-diff +```markdown +- {+ addition 1 +} +- [+ addition 2 +] +- {- deletion 3 -} +- [- deletion 4 -] +``` -With inline diffs tags you can display {+ additions +} or [- deletions -]. +- {+ addition 1 +} +- [+ addition 2 +] +- {- deletion 3 -} +- [- deletion 4 -] -The wrapping tags can be either curly braces or square brackets. +--- -Examples: +However the wrapping tags cannot be mixed: +```markdown +- {+ addition +] +- [+ addition +} +- {- deletion -] +- [- deletion -} ``` -- {+ additions +} -- [+ additions +] -- {- deletions -} -- [- deletions -] -``` -becomes: +### Math + +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#math). + +It is possible to have math written with LaTeX syntax rendered using [KaTeX](https://github.com/Khan/KaTeX). + +Math written between dollar signs `$` will be rendered inline with the text. Math written +inside a [code block](#code-spans-and-blocks) with the language declared as `math`, will be rendered +on a separate line: - +~~~ +This math is inline $`a^2+b^2=c^2`$. -However the wrapping tags cannot be mixed as such: +This is on a separate line +```math +a^2+b^2=c^2 ``` -- {+ additions +] -- [+ additions +} -- {- deletions -] -- [- deletions -} +~~~ + +This math is inline $`a^2+b^2=c^2`$. + +This is on a separate line + +```math +a^2+b^2=c^2 ``` -### Emoji +_Be advised that KaTeX only supports a [subset](https://katex.org/docs/supported.html) of LaTeX._ -```md -Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: +NOTE: **Note:** This also works for the asciidoctor `:stem: latexmath`. For details see +the [asciidoctor user manual](http://asciidoctor.org/docs/user-manual/#activating-stem-support). -:zap: You can use emoji anywhere GFM is supported. :v: +### Special GitLab references -You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People will :heart: you for that. +GFM recognizes special GitLab related references. For example, you can easily reference +an issue, a commit, a team member or even the whole team within a project. GFM will turn +that reference into a link so you can navigate between them easily. -If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up one of the supported codes. +Additionally, GFM recognizes certain cross-project references, and also has a shorthand +version to reference other projects from the same namespace. -Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: +GFM will recognize the following: -Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. +| references | input | cross-project reference | shortcut within same namespace | +| :------------------------------ | :------------------------- | :-------------------------------------- | :----------------------------- | +| specific user | `@user_name` | | | +| specific group | `@group_name` | | | +| entire team | `@all` | | | +| project | `namespace/project>` | | | +| issue | ``#123`` | `namespace/project#123` | `project#123` | +| merge request | `!123` | `namespace/project!123` | `project!123` | +| snippet | `$123` | `namespace/project$123` | `project$123` | +| epic **[ULTIMATE]** | `&123` | `group1/subgroup&123` | | +| label by ID | `~123` | `namespace/project~123` | `project~123` | +| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | +| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | +| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | +| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | +| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | +| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | +| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | +| repository file references | `[README](doc/README)` | | | +| repository file line references | `[README](doc/README#L13)` | | | -On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) to get full native emoji support. +### Task lists + +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#task-lists). + +You can add task lists anywhere markdown is supported, but you can only "click" to +toggle the boxes if they are in issues, merge requests, or comments. In other places +you must edit the markdown manually to change the status by adding or removing the `x`. -Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. +To create a task list, add a specially-formatted Markdown list. You can use either +unordered or ordered lists: + +```markdown +- [x] Completed task +- [ ] Incomplete task + - [ ] Sub-task 1 + - [x] Sub-task 2 + - [ ] Sub-task 3 +1. [x] Completed task +1. [ ] Incomplete task + 1. [ ] Sub-task 1 + 1. [x] Sub-task 2 ``` -Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: +- [x] Completed task +- [ ] Incomplete task + - [ ] Sub-task 1 + - [x] Sub-task 2 + - [ ] Sub-task 3 +1. [x] Completed task +1. [ ] Incomplete task + 1. [ ] Sub-task 1 + 1. [x] Sub-task 2 -<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0"> +### Wiki-specific Markdown -You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. +The following examples show how links inside wikis behave. -If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. +#### Wiki - Direct page link -Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> +A link which just includes the slug for a page will point to that page, +_at the base level of the wiki_. -Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. +This snippet would link to a `documentation` page at the root of your wiki: -On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) to get full native emoji support. +```markdown +[Link to Documentation](documentation) +``` -Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. +#### Wiki - Direct file link -### Special GitLab references +Links with a file extension point to that file, _relative to the current page_. -GFM recognizes special references. +If the snippet below was placed on a page at `<your_wiki>/documentation/related`, +it would link to `<your_wiki>/documentation/file.md`: -You can easily reference e.g. an issue, a commit, a team member or even the whole team within a project. +```markdown +[Link to File](file.md) +``` -GFM will turn that reference into a link so you can navigate between them easily. +#### Wiki - Hierarchical link -GFM will recognize the following: +A link can be constructed relative to the current wiki page using `./<page>`, +`../<page>`, etc. -| input | references | -|:---------------------------|:--------------------------------| -| `@user_name` | specific user | -| `@group_name` | specific group | -| `@all` | entire team | -| `namespace/project>` | project | -| `#12345` | issue | -| `!123` | merge request | -| `$123` | snippet | -| `&123` | epic **[ULTIMATE]** | -| `~123` | label by ID | -| `~bug` | one-word label by name | -| `~"feature request"` | multi-word label by name | -| `%123` | project milestone by ID | -| `%v1.23` | one-word milestone by name | -| `%"release candidate"` | multi-word milestone by name | -| `9ba12248` | specific commit | -| `9ba12248...b19a04f5` | commit range comparison | -| `[README](doc/README)` | repository file references | -| `[README](doc/README#L13)` | repository file line references | - -GFM also recognizes certain cross-project references: - -| input | references | -|:----------------------------------------|:------------------------| -| `namespace/project#123` | issue | -| `namespace/project!123` | merge request | -| `namespace/project%123` | project milestone | -| `namespace/project$123` | snippet | -| `namespace/project@9ba12248` | specific commit | -| `group1/subgroup&123` | epic **[ULTIMATE]** | -| `namespace/project@9ba12248...b19a04f5` | commit range comparison | -| `namespace/project~"Some label"` | issues with given label | - -It also has a shorthand version to reference other projects from the same namespace: - -| input | references | -|:------------------------------|:------------------------| -| `project#123` | issue | -| `project!123` | merge request | -| `project%123` | project milestone | -| `project$123` | snippet | -| `project@9ba12248` | specific commit | -| `project@9ba12248...b19a04f5` | commit range comparison | -| `project~"Some label"` | issues with given label | +If this snippet was placed on a page at `<your_wiki>/documentation/main`, +it would link to `<your_wiki>/documentation/related`: -### Task lists +```markdown +[Link to Related Page](./related) +``` + +If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, +it would link to `<your_wiki>/documentation/main`: -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#task-lists +```markdown +[Link to Related Page](../main) +``` -You can add task lists to issues, merge requests and comments. To create a task list, add a specially-formatted Markdown list, like so: +If this snippet was placed on a page at `<your_wiki>/documentation/main`, +it would link to `<your_wiki>/documentation/related.md`: +```markdown +[Link to Related Page](./related.md) ``` -- [x] Completed task -- [ ] Incomplete task - - [ ] Sub-task 1 - - [x] Sub-task 2 - - [ ] Sub-task 3 + +If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, +it would link to `<your_wiki>/documentation/main.md`: + +```markdown +[Link to Related Page](../main.md) ``` - +#### Wiki - Root link + +A link starting with a `/` is relative to the wiki root. -Tasks formatted as ordered lists are supported as well: +This snippet links to `<wiki_root>/documentation`: +```markdown +[Link to Related Page](/documentation) ``` -1. [x] Completed task -1. [ ] Incomplete task - 1. [ ] Sub-task 1 - 1. [x] Sub-task 2 + +This snippet links to `<wiki_root>/miscellaneous.md`: + +```markdown +[Link to Related Page](/miscellaneous.md) ``` - +## Standard markdown and extensions in GitLab -Task lists can only be created in descriptions, not in titles. Task item state can be managed by editing the description's Markdown or by toggling the rendered check boxes. +All standard markdown formatting should work as expected within GitLab. Some standard +functionality is extended with additional features, without affecting the standard usage. +If a functionality is extended, the new option will be listed as a sub-section. -### Videos +### Blockquotes -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#videos +Blockquotes are an easy way to highlight information, such as a side-note. It is generated +by starting the lines of the blockquote with `>`: -Image tags with a video extension are automatically converted to a video player. +```markdown +> Blockquotes are very handy to emulate reply text. +> This line is part of the same quote. -The valid video extensions are `.mp4`, `.m4v`, `.mov`, `.webm`, and `.ogv`. +Quote break. - Here's a sample video: +> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. +``` -  +> Blockquotes are very handy to emulate reply text. +> This line is part of the same quote. -Here's a sample video: +Quote break. -<div class="video-container"> - <video src="img/markdown_video.mp4" width="400" controls="true" data-setup="{}" data-title="Sample Video"></video> - <p><a href="img/markdown_video.mp4" target="_blank" rel="noopener noreferrer" title="Download 'Sample Video'">Sample Video</a></p> -</div> +> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. -### Math +#### Multiline blockquote -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#math +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiline-blockquote). -It is possible to have math written with the LaTeX syntax rendered using [KaTeX][katex]. +GFM extends the standard markdown standard by also supporting multiline blockquotes +fenced by `>>>`: -Math written inside ```$``$``` will be rendered inline with the text. +``` +>>> +If you paste a message from somewhere else -Math written inside triple back quotes, with the language declared as `math`, will be rendered on a separate line. +that spans multiple lines, -Example: +you can quote that without having to manually prepend `>` to every line! +>>> +``` + +>>> +If you paste a message from somewhere else - This math is inline $`a^2+b^2=c^2`$. +that spans multiple lines, - This is on a separate line - ```math - a^2+b^2=c^2 - ``` +you can quote that without having to manually prepend `>` to every line! +>>> -Becomes: +### Code spans and blocks -This math is inline . +You can easily highlight anything that should be viewed as code and not simple text. -This is on a separate line +Simple inline code is easily highlighted with single backticks `` ` ``: -<img src="./img/math_inline_sup_render_gfm.png" > +```markdown +Inline `code` has `back-ticks around` it. +``` -_Be advised that KaTeX only supports a [subset][katex-subset] of LaTeX._ +Inline `code` has `back-ticks around` it. ->**Note:** -This also works for the asciidoctor `:stem: latexmath`. For details see the [asciidoctor user manual][asciidoctor-manual]. +--- -### Colors +Similarly, a whole block of code can be fenced with triple backticks ```` ``` ````, +triple tildes (`~~~`), or indended 4 or more spaces to achieve a similar effect for +a larger body of code. test. -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colors +~~~ +``` +def function(): + #indenting works just fine in the fenced code block + s = "Python code" + print s +``` -It is possible to have color written in HEX, RGB or HSL format rendered with a color indicator. + Using 4 spaces + is like using + 3-backtick fences. +~~~ -Color written inside backticks will be followed by a color "chip". +``` +~~~ +Tildes are OK too. +~~~ +``` -Examples: +The three examples above render as: - `#F00` - `#F00A` - `#FF0000` - `#FF0000AA` - `RGB(0,255,0)` - `RGB(0%,100%,0%)` - `RGBA(0,255,0,0.7)` - `HSL(540,70%,50%)` - `HSLA(540,70%,50%,0.7)` +``` +def function(): + #indenting works just fine in the fenced code block + s = "Python code" + print s +``` -Becomes: + Using 4 spaces + is like using + 3-backtick fences. - +~~~ +Tildes are OK too. +~~~ -#### Supported formats: +#### Colored code and syntax highlighting -* HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` -* RGB: `` `RGB[A](R, G, B[, A])` `` -* HSL: `` `HSL[A](H, S, L[, A])` `` +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colored-code-and-syntax-highlighting). -### Mermaid +GitLab uses the [Rouge Ruby library](http://rouge.jneen.net/) for more colorful syntax +highlighting in code blocks. For a list of supported languages visit the +[Rouge project wiki](https://github.com/jneen/rouge/wiki/List-of-supported-languages-and-lexers). +Syntax highlighting is only supported in code blocks, it is not possible to highlight +code when it is inline. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15107) in -GitLab 10.3. -> -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#mermaid +Blocks of code are fenced by lines with three back-ticks ```` ``` ```` or three tildes `~~~`, and have +the language identified at the end of the first fence: -It is possible to generate diagrams and flowcharts from text using [Mermaid](https://mermaidjs.github.io/). +~~~ +```javascript +var s = "JavaScript syntax highlighting"; +alert(s); +``` -In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block. +```python +def function(): + #indenting works just fine in the fenced code block + s = "Python syntax highlighting" + print s +``` -Example: +```ruby +require 'redcarpet' +markdown = Redcarpet.new("Hello World!") +puts markdown.to_html +``` - ```mermaid - graph TD; - A-->B; - A-->C; - B-->D; - C-->D; - ``` +``` +No language indicated, so no syntax highlighting. +s = "There is no highlighting for this." +But let's throw in a <b>tag</b>. +``` +~~~ -Becomes: +The four examples above render as: -```mermaid -graph TD; - A-->B; - A-->C; - B-->D; - C-->D; +```javascript +var s = "JavaScript syntax highlighting"; +alert(s); ``` -For details see the [Mermaid official page](https://mermaidjs.github.io/). +```python +def function(): + #indenting works just fine in the fenced code block + s = "Python syntax highlighting" + print s +``` -### Front matter +```ruby +require 'redcarpet' +markdown = Redcarpet.new("Hello World!") +puts markdown.to_html +``` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) - in GitLab 11.6. +``` +No language indicated, so no syntax highlighting. +s = "There is no highlighting for this." +But let's throw in a <b>tag</b>. +``` -Front matter is metadata included at the beginning of a markdown document, preceding -its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/) and [Hugo](https://gohugo.io/content-management/front-matter/), -and many other applications. +### Emphasis -In GitLab, front matter is only used in Markdown files and wiki pages, not the other places where Markdown formatting is supported. -When you view a Markdown file rendered by GitLab, any front matter is displayed as-is, in a box at the top of the document, before the rendered HTML content. -To view an example, you can toggle between the source and rendered version of a [GitLab documentation file](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/README.md). +There are multiple ways to emphasize text in markdown. You can italicize, bold, strikethrough, +as well as combine these emphasis styles together. -The following delimeters are supported: +Examples: -- YAML (`---`): +```markdown +Emphasis, aka italics, with *asterisks* or _underscores_. - ``` - --- - title: About Front Matter - example: - language: yaml - --- - ``` +Strong emphasis, aka bold, with double **asterisks** or __underscores__. -- TOML (`+++`): +Combined emphasis with **asterisks and _underscores_**. - ``` - +++ - title = "About Front Matter" - [example] - language = "toml" - +++ - ``` +Strikethrough uses two tildes. ~~Scratch this.~~ +``` -- JSON (`;;;`): +Emphasis, aka italics, with *asterisks* or _underscores_. - ``` - ;;; - { - "title": "About Front Matter" - "example": { - "language": "json" - } - } - ;;; - ``` +Strong emphasis, aka bold, with double **asterisks** or __underscores__. -Other languages are supported by adding a specifier to any of the existing -delimiters. For example: +Combined emphasis with **asterisks and _underscores_**. + +Strikethrough uses two tildes. ~~Scratch this.~~ + +NOTE: **Note:** Strikethrough is not part of the core Markdown standard, but is part of GFM. + +#### Multiple underscores in words and mid-word emphasis + +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiple-underscores-in-words). +It is not usually useful to italicize just _part_ of a word, especially when you're +dealing with code and names that often appear with multiple underscores. As a result, +GFM extends the standard markdown standard by ignoring multiple underlines in words, +to allow better rendering of markdown documents discussing code: + +```md +perform_complicated_task + +do_this_and_do_that_and_another_thing + +but_emphasis is_desired _here_ ``` ----php -$title = "About Front Matter"; -$example = array( - 'language' => "php", -); + +perform_complicated_task + +do_this_and_do_that_and_another_thing + +but_emphasis is_desired _here_ + --- + +If you wish to emphasize only a part of a word, it can still be done with asterisks: + +```md +perform*complicated*task + +do*this*and*do*that*and*another thing ``` -## Standard Markdown +perform*complicated*task -### Headers +do*this*and*do*that*and*another thing + +### Footnotes + +Footnotes add a link to a note rendered at the end of a markdown file: +```markdown +You can add footnotes to your text as follows.[^1] + +[^1]: This is my awesome footnote (later in file). ``` + +You can add footnotes to your text as follows.[^1] + +[^1]: This is my awesome footnote (later in file). + +### Headers + +```markdown # H1 ## H2 ### H3 @@ -593,9 +758,11 @@ Alt-H2 #### Header IDs and links -All Markdown-rendered headers automatically get IDs, which can be linked to, except in comments. +GFM extends the standard markdown standard so that all Markdown-rendered headers automatically +get IDs, which can be linked to, except in comments. -On hover, a link to those IDs becomes visible to make it easier to copy the link to the header to use it somewhere else. +On hover, a link to those IDs becomes visible to make it easier to copy the link to +the header to use it somewhere else. The IDs are generated from the content of the header according to the following rules: @@ -606,7 +773,7 @@ The IDs are generated from the content of the header according to the following 1. If a header with the same ID has already been generated, a unique incrementing number is appended, starting at 1. -For example: +Example: ``` # This header has spaces in it @@ -626,216 +793,165 @@ Would generate the following link IDs: 1. `this-header-has-spaces-in-it-2` 1. `this-header-has-3-5-in-it-and-parentheses` -Note that the Emoji processing happens before the header IDs are generated, so the Emoji is converted to an image which then gets removed from the ID. +Note that the Emoji processing happens before the header IDs are generated, so the +Emoji is converted to an image which is then removed from the ID. -### Emphasis - -Examples: - -``` -Emphasis, aka italics, with *asterisks* or _underscores_. - -Strong emphasis, aka bold, with **asterisks** or __underscores__. - -Combined emphasis with **asterisks and _underscores_**. - -Strikethrough uses two tildes. ~~Scratch this.~~ -``` - -Becomes: - -Emphasis, aka italics, with *asterisks* or _underscores_. +### Horizontal Rule -Strong emphasis, aka bold, with **asterisks** or __underscores__. +It's very simple to create a horizontal rule, by using three or more hyphens, asterisks, +or underscores: -Combined emphasis with **asterisks and _underscores_**. +```markdown +Three or more hyphens, -Strikethrough uses two tildes. ~~Scratch this.~~ +--- -### Lists +asterisks, -Examples: +*** -``` -1. First ordered list item -2. Another item - * Unordered sub-list. -1. Actual numbers don't matter, just that it's a number - 1. Ordered sub-list -4. And another item. +or underscores -* Unordered list can use asterisks -- Or minuses -+ Or pluses +___ ``` -Becomes: - -1. First ordered list item -2. Another item - * Unordered sub-list. -1. Actual numbers don't matter, just that it's a number - 1. Ordered sub-list -4. And another item. - -* Unordered list can use asterisks -- Or minuses -+ Or pluses - -If a list item contains multiple paragraphs, -each subsequent paragraph should be indented to the same level as the start of the list item text +Three or more hyphens, -Example: +--- -``` -1. First ordered list item +asterisks, - Second paragraph of first item. - -2. Another item -``` +*** -Becomes: +or underscores -1. First ordered list item +___ - Paragraph of first item. +### Images -2. Another item +Examples: -If the paragraph of the first item is not indented with the proper number of spaces, -the paragraph will appear outside the list, instead of properly indented under the list item. +```markdown +Inline-style (hover to see title text): -Example: + -``` -1. First ordered list item +Reference-style (hover to see title text): - Paragraph of first item. +![alt text1][logo] -2. Another item +[logo]: img/markdown_logo.png "Title Text" ``` -Becomes: +Inline-style (hover to see title text): -1. First ordered list item + - Paragraph of first item. +Reference-style (hover to see title text): -2. Another item - -### Links +![alt text][logo] -There are two ways to create links, inline-style and reference-style. +[logo]: img/markdown_logo.png "Title Text" -```markdown -[I'm an inline-style link](https://www.google.com) -[I'm a link to a repository file in the same directory](index.md) -[I am an absolute reference within the repository](/doc/user/index.md) -[I'm a relative link to the Milestones page](../README.md) +#### Videos -[I link to a section on a different markdown page, using a header ID](index.md#overview) -[I link to a different section on the same page, using the header ID](#header-ids-and-links) +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#videos). -[I'm a reference-style link][Arbitrary case-insensitive reference text] -[You can use numbers for reference-style link definitions][1] -Or leave it empty and use the [link text itself][] +Image tags that link to files with a video extension are automatically converted to +a video player. The valid video extensions are `.mp4`, `.m4v`, `.mov`, `.webm`, and `.ogv`: -Some text to show that the reference links can follow later. +```md +Here's a sample video: -[arbitrary case-insensitive reference text]: https://www.mozilla.org -[1]: http://slashdot.org -[link text itself]: https://www.reddit.com + ``` ->**Note:** -Relative links do not allow referencing project files in a wiki page or wiki -page in a project file. The reason for this is that, in GitLab, wiki is always -a separate Git repository. For example, `[I'm a reference-style link](style)` -will point the link to `wikis/style` when the link is inside of a wiki markdown file. - -### Images +Here's a sample video: -Examples: + - Here's our logo (hover to see the title text): +### Inline HTML - Inline-style: -  +> To see the markdown rendered within HTML in the second example, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-html). - Reference-style: - ![alt text1][logo] +You can also use raw HTML in your Markdown, and it'll usually work pretty well. - [logo]: img/markdown_logo.png +See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubydoc.info/gems/html-pipeline/1.11.0/HTML/Pipeline/SanitizationFilter#WHITELIST-constant) +class for the list of allowed HTML tags and attributes. In addition to the default +`SanitizationFilter` whitelist, GitLab allows `span`, `abbr`, `details` and `summary` elements. -Becomes: +```html +<dl> + <dt>Definition list</dt> + <dd>Is something people use sometimes.</dd> -Here's our logo: + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML <em>tags</em> will <b>always</b> work.</dd> +</dl> +``` -Inline-style: +<dl> + <dt>Definition list</dt> + <dd>Is something people use sometimes.</dd> - + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML <em>tags</em> will <b>always</b> work.</dd> +</dl> -Reference-style: +--- -![alt text][logo] +It is still possible to use markdown inside HTML tags, but only if the lines containing markdown +are separated into their own lines: -[logo]: img/markdown_logo.png +```html +<dl> + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML tags will always work.</dd> -### Blockquotes + <dt>Markdown in HTML</dt> + <dd> -Examples: + Does *not* work **very** well. HTML tags will always work. + </dd> +</dl> ``` -> Blockquotes are very handy in email to emulate reply text. -> This line is part of the same quote. -Quote break. +<!-- Note: The example below uses HTML to force correct rendering on docs.gitlab.com, markdown will be fine in GitLab --> -> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. -``` +<dl> + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML tags will always work.</dd> -Becomes: + <dt>Markdown in HTML</dt> + <dd> -> Blockquotes are very handy in email to emulate reply text. -> This line is part of the same quote. + Does <em>not</em> work <b>very</b> well. HTML tags will always work. -Quote break. + </dd> +</dl> -> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. +#### Details and Summary -### Inline HTML +> To see the markdown rendered within HTML in the second example, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#details-and-summary). -You can also use raw HTML in your Markdown, and it'll mostly work pretty well. +Content can be collapsed using HTML's [`<details>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details) +and [`<summary>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/summary) +tags. This is especially useful for collapsing long logs so they take up less screen space. -See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubydoc.info/gems/html-pipeline/1.11.0/HTML/Pipeline/SanitizationFilter#WHITELIST-constant) class for the list of allowed HTML tags and attributes. In addition to the default `SanitizationFilter` whitelist, GitLab allows `span`, `abbr`, `details` and `summary` elements. +```html +<p> +<details> +<summary>Click me to collapse/fold.</summary> -Examples: +These details <em>will</em> remain <strong>hidden</strong> until expanded. -``` -<dl> - <dt>Definition list</dt> - <dd>Is something people use sometimes.</dd> +<pre><code>PASTE LOGS HERE</code></pre> - <dt>Markdown in HTML</dt> - <dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd> -</dl> +</details> +</p> ``` -Becomes: - -<dl> - <dt>Definition list</dt> - <dd>Is something people use sometimes.</dd> - - <dt>Markdown in HTML</dt> - <dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd> -</dl> - -#### Details and Summary - -Content can be collapsed using HTML's [`<details>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details) and [`<summary>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/summary) tags. This is especially useful for collapsing long logs so they take up less screen space. - <p> <details> <summary>Click me to collapse/fold.</summary> @@ -847,7 +963,10 @@ These details <em>will</em> remain <strong>hidden</strong> until expanded. </details> </p> -**Note:** Markdown inside these tags is supported, as long as you have a blank line after the `</summary>` tag and before the `</details>` tag, as shown in the example. +--- + +Markdown inside these tags is supported as well, as long as you have a blank line +after the `</summary>` tag and before the `</details>` tag, as shown in the example: ```html <details> @@ -860,232 +979,302 @@ These details _will_ remain **hidden** until expanded. </details> ``` -### Horizontal Rule +<!-- Note: The example below uses HTML to force correct rendering on docs.gitlab.com, markdown will be fine in GitLab --> -Examples: +<details> +<summary>Click me to collapse/fold.</summary> -``` -Three or more... +These details <em>will</em> remain <b>hidden</b> until expanded. ---- + PASTE LOGS HERE -Hyphens +</details> -*** +### Line Breaks -Asterisks +A line break will be inserted (a new paragraph will start) if the previous text is +ended with two newlines, i.e. you hit <kbd>Enter</kbd> twice in a row. If you only +use one newline (hit <kbd>Enter</kbd> once), the next sentence will be part of the +same paragraph. This is useful if you want to keep long lines from wrapping, and keep +them easily editable: -___ +```markdown +Here's a line for us to start with. -Underscores -``` +This longer line is separated from the one above by two newlines, so it will be a *separate paragraph*. -Becomes: +This line is also a separate paragraph, but... +These lines are only separated by single newlines, +so they *do not break* and just follow the previous lines +in the *same paragraph*. +``` -Three or more... +Here's a line for us to start with. ---- +This longer line is separated from the one above by two newlines, so it will be a *separate paragraph*. -Hyphens +This line is also a separate paragraph, but... +These lines are only separated by single newlines, +so they *do not break* and just follow the previous lines +in the *same paragraph*. -*** +#### Newlines -Asterisks +GFM adheres to the markdown specification in how [paragraphs and line breaks are handled](https://spec.commonmark.org/current/). -___ +A paragraph is simply one or more consecutive lines of text, separated by one or +more blank lines (i.e. two newlines at the end of the first paragraph), as [explained above](#line-breaks). -Underscores +If you need more control over line-breaks or soft returns, you can add a single line-break +by ending a line with a backslash, or two or more spaces. Two newlines in a row will create a new +paragraph, with a blank line in between: -### Line Breaks +```markdown +First paragraph. +Another line in the same paragraph. +A third line in the same paragraph, but this time ending with two spaces.{space}{space} +A new line directly under the first paragraph. + +Second paragraph. +Another line, this time ending with a backslash.\ +A new line due to the previous backslash. +``` -A good way to learn how line breaks work is to experiment and discover -- hit <kbd>Enter</kbd> once (i.e., insert one newline), then hit it twice (i.e., insert two newlines), see what happens. You'll soon learn to get what you want. The "Preview" tab is your friend. +<!-- (Do *NOT* remove the two ending whitespaces in the third line) --> +<!-- (They are needed for the Markdown text to render correctly) --> -Here are some things to try out: +First paragraph. +Another line in the same paragraph. +A third line in the same paragraph, but this time ending with two spaces. +A new line directly under the first paragraph. -Examples: +<!-- (Do *NOT* remove the two ending whitespaces in the second line) --> +<!-- (They are needed for the Markdown text to render correctly on docs.gitlab.com, the backslash works fine inside GitLab itself) --> -``` -Here's a line for us to start with. +Second paragraph. +Another line, this time ending with a backslash. +A new line due to the previous backslash. -This line is separated from the one above by two newlines, so it will be a *separate paragraph*. +### Links -This line is also a separate paragraph, but... -This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. +There are two ways to create links, inline-style and reference-style: -This line is also a separate paragraph, and... -This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) +```md +- This is an [inline-style link](https://www.google.com) +- This is a [link to a repository file in the same directory](index.md) +- This is a [relative link to a readme one directory higher](../README.md) +- This is a [link that also has title text](https://www.google.com "This link takes you to Google!") -spaces. -``` +Using header ID anchors: -Becomes: +- This links to [a section on a different markdown page, using a "#" and the header ID](index.md#overview) +- This links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) -Here's a line for us to start with. +Using references: -This line is separated from the one above by two newlines, so it will be a *separate paragraph*. +- This is a [reference-style link, see below][Arbitrary case-insensitive reference text] +- You can [use numbers for reference-style link definitions, see below][1] +- Or leave it empty and use the [link text itself][], see below. -This line is also a separate paragraph, but... -This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. +Some text to show that the reference links can follow later. -This line is also a separate paragraph, and... -This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) +[arbitrary case-insensitive reference text]: https://www.mozilla.org +[1]: http://slashdot.org +[link text itself]: https://www.reddit.com +``` -spaces. +- This is an [inline-style link](https://www.google.com) +- This is a [link to a repository file in the same directory](index.md) +- This is a [relative link to a readme one directory higher](../README.md) +- This is a [link that also has title text](https://www.google.com "This link takes you to Google!") -### Tables +Using header ID anchors: -Tables aren't part of the core Markdown spec, but they are part of GFM. +- This links to [a section on a different markdown page, using a "#" and the header ID](index.md#overview) +- This links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) -Example: +Using references: -``` -| header 1 | header 2 | -| -------- | -------- | -| cell 1 | cell 2 | -| cell 3 | cell 4 | -``` +- This is a [reference-style link, see below][Arbitrary case-insensitive reference text] +- You can [use numbers for reference-style link definitions, see below][1] +- Or leave it empty and use the [link text itself][], see below. -Becomes: +Some text to show that the reference links can follow later. -| header 1 | header 2 | -| -------- | -------- | -| cell 1 | cell 2 | -| cell 3 | cell 4 | +[arbitrary case-insensitive reference text]: https://www.mozilla.org +[1]: http://slashdot.org +[link text itself]: https://www.reddit.com -**Note:** The row of dashes between the table header and body must have at least three dashes in each column. +NOTE: **Note:** Relative links do not allow the referencing of project files in a wiki +page, or a wiki page in a project file. The reason for this is that a wiki is always +in a separate Git repository in GitLab. For example, `[I'm a reference-style link](style)` +will point the link to `wikis/style` only when the link is inside of a wiki markdown file. -By including colons in the header row, you can align the text within that column. +#### URL auto-linking -Example: +GFM will autolink almost any URL you put into your text: -``` -| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | -| :----------- | :------: | ------------: | :----------- | :------: | ------------: | -| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | -| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +```markdown +- https://www.google.com +- https://google.com/ +- ftp://ftp.us.debian.org/debian/ +- smb://foo/bar/baz +- irc://irc.freenode.net/gitlab +- http://localhost:3000 ``` -Becomes: +- <https://www.google.com> +- <https://google.com/> +- <ftp://ftp.us.debian.org/debian/> +- <smb://foo/bar/baz> +- <irc://irc.freenode.net/gitlab> +- <http://localhost:3000> -| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | -| :----------- | :------: | ------------: | :----------- | :------: | ------------: | -| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | -| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +### Lists -### Footnotes +Ordered and unordered lists can be easily created. Add the number you want the list +to start with, like `1. ` (with a space) at the start of each line for ordered lists. +After the first number, it does not matter what number you use, ordered lists will be +numbered automatically by vertical order, so repeating `1. ` for all items in the +same list is common. If you start with a number other than `1. `, it will use that as the first +number, and count up from there. -Example: +Add a `* `, `- ` or `+ ` (with a space) at the start of each line for unordered lists, but +you should not use a mix of them. -``` -You can add footnotes to your text as follows.[^2] -[^2]: This is my awesome footnote. +Examples: + +```md +1. First ordered list item +2. Another item + - Unordered sub-list. +1. Actual numbers don't matter, just that it's a number + 1. Ordered sub-list + 1. Next ordered sub-list item +4. And another item. + +* Unordered lists can use asterisks +- Or minuses ++ Or pluses ``` -Becomes: +1. First ordered list item +2. Another item + - Unordered sub-list. +1. Actual numbers don't matter, just that it's a number + 1. Ordered sub-list + 1. Next ordered sub-list item +4. And another item. -You can add footnotes to your text as follows.[^2] +* Unordered lists can use asterisks +- Or minuses ++ Or pluses -### Superscripts / Subscripts +--- -CommonMark and GFM currently do not support the superscript syntax ( `x^2` ) that Redcarpet does. You can use the standard HTML syntax for superscripts and subscripts. +If a list item contains multiple paragraphs, each subsequent paragraph should be indented +to the same level as the start of the list item text. -``` -The formula for water is H<sub>2</sub>O -while the equation for the theory of relativity is E = mc<sup>2</sup>. -``` +Example: -The formula for water is H<sub>2</sub>O while the equation for the theory of relativity is E = mc<sup>2</sup>. +```markdown +1. First ordered list item -## Wiki-specific Markdown + Second paragraph of first item. -The following examples show how links inside wikis behave. +2. Another item +``` -### Wiki - Direct page link +1. First ordered list item -A link which just includes the slug for a page will point to that page, -_at the base level of the wiki_. + Second paragraph of first item. -This snippet would link to a `documentation` page at the root of your wiki: +2. Another item -```markdown -[Link to Documentation](documentation) -``` +--- -### Wiki - Direct file link +If the paragraph of the first item is not indented with the proper number of spaces, +the paragraph will appear outside the list, instead of properly indented under the list item. -Links with a file extension point to that file, _relative to the current page_. +Example: -If this snippet was placed on a page at `<your_wiki>/documentation/related`, -it would link to `<your_wiki>/documentation/file.md`: +``` +1. First ordered list item -```markdown -[Link to File](file.md) + Paragraph of first item. + +2. Another item ``` -### Wiki - Hierarchical link +1. First ordered list item -A link can be constructed relative to the current wiki page using `./<page>`, -`../<page>`, etc. + Paragraph of first item. -- If this snippet was placed on a page at `<your_wiki>/documentation/main`, - it would link to `<your_wiki>/documentation/related`: +2. Another item - ```markdown - [Link to Related Page](related) - ``` +### Superscripts / Subscripts -- If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, - it would link to `<your_wiki>/documentation/main`: +CommonMark and GFM currently do not support the superscript syntax ( `x^2` ) that +Redcarpet does. You can use the standard HTML syntax for superscripts and subscripts: + +```html +The formula for water is H<sub>2</sub>O +while the equation for the theory of relativity is E = mc<sup>2</sup>. +``` - ```markdown - [Link to Related Page](../main) - ``` +The formula for water is H<sub>2</sub>O +while the equation for the theory of relativity is E = mc<sup>2</sup>. -- If this snippet was placed on a page at `<your_wiki>/documentation/main`, - it would link to `<your_wiki>/documentation/related.md`: +### Tables - ```markdown - [Link to Related Page](related.md) - ``` +Tables aren't part of the core Markdown spec, but they are part of GFM. -- If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, - it would link to `<your_wiki>/documentation/main.md`: +1. The first line contains the headers, separated by "pipes" (`|`). +1. The second line separates the headers from the cells, and must contain three or more dashes. +1. The third, and any following lines, contain the cell values. + - You **can't** have cells separated over many lines in the markdown, they must be kept to single lines, + but they can be very long. You can also include HTML `<br>` tags to force newlines if needed. + - The cell sizes **don't** have to match each other. They are flexible, but must be separated + by pipes (`|`). + - You **can** have blank cells. - ```markdown - [Link to Related Page](../main.md) - ``` +Example: -### Wiki - Root link +```markdown +| header 1 | header 2 | header 3 | +| --- | ------ |----------| +| cell 1 | cell 2 | cell 3 | +| cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It will eventually wrap the text when the cell is too large for the display size. | +| cell 7 | | cell <br> 9 | +``` -A link starting with a `/` is relative to the wiki root. +| header 1 | header 2 | header 3 | +| --- | ------ |---------:| +| cell 1 | cell 2 | cell 3 | +| cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It will eventually wrap the text when the cell is too large for the display size. | +| cell 7 | | cell <br> 9 | -- This snippet links to `<wiki_root>/documentation`: +Additionally, you can choose the alignment of text within columns by adding colons (`:`) +to the sides of the "dash" lines in the second row. This will affect every cell in the column. - ```markdown - [Link to Related Page](/documentation) - ``` +> Note that the headers are always right aligned [within GitLab itself itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#tables). -- This snippet links to `<wiki_root>/miscellaneous.md`: +```markdown +| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | +| :--- | :---: | ---: | :----------- | :------: | ------------: | +| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | +| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +``` - ```markdown - [Link to Related Page](/miscellaneous.md) - ``` +| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | +| :--- | :---: | ---: | :----------- | :------: | ------------: | +| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | +| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | ## References - This document leveraged heavily from the [Markdown-Cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet). -- The original [Markdown Syntax Guide](https://daringfireball.net/projects/markdown/syntax) at Daring Fireball is an excellent resource for a detailed explanation of standard markdown. -- The detailed specification for CommonMark can be found in the [CommonMark Spec][commonmark-spec] +- The original [Markdown Syntax Guide](https://daringfireball.net/projects/markdown/syntax) + at Daring Fireball is an excellent resource for a detailed explanation of standard markdown. +- The detailed specification for CommonMark can be found in the [CommonMark Spec](https://spec.commonmark.org/current/) - The [CommonMark Dingus](http://try.commonmark.org) is a handy tool for testing CommonMark syntax. - -[^1]: This link will be broken if you see this document from the Help page or docs.gitlab.com -[^2]: This is my awesome footnote. - -[markdown.md]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md -[rouge]: http://rouge.jneen.net/ "Rouge website" -[redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" -[katex]: https://github.com/Khan/KaTeX "KaTeX website" -[katex-subset]: https://katex.org/docs/supported.html "Macros supported by KaTeX" -[asciidoctor-manual]: http://asciidoctor.org/docs/user-manual/#activating-stem-support "Asciidoctor user manual" -[commonmarker]: https://github.com/gjtorikian/commonmarker -[commonmark-spec]: https://spec.commonmark.org/current/ diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index 66362f27299..54bf3ff8a40 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -1,9 +1,6 @@ # Operations Dashboard **[PREMIUM]** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) -in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. -[Moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/9218) to -[GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. [Moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/9218) to [GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. The Operations Dashboard provides a summary of each project's operational health, including pipeline and alert status. @@ -16,9 +13,9 @@ dashboard icon: ## Adding a project to the dashboard NOTE: **Note:** -For GitLab.com, the Operations Dashboard is available for free for public projects. -If your project is private, the group it belongs to must have a -[Gold](https://about.gitlab.com/pricing/) plan. +For GitLab.com, you can add your project to the Operations Dashboard for free if +your project is public. If your project is private, the group it belongs to must +have a [Silver](https://about.gitlab.com/pricing/) plan. To add a project to the dashboard: diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 7af3d4a0ac3..33fff0fed74 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -122,8 +122,8 @@ The following table depicts the various user permission levels in a project. | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | | Delete issues | | | | | ✓ | -| Force push to protected branches [^4] | | | | | | -| Remove protected branches [^4] | | | | | | +| Force push to protected branches (*4*) | | | | | | +| Remove protected branches (*4*) | | | | | | - (*1*): All users are able to perform this action on public and internal projects, but not private projects. - (*2*): Guest users can only view the confidential issues they created themselves @@ -365,3 +365,8 @@ for details about the pipelines security model. Since GitLab 8.15, LDAP user permissions can now be manually overridden by an admin user. Read through the documentation on [LDAP users permissions](../administration/auth/how_to_configure_ldap_gitlab_ee/index.html) to learn more. + +## Project aliases + +Project aliases can only be read, created and deleted by a GitLab administrator. +Read through the documentation on [Project aliases](../user/project/index.md#project-aliases-premium-only) to learn more. diff --git a/doc/user/profile/account/img/2fa.png b/doc/user/profile/account/img/2fa.png Binary files differdeleted file mode 100644 index bb464efa685..00000000000 --- a/doc/user/profile/account/img/2fa.png +++ /dev/null diff --git a/doc/user/profile/account/img/2fa_auth.png b/doc/user/profile/account/img/2fa_auth.png Binary files differdeleted file mode 100644 index 0caaed10805..00000000000 --- a/doc/user/profile/account/img/2fa_auth.png +++ /dev/null diff --git a/doc/user/profile/account/img/2fa_u2f_authenticate.png b/doc/user/profile/account/img/2fa_u2f_authenticate.png Binary files differdeleted file mode 100644 index ff2e936764d..00000000000 --- a/doc/user/profile/account/img/2fa_u2f_authenticate.png +++ /dev/null diff --git a/doc/user/profile/account/img/2fa_u2f_register.png b/doc/user/profile/account/img/2fa_u2f_register.png Binary files differdeleted file mode 100644 index 1cc142aa851..00000000000 --- a/doc/user/profile/account/img/2fa_u2f_register.png +++ /dev/null diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index df413a11af0..e3e8c9a0d6d 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -10,17 +10,18 @@ is to know your username and password *and* have access to your one time passwor ## Overview -> **Note:** -When you enable 2FA, don't forget to back up your recovery codes. - -In addition to one time authenticators (TOTP), GitLab supports U2F (universal 2nd factor) devices as -the second factor of authentication. Once enabled, in addition to supplying your username and -password to login, you'll be prompted to activate your U2F device (usually by pressing -a button on it), and it will perform secure authentication on your behalf. - -The U2F workflow is [supported by](https://caniuse.com/#search=U2F) Google Chrome, Opera, and Firefox. - -We recommend that you set up 2FA with both a [one-time password authenticator](#enable-2fa-via-one-time-password-authenticator) and a [U2F device](#enable-2fa-via-u2f-device), so you can still access your account +TIP: **Tip:** +When you enable 2FA, don't forget to back up your [recovery codes](#recovery-codes)! + +In addition to time-based one time passwords (TOTP), GitLab supports U2F +(universal 2nd factor) devices as the second factor of authentication. Once +enabled, in addition to supplying your username and password to login, you'll +be prompted to activate your U2F device (usually by pressing a button on it), +and it will perform secure authentication on your behalf. + +It is highly recommended that you set up 2FA with both a +[one-time password authenticator](#enable-2fa-via-one-time-password-authenticator) +and a [U2F device](#enable-2fa-via-u2f-device), so you can still access your account if you lose your U2F device. ## Enabling 2FA @@ -30,41 +31,52 @@ or a U2F device. ### Enable 2FA via one time password authenticator -**In GitLab:** - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Account**. -1. Click **Enable Two-factor Authentication**. - - - -**On your phone:** - -1. Install a compatible application. We recommend [Google Authenticator] - \(proprietary\) or [FreeOTP] \(open source\). -1. In the application, add a new entry in one of two ways: - - Scan the code with your phone's camera to add the entry automatically. - - Enter the details provided to add the entry manually. - -**In GitLab:** - -1. Enter the six-digit pin number from the entry on your phone into the **Pin - code** field. -1. Click **Submit**. +To enable 2FA: + +1. **In GitLab:** + 1. Log in to your GitLab account. + 1. Go to your **Profile Settings**. + 1. Go to **Account**. + 1. Click **Enable Two-factor Authentication**. +1. **On your device (usually your phone):** + 1. Install a compatible application, like: + - [Authenticator](https://mattrubin.me/authenticator/): open source app for iOS devices. + - [andOTP](https://github.com/andOTP/andOTP): feature rich open source app for Android which supports PGP encrypted backups. + - [FreeOTP](https://freeotp.github.io/): open source app for Android. + - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en): proprietary app for iOS and Android. + 1. In the application, add a new entry in one of two ways: + - Scan the code presented in GitLab with your device's camera to add the + entry automatically. + - Enter the details provided to add the entry manually. +1. **In GitLab:** + 1. Enter the six-digit pin number from the entry on your device into the **Pin + code** field. + 1. Click **Submit**. If the pin you entered was correct, you'll see a message indicating that Two-Factor Authentication has been enabled, and you'll be presented with a list -of recovery codes. +of [recovery codes](#recovery-codes). Make sure you download them and keep them +in a safe place. ### Enable 2FA via U2F device -> **Notes:** -> -> - GitLab officially only supports [Yubikey] U2F devices. -> - Support for U2F devices was added in GitLab 8.8. +GitLab officially only supports [YubiKey](https://www.yubico.com/products/yubikey-hardware/) +U2F devices, but users have successfully used [SoloKeys](https://solokeys.com/). + +The U2F workflow is [supported by](https://caniuse.com/#search=U2F) the +following desktop browsers: -**In GitLab:** +- Chrome +- Edge +- Firefox (disabled by default) +- Opera + +NOTE: **Note:** +For Firefox, you can enable the FIDO U2F API in +[about:config](https://support.mozilla.org/en-US/kb/about-config-editor-firefox). +Search for `security.webauth.u2f` and double click on it to toggle to `true`. + +To set up 2FA with a U2F device: 1. Log in to your GitLab account. 1. Go to your **Profile Settings**. @@ -77,20 +89,21 @@ of recovery codes. You will see a message indicating that your device was successfully set up. Click on **Register U2F Device** to complete the process. - +## Recovery codes -## Recovery Codes - -> **Note:** +NOTE: **Note:** Recovery codes are not generated for U2F devices. -Should you ever lose access to your one time password authenticator, you can use one of the ten provided -backup codes to login to your account. We suggest copying them, printing them, or downloading them using -the **Download codes** button for storage in a safe place. - CAUTION: **Caution:** Each code can be used only once to log in to your account. +Immediately after successfully enabling two-factor authentication, you'll be +prompted to download a set of set recovery codes. Should you ever lose access +to your one time password authenticator, you can use one of them to log in to +your account. We suggest copying them, printing them, or downloading them using +the **Download codes** button for storage in a safe place. If you choose to +download them, the file will be called **gitlab-recovery-codes.txt**. + If you lose the recovery codes or just want to generate new ones, you can do so [using SSH](#generate-new-recovery-codes-using-ssh). @@ -100,24 +113,26 @@ Logging in with 2FA enabled is only slightly different than a normal login. Enter your username and password credentials as you normally would, and you'll be presented with a second prompt, depending on which type of 2FA you've enabled. -### Log in via mobile application - -Enter the pin from your one time password authenticator's application or a recovery code to log in. +### Log in via a one-time password - +When asked, enter the pin from your one time password authenticator's application or a +recovery code to log in. ### Log in via U2F device -1. Click **Login via U2F Device**. -1. A light will start blinking on your device. Activate it by pressing its button. +To log in via a U2F device: -You will see a message indicating that your device responded to the authentication request. -Click on **Authenticate via U2F Device** to complete the process. +1. Click **Login via U2F Device**. +1. A light will start blinking on your device. Activate it by touching/pressing + its button. - +You will see a message indicating that your device responded to the authentication +request and you will be automatically logged in. ## Disabling 2FA +If you ever need to disable 2FA: + 1. Log in to your GitLab account. 1. Go to your **Profile Settings**. 1. Go to **Account**. @@ -130,7 +145,8 @@ applications and U2F devices. When 2FA is enabled, you can no longer use your normal account password to authenticate with Git over HTTPS on the command line or when using -[GitLab's API][api], you must use a [personal access token][pat] instead. +[GitLab's API](../../../api/README.md). You must use a +[personal access token](../personal_access_tokens.md) instead. ## Recovery options @@ -149,7 +165,6 @@ codes. If you saved these codes, you can use one of them to sign in. To use a recovery code, enter your username/email and password on the GitLab sign-in page. When prompted for a two-factor code, enter the recovery code. -> **Note:** Once you use a recovery code, you cannot re-use it. You can still use the other recovery codes you saved. @@ -157,13 +172,18 @@ recovery codes you saved. Users often forget to save their recovery codes when enabling two-factor authentication. If an SSH key is added to your GitLab account, you can generate -a new set of recovery codes with SSH. +a new set of recovery codes with SSH: -1. Run `ssh git@gitlab.example.com 2fa_recovery_codes`. -1. You are prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes. +1. Run: + + ```sh + ssh git@gitlab.example.com 2fa_recovery_codes + ``` + +1. You will then be prompted to confirm that you want to generate new codes. + Continuing this process invalidates previously saved codes: ```sh - $ ssh git@gitlab.example.com 2fa_recovery_codes Are you sure you want to generate new two-factor recovery codes? Any existing recovery codes you saved will be invalidated. (yes/no) @@ -191,7 +211,6 @@ a new set of recovery codes with SSH. When prompted for a two-factor code, enter one of the recovery codes obtained from the command-line output. -> **Note:** After signing in, visit your **Profile settings > Account** immediately to set up two-factor authentication with a new device. @@ -219,9 +238,3 @@ Sign in and re-enable two-factor authentication as soon as possible. - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because the U2F key has only been registered on `first.host.xyz`. - -[Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en -[FreeOTP]: https://freeotp.github.io/ -[YubiKey]: https://www.yubico.com/products/yubikey-hardware/ -[api]: ../../../api/README.md -[pat]: ../personal_access_tokens.md diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md index cce862b0911..9bb282f1b78 100644 --- a/doc/user/project/canary_deployments.md +++ b/doc/user/project/canary_deployments.md @@ -44,7 +44,7 @@ Canary deployments require that you properly configure Deploy Boards: 1. Follow the steps to [enable Deploy Boards](deploy_boards.md#enabling-deploy-boards). 1. To track canary deployments you need to label your Kubernetes deployments and - pods with `track: canary`. To get started quickly, you can use the [Auto Deploy] + pods with `track: canary`. To get started quickly, you can use the [Auto Deploy](../../ci/autodeploy/index.md) template for canary deployments that GitLab provides. Depending on the deploy, the label should be either `stable` or `canary`. @@ -62,7 +62,6 @@ can easily notice them.  -[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" [eep]: https://about.gitlab.com/pricing/ [ee-1659]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1659 [kube-canary]: https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index a0fe97f2b9d..d21455fb5ca 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -222,7 +222,7 @@ functionalities needed to successfully build and deploy a containerized application. Bear in mind that the same credentials are used for all the applications running on the cluster. -## Gitlab-managed clusters +## GitLab-managed clusters > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22011) in GitLab 11.5. > Became [optional](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26565) in GitLab 11.11. @@ -519,8 +519,11 @@ service account of the cluster integration. ### Troubleshooting failed deployment jobs -GitLab will create a namespace and service account specifically for your -deployment jobs. This happens immediately before the deployment job starts. +Before the deployment jobs starts, GitLab creates the following specifically for +the deployment job: + +- A namespace. +- A service account. However, sometimes GitLab can not create them. In such instances, your job will fail with the message: @@ -530,22 +533,20 @@ This job failed because the necessary resources were not successfully created. To find the cause of this error when creating a namespace and service account, check the [logs](../../../administration/logs.md#kuberneteslog). -NOTE: **NOTE:** -As of GitLab 12.1 we require [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) -tokens for all project level clusters unless you unselect the -[GitLab-managed cluster](#gitlab-managed-clusters) option. If you -want to manage namespaces and service accounts yourself and don't -want to provide a `cluster-admin` token to GitLab you must unselect this -option or you will get the above error. - -Common reasons for failure include: +Reasons for failure include: -- The token you gave GitLab did not have [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) +- The token you gave GitLab does not have [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) privileges required by GitLab. - Missing `KUBECONFIG` or `KUBE_TOKEN` variables. To be passed to your job, they must have a matching [`environment:name`](../../../ci/environments.md#defining-environments). If your job has no `environment:name` set, it will not be passed the Kubernetes credentials. +NOTE: **NOTE:** +Project-level clusters upgraded from GitLab 12.0 or older may be configured +in a way that causes this error. Ensure you deselect the +[GitLab-managed cluster](#gitlab-managed-clusters) option if you want to manage +namespaces and service accounts yourself. + ## Monitoring your Kubernetes cluster **[ULTIMATE]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4701) in [GitLab Ultimate][ee] 10.6. diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index 368031070c1..25d8abebf07 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -12,9 +12,9 @@ By displaying the logs directly in GitLab, developers can avoid having to manage 1. Go to **Operations > Environments** and find the environment which contains the desired pod, like `production`. 1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). 1. When mousing over the list of pods, a tooltip will appear with the exact pod name and status. - +  1. Click on the desired pod to bring up the logs view, which will contain the last 500 lines for that pod. Support for pods with multiple containers is coming [in a future release](https://gitlab.com/gitlab-org/gitlab-ee/issues/6502). - +  ## Requirements diff --git a/doc/user/project/clusters/serverless/img/function-endpoint.png b/doc/user/project/clusters/serverless/img/function-endpoint.png Binary files differnew file mode 100644 index 00000000000..f3c7ae7a00d --- /dev/null +++ b/doc/user/project/clusters/serverless/img/function-endpoint.png diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index 3be5bfeaddc..a06c3d3c662 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -94,10 +94,55 @@ adding an existing installation of Knative. It is also possible to use GitLab Serverless with an existing Kubernetes cluster which already has Knative installed. -Simply: +You must do the following: 1. Follow the steps to [add an existing Kubernetes cluster](../index.md#adding-an-existing-kubernetes-cluster). + +1. Ensure GitLab can manage Knative: + - For a non-GitLab managed cluster, ensure that the service account for the token + provided can manage resources in the `serving.knative.dev` API group. + - For a GitLab managed cluster, + GitLab uses a service account with the `edit` cluster role. This account needs + the ability to manage resources in the `serving.knative.dev` API group. + We suggest you do this with an [aggregated ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) + adding rules to the default `edit` cluster role: + First, save the following YAML as `knative-serving-only-role.yaml`: + + ```yaml + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: knative-serving-only-role + labels: + rbac.authorization.k8s.io/aggregate-to-edit: "true" + rules: + - apiGroups: + - serving.knative.dev + resources: + - configurations + - configurationgenerations + - routes + - revisions + - revisionuids + - autoscalers + - services + verbs: + - get + - list + - create + - update + - delete + - patch + - watch + ``` + + Then run the following command: + + ```bash + kubectl apply -f knative-serving-only-role.yaml + ``` + 1. Follow the steps to deploy [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) onto your cluster. @@ -325,3 +370,275 @@ loading or is not available at this time._ It will appear upon the first access page, but should go away after a few seconds. If the message does not disappear, then it is possible that GitLab is unable to connect to the Prometheus instance running on the cluster. + +## Enabling TLS for Knative services + +By default, a GitLab serverless deployment will be served over `http`. In order to serve over `https` you +must manually obtain and install TLS certificates. + +The simplest way to accomplish this is to +use [Certbot to manually obtain Let's Encrypt certificates](https://knative.dev/docs/serving/using-a-tls-cert/#using-certbot-to-manually-obtain-let-s-encrypt-certificates). Certbot is a free, open source software tool for automatically using Let’s Encrypt certificates on manually-administrated websites to enable HTTPS. + +NOTE: **Note:** +The instructions below relate to installing and running Certbot on a Linux server and may not work on other operating systems. + +1. Install Certbot by running the + [`certbot-auto` wrapper script](https://certbot.eff.org/docs/install.html#certbot-auto). + On the command line of your server, run the following commands: + + ```sh + wget https://dl.eff.org/certbot-auto + sudo mv certbot-auto /usr/local/bin/certbot-auto + sudo chown root /usr/local/bin/certbot-auto + chmod 0755 /usr/local/bin/certbot-auto + /usr/local/bin/certbot-auto --help + ``` + + To check the integrity of the `certbot-auto` script, run: + + ```sh + wget -N https://dl.eff.org/certbot-auto.asc + gpg2 --keyserver ipv4.pool.sks-keyservers.net --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc /usr/local/bin/certbot-auto + ``` + + The output of the last command should look something like: + + ```sh + gpg: Signature made Mon 10 Jun 2019 06:24:40 PM EDT + gpg: using RSA key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + gpg: key 4D17C995CD9775F2 marked as ultimately trusted + gpg: checking the trustdb + gpg: marginals needed: 3 completes needed: 1 trust model: pgp + gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u + gpg: next trustdb check due at 2027-11-22 + gpg: Good signature from "Let's Encrypt Client Team <letsencrypt-client@eff.org>" [ultimate] + ``` + +1. Run the following command to use Certbot to request a certificate + using DNS challenge during authorization: + + + ```sh + ./certbot-auto certonly --manual --preferred-challenges dns -d '*.<namespace>.example.com' + ``` + + Where `<namespace>` is the namespace created by GitLab for your serverless project (composed of `<projectname+id>`) and + `example.com` is the domain being used for your project. If you are unsure what the namespace of your project is, navigate + to the **Operations > Serverless** page of your project and inspect + the endpoint provided for your function/app. + +  + + In the above image, the namespace for the project is `node-function-11909507` and the domain is `knative.info`, thus + certificate request line would look like this: + + ```sh + ./certbot-auto certonly --manual --preferred-challenges dns -d '*.node-function-11909507.knative.info' + ``` + + The Certbot tool walks you through the steps of validating that you own each domain that you specify by creating TXT records in those domains. + After this process is complete, the output should look something like this: + + ```sh + IMPORTANT NOTES: + - Congratulations! Your certificate and chain have been saved at: + /etc/letsencrypt/live/namespace.example.com/fullchain.pem + Your key file has been saved at: + /etc/letsencrypt/live/namespace.example/privkey.pem + Your cert will expire on 2019-09-19. To obtain a new or tweaked + version of this certificate in the future, simply run certbot-auto + again. To non-interactively renew *all* of your certificates, run + "certbot-auto renew" + -----BEGIN PRIVATE KEY----- + - Your account credentials have been saved in your Certbot + configuration directory at /etc/letsencrypt. You should make a + secure backup of this folder now. This configuration directory will + also contain certificates and private keys obtained by Certbot so + making regular backups of this folder is ideal. + ``` + +1. Create certificate and private key files. Using the contents of the files + returned by Certbot, we'll create two files in order to create the + Kubernetes secret: + + Run the following command to see the contents of `fullchain.pem`: + + ```sh + sudo cat /etc/letsencrypt/live/node-function-11909507.knative.info/fullchain.pem + ``` + + Output should look like this: + + ```sh + -----BEGIN CERTIFICATE----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b4ag== + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + K2fcb195768c39e9a94cec2c2e30Qg== + -----END CERTIFICATE----- + ``` + + Create a file with the name `cert.pem` with the contents of the entire output. + + Once `cert.pem` is created, run the following command to see the contents of `privkey.pem`: + + ```sh + sudo cat /etc/letsencrypt/live/namespace.example/privkey.pem + ``` + + Output should look like this: + + ```sh + -----BEGIN PRIVATE KEY----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + -----BEGIN CERTIFICATE----- + fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 4f294d1eaca42b8692017b4262== + -----END PRIVATE KEY----- + ``` + + Create a new file with the name `cert.pk` with the contents of the entire output. + +1. Create a Kubernetes secret to hold your TLS certificate, `cert.pem`, and + the private key `cert.pk`: + + NOTE: **Note:** + Running `kubectl` commands on your cluster requires setting up access to the cluster first. + For clusters created on GKE, see + [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl). + For other platforms, [install `kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + + ```sh + kubectl create --namespace istio-system secret tls istio-ingressgateway-certs \ + --key cert.pk \ + --cert cert.pem + ``` + + Where `cert.pem` and `cert.pk` are your certificate and private key files. Note that the `istio-ingressgateway-certs` secret name is required. + +1. Configure Knative to use the new secret that you created for HTTPS + connections. Run the + following command to open the Knative shared `gateway` in edit mode: + + ```sh + kubectl edit gateway knative-ingress-gateway --namespace knative-serving + ``` + + Update the gateway to include the following tls: section and configuration: + + ```sh + tls: + mode: SIMPLE + privateKey: /etc/istio/ingressgateway-certs/tls.key + serverCertificate: /etc/istio/ingressgateway-certs/tls.crt + ``` + + Example: + + ```sh + apiVersion: networking.istio.io/v1alpha3 + kind: Gateway + metadata: + # ... skipped ... + spec: + selector: + istio: ingressgateway + servers: + - hosts: + - "*" + port: + name: http + number: 80 + protocol: HTTP + - hosts: + - "*" + port: + name: https + number: 443 + protocol: HTTPS + tls: + mode: SIMPLE + privateKey: /etc/istio/ingressgateway-certs/tls.key + serverCertificate: /etc/istio/ingressgateway-certs/tls.crt + ``` + + After your changes are running on your Knative cluster, you can begin using the HTTPS protocol for secure access your deployed Knative services. + In the event a mistake is made during this process and you need to update the cert, you will need to edit the gateway `knative-ingress-gateway` + to switch back to `PASSTHROUGH` mode. Once corrections are made, edit the file again so the gateway will use the new certificates.
\ No newline at end of file diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md index 58b7fe33906..7d567da1c9a 100644 --- a/doc/user/project/container_registry.md +++ b/doc/user/project/container_registry.md @@ -113,6 +113,7 @@ This feature requires GitLab 8.8 and GitLab Runner 1.2. Make sure that your GitLab Runner is configured to allow building Docker images by following the [Using Docker Build](../../ci/docker/using_docker_build.md) and [Using the GitLab Container Registry documentation](../../ci/docker/using_docker_build.md#using-the-gitlab-container-registry). +Alternatively, you can [build images with Kaniko](../../ci/docker/using_kaniko.md) if the Docker builds are not an option for you. ## Using with private projects @@ -167,6 +168,19 @@ curl localhost:5001/debug/health curl localhost:5001/debug/vars ``` +#### Docker connection error + +A Docker connection error can occur when there are special characters in either the group, +project or branch name. Special characters can include: + +* Leading underscore +* Trailing hyphen/dash +* Double hyphen/dash + +To get around this, you can [change the group path](../group/index.md#changing-a-groups-path), +[change the project path](../project/settings/index.md#renaming-a-repository) or chanage the branch +name. + ### Advanced Troubleshooting >**NOTE:** The following section is only recommended for experts. diff --git a/doc/user/project/deploy_tokens/img/deploy_tokens.png b/doc/user/project/deploy_tokens/img/deploy_tokens.png Binary files differindex 55c537fd1d3..421aa1ab3e5 100644 --- a/doc/user/project/deploy_tokens/img/deploy_tokens.png +++ b/doc/user/project/deploy_tokens/img/deploy_tokens.png diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 92a29b68a22..5e11e7c0203 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -15,7 +15,7 @@ You can create as many deploy tokens as you like from the settings of your proje 1. Go to the project you want to create Deploy Tokens for. 1. Go to **Settings** > **Repository**. 1. Click on "Expand" on **Deploy Tokens** section. -1. Choose a name and optionally an expiry date for the token. +1. Choose a name, expiry date (optional), and username (optional) for the token. 1. Choose the [desired scopes](#limiting-scopes-of-a-deploy-token). 1. Click on **Create deploy token**. 1. Save the deploy token somewhere safe. Once you leave or refresh @@ -39,6 +39,13 @@ the following table. | `read_repository` | Allows read-access to the repository through `git clone` | | `read_registry` | Allows read-access to [container registry] images if a project is private and authorization is required. | +## Deploy token custom username + +> [Introduced][https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29639] in GitLab 12.1. + +The default username format is `gitlab+deploy-token-#{n}`. Some tools or platforms may not support this format, +in such case you can specify custom username to be used when creating the deploy token. + ## Usage ### Git clone a repository diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index ca59fe3cc7d..7520237251a 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -25,14 +25,14 @@ templates of the default branch will be taken into account. ## Use-cases - Add a template to be used in every issue for a specific project, -giving instructions and guidelines, requiring for information specific to that subject. -For example, if you have a project for tracking new blog posts, you can require the -title, outlines, author name, author social media information, etc. + giving instructions and guidelines, requiring for information specific to that subject. + For example, if you have a project for tracking new blog posts, you can require the + title, outlines, author name, author social media information, etc. - Following the previous example, you can make a template for every MR submitted -with a new blog post, requiring information about the post date, frontmatter data, -images guidelines, link to the related issue, reviewer name, etc. + with a new blog post, requiring information about the post date, frontmatter data, + images guidelines, link to the related issue, reviewer name, etc. - You can also create issues and merge request templates for different -stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. + stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. ## Creating issue templates diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md index 7f79ebf6353..3b071ff590f 100644 --- a/doc/user/project/import/gemnasium.md +++ b/doc/user/project/import/gemnasium.md @@ -40,14 +40,14 @@ some steps to migrate your projects. There is no automatic import since GitLab doesn't know anything about any projects which existed on Gemnasium.com. Security features are free for public (open-source) projects hosted on GitLab.com. -### If your project is hosted on GitLab (https://gitlab.com / self-hosted) +### If your project is hosted on GitLab (`https://gitlab.com` / self-hosted) You're almost set! If you're already using [Auto DevOps](../../../topics/autodevops/), you are already covered. Otherwise, you must configure your `.gitlab-ci.yml` according to the [dependency scanning page](../../application_security/dependency_scanning/index.md). -### If your project is hosted on GitHub (https://github.com / GitHub Enterprise) +### If your project is hosted on GitHub (`https://github.com` / GitHub Enterprise) Since [GitLab 10.6 comes with GitHub integration](https://about.gitlab.com/features/github/), GitLab users can now create a CI/CD project in GitLab connected to an external diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 587b4121e4e..06286951e20 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -193,6 +193,28 @@ password <personal_access_token> To quickly access a project from the GitLab UI using the project ID, visit the `/projects/:id` URL in your browser or other tool accessing the project. +## Project aliases **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3264) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.1. + +When migrating repositories to GitLab and they are being accessed by other systems, +it's very useful to be able to access them using the same name especially when +they are a lot. It reduces the risk of changing significant number of Git URLs in +a large number of systems. + +GitLab provides a functionality to help with this. In GitLab, repositories are +usually accessed with a namespace and project name. It is also possible to access +them via a project alias. This feature is only available on Git over SSH. + +A project alias can be only created via API and only by GitLab administrators. +Follow the [Project Aliases API documentation](../../api/project_aliases.md) for +more details. + +Once an alias has been created for a project (e.g., an alias `gitlab-ce` for the +project `https://gitlab.com/gitlab-org/gitlab-ce`), the repository can be cloned +using the alias (e.g `git clone git@gitlab.com:gitlab-ce.git` instead of +`git clone git@gitlab.com:gitlab-org/gitlab-ce.git`). + ## Project APIs There are numerous [APIs](../../api/README.md) to use with your projects: @@ -212,3 +234,4 @@ There are numerous [APIs](../../api/README.md) to use with your projects: - [Templates](../../api/project_templates.md) - [Traffic](../../api/project_statistics.md) - [Variables](../../api/project_level_variables.md) +- [Aliases](../../api/project_aliases.md) diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md index 2e2a27f112e..1c6ad0b8b2b 100644 --- a/doc/user/project/insights/index.md +++ b/doc/user/project/insights/index.md @@ -1,7 +1,6 @@ # Insights **[ULTIMATE]** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. -> **Generally Available** (GA) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. Configure the Insights that matter for your projects to explore data such as triage hygiene, issues created/closed per a given period, average time for merge diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index cdb0e34fdf6..680fcdb78bb 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -17,7 +17,7 @@ and is automatically configured on [GitHub import](../../../integration/github.m This integration requires a [GitHub API token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) with `repo:status` access granted: -1. Go to your "Personal access tokens" page at https://github.com/settings/tokens +1. Go to your "Personal access tokens" page at <https://github.com/settings/tokens> 1. Click "Generate New Token" 1. Ensure that `repo:status` is checked and click "Generate token" 1. Copy the generated token to use on GitLab diff --git a/doc/user/project/integrations/hipchat.md b/doc/user/project/integrations/hipchat.md index 0fd847d415f..7a0540aa9e3 100644 --- a/doc/user/project/integrations/hipchat.md +++ b/doc/user/project/integrations/hipchat.md @@ -23,7 +23,7 @@ allow GitLab to send messages only to *one* room. 1. Find "Build Your Own!" and click "Create". 1. Select the desired room, name the integration "GitLab", and click "Create". 1. In the "Send messages to this room by posting this URL" column, you should -see a URL in the format: + see a URL in the format: ``` https://api.hipchat.com/v2/room/<room>/notification?auth_token=<token> diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index 234e3ad31cc..8f2e5a55b5f 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -39,7 +39,7 @@ a GitLab project with any single Jira project. If you have one Jira instance, you can pre-fill the settings page with a default template. See the [Services Templates][services-templates] docs. -Configuration happens via user name and password. Connecting to a Jira server +Configuration happens via user name and password. Connecting to a Jira Server via CAS is not possible. In order to enable the Jira service in GitLab, you need to first configure the @@ -47,13 +47,13 @@ project in Jira and then enter the correct values in GitLab. ### Configuring Jira -When connecting to **JIRA Server**, which supports basic authentication, a **username and password** are required. Check the link below and proceed to the next step: +When connecting to **Jira Server**, which supports basic authentication, a **username and password** are required. Check the link below and proceed to the next step: -- [Setting up a user in JIRA server](jira_server_configuration.md) +- [Setting up a user in Jira Server](jira_server_configuration.md) -When connecting to **JIRA Cloud**, which supports authentication via API token, an **email and API token**, are required. Check the link below and proceed to the next step: +When connecting to **Jira Cloud**, which supports authentication via API token, an **email and API token**, are required. Check the link below and proceed to the next step: -- [Setting up a user in JIRA cloud](jira_cloud_configuration.md) +- [Setting up a user in Jira Cloud](jira_cloud_configuration.md) ### Configuring GitLab @@ -77,8 +77,8 @@ in the table below. | ----- | ----------- | | `Web URL` | The base URL to the Jira instance web interface which is being linked to this GitLab project. E.g., `https://Jira.example.com`. | | `Jira API URL` | The base URL to the Jira instance API. Web URL value will be used if not set. E.g., `https://jira-api.example.com`. | -| `Username/Email` | Created when [configuring Jira step](#configuring-jira). Use `username` for **JIRA server** or `email` for **JIRA cloud**. | -| `Password/API token` |Created in [configuring Jira step](#configuring-jira). Use `password` for **JIRA server** or `API token` for **JIRA cloud**. | +| `Username/Email` | Created when [configuring Jira step](#configuring-jira). Use `username` for **Jira Server** or `email` for **Jira Cloud**. | +| `Password/API token` |Created in [configuring Jira step](#configuring-jira). Use `password` for **Jira Server** or `API token` for **Jira Cloud**. | | `Transition ID` | This is the ID of a transition that moves issues to the desired state. It is possible to insert transition ids separated by `,` or `;` which means the issue will be moved to each state after another using the given order. **Closing Jira issues via commits or Merge Requests won't work if you don't set the ID correctly.** | ### Obtaining a transition ID diff --git a/doc/user/project/integrations/jira_cloud_configuration.md b/doc/user/project/integrations/jira_cloud_configuration.md index 849df707521..614f05d5b7e 100644 --- a/doc/user/project/integrations/jira_cloud_configuration.md +++ b/doc/user/project/integrations/jira_cloud_configuration.md @@ -1,18 +1,18 @@ -# Creating an API token in JIRA cloud +# Creating an API token in Jira Cloud -An API token is needed when integrating with JIRA Cloud, follow the steps +An API token is needed when integrating with Jira Cloud, follow the steps below to create one: 1. Log in to <https://id.atlassian.com> with your email. 1. **Click API tokens**, then **Create API token**. - + - + 1. Make sure to write down your new API token as you will need it in the next [steps](jira.md#configuring-gitlab). NOTE: **Note** -It is important that the user associated with this email has 'write' access to projects in JIRA. +It is important that the user associated with this email has 'write' access to projects in Jira. -The JIRA configuration is complete. You are going to need this new created token and the email you used to log in when [configuring GitLab in the next section](jira.md#configuring-gitlab). +The Jira configuration is complete. You are going to need this newly created token and the email you used to log in, when [configuring GitLab in the next section](jira.md#configuring-gitlab). diff --git a/doc/user/project/integrations/jira_server_configuration.md b/doc/user/project/integrations/jira_server_configuration.md index 13d65c4d8e4..32991714973 100644 --- a/doc/user/project/integrations/jira_server_configuration.md +++ b/doc/user/project/integrations/jira_server_configuration.md @@ -1,4 +1,4 @@ -# Creating a username and password for JIRA server +# Creating a username and password for Jira Server We need to create a user in Jira which will have access to all projects that need to integrate with GitLab. diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index 0bfee3bac99..0e4c71a9d3e 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -38,7 +38,7 @@ Click on the service links to see further configuration instructions and details | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | | [HipChat](hipchat.md) | Private group chat and IM | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | -| [JIRA](jira.md) | JIRA issue tracker | +| [Jira](jira.md) | Jira issue tracker | | [Jenkins](../../../integration/jenkins.md) **[STARTER]** | An extendable open source continuous integration server | | JetBrains TeamCity CI | A continuous integration and build server | | [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 751e8e44e60..aab7131e353 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -160,7 +160,7 @@ receivers: > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. -Alerts can be used to trigger actions, like open an issue automatically. To configure the actions: +Alerts can be used to trigger actions, like open an issue automatically (enabled by default since `12.1`). To configure the actions: 1. Navigate to your project's **Settings > Operations > Incidents**. 1. Enable the option to create issues. diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md index abb0c01ad18..6be8fc82431 100644 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -11,8 +11,8 @@ The [Prometheus service](../prometheus.md) must be enabled. | Name | Query | | ---- | ----- | -| Throughput (req/sec) | sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) by (code) | -| HTTP Error Rate (%) | sum(rate(haproxy_frontend_http_requests_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) | +| Throughput (req/sec) | `sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) by (code)` | +| HTTP Error Rate (%) | `sum(rate(haproxy_frontend_http_requests_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m]))` | ## Configuring Prometheus to monitor for HAProxy metrics diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md index c4fea178ab5..7db9629c002 100644 --- a/doc/user/project/integrations/prometheus_library/nginx.md +++ b/doc/user/project/integrations/prometheus_library/nginx.md @@ -14,9 +14,9 @@ NGINX server metrics are detected, which tracks the pages and content directly s | Name | Query | | ---- | ----- | -| Throughput (req/sec) | sum(rate(nginx_server_requests{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) by (code) | -| Latency (ms) | avg(nginx_server_requestMsec{%{environment_filter}}) | -| HTTP Error Rate (HTTP Errors / sec) | sum(rate(nginx_server_requests{code="5xx", %{environment_filter}}[2m])) | +| Throughput (req/sec) | `sum(rate(nginx_server_requests{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) by (code)` | +| Latency (ms) | `avg(nginx_server_requestMsec{%{environment_filter}})` | +| HTTP Error Rate (HTTP Errors / sec) | `sum(rate(nginx_server_requests{code="5xx", %{environment_filter}}[2m]))` | ## Configuring Prometheus to monitor for NGINX metrics diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index de7fc93f0a4..fd743855a8c 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -14,9 +14,9 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI | Name | Query | | ---- | ----- | -| Throughput (req/sec) | sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code) | -| Latency (ms) | sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_count{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 1000 | -| HTTP Error Rate (%) | sum(rate(nginx_ingress_controller_requests{status=~"5.*",namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 100 | +| Throughput (req/sec) | `sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code)` | +| Latency (ms) | `sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_count{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 1000` | +| HTTP Error Rate (%) | `sum(rate(nginx_ingress_controller_requests{status=~"5.*",namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 100` | ## Configuring NGINX ingress monitoring diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index 31ac53c0d14..b03787b3e0d 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -14,9 +14,9 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI | Name | Query | | ---- | ----- | -| Throughput (req/sec) | sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) | -| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) | -| HTTP Error Rate (%) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) / sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) * 100 | +| Throughput (req/sec) | `sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code)` | +| Latency (ms) | `avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"})` | +| HTTP Error Rate (%) | `sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) / sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) * 100` | ## Configuring NGINX ingress monitoring diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 76dc6e49bce..4acbb4cc3f6 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -134,4 +134,4 @@ For more information, see [Crosslinking issues](crosslinking_issues.md). - [Export issues](csv_export.md) **[STARTER]** - [Issues API](../../../api/issues.md) - Configure an [external issue tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, -or Bugzilla. + or Bugzilla. diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index fd151a6df45..8e8ec26daf2 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -12,9 +12,9 @@ to approve a merge request before it can be unblocked for merging. ## Use cases 1. Enforcing review of all code that gets merged into a repository. -2. Specifying code maintainers for an entire repository. -3. Specifying reviewers for a given proposed code change. -4. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. +1. Specifying code maintainers for an entire repository. +1. Specifying reviewers for a given proposed code change. +1. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. ## Enabling the new approvals interface @@ -246,7 +246,7 @@ restrictions (compared to [GitLab Starter](#overriding-the-merge-request-approva - Approval rules can be added to an MR with no restriction. - For project sourced approval rules, editing and removing approvers is not allowed. - The approvals required of all approval rules is configurable, but if a rule is backed by a project rule, then it is restricted -to the minimum approvals required set in the project's corresponding rule. + to the minimum approvals required set in the project's corresponding rule. ## Resetting approvals on push diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index c93c7a5fe08..0dd60d84c42 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -42,6 +42,8 @@ Navigate to your project's settings page and expand the **Merge requests** secti In the **Merge checks** subsection, select the **Pipelines must succeed** check box and hit **Save** for the changes to take effect. +NOTE: **Note:** This setting also prevents merge requests from being merged if there is no pipeline. +  From now on, every time the pipeline fails you will not be able to merge the @@ -49,6 +51,21 @@ merge request from the UI, until you make all relevant jobs pass.  +### Limitations + +When this setting is enabled, a merge request is prevented from being merged if there is no pipeline. This may conflict with some use cases where [`only/except`](../../../ci/yaml/README.md#onlyexcept-advanced) rules are used and they don't generate any pipelines. + +Users that expect to be able to merge a merge request in this scenario should ensure that [there is always a pipeline](https://gitlab.com/gitlab-org/gitlab-ce/issues/54226) and that it's succesful. + +For example, to that on merge requests there is always a passing job even though `only/except` rules may not generate any other jobs: + +```yaml +enable_merge: + only: merge_requests + script: + - echo true +``` + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index 39fd2588811..38459584eed 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -50,7 +50,10 @@ all you have to do is enable squashing before you press merge to join the commits in the merge request into a single commit. This way, the history of your base branch remains clean with -meaningful commit messages and is simpler to [revert](revert_changes.md) if necessary. +meaningful commit messages and: + +- It's simpler to [revert](revert_changes.md) if necessary. +- The merged branch will retain the full commit history. ## Enabling squash for a merge request diff --git a/doc/user/project/merge_requests/work_in_progress_merge_requests.md b/doc/user/project/merge_requests/work_in_progress_merge_requests.md index 9cd868067bc..142178ba241 100644 --- a/doc/user/project/merge_requests/work_in_progress_merge_requests.md +++ b/doc/user/project/merge_requests/work_in_progress_merge_requests.md @@ -15,7 +15,7 @@ being merged, and it will stay disabled until the "WIP" flag has been removed. There are several ways to flag a merge request as a Work In Progress: -- Add "[WIP]" or "WIP:" to the start of the merge request's title. Clicking on +- Add `[WIP]` or `WIP:` to the start of the merge request's title. Clicking on **Start the title with WIP:**, under the title box, when editing the merge request's description will have the same effect. - Add the `/wip` [quick action](../quick_actions.md#quick-actions-for-issues-and-merge-requests) @@ -30,7 +30,7 @@ There are several ways to flag a merge request as a Work In Progress: Similar to above, when a Merge Request is ready to be merged, you can remove the "Work in Progress" flag in several ways: -- Remove "[WIP]" or "WIP:" from the start of the merge request's title. Clicking on +- Remove `[WIP]` or `WIP:` from the start of the merge request's title. Clicking on **Remove the WIP: prefix from the title**, under the title box, when editing the merge request's description, will have the same effect. - Add the `/wip` [quick action](../quick_actions.md#quick-actions-for-issues-and-merge-requests) diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index d585c19fc5c..bc9a11504cd 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1,5 +1,5 @@ --- -last_updated: 2019-06-04 +last_updated: 2019-06-25 type: concepts, reference, howto --- @@ -138,9 +138,9 @@ verify your domain's ownership with a TXT record: > - **Do not** add any special chars after the default Pages domain. E.g., **do not** point your `subdomain.domain.com` to `namespace.gitlab.io.` or `namespace.gitlab.io/`. -> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017 +> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017. > - GitLab Pages IP on GitLab.com [has been changed](https://about.gitlab.com/2018/07/19/gcp-move-update/#gitlab-pages-and-custom-domains) - from `52.167.214.135` to `35.185.44.232` in 2018 + from `52.167.214.135` to `35.185.44.232` in 2018. ### Add your custom domain to GitLab Pages settings @@ -199,7 +199,7 @@ Certificates are NOT required to add to your custom highly recommendable. Let's start with an introduction to the importance of HTTPS. -Alternatively, jump ahead to [adding certificates to your project](#adding-certificates-to-your-project). +Alternatively, jump ahead to [adding certificates to your project](#adding-certificates-to-pages). ### Why should I care about HTTPS? @@ -255,12 +255,12 @@ which also offers a [free CDN service](https://blog.cloudflare.com/cloudflares-f Their certs are valid up to 15 years. See the tutorial on [how to add a CloudFlare Certificate to your GitLab Pages website](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/). -### Adding certificates to your project +### Adding certificates to Pages Regardless the CA you choose, the steps to add your certificate to your Pages project are the same. -### What do you need +#### Requirements 1. A PEM certificate 1. An intermediate certificate @@ -270,7 +270,7 @@ your Pages project are the same. These fields are found under your **Project**'s **Settings** > **Pages** > **New Domain**. -### What's what? +#### Certificate types - A PEM certificate is the certificate generated by the CA, which needs to be added to the field **Certificate (PEM)**. @@ -283,21 +283,32 @@ These fields are found under your **Project**'s **Settings** > **Pages** > **New - A private key is an encrypted key which validates your PEM against your domain. -### Now what? +#### Add the certificate to your project -Now that you hopefully understand why you need all -of this, it's simple: +Once you've met the requirements: -- Your PEM certificate needs to be added to the first field +- Your PEM certificate needs to be added to the first field. - If your certificate is missing its intermediate, copy and paste the root certificate (usually available from your CA website) and paste it in the [same field as your PEM certificate](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/), just jumping a line between them. -- Copy your private key and paste it in the last field +- Copy your private key and paste it in the last field. ->**Note:** +NOTE: **Note:** **Do not** open certificates or encryption keys in regular text editors. Always use code editors (such as Sublime Text, Atom, Dreamweaver, Brackets, etc). -_Read on about [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md)_ +## Force HTTPS for GitLab Pages websites + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28857) in GitLab 10.7. + +To make your website's visitors even more secure, you can choose to +force HTTPS for GitLab Pages. By doing so, all attempts to visit your +website via HTTP will be automatically redirected to HTTPS via 301. + +It works with both GitLab's default domain and with your custom +domain (as long as you've set a valid certificate for it). + +To enable this setting, navigate to your project's **Settings > Pages** +and tick the checkbox **Force HTTPS (requires valid certificates)**. diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index 3e50cd4887c..fe92d19567d 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -77,10 +77,10 @@ containing the most popular SSGs templates to get you started. 1. [Fork](../../../gitlab-basics/fork-project.md) a sample project from the [GitLab Pages examples](https://gitlab.com/pages) group. 1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** -and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your -site to the server. + and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your + site to the server. 1. Once the pipeline has finished successfully, find the link to visit your -website from your project's **Settings > Pages**. + website from your project's **Settings > Pages**. You can also take some **optional** further steps: @@ -89,14 +89,14 @@ You can also take some **optional** further steps:  - _Make it a user or group website._ To turn a **project website** forked -from the Pages group into a **user/group** website, you'll need to: + from the Pages group into a **user/group** website, you'll need to: - Rename it to `namespace.gitlab.io`: go to your project's - **Settings > General** and expand **Advanced**. Scroll down to - **Rename repository** and change the path to `namespace.gitlab.io`. + **Settings > General** and expand **Advanced**. Scroll down to + **Rename repository** and change the path to `namespace.gitlab.io`. - Adjust your SSG's [base URL](#urls-and-baseurls) from `"project-name"` to - `""`. This setting will be at a different place for each SSG, as each of them - have their own structure and file tree. Most likely, it will be in the SSG's - config file. + `""`. This setting will be at a different place for each SSG, as each of them + have their own structure and file tree. Most likely, it will be in the SSG's + config file. ### Create a project from scratch diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 04bda212128..fa79c393b72 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -12,7 +12,6 @@ type: index, reference > - Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. > - Bundled project templates were [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47857) in GitLab 11.8. - **GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab.** @@ -105,10 +104,10 @@ To get started with GitLab Pages, you can either:  1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** -and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your -site to the server. + and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your + site to the server. 1. Once the pipeline has finished successfully, find the link to visit your -website from your project's **Settings > Pages**. + website from your project's **Settings > Pages**. Your website is then visible on your domain, and you can modify yourfiles as you wish. For every modification pushed to your repository, GitLab CI/CD diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 4fab7f79e0c..4ea3bd9be9b 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -13,17 +13,17 @@ To familiarize yourself with GitLab Pages first: - Read an [introduction to GitLab Pages](index.md#overview). - Learn [how to get started with Pages](index.md#getting-started). - Learn how to enable GitLab Pages -across your GitLab instance on the [administrator documentation](../../../administration/pages/index.md). + across your GitLab instance on the [administrator documentation](../../../administration/pages/index.md). ## GitLab Pages requirements In brief, this is what you need to upload your website in GitLab Pages: 1. Domain of the instance: domain name that is used for GitLab Pages -(ask your administrator). + (ask your administrator). 1. GitLab CI/CD: a `.gitlab-ci.yml` file with a specific job named [`pages`][pages] in the root directory of your repository. 1. A directory called `public` in your site's repo containing the content -to be published. + to be published. 1. GitLab Runner enabled for the project. ## GitLab Pages on GitLab.com diff --git a/doc/user/project/pipelines/job_artifacts.md b/doc/user/project/pipelines/job_artifacts.md index 002addfc043..f9feae36dbc 100644 --- a/doc/user/project/pipelines/job_artifacts.md +++ b/doc/user/project/pipelines/job_artifacts.md @@ -187,7 +187,8 @@ information in the UI. DANGER: **Warning:** This is a destructive action that leads to data loss. Use with caution. -If you have at least Developer [permissions](../../permissions.md#gitlab-cicd-permissions) +If you are either the owner of a given job or have Master +[permissions](../../permissions.md#gitlab-cicd-permissions) on the project, you can erase a single job via the UI which will also remove the artifacts and the job's trace. diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 7215b86ddeb..aa15fe987db 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -57,6 +57,7 @@ discussions, and descriptions: | `/approve` | Approve the merge request | | ✓ | | `/merge` | Merge (when pipeline succeeds) | | ✓ | | `/create_merge_request <branch name>` | Create a new merge request starting from the current issue | ✓ | | +| `/relate #issue1 #issue2` | Mark issues as related **[STARTER]** | ✓ | | ## Quick actions for commit messages diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index ba890c5ac01..2bf8d4dfe7b 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -18,7 +18,7 @@ Adjust your project's name, description, avatar, [default branch](../repository/  -The project description also partially supports [standard markdown](../../markdown.md#standard-markdown). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. +The project description also partially supports [standard markdown](../../markdown.md#standard-markdown-and-extensions-in-gitlab). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. ### Sharing and permissions @@ -26,10 +26,10 @@ Set up your project's access, [visibility](../../../public_access/public_access.  -If Issues are disabled, or you can't access Issues because you're not a project member, then Lables and Milestones +If Issues are disabled, or you can't access Issues because you're not a project member, then Lables and Milestones links will be missing from the sidebar UI. -You can still access them with direct links if you can access Merge Requests. This is deliberate, if you can see +You can still access them with direct links if you can access Merge Requests. This is deliberate, if you can see Issues or Merge Requests, both of which use Labels and Milestones, then you shouldn't be denied access to Labels and Milestones pages. ### Issue settings @@ -109,8 +109,8 @@ You can transfer an existing project into a [group](../../group/index.md) if: 1. You have at least **Maintainer** [permissions] to that group. 1. The project is in a subgroup you own. 1. You are at least a **Maintainer** of the project under your personal namespace. -Similarly, if you are an owner of a group, you can transfer any of its projects -under your own user. + Similarly, if you are an owner of a group, you can transfer any of its projects + under your own user. To transfer a project: diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index b2bf85335c5..7d85f4adfed 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -171,13 +171,13 @@ syntax but with some restrictions: - No global blocks can be defined (ie: `before_script` or `after_script`) - Only one job named `terminal` can be added to this file. - Only the keywords `image`, `services`, `tags`, `before_script`, `script`, and -`variables` are allowed to be used to configure the job. + `variables` are allowed to be used to configure the job. - To connect to the interactive terminal, the `terminal` job must be still alive -and running, otherwise the terminal won't be able to connect to the job's session. -By default the `script` keyword has the value `sleep 60` to prevent -the job from ending and giving the Web IDE enough time to connect. This means -that, if you override the default `script` value, you'll have to add a command -which would keep the job running, like `sleep`. + and running, otherwise the terminal won't be able to connect to the job's session. + By default the `script` keyword has the value `sleep 60` to prevent + the job from ending and giving the Web IDE enough time to connect. This means + that, if you override the default `script` value, you'll have to add a command + which would keep the job running, like `sleep`. In the code below there is an example of this configuration file: diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 3e24557591c..0cb390e1242 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -175,7 +175,7 @@ A merge request is an online place to discuss the change and review the code. If you open the merge request but do not assign it to anyone, it is a "Work In Progress" merge request. These are used to discuss the proposed implementation but are not ready for inclusion in the `master` branch yet. -Start the title of the merge request with "[WIP]" or "WIP:" to prevent it from being merged before it's ready. +Start the title of the merge request with `[WIP]` or `WIP:` to prevent it from being merged before it's ready. When you think the code is ready, assign the merge request to a reviewer. The reviewer can merge the changes when they think the code is ready for inclusion in the `master` branch. diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 5d560f2e000..d49d29c805a 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -64,18 +64,18 @@ These settings can be configured on project page under the name of the project. Below is the table of events users can be notified of: -| Event | Sent to | Settings level | -|------------------------------|-------------------------------------------------------------------|------------------------------| -| New SSH key added | User | Security email, always sent. | -| New email added | User | Security email, always sent. | -| Email changed | User | Security email, always sent. | -| Password changed | User | Security email, always sent. | -| New user created | User | Sent on user creation, except for omniauth (LDAP)| -| User added to project | User | Sent when user is added to project | -| Project access level changed | User | Sent when user project access level is changed | -| User added to group | User | Sent when user is added to group | -| Group access level changed | User | Sent when user group access level is changed | -| Project moved | Project members [1] | [1] not disabled | +| Event | Sent to | Settings level | +|------------------------------|---------------------|------------------------------| +| New SSH key added | User | Security email, always sent. | +| New email added | User | Security email, always sent. | +| Email changed | User | Security email, always sent. | +| Password changed | User | Security email, always sent. | +| New user created | User | Sent on user creation, except for omniauth (LDAP)| +| User added to project | User | Sent when user is added to project | +| Project access level changed | User | Sent when user project access level is changed | +| User added to group | User | Sent when user is added to group | +| Group access level changed | User | Sent when user group access level is changed | +| Project moved | Project members (1) | (1) not disabled | ### Issue / Epics / Merge request events diff --git a/doc/workflow/time_tracking.md b/doc/workflow/time_tracking.md index c03dffa967d..4286a3625a2 100644 --- a/doc/workflow/time_tracking.md +++ b/doc/workflow/time_tracking.md @@ -73,7 +73,15 @@ The following time units are available: Default conversion rates are 1mo = 4w, 1w = 5d and 1d = 8h. -Other interesting links: +### Limit displayed units to hours + +> Introduced in GitLab 12.0. + +The display of time units can be limited to hours through the option in **Admin Area > Settings > Preferences** under 'Localization'. + +With this option enabled, `75h` is displayed instead of `1w 4d 3h`. + +## Other interesting links - [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/solutions/time-tracking/) |
