diff options
Diffstat (limited to 'doc')
254 files changed, 5388 insertions, 2901 deletions
diff --git a/doc/README.md b/doc/README.md index 3863e17c268..489c8117b9d 100644 --- a/doc/README.md +++ b/doc/README.md @@ -54,7 +54,7 @@ GitLab provides solutions for [all the stages of the DevOps lifecycle](https://a  GitLab is like a top-of-the-line kitchen for making software. As the executive -chef, you decide what software you want serve. Using your recipe, GitLab handles +chef, you decide what software you want to serve. Using your recipe, GitLab handles all the prep work, cooking, and delivery, so you can turn around orders faster than ever. @@ -209,7 +209,7 @@ The following documentation relates to the DevOps **Create** stage: | [GitLab API](api/README.md) | Integrate GitLab via a simple and powerful API. | | [GitLab Integration](integration/README.md) | Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. | | [GitLab Webhooks](user/project/integrations/webhooks.md) | Let GitLab notify you when new code has been pushed to your project. | -| [JIRA Development Panel](integration/jira_development_panel.md) **[PREMIUM]** | See GitLab information in the JIRA Development Panel. | +| [Jira Development Panel](integration/jira_development_panel.md) **[PREMIUM]** | See GitLab information in the Jira Development Panel. | | [Project Services](user/project/integrations/project_services.md) | Integrate a project with external services, such as CI and chat. | | [Trello Power-Up](integration/trello_power_up.md) | Integrate with GitLab's Trello Power-Up. | diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index 760af0cfd1a..c668f19ca7d 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -13,7 +13,7 @@ The steps below cover: ## Configuring Google LDAP client -1. Navigate to https://admin.google.com and sign in as a GSuite domain administrator. +1. Navigate to <https://admin.google.com> and sign in as a GSuite domain administrator. 1. Go to **Apps > LDAP > Add Client**. diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 30095d35705..b45966fa920 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -183,20 +183,29 @@ group, as opposed to the full DN. 1. [Restart GitLab][restart] for the changes to take effect. +## Global group memberships lock + +"Lock memberships to LDAP synchronization" setting allows instance administrators +to lock down user abilities to invite new members to a group. When enabled following happens: + +1. Only administrator can manage memberships of any group including access levels. +2. Users are not allowed to share project with other groups or invite members to a project created in a group. + + ## Adjusting LDAP user sync schedule > Introduced in GitLab Enterprise Edition Starter. NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. You can manually configure LDAP user sync times by setting the following configuration values. The example below shows how to set LDAP user -sync to run once every 12 hours at the top of the hour. +sync to run once every 12 hours at the top of the hour. **Omnibus installations** @@ -224,7 +233,7 @@ sync to run once every 12 hours at the top of the hour. NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a group sync process every hour, on the hour. @@ -236,8 +245,8 @@ for installations with a large number of LDAP users. Please review the your installation compares before proceeding. You can manually configure LDAP group sync times by setting the -following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. **Omnibus installations** diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 54279897e04..1f2961ea39a 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -396,21 +396,34 @@ omniauth-ldap. ### Escaping special characters -If the `user_filter` DN contains special characters. For example, a comma: +The `user_filter` DN can contain special characters. For example: -``` -OU=GitLab, Inc,DC=gitlab,DC=com -``` +- A comma: -This character needs to be escaped as documented in [RFC 4515](https://tools.ietf.org/search/rfc4515). + ``` + OU=GitLab, Inc,DC=gitlab,DC=com + ``` -Due to the way the string is parsed, the special character needs to be converted -to hex and `\\5C\\` (`5C` = `\` in hex) added before it. -As an example the above DN would look like +- Open and close brackets: -``` -OU=GitLab\\5C\\2C Inc,DC=gitlab,DC=com -``` + ``` + OU=Gitlab (Inc),DC=gitlab,DC=com + ``` + + These characters must be escaped as documented in + [RFC 4515](https://tools.ietf.org/search/rfc4515). + +- Escape commas with `\2C`. For example: + + ``` + OU=GitLab\2C Inc,DC=gitlab,DC=com + ``` + +- Escape open and close brackets with `\28` and `\29`, respectively. For example: + + ``` + OU=Gitlab \28Inc\29,DC=gitlab,DC=com + ``` ## Enabling LDAP sign-in for existing GitLab users diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 00422ec347c..6e48add6930 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -144,20 +144,20 @@ for more details: If you're having trouble, here are some tips: 1. Ensure `discovery` is set to `true`. Setting it to `false` requires -specifying all the URLs and keys required to make OpenID work. + specifying all the URLs and keys required to make OpenID work. 1. Check your system clock to ensure the time is synchronized properly. 1. As mentioned in [the -documentation](https://github.com/m0n9oose/omniauth_openid_connect), -make sure `issuer` corresponds to the base URL of the Discovery URL. For -example, `https://accounts.google.com` is used for the URL -`https://accounts.google.com/.well-known/openid-configuration`. + documentation](https://github.com/m0n9oose/omniauth_openid_connect), + make sure `issuer` corresponds to the base URL of the Discovery URL. For + example, `https://accounts.google.com` is used for the URL + `https://accounts.google.com/.well-known/openid-configuration`. 1. The OpenID Connect client uses HTTP Basic Authentication to send the -OAuth2 access token. For example, if you are seeing 401 errors upon -retrieving the `userinfo` endpoint, you may want to check your OpenID -Web server configuration. For example, for -[oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you -may need to [add a configuration parameter to -Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). + OAuth2 access token. For example, if you are seeing 401 errors upon + retrieving the `userinfo` endpoint, you may want to check your OpenID + Web server configuration. For example, for + [oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you + may need to [add a configuration parameter to + Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 7f3be402b84..98404ff2a10 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -40,16 +40,16 @@ For example, say you have a primary (`db1.gitlab.com`) and two secondaries, `db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 load balancers, one for every host. For example: -* `primary.gitlab.com` forwards to `db1.gitlab.com` -* `secondary1.gitlab.com` forwards to `db2.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db1.gitlab.com` +- `secondary1.gitlab.com` forwards to `db2.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` Now let's say that a failover happens and db2 becomes the new primary. This means forwarding should now happen as follows: -* `primary.gitlab.com` forwards to `db2.gitlab.com` -* `secondary1.gitlab.com` forwards to `db1.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db2.gitlab.com` +- `secondary1.gitlab.com` forwards to `db1.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` GitLab does not take care of this for you, so you will need to do so yourself. @@ -209,9 +209,9 @@ without it immediately leading to errors being presented to the users. The load balancer logs various messages, such as: -* When a host is marked as offline -* When a host comes back online -* When all secondaries are offline +- When a host is marked as offline +- When a host comes back online +- When all secondaries are offline Each log message contains the tag `[DB-LB]` to make searching/filtering of such log entries easier. For example: diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index e6c8f59549f..874b1f3c80d 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -42,7 +42,7 @@ The list of `GITLAB_DATABASE_XXX` variables that you can set is: Variable | Default value | Overridden by `DATABASE_URL`? -------- | ------------- | ----------------------------- -`GITLAB_DATABASE_ADAPTER` | `postgresql` (for MySQL use `mysql2`) | Yes +`GITLAB_DATABASE_ADAPTER` | `postgresql` | Yes `GITLAB_DATABASE_DATABASE` | `gitlab_#{ENV['RAILS_ENV']` | Yes `GITLAB_DATABASE_USERNAME` | `root` | Yes `GITLAB_DATABASE_PASSWORD` | None | Yes @@ -64,4 +64,4 @@ instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.htm It's possible to preconfigure the GitLab docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](http://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). +For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](https://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 1e5a56c3f4e..6658c37752a 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -489,10 +489,6 @@ work: gitlab-ctl reconfigure ``` -## MySQL replication - -MySQL replication is not supported for Geo. - ## Troubleshooting Read the [troubleshooting document](troubleshooting.md). diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index dd1af0dbf9c..c527248bc72 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -6,8 +6,8 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin ## How does Geo know which projects to sync? -On each **secondary** node, there is a read-only replicated copy of the GitLab database. -A **secondary** node also has a tracking database where it stores which projects have been synced. +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. @@ -15,19 +15,19 @@ At the start, this tracking database is empty, so Geo will start trying to updat For each project to sync: 1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. -If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. + If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. 1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. -When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when -the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. -Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. -If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. -So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. ## Can I use Geo in a disaster recovery situation? diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 921a3ef1c7a..28ad89c4446 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -79,9 +79,9 @@ The **primary** database will require modification later, as part of A **secondary** cluster is similar to any other GitLab HA cluster, with two major differences: -* The main PostgreSQL database is a read-only replica of the **primary** node's +- The main PostgreSQL database is a read-only replica of the **primary** node's PostgreSQL database. -* There is also a single PostgreSQL database for the **secondary** cluster, +- There is also a single PostgreSQL database for the **secondary** cluster, called the "tracking database", which tracks the synchronization state of various resources. @@ -93,9 +93,9 @@ from the normal HA setup. Configure the following services, again using the non-Geo high availability documentation: -* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high +- [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high availability. -* [NFS](../../high_availability/nfs.md) which will store data that is +- [NFS](../../high_availability/nfs.md) which will store data that is synchronized from the **primary** node. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node @@ -270,15 +270,15 @@ After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the chang On the secondary the following GitLab frontend services will be enabled: -* geo-logcursor -* gitlab-pages -* gitlab-workhorse -* logrotate -* nginx -* registry -* remote-syslog -* sidekiq -* unicorn +- geo-logcursor +- gitlab-pages +- gitlab-workhorse +- logrotate +- nginx +- registry +- remote-syslog +- sidekiq +- unicorn Verify these services by running `sudo gitlab-ctl status` on the frontend application servers. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index c5bdd36ba70..5bd6cc81362 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,15 +1,23 @@ # Geo Troubleshooting **[PREMIUM ONLY]** -NOTE: **Note:** -This list is an attempt to document all the moving parts that can go wrong. -We are working into getting all this steps verified automatically in a -rake task in the future. - Setting up Geo requires careful attention to details and sometimes it's easy to -miss a step. Here is a list of questions you should ask to try to detect -what you need to fix (all commands and path locations are for Omnibus installs): +miss a step. + +Here is a list of steps you should take to attempt to fix problem: + +- Perform [basic troubleshooting](#basic-troubleshooting). +- Fix any [replication errors](#fixing-replication-errors). +- Fix any [Foreign Data Wrapper](#fixing-foreign-data-wrapper-errors) errors. +- Fix any [common](#fixing-common-errors) errors. + +## Basic troubleshooting -## First check the health of the **secondary** node +Before attempting more advanced troubleshooting: + +- Check [the health of the **secondary** node](#check-the-health-of-the-secondary-node). +- Check [if PostgreSQL replication is working](#check-if-postgresql-replication-is-working). + +### Check the health of the **secondary** node Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in your browser. We perform the following health checks on each **secondary** node @@ -23,10 +31,12 @@ to help identify if something is wrong:  -For information on how to resolve common errors reported from the UI, see [common errors](#common-errors). +For information on how to resolve common errors reported from the UI, see +[Fixing Common Errors](#fixing-common-errors). If the UI is not working, or you are unable to log in, you can run the Geo health check manually to get this information as well as a few more details. + This rake task can be run on an app node in the **primary** or **secondary** Geo nodes: @@ -36,7 +46,7 @@ sudo gitlab-rake gitlab:geo:check Example output: -``` +```text Checking Geo ... GitLab Geo is available ... yes @@ -68,7 +78,7 @@ sudo gitlab-rake geo:status Example output: -``` +```text http://secondary.example.com/ ----------------------------------------------------- GitLab Version: 11.10.4-ee @@ -89,16 +99,21 @@ http://secondary.example.com/ Last status report was: 2 minutes ago ``` -## Is Postgres replication working? +### Check if PostgreSQL replication is working + +To check if PostgreSQL replication is working, check if: + +- [Nodes are pointing to the correct database instance](#are-nodes-pointing-to-the-correct-database-instance). +- [Geo can detect the current node correctly](#can-geo-detect-the-current-node-correctly). -### Are my nodes pointing to the correct database instance? +#### Are nodes pointing to the correct database instance? You should make sure your **primary** Geo node points to the instance with writing permissions. Any **secondary** nodes should point only to read-only instances. -### Can Geo detect my current node correctly? +#### Can Geo detect the current node correctly? Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. @@ -112,29 +127,38 @@ sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" and expect something like: -``` +```ruby #<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...> ``` By running the command above, `primary` should be `true` when executed in the **primary** node, and `false` on any **secondary** node. -## How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"? +## Fixing replication errors + +The following sections outline troubleshooting steps for fixing replication +errors. + +### Message: "ERROR: replication slots can only be used if max_replication_slots > 0"? This means that the `max_replication_slots` PostgreSQL variable needs to be set on the **primary** database. In GitLab 9.4, we have made this setting default to 1. You may need to increase this value if you have more -**secondary** nodes. Be sure to restart PostgreSQL for this to take +**secondary** nodes. + +Be sure to restart PostgreSQL for this to take effect. See the [PostgreSQL replication setup][database-pg-replication] guide for more details. -## How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? +### Message: "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? This occurs when PostgreSQL does not have a replication slot for the -**secondary** node by that name. You may want to rerun the [replication +**secondary** node by that name. + +You may want to rerun the [replication process](database.md) on the **secondary** node . -## How do I fix the message, "Command exceeded allowed execution time" when setting up replication? +### Message: "Command exceeded allowed execution time" when setting up replication? This may happen while [initiating the replication process][database-start-replication] on the **secondary** node, and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). @@ -153,7 +177,7 @@ sudo gitlab-ctl \ This will give the initial replication up to six hours to complete, rather than the default thirty minutes. Adjust as required for your installation. -## How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" +### Message: "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" Determine if you have any unused replication slots in the **primary** database. This can cause large amounts of log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`. @@ -184,11 +208,12 @@ Slots where `active` is `f` are not active. SELECT pg_drop_replication_slot('<name_of_extra_slot>'); ``` -## Very large repositories never successfully synchronize on the **secondary** node +### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh `git clone` of a repository on the primary takes more than a few minutes, you may be affected by this. + To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` on the **secondary** node: @@ -205,7 +230,7 @@ sudo gitlab-ctl reconfigure This will increase the timeout to three hours (10800 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. -## How to reset Geo **secondary** node replication +### Reseting Geo **secondary** node replication If you get a **secondary** node in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you: @@ -289,12 +314,16 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start ``` -## How do I fix a "Foreign Data Wrapper (FDW) is not configured" error? +## Fixing Foreign Data Wrapper errors + +This section documents ways to fix potential Foreign Data Wrapper errors. + +### "Foreign Data Wrapper (FDW) is not configured" error When setting up Geo, you might see this warning in the `gitlab-rake gitlab:geo:check` output: -``` +```text GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured ``` @@ -302,12 +331,12 @@ There are a few key points to remember: 1. The FDW settings are configured on the Geo **tracking** database. 1. The configured foreign server enables a login to the Geo -**secondary**, read-only database. + **secondary**, read-only database. By default, the Geo secondary and tracking database are running on the same host on different ports. That is, 5432 and 5431 respectively. -### Checking configuration +#### Checking configuration NOTE: **Note:** The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5. @@ -321,7 +350,7 @@ To check the configuration: ``` 1. Check whether any tables are present. If everything is working, you -should see something like this: + should see something like this: ```sql gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; @@ -419,7 +448,7 @@ should see something like this: - `geo_postgresql['fdw_external_user']` - `geo_postgresql['fdw_external_password']` -### Manual reload of FDW schema +#### Manual reload of FDW schema If you're still unable to get FDW working, you may want to try a manual reload of the FDW schema. To manually reload the FDW schema: @@ -459,9 +488,34 @@ reload of the FDW schema. To manually reload the FDW schema: [database-start-replication]: database.md#step-3-initiate-the-replication-process [database-pg-replication]: database.md#postgresql-replication -## Common errors +### "Geo database has an outdated FDW remote schema" error + +GitLab can error with a `Geo database has an outdated FDW remote schema` message. -This section documents common errors reported in the admin UI and how to fix them. +For example: + +```text +Geo database has an outdated FDW remote schema. It contains 229 of 236 expected tables. Please refer to Geo Troubleshooting. +``` + +To resolve this, run the following command: + +```sh +sudo gitlab-rake geo:db:refresh_foreign_tables +``` + +## Expired artifacts + +If you notice for some reason there are more artifacts on the Geo +secondary node than on the Geo primary node, you can use the rake task +to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). + +On a Geo **secondary** node, this command will also clean up all Geo +registry record related to the orphan files on disk. + +## Fixing common errors + +This section documents common errors reported in the Admin UI and how to fix them. ### Geo database configuration file is missing @@ -470,7 +524,6 @@ GitLab cannot find or doesn't have permission to access the `database_geo.yml` c In an Omnibus GitLab installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. - If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node. @@ -503,7 +556,7 @@ Make sure you follow the [Geo database replication](database.md) instructions fo If you are using GitLab Omnibus installation, something might have failed during upgrade. You can: -- Run `sudo gitlab-ctl reconfigure`. +- Run `sudo gitlab-ctl reconfigure`. - Manually trigger the database migration by running: `sudo gitlab-rake geo:db:migrate` as root on the **secondary** node. ### Geo database is not configured to use Foreign Data Wrapper @@ -511,4 +564,4 @@ If you are using GitLab Omnibus installation, something might have failed during This error means the Geo Tracking Database doesn't have the FDW server and credentials configured. -See [How do I fix a "Foreign Data Wrapper (FDW) is not configured" error?](#how-do-i-fix-a-foreign-data-wrapper-fdw-is-not-configured-error). +See ["Foreign Data Wrapper (FDW) is not configured" error?](#foreign-data-wrapper-fdw-is-not-configured-error). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 53a85dfad6c..0bffe2f7472 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -48,7 +48,7 @@ used by Omnibus and the GitLab source installation guide. Starting with GitLab 11.4, Gitaly is able to serve all Git requests without needed a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the -[Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still be leveraged for redudancy on block level of the Git data. But only has to be mounted on the Gitaly server. @@ -432,6 +432,24 @@ gitaly_enabled=false When you run `service gitlab restart` Gitaly will be disabled on this particular machine. +## Eliminating NFS altogether + +If you are planning to use Gitaly without NFS for your storage needs +and want to eliminate NFS from your environment altogether, there are +a few things that you need to do: + + 1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. + 1. Configure [database lookup of SSH keys](https://docs.gitlab.com/ce/administration/operations/fast_ssh_key_lookup.html) + to eliminate the need for a shared authorized_keys file. + 1. Configure [object storage for job artifacts](https://docs.gitlab.com/ce/administration/job_artifacts.html#using-object-storage) + including [live tracing](https://docs.gitlab.com/ce/administration/job_traces.html#new-live-trace-architecture). + 1. Configure [object storage for LFS objects](https://docs.gitlab.com/ce/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage). + 1. Configure [object storage for uploads](https://docs.gitlab.com/ce/administration/uploads.html#using-object-storage-core-only). + +NOTE: **Note:** One current feature of GitLab still requires a shared directory (NFS): [GitLab Pages](../../user/project/pages/index.md). +There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) +to eliminate the need for NFS to support GitLab Pages. + ## Troubleshooting Gitaly in production Since GitLab 11.6, Gitaly comes with a command-line tool called diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index d9c80b1ec59..e81d2741082 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -65,6 +65,7 @@ larger one. - 1 Redis node - 1 NFS/Gitaly storage server - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) +- 1 Monitoring node (Prometheus, Grafana) #### Installation Instructions @@ -76,6 +77,7 @@ you can continue with the next step. 1. [Redis](redis.md#redis-in-a-scaled-environment) 1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md) 1. [GitLab application nodes](gitlab.md) +1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) ### Full Scaling @@ -90,6 +92,7 @@ in size, indicating that there is contention or not enough resources. - 2 or more NFS/Gitaly storage servers - 2 or more Sidekiq nodes - 2 or more GitLab application nodes (Unicorn, Workhorse) +- 1 Monitoring node (Prometheus, Grafana) ## High Availability Architecture Examples @@ -133,6 +136,7 @@ the contention. - 3 Consul/Sentinel nodes - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer) - 1 NFS/Gitaly server +- 1 Monitoring node (Prometheus, Grafana)  @@ -167,12 +171,12 @@ are a work-in-progress representation of the work so far. Quality will be certifying this environment in FY20-Q2. The specifications may be adjusted prior to certification based on performance testing. -- 3 PostgreSQL - 4 CPU, 8GB RAM +- 3 PostgreSQL - 4 CPU, 8GB RAM per node - 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM -- 3 Consul/Sentinel - 2 CPU, 2GB RAM -- 4 Sidekiq - 4 CPU, 8GB RAM -- 5 GitLab application nodes - 20 CPU, 64GB RAM +- 2 Redis - 2 CPU, 8GB RAM per node +- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node +- 4 Sidekiq - 4 CPU, 8GB RAM per node +- 5 GitLab application nodes - 20 CPU, 64GB RAM per node - 1 Gitaly - 20 CPU, 64GB RAM - 1 Monitoring node - 4 CPU, 8GB RAM @@ -192,6 +196,7 @@ with the added complexity of many more nodes to configure, manage and monitor. - 2 or more API nodes (All requests to `/api`) - 2 or more Web nodes (All other web requests) - 2 or more NFS/Gitaly servers +- 1 Monitoring node (Prometheus, Grafana)  @@ -205,4 +210,5 @@ separately: 1. [NFS Client and Host setup](nfs_host_client_setup.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) +1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 3b874e5d312..20bbfdb2603 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -82,6 +82,7 @@ deploy the bundled PostgreSQL. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These will be necessary when configuring the GitLab application servers later. +1. [Enable monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -111,7 +112,7 @@ deploy the bundled PostgreSQL. > - If you are a Community Edition or Starter user, consider using a cloud hosted solution. > - This document will not cover installations from source. > -> - If HA setup is not what you were looking for, see the [database configuration document](http://docs.gitlab.com/omnibus/settings/database.html) +> - If HA setup is not what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) > for the Omnibus GitLab packages. > > Please read this document fully before attempting to configure PostgreSQL HA @@ -203,9 +204,9 @@ Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - If you are using a different username, you will have to specify it. We -will refer to it with `CONSUL_USERNAME`, + will refer to it with `CONSUL_USERNAME`, - There will be a database user created with read only access to the repmgr -database + database - Passwords will be stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -399,6 +400,7 @@ check the [Troubleshooting section](#troubleshooting) before proceeding. ``` 1. [Reconfigure GitLab] for the changes to take effect. +1. [Enable Monitoring](#enable-monitoring) > Please note: > @@ -1086,6 +1088,25 @@ the previous section: the `gitlab` database user 1. [Reconfigure GitLab] for the changes to take effect +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** database servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ## Troubleshooting #### Consul and PostgreSQL changes not taking effect. @@ -1146,7 +1167,7 @@ postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. - [Consul](consul.md#troubleshooting) -- [PostgreSQL](http://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) +- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) - [GitLab application](gitlab.md#troubleshooting) ## Configure using Omnibus diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 1d8e6c999cb..b7eaa4ce105 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -2,13 +2,13 @@ Gitaly does not yet support full high availability. However, Gitaly is quite stable and is in use on GitLab.com. Scaled and highly available GitLab environments -should consider using Gitaly on a separate node. +should consider using Gitaly on a separate node. -See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to -track plans and progress toward high availability support. +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. This document is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) -environments and [High Availability Architecture](README.md#high-availability-architecture-examples). +environments and [High Availability Architecture](README.md#high-availability-architecture-examples). ## Running Gitaly on its own server @@ -19,3 +19,30 @@ Continue configuration of other components by going back to: - [Scaled Architectures](README.md#scalable-architecture-examples) - [High Availability Architectures](README.md#high-availability-architecture-examples) + +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 888426ece5c..3045be616a6 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -66,7 +66,7 @@ gitlab_rails['redis_port'] = '6379' gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server gitlab_rails['redis_password'] = 'Redis Password' - + # Ensure UIDs and GIDs match between servers for permissions via NFS user['uid'] = 9000 user['gid'] = 9000 @@ -76,19 +76,22 @@ registry['gid'] = 9002 ``` +1. [Enable monitoring](#enable-monitoring) + > **Note:** To maintain uniformity of links across HA clusters, the `external_url` on the first application server as well as the additional application servers should point to the external url that users will use to access GitLab. In a typical HA setup, this will be the url of the load balancer which will route traffic to all GitLab application servers in the HA cluster. - > + > > **Note:** When you specify `https` in the `external_url`, as in the example above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If certificates are not present, Nginx will fail to start. See - [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) + [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for more information. > - > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. + > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure + of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. ## First GitLab application server @@ -129,6 +132,47 @@ need some extra configuration. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** GitLab servers. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + unicorn['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with + # the address and/or subnets gathered from the monitoring node(s). + gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`, + it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`. + For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401). + ## Troubleshooting - `mount: wrong fs type, bad option, bad superblock on` diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md new file mode 100644 index 00000000000..ef415dde10a --- /dev/null +++ b/doc/administration/high_availability/monitoring_node.md @@ -0,0 +1,69 @@ +# Configuring a Monitoring node for Scaling and High Availability + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +## Standalone Monitoring node using GitLab Omnibus + +The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana. +The monitoring node is not highly available. See [Scaling and High Availability](README.md) +for an overview of GitLab scaling and high availability options. + +The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with +Omnibus: + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +## Migrating to Service Discovery + +Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, +ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both +`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` +will result in errors. diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 3013ce63bd5..561ba214686 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -235,4 +235,4 @@ Read more on high-availability configuration: 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) -[udp-log-shipping]: http://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" +[udp-log-shipping]: https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 762179cf756..053dae25823 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -62,6 +62,33 @@ See our [HA documentation for PostgreSQL](database.md) for information on runnin 1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + + If you enable Monitoring, it must be enabled on **all** pgbouncer servers. + + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' + ``` + + 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ### Interacting with pgbouncer #### Administrative console diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 1aaa709fc8f..cc338725e1b 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -22,10 +22,10 @@ environments including [Basic Scaling](README.md#basic-scaling) and ### Provide your own Redis instance **[CORE ONLY]** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. ### Standalone Redis using GitLab Omnibus **[CORE ONLY]** @@ -62,11 +62,11 @@ Omnibus: pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false - + redis['bind'] = '0.0.0.0' redis['port'] = '6379' redis['password'] = 'SECRET_PASSWORD_HERE' - + gitlab_rails['auto_migrate'] = false ``` @@ -74,6 +74,7 @@ Omnibus: 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when configuring the GitLab application servers later. +1. [Enable Monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -90,10 +91,10 @@ environments including [Horizontal](README.md#horizontal), ### Provide your own Redis instance **[CORE ONLY]** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. ### High Availability with GitLab Omnibus **[PREMIUM ONLY]** @@ -367,7 +368,7 @@ The prerequisites for a HA Redis setup are the following: ```ruby # Specify server role as 'redis_master_role' roles ['redis_master_role'] - + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -381,7 +382,6 @@ The prerequisites for a HA Redis setup are the following: # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'redis-password-goes-here' ``` - 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following @@ -393,9 +393,9 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. ### Step 2. Configuring the slave Redis instances @@ -411,7 +411,7 @@ The prerequisites for a HA Redis setup are the following: ```ruby # Specify server role as 'redis_slave_role' roles ['redis_slave_role'] - + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -442,9 +442,9 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. --- @@ -749,6 +749,35 @@ gitlab_rails['redis_sentinels'] = [ [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** Redis servers. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ## Advanced configuration Omnibus GitLab configures some things behind the curtains to make the sysadmins' diff --git a/doc/administration/index.md b/doc/administration/index.md index 06d900b152d..602eecb9746 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -33,7 +33,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Install](../install/README.md): Requirements, directory structures, and installation methods. - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **[STARTER ONLY]** - - [Omnibus support for external MySQL DB](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only): Omnibus package supports configuring an external MySQL database. **[STARTER ONLY]** - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **[STARTER ONLY]** - [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability. - [Installing GitLab HA on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab High Availability on Amazon AWS. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index c5cfb8d5016..9921ffd8ea0 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -4,7 +4,7 @@ GitLab has an advanced log system where everything is logged so that you can analyze your instance using various system log files. In addition to system log files, GitLab Enterprise Edition comes with Audit Events. Find more about them [in Audit Events -documentation](http://docs.gitlab.com/ee/administration/audit_events.html) +documentation](https://docs.gitlab.com/ee/administration/audit_events.html) System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. @@ -125,7 +125,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/integrations_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/integrations_json.log` for installations from source. -It contains information about [integrations](../user/project/integrations/project_services.md) activities such as JIRA, Asana and Irker services. It uses JSON format like the example below: +It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below: ``` json {"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"} diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 286b99aceb5..7297507f599 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,70 +1,132 @@ # Extra Sidekiq processes **[STARTER ONLY]** -GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +NOTE: **Note:** +The information in this page applies only to Omnibus GitLab. + +GitLab Starter allows one to start an extra set of Sidekiq processes besides the default one. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. -## Starting extra processes via Omnibus GitLab +## Available Sidekiq queues -To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true` -setting `/etc/gitlab/gitlab.rb`: +For a list of the existing Sidekiq queues, check the following files: -```ruby -sidekiq_cluster['enable'] = true -``` +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/workers/all_queues.yml) -You will then specify how many additional processes to create via `sidekiq-cluster` -as well as which queues for them to handle. This is done via the -`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain -which queues to process. Each item in the array will equate to one additional -sidekiq process. +Each entry in the above files represents a queue on which extra Sidekiq processes +can be started. -As an example, to make additional sidekiq processes that process the -`elastic_indexer` and `mailers` queues, you would apply the following: +## Starting extra processes -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer", - "mailers" -] -``` +To start extra Sidekiq processes, you must enable `sidekiq-cluster`: -To have an additional sidekiq process handle multiple queues, you simply put a -comma after the first queue name and then put the next queue name: +1. Edit `/etc/gitlab/gitlab.rb` and add: -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer,elastic_commit_indexer", - "mailers" -] -``` + ```ruby + sidekiq_cluster['enable'] = true + ``` -Keep in mind, all changes must be followed by reconfiguring your GitLab -application via `sudo gitlab-ctl reconfigure`. +1. You will then need to specify how many additional processes to create via `sidekiq-cluster` + and which queue they should handle via the `sidekiq_cluster['queue_groups']` + array setting. Each item in the array equates to one additional Sidekiq + process, and values in each item determine the queues it works on. -### Monitoring + For example, the following setting adds additional Sidekiq processes to two + queues, one to `elastic_indexer` and one to `mailers`: -Once the Sidekiq processes are added, you can visit the "Background Jobs" + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer", + "mailers" + ] + ``` + + To have an additional Sidekiq process handle multiple queues, add multiple + queue names to its item delimited by commas. For example: + + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer, elastic_commit_indexer", + "mailers" + ] + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +Once the extra Sidekiq processes are added, you can visit the "Background Jobs" section under the admin area in GitLab (`/admin/background_jobs`). - + -### All queues with exceptions +## Negating settings -To have the additional sidekiq processes work on every queue EXCEPT the ones +To have the additional Sidekiq processes work on every queue **except** the ones you list: +1. After you follow the steps for [starting extra processes](#starting-extra-processes), + edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['negate'] = true + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +## Ignore all GitHub import queues + +When [importing from GitHub](../../user/project/import/github.md), Sidekiq might +use all of its resources to perform those operations. To set up a separate +`sidekiq-cluster` process to ignore all GitHub import-related queues: + 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby + sidekiq_cluster['enable'] = true sidekiq_cluster['negate'] = true + sidekiq_cluster['queue_groups'] = [ + "github_import_advance_stage", + "github_importer:github_import_import_diff_note", + "github_importer:github_import_import_issue", + "github_importer:github_import_import_note", + "github_importer:github_import_import_lfs_object", + "github_importer:github_import_import_pull_request", + "github_importer:github_import_refresh_import_jid", + "github_importer:github_import_stage_finish_import", + "github_importer:github_import_stage_import_base_data", + "github_importer:github_import_stage_import_issues_and_diff_notes", + "github_importer:github_import_stage_import_notes", + "github_importer:github_import_stage_import_lfs_objects", + "github_importer:github_import_stage_import_pull_requests", + "github_importer:github_import_stage_import_repository" + ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: + ```sh + sudo gitlab-ctl reconfigure + ``` -### Limiting concurrency +## Number of threads + +Each process defined under `sidekiq_cluster` starts with a +number of threads that equals the number of queues, plus one spare thread. +For example, a process that handles the `process_commit` and `post_receive` +queues will use three threads in total. + +## Limiting concurrency + +To limit the concurrency of the Sidekiq processes: 1. Edit `/etc/gitlab/gitlab.rb` and add: @@ -72,11 +134,22 @@ you list: sidekiq_cluster['concurrency'] = 25 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: -Keep in mind, this normally would not exceed the number of CPU cores available. + ```sh + sudo gitlab-ctl reconfigure + ``` -### Modifying the check interval +For each queue group, the concurrency factor will be set to `min(number of queues, N)`. +Setting the value to 0 will disable the limit. Keep in mind this normally would +not exceed the number of CPU cores available. + +Each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) +for more details. + +## Modifying the check interval To modify the check interval for the additional Sidekiq processes: @@ -90,9 +163,14 @@ To modify the check interval for the additional Sidekiq processes: This tells the additional processes how often to check for enqueued jobs. -## Starting extra processes via command line +## Troubleshooting using the CLI -Starting extra Sidekiq processes can be done using the command +CAUTION: **Warning:** +It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. +If you experience a problem, you should contact GitLab support. Use the command +line at your own risk. + +For debugging purposes, you can start extra Sidekiq processes by using the command `/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster`. This command takes arguments using the following syntax: @@ -111,29 +189,29 @@ see the relevant section in the [Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). For example, say you want to start 2 extra processes: one to process the -"process_commit" queue, and one to process the "post_receive" queue. This can be +`process_commit` queue, and one to process the `post_receive` queue. This can be done as follows: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit post_receive ``` -If you instead want to start one process processing both queues you'd use the +If you instead want to start one process processing both queues, you'd use the following syntax: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive ``` -If you want to have one Sidekiq process process the "process_commit" and -"post_receive" queues, and one process to process the "gitlab_shell" queue, +If you want to have one Sidekiq process dealing with the `process_commit` and +`post_receive` queues, and one process to process the `gitlab_shell` queue, you'd use the following: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive gitlab_shell ``` -### Monitoring +### Monitoring the `sidekiq-cluster` command The `sidekiq-cluster` command will not terminate once it has started the desired amount of Sidekiq processes. Instead, the process will continue running and @@ -172,24 +250,24 @@ command and not the PID(s) of the started Sidekiq processes. The Rails environment can be set by passing the `--environment` flag to the `sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The -default value is "development". +default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`. -### All queues with exceptions +### Using negation You're able to run all queues in `sidekiq_queues.yml` file on a single or multiple processes with exceptions using the `--negate` flag. For example, say you want to run a single process for all queues, -except "process_commit" and "post_receive". You can do so by executing: +except `process_commit` and `post_receive`: ```bash -sidekiq-cluster process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive --negate ``` -For multiple processes of all queues (except "process_commit" and "post_receive"): +For multiple processes of all queues (except `process_commit` and `post_receive`): ```bash -sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate ``` ### Limiting concurrency @@ -201,18 +279,3 @@ the `-m N` option. For example, this would cap the maximum number of threads to ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive -m 1 ``` - -For each queue group, the concurrency factor will be set to min(number of -queues, N). Setting the value to 0 will disable the limit. - -Note that each thread requires a Redis connection, so adding threads may -increase Redis latency and potentially cause client timeouts. See the [Sidekiq -documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) -for more details. - -## Number of threads - -Each process started using `sidekiq-cluster` (whether it be via command line or -via the gitlab.rb file) starts with a number of threads that equals the number -of queues, plus one spare thread. For example, a process that handles the -"process_commit" and "post_receive" queues will use 3 threads in total. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 0b4c1ae15d6..becd480a08f 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -62,7 +62,7 @@ It will check that each component was set up according to the installation guide You may also have a look at our Troubleshooting Guides: - [Troubleshooting Guide (GitLab)](http://docs.gitlab.com/ee/README.html#troubleshooting) -- [Troubleshooting Guide (Omnibus Gitlab)](http://docs.gitlab.com/omnibus/README.html#troubleshooting) +- [Troubleshooting Guide (Omnibus Gitlab)](https://docs.gitlab.com/omnibus/README.html#troubleshooting) **Omnibus Installation** diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 38842693d73..834b41b3a2c 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -47,18 +47,12 @@ Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, especially if using any type of network based filesystem. -CAUTION: **Caution:** -For Geo in particular: Geo does work with legacy storage, but in some -edge cases due to race conditions it can lead to errors when a project is -renamed multiple times in short succession, or a project is deleted and -recreated under the same name very quickly. We expect these race events to be -rare, and we have not observed a race condition side-effect happening yet. -This pattern also exists in other objects stored in GitLab, like issue -Attachments, GitLab Pages artifacts, Docker Containers for the integrated -Registry, etc. Hashed storage is a requirement for Geo. - ## Hashed Storage +CAUTION: **Important:** +Geo requires Hashed Storage since 12.0. If you haven't migrated yet, +check the [migration instructions](#how-to-migrate-to-hashed-storage) ASAP. + Hashed Storage is the new storage behavior we rolled out with 10.0. Instead of coupling project URL and the folder structure where the repository will be stored on disk, we are coupling a hash, based on the project's ID. This makes diff --git a/doc/api/README.md b/doc/api/README.md index 3a1064b787e..23c69deef23 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -35,6 +35,7 @@ The following API resources are available in the project context: | [Environments](environments.md) | `/projects/:id/environments` | | [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | | [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | +| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | | [Issue boards](boards.md) | `/projects/:id/boards` | | [Issue links](issue_links.md) **[STARTER]** | `/projects/:id/issues/.../links` | | [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | @@ -92,6 +93,7 @@ The following API resources are available in the group context: | [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | | [Group milestones](group_milestones.md) | `/groups/:id/milestones` | | [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | +| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | | [Members](members.md) | `/groups/:id/members` (also available for projects) | | [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | | [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | @@ -116,6 +118,7 @@ The following API resources are available outside of project and group contexts | [Geo Nodes](geo_nodes.md) **[PREMIUM ONLY]** | `/geo_nodes` | | [Import repository from GitHub](import.md) | `/import/github` | | [Issues](issues.md) | `/issues` (also available for groups and projects) | +| [Issues Statistics](issues_statistics.md) | `/issues_statistics` (also available for groups and projects) | | [Keys](keys.md) | `/keys` | | [License](license.md) **[CORE ONLY]** | `/license` | | [Markdown](markdown.md) | `/markdown` | diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index 1f17af1f1e9..64ea15bca93 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -145,6 +145,9 @@ DELETE /projects/:id/registry/repositories/:repository_id/tags/:tag_name curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags/v10.0.0" ``` +This action does not delete blobs. In order to delete them and recycle disk space, +[run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/README.html#removing-unused-layers-not-referenced-by-manifests). + ## Delete repository tags in bulk Delete repository tags in bulk based on given criteria. @@ -174,6 +177,8 @@ This API call performs the following operations: These operations are executed asynchronously and it might take time to get executed. You can run this at most once an hour for a given container repository. +This action does not delete blobs. In order to delete them and recycle disk space, +[run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/README.html#removing-unused-layers-not-referenced-by-manifests). NOTE: **Note:** Due to a [Docker Distribution deficiency](https://gitlab.com/gitlab-org/gitlab-ce/issues/21405), diff --git a/doc/api/environments.md b/doc/api/environments.md index ebcdc546d08..44f86861ff7 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -11,9 +11,11 @@ GET /projects/:id/environments | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | no | Return the environment with this name. Mutually exclusive with `search` | +| `search` | string | no | Return list of environments matching the search criteria. Mutually exclusive with `name` | ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/environments +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/environments?name=review%2Ffix-foo ``` Example response: diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 88e657a5d2f..6c1cce620ca 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -46,7 +46,7 @@ curl --data "value=100" --header "PRIVATE-TOKEN: <your_access_token>" https://gi A first iteration of a GraphQL API includes the following queries 1. `project` : Within a project it is also possible to fetch a `mergeRequest` by IID. -1. `group` : Only basic group information is currently supported. +1. `group` : Basic group information and epics **[ULTIMATE]** are currently supported. 1. `namespace` : Within a namespace it is also possible to fetch `projects`. ### Multiplex queries diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index ddac81328b9..49aaac06b46 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -222,7 +222,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/approvals "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } } ], @@ -314,7 +314,7 @@ PUT /projects/:id/merge_requests/:merge_request_iid/approvers "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } } ], @@ -387,7 +387,7 @@ does not match, the response code will be `409`. "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } }, { @@ -397,7 +397,7 @@ does not match, the response code will be `409`. "id": 2, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/cf7ad14b34162a76d593e3affca2adca?s=80\u0026d=identicon", - "web_url": "http://localhost:3000/u/ryley" + "web_url": "http://localhost:3000/ryley" } } ], diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index dd7810c3403..85a07589956 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1191,33 +1191,29 @@ Parameters: } ``` -## Merge to default merge ref path +## Returns the up to date merge-ref HEAD commit Merge the changes between the merge request source and target branches into `refs/merge-requests/:iid/merge` -ref, of the target project repository. This ref will have the state the target branch would have if +ref, of the target project repository, if possible. This ref will have the state the target branch would have if a regular merge action was taken. -This is not a regular merge action given it doesn't change the merge request state in any manner. +This is not a regular merge action given it doesn't change the merge request target branch state in any manner. -This ref (`refs/merge-requests/:iid/merge`) is **always** overwritten when submitting -requests to this API, so none of its state is kept or used in the process. +This ref (`refs/merge-requests/:iid/merge`) isn't necessarily overwritten when submitting +requests to this API, though it'll make sure the ref has the latest possible state. -If the merge request has conflicts, is empty or already merged, -you'll get a `400` and a descriptive error message. If you don't have permissions to do so, -you'll get a `403`. +If the merge request has conflicts, is empty or already merged, you'll get a `400` and a descriptive error message. -It returns the HEAD commit of `refs/merge-requests/:iid/merge` in the response body in -case of `200`. +It returns the HEAD commit of `refs/merge-requests/:iid/merge` in the response body in case of `200`. ``` -PUT /projects/:id/merge_requests/:merge_request_iid/merge_to_ref +GET /projects/:id/merge_requests/:merge_request_iid/merge_ref ``` Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `merge_request_iid` (required) - Internal ID of MR -- `merge_commit_message` (optional) - Custom merge commit message ```json { @@ -1477,7 +1473,7 @@ Example response when the GitLab issue tracker is used: ] ``` -Example response when an external issue tracker (e.g. JIRA) is used: +Example response when an external issue tracker (e.g. Jira) is used: ```json [ diff --git a/doc/api/project_aliases.md b/doc/api/project_aliases.md new file mode 100644 index 00000000000..65845579409 --- /dev/null +++ b/doc/api/project_aliases.md @@ -0,0 +1,105 @@ +# Project Aliases API **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3264) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.1. + +All methods require administrator authorization. + +## List all project aliases + +Get a list of all project aliases: + +``` +GET /project_aliases +``` + +``` +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases" +``` + +Example response: + +```json +[ + { + "id": 1, + "project_id": 1, + "name": "gitlab-ce" + }, + { + "id": 2, + "project_id": 2, + "name": "gitlab-ee" + } +] +``` + +## Get project alias' details + +Get details of a project alias: + +``` +GET /project_aliases/:name +``` + +| Attribute | Type | Required | Description | +|-----------|--------|----------|-----------------------| +| `name` | string | yes | The name of the alias | + +``` +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases/gitlab-ee" +``` + +Example response: + +```json +{ + "id": 1, + "project_id": 1, + "name": "gitlab-ee" +} +``` + +## Create a project alias + +Add a new alias for a project. Responds with a 201 when successful, +400 when there are validation errors (e.g. alias already exists): + +``` +POST /project_aliases +``` + +| Attribute | Type | Required | Description | +|--------------|--------|----------|-----------------------------------------------| +| `project_id` | string | yes | The ID or URL-encoded path of the project. | +| `name` | string | yes | The name of the alias. Must be unique. | + +``` +curl --request POST "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=gitlab-org%2Fgitlab-ee" --form "name=gitlab-ee" +``` + +Example response: + +```json +{ + "id": 1, + "project_id": 1, + "name": "gitlab-ee" +} +``` + +## Delete a project alias + +Removes a project aliases. Responds with a 204 when project alias +exists, 404 when it doesn't: + +``` +DELETE /project_aliases/:name +``` + +| Attribute | Type | Required | Description | +|-----------|--------|----------|-----------------------| +| `name` | string | yes | The name of the alias | + +``` +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases/gitlab-ee" +``` diff --git a/doc/api/search.md b/doc/api/search.md index da81c8321c9..abb77ae05dc 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -285,7 +285,7 @@ Example response: ### Scope: wiki_blobs **[STARTER]** -This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. ```bash curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=wiki_blobs&search=bye @@ -346,6 +346,7 @@ Example response: This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. Filters are available for this scope: + - filename - path - extension @@ -679,6 +680,7 @@ Example response: This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. Filters are available for this scope: + - filename - path - extension diff --git a/doc/api/services.md b/doc/api/services.md index 898cfad7254..c811d0e84ca 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -22,6 +22,7 @@ Parameters: | --------- | ---- | -------- | ----------- | | `api_key` | string | true | User API token. User must have access to task, all comments will be attributed to this user. | | `restrict_to_branch` | string | false | Comma-separated list of branches which will be automatically inspected. Leave blank to include all branches. | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Asana service @@ -57,6 +58,7 @@ Parameters: | --------- | ---- | -------- | ----------- | | `token` | string | true | The authentication token | `subdomain` | string | false | The subdomain setting | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Assembla service @@ -96,6 +98,7 @@ Parameters: | `build_key` | string | true | Bamboo build plan key like KEY | | `username` | string | true | A user with API access, if applicable | | `password` | string | true | Password of the user | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Atlassian Bamboo CI service @@ -134,6 +137,7 @@ Parameters: | `project_url` | string | true | Project url | | `description` | string | false | Description | | `title` | string | false | Title | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Bugzilla Service @@ -170,6 +174,7 @@ Parameters: | `token` | string | true | Buildkite project GitLab token | | `project_url` | string | true | `https://buildkite.com/example/project` | | `enable_ssl_verification` | boolean | false | Enable SSL verification | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Buildkite service @@ -206,6 +211,7 @@ Parameters: | `token` | string | true | Campfire token | | `subdomain` | string | false | Campfire subdomain | | `room` | string | false | Campfire room | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Campfire service @@ -244,6 +250,7 @@ Parameters: | `project_url` | string | true | Project url | `description` | string | false | Description | `title` | string | false | Title +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Custom Issue Tracker service @@ -280,6 +287,9 @@ Parameters: | `token` | string | true | Drone CI project specific token | | `drone_url` | string | true | `http://drone.example.com` | | `enable_ssl_verification` | boolean | false | Enable SSL verification | +| `push_events` | boolean | false | Enable notifications for push events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | ### Delete Drone CI service @@ -316,6 +326,8 @@ Parameters: | `recipients` | string | true | Emails separated by whitespace | | `disable_diffs` | boolean | false | Disable code diffs | | `send_from_committer_email` | boolean | false | Send from committer | +| `push_events` | boolean | false | Enable notifications for push events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | ### Delete Emails on push service @@ -384,6 +396,7 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `token` | string | true | Flowdock Git source token | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Flowdock service @@ -471,6 +484,14 @@ Parameters: | `room` | string | false |Room name or ID | | `api_version` | string | false | Leave blank for default (v2) | | `server` | string | false | Leave blank for default. For example, `https://hipchat.example.com`. | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidental_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | ### Delete HipChat service @@ -511,6 +532,7 @@ Parameters: | `server_host` | string | false | localhost | | `server_port` | integer | false | 6659 | | `colorize_messages` | boolean | false | Colorize messages | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Irker (IRC gateway) service @@ -528,21 +550,21 @@ Get Irker (IRC gateway) service settings for a project. GET /projects/:id/services/irker ``` -## JIRA +## Jira -JIRA issue tracker. +Jira issue tracker. -### Get JIRA service settings +### Get Jira service settings -Get JIRA service settings for a project. +Get Jira service settings for a project. ``` GET /projects/:id/services/jira ``` -### Create/Edit JIRA service +### Create/Edit Jira service -Set JIRA service for a project. +Set Jira service for a project. > Starting with GitLab 8.14, `api_url`, `issues_url`, `new_issue_url` and > `project_url` are replaced by `url`. If you are using an @@ -556,16 +578,18 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `url` | string | yes | The URL to the JIRA project which is being linked to this GitLab project. For example, `https://jira.example.com`. | -| `api_url` | string | no | The base URL to the JIRA instance API. Web URL value will be used if not set. For example, `https://jira-api.example.com`. | -| `username` | string | yes | The username of the user created to be used with GitLab/JIRA. | -| `password` | string | yes | The password of the user created to be used with GitLab/JIRA. | +| `url` | string | yes | The URL to the Jira project which is being linked to this GitLab project. For example, `https://jira.example.com`. | +| `api_url` | string | no | The base URL to the Jira instance API. Web URL value will be used if not set. For example, `https://jira-api.example.com`. | +| `username` | string | yes | The username of the user created to be used with GitLab/Jira. | +| `password` | string | yes | The password of the user created to be used with GitLab/Jira. | | `active` | boolean | no | Activates or deactivates the service. Defaults to false (deactivated). | -| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the JIRA workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | +| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the Jira workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | +| `commit_events` | boolean | false | Enable notifications for commit events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -### Delete JIRA service +### Delete Jira service -Remove all previously JIRA settings from a project. +Remove all previously Jira settings from a project. ``` DELETE /projects/:id/services/jira @@ -715,9 +739,14 @@ PUT /projects/:id/services/packagist Parameters: -- `username` (**required**) -- `token` (**required**) -- `server` (optional) +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | The username of a Packagist account | +| `token` | string | yes | API token to the Packagist server | +| `server` | boolean | no | URL of the Packagist server. Leave blank for default: <https://packagist.org> | +| `push_events` | boolean | false | Enable notifications for push events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | ### Delete Packagist service @@ -755,6 +784,7 @@ Parameters: | `add_pusher` | boolean | no | Add pusher to recipients list | | `notify_only_broken_pipelines` | boolean | no | Notify only broken pipelines | | `notify_only_default_branch` | boolean | no | Send notifications only for the default branch ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28271)) | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | ### Delete Pipeline-Emails service @@ -790,6 +820,7 @@ Parameters: | --------- | ---- | -------- | ----------- | | `token` | string | true | The PivotalTracker token | | `restrict_to_branch` | boolean | false | Comma-separated list of branches which will be automatically inspected. Leave blank to include all branches. | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete PivotalTracker service @@ -862,6 +893,7 @@ Parameters: | `priority` | string | true | The priority | | `device` | string | false | Leave blank for all active devices | | `sound` | string | false | The sound of the notification | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Pushover service @@ -899,6 +931,7 @@ Parameters: | `project_url` | string | true | Project url | | `issues_url` | string | true | Issue url | | `description` | string | false | Description | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete Redmine service @@ -989,6 +1022,17 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `webhook` | string | true | The Microsoft Teams webhook. For example, `https://outlook.office.com/webhook/...` | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `notify_only_default_branch` | boolean | false | Send notifications only for the default branch | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidental_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | ### Delete Microsoft Teams service @@ -1084,6 +1128,7 @@ Parameters: | `build_type` | string | true | Build configuration ID | | `username` | string | true | A user with permissions to trigger a manual build | | `password` | string | true | The password of the user | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete JetBrains TeamCity CI service @@ -1115,7 +1160,7 @@ PUT /projects/:id/services/jenkins Parameters: -- `jenkins_url` (**required**) - Jenkins URL like http://jenkins.example.com +- `jenkins_url` (**required**) - Jenkins URL like `http://jenkins.example.com` - `project_name` (**required**) - The URL-friendly project name. Example: my_project_name - `username` (optional) - A user with access to the Jenkins server, if applicable - `password` (optional) - The password of the user @@ -1150,7 +1195,7 @@ PUT /projects/:id/services/jenkins-deprecated Parameters: -- `project_url` (**required**) - Jenkins project URL like http://jenkins.example.com/job/my-project/ +- `project_url` (**required**) - Jenkins project URL like `http://jenkins.example.com/job/my-project/` - `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin - `pass_unstable` (optional) - Unstable builds will be treated as passing @@ -1230,6 +1275,7 @@ Parameters: | `issues_url` | string | true | Issue url | | `project_url` | string | true | Project url | | `description` | string | false | Description | +| `push_events` | boolean | false | Enable notifications for push events | ### Delete YouTrack Service diff --git a/doc/api/settings.md b/doc/api/settings.md index c2a1f7feefd..b01cec64837 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -142,8 +142,6 @@ are listed in the descriptions of the relevant settings. | `authorized_keys_enabled` | boolean | no | By default, we write to the `authorized_keys` file to support Git over SSH without additional configuration. GitLab can be optimized to authenticate SSH keys via the database file. Only disable this if you have configured your OpenSSH server to use the AuthorizedKeysCommand. | | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It will automatically build, test, and deploy applications based on a predefined CI/CD configuration. | -| `clientside_sentry_dsn` | string | required by: `clientside_sentry_enabled` | Clientside Sentry Data Source Name. | -| `clientside_sentry_enabled` | boolean | no | (**If enabled, requires:** `clientside_sentry_dsn`) Enable Sentry error reporting for the client side. | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | | `default_branch_protection` | integer | no | Determine if developers can push to master. Can take: `0` _(not protected, both developers and maintainers can push new commits, force push, or delete the branch)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push or delete the branch)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no-one can force push or delete the branch)_ as a parameter. Default is `2`. | @@ -212,8 +210,6 @@ are listed in the descriptions of the relevant settings. | `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-admin users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | | `rsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded RSA key. Default is `0` (no restriction). `-1` disables RSA keys. | | `send_user_confirmation_email` | boolean | no | Send confirmation email on sign-up. | -| `sentry_dsn` | string | required by: `sentry_enabled` | Sentry Data Source Name. | -| `sentry_enabled` | boolean | no | (**If enabled, requires:** `sentry_dsn`) Sentry is an error reporting and logging tool which is currently not shipped with GitLab, available at <https://sentry.io>. | | `session_expire_delay` | integer | no | Session duration in minutes. GitLab restart is required to apply changes | | `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text`) Enable shared runners for new projects. | | `shared_runners_text` | string | required by: `shared_runners_enabled` | Shared runners text. | @@ -231,6 +227,7 @@ are listed in the descriptions of the relevant settings. | `throttle_unauthenticated_enabled` | boolean | no | (**If enabled, requires:** `throttle_unauthenticated_period_in_seconds` and `throttle_unauthenticated_requests_per_period`) Enable unauthenticated request rate limit. Helps reduce request volume (e.g. from crawlers or abusive bots). | | `throttle_unauthenticated_period_in_seconds` | integer | required by: `throttle_unauthenticated_enabled` | Rate limit period in seconds. | | `throttle_unauthenticated_requests_per_period` | integer | required by: `throttle_unauthenticated_enabled` | Max requests per period per IP. | +| `time_tracking_limit_to_hours` | boolean | no | Limit display of time tracking units to hours. Default is `false`. | | `two_factor_grace_period` | integer | required by: `require_two_factor_authentication` | Amount of time (in hours) that users are allowed to skip forced configuration of two-factor authentication. | | `unique_ips_limit_enabled` | boolean | no | (**If enabled, requires:** `unique_ips_limit_per_user` and `unique_ips_limit_time_window`) Limit sign in from multiple ips. | | `unique_ips_limit_per_user` | integer | required by: `unique_ips_limit_enabled` | Maximum number of ips per user. | diff --git a/doc/api/users.md b/doc/api/users.md index 47028c679b8..4667a985eb9 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -2,6 +2,8 @@ ## List users +Active users = Total accounts - Blocked users + Get a list of users. This function takes pagination parameters `page` and `per_page` to restrict the list of users. @@ -257,7 +259,8 @@ Parameters: "two_factor_enabled": true, "external": false, "private_profile": false, - "highest_role":10 + "shared_runners_minutes_limit": 133 + "extra_shared_runners_minutes_limit": 133 } ``` @@ -269,7 +272,14 @@ GET /users/:id?with_custom_attributes=true ## User creation -Creates a new user. Note only administrators can create new users. Either `password` or `reset_password` should be specified (`reset_password` takes priority). If `reset_password` is `false`, then `password` is required. +Creates a new user. Note only administrators can create new +users. Either `password`, `reset_password`, or `force_random_password` +must be specified. If `reset_password` and `force_random_password` are +both `false`, then `password` is required. + +Note that `force_random_password` and `reset_password` take priority +over `password`. In addition, `reset_password` and +`force_random_password` can be used together. ``` POST /users @@ -277,28 +287,32 @@ POST /users Parameters: -- `email` (required) - Email -- `password` (optional) - Password -- `reset_password` (optional) - Send user password reset link - true or false(default) -- `username` (required) - Username -- `name` (required) - Name -- `skype` (optional) - Skype ID -- `linkedin` (optional) - LinkedIn -- `twitter` (optional) - Twitter account -- `website_url` (optional) - Website URL -- `organization` (optional) - Organization name -- `projects_limit` (optional) - Number of projects user can create -- `extern_uid` (optional) - External UID -- `provider` (optional) - External provider name -- `bio` (optional) - User's biography -- `location` (optional) - User's location -- `public_email` (optional) - The public email of the user -- `admin` (optional) - User is admin - true or false (default) -- `can_create_group` (optional) - User can create groups - true or false -- `skip_confirmation` (optional) - Skip confirmation - true or false (default) -- `external` (optional) - Flags the user as external - true or false(default) -- `avatar` (optional) - Image file for user's avatar -- `private_profile` (optional) - User's profile is private - true or false +- `email` (required) - Email +- `password` (optional) - Password +- `reset_password` (optional) - Send user password reset link - true or false (default) +- `force_random_password` (optional) - Set user password to a random value - true or false (default) +- `username` (required) - Username +- `name` (required) - Name +- `skype` (optional) - Skype ID +- `linkedin` (optional) - LinkedIn +- `twitter` (optional) - Twitter account +- `website_url` (optional) - Website URL +- `organization` (optional) - Organization name +- `projects_limit` (optional) - Number of projects user can create +- `extern_uid` (optional) - External UID +- `provider` (optional) - External provider name +- `group_id_for_saml` (optional) - ID of group where SAML has been configured +- `bio` (optional) - User's biography +- `location` (optional) - User's location +- `public_email` (optional) - The public email of the user +- `admin` (optional) - User is admin - true or false (default) +- `can_create_group` (optional) - User can create groups - true or false +- `skip_confirmation` (optional) - Skip confirmation - true or false (default) +- `external` (optional) - Flags the user as external - true or false(default) +- `avatar` (optional) - Image file for user's avatar +- `private_profile` (optional) - User's profile is private - true or false +- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user +- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user ## User modification @@ -322,6 +336,7 @@ Parameters: - `projects_limit` - Limit projects each user can create - `extern_uid` - External UID - `provider` - External provider name +- `group_id_for_saml` (optional) - ID of group where SAML has been configured - `bio` - User's biography - `location` (optional) - User's location - `public_email` (optional) - The public email of the user @@ -329,6 +344,8 @@ Parameters: - `can_create_group` (optional) - User can create groups - true or false - `skip_reconfirmation` (optional) - Skip reconfirmation - true or false (default) - `external` (optional) - Flags the user as external - true or false(default) +- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user +- `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user - `avatar` (optional) - Image file for user's avatar - `private_profile` (optional) - User's profile is private - true or false @@ -1150,8 +1167,6 @@ settings page. POST /users/:user_id/impersonation_tokens ``` -Parameters: - | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `user_id` | integer | yes | The ID of the user | @@ -1255,4 +1270,4 @@ Example response: Please note that `last_activity_at` is deprecated, please use `last_activity_on`. -[gemojione-index]: https://github.com/jonathanwiesel/gemojione/blob/master/config/index.json +[gemojione-index]: https://github.com/jonathanwiesel/gemojione/blob/master/config/index.json
\ No newline at end of file diff --git a/doc/ci/README.md b/doc/ci/README.md index 8b70699cdb5..da864a0b3cc 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -63,7 +63,7 @@ Once you're familiar with how GitLab CI/CD works, see the for all the attributes you can set and use. NOTE: **Note:** -GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#extra-shared-runners-pipeline-minutes-quota). +GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/admin_area/settings/continuous_integration.md#extra-shared-runners-pipeline-minutes-quota-free-only). ## Configuration @@ -101,10 +101,10 @@ Its feature set is listed on the table below according to DevOps stages. | [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | |---+---| | **Verify** || -| [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html) | Quickly determine the performance impact of pending code changes. | +| [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the performance impact of pending code changes. | | [CI services](services/README.md) | Link Docker containers with your base image.| -| [Code Quality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** | Analyze your source code quality. | -| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | +| [Code Quality](../user/project/merge_requests/code_quality.md) **[STARTER]** | Analyze your source code quality. | +| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | | [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | | [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | | [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | @@ -112,18 +112,18 @@ Its feature set is listed on the table below according to DevOps stages. | **Release** || | [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | | [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | -| [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) **[PREMIUM]** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) **[PREMIUM]** | Deploy your features behind Feature Flags. | +| [Canary Deployments](../user/project/canary_deployments.md) **[PREMIUM]** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | +| [Deploy Boards](../user/project/deploy_boards.md) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [Feature Flags](../user/project/operations/feature_flags.md) **[PREMIUM]** | Deploy your features behind Feature Flags. | | [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | |---+---| | **Secure** || -| [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]** | Check your Docker containers for known vulnerabilities.| -| [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | -| [License Management](https://docs.gitlab.com/ee/user/application_security/license_management/) **[ULTIMATE]** | Search your project dependencies for their licenses. | -| [Security Test reports](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate) **[ULTIMATE]** | Check for app vulnerabilities. | +| [Container Scanning](../user/application_security/container_scanning/index.md) **[ULTIMATE]** | Check your Docker containers for known vulnerabilities.| +| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [License Management](../user/application_security/license_management/index.md) **[ULTIMATE]** | Search your project dependencies for their licenses. | +| [Security Test reports](../user/project/merge_requests/index.md#security-reports-ultimate) **[ULTIMATE]** | Check for app vulnerabilities. | ## Examples diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 9126eb65f07..bbb25c78ec5 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -6,7 +6,7 @@ type: howto GitLab CI/CD can be used with Bitbucket Cloud by: -1. Creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html). +1. Creating a [CI/CD project](../../user/project/ci_cd_for_external_repo.md). 1. Connecting your Git repository via URL. To use GitLab CI/CD with a Bitbucket Cloud repository: diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 2a8ddb610f9..53b36181062 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -32,7 +32,7 @@ GitLab will: 1. Import the project. 1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter). -1. Enable [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html). +1. Enable [GitHub project integration](../../user/project/integrations/github.md). 1. Create a web hook on GitHub to notify GitLab of new commits. CAUTION: **Caution:** @@ -69,12 +69,9 @@ repositories: 1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). -GitLab will: - -1. Import the project. -1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter). -1. Enable [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html). -1. Create a web hook on GitHub to notify GitLab of new commits. +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](../../user/project/integrations/github.md), and create a web hook +on GitHub to notify GitLab of new commits. ## Connect manually @@ -99,7 +96,7 @@ your repository: GitLab will automatically configure polling-based pull mirroring. -1. Still in GitLab, enable the [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html) +1. Still in GitLab, enable the [GitHub project integration](../../user/project/integrations/github.md) from **Settings > Integrations.** Check the **Active** checkbox to enable the integration, paste your diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index b4c4bea6447..efdcaf5a6f5 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -205,7 +205,14 @@ An example project using this approach can be found here: <https://gitlab.com/gi ### Use Docker socket binding -The third approach is to bind-mount `/var/run/docker.sock` into the container so that docker is available in the context of that image. +The third approach is to bind-mount `/var/run/docker.sock` into the +container so that Docker is available in the context of that image. + +NOTE: **Note:** +If you bind the Docker socket [when using GitLab Runner 11.11 or +newer](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1261), +you can no longer use `docker:dind` as a service because volume bindings +are done to the services as well, making these incompatible. In order to do that, follow the steps: diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 29578efacbb..e012f4f8595 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -463,8 +463,6 @@ that runner. > support for using private registries, which required manual configuration > of credentials on runner's host. We recommend to upgrade your Runner to > at least version **1.8** if you want to use private registries. -> - If the repository is private you need to authenticate your GitLab Runner in the -> registry. Learn more about how [GitLab Runner works in this case][runner-priv-reg]. To access private container registries, the GitLab Runner process can use: @@ -478,17 +476,31 @@ To define which should be used, the GitLab Runner process reads the configuratio - A [variable](../variables/README.md#gitlab-cicd-environment-variables) in `.gitlab-ci.yml`. - A project's variables stored on the projects **Settings > CI/CD** page. - `DOCKER_AUTH_CONFIG` variable provided as environment variable in `config.toml` of the Runner. -- `config.json` file placed in `$HOME/docker` directory of the user running GitLab Runner process. +- `config.json` file placed in `$HOME/.docker` directory of the user running GitLab Runner process. If the `--user` flag is provided to run the GitLab Runner child processes as unprivileged user, the home directory of the main GitLab Runner process user will be used. -NOTE: **Note:** -GitLab Runner reads this configuration **only** from `config.toml` and ignores it if +NOTE: **Note:** +GitLab Runner reads this configuration **only** from `config.toml` and ignores it if it's provided as an environment variable. This is because GitLab Runnner uses **only** `config.toml` configuration and doesn't interpolate **ANY** environment variables at runtime. ### Using statically-defined credentials +There are two approaches that you can take in order to access a +private registry. Both require setting the environment variable +`DOCKER_AUTH_LOGIN` with appropriate authentication info. + +1. Per-job: To configure one job to access a private registry, add + `DOCKER_AUTH_LOGIN` as a job variable. +1. Per-runner: To configure a Runner so all its jobs can access a + private registry, add `DOCKER_AUTH_LOGIN` to the environment in the + Runner's configuration. + +See below for examples of each. + +#### Determining your `DOCKER_AUTH_LOGIN` data + As an example, let's assume that you want to use the `registry.example.com:5000/private/image:latest` image which is private and requires you to login into a private container registry. @@ -500,30 +512,41 @@ Let's also assume that these are the login credentials: | username | `my_username` | | password | `my_password` | -To configure access for `registry.example.com:5000`, follow these steps: +There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: + +- **First way -** Do a `docker login` on your local machine: + + ```bash + docker login registry.example.com:5000 --username my_username --password my_password + ``` + + Then copy the content of `~/.docker/config.json`. -1. Find what the value of `DOCKER_AUTH_CONFIG` should be. There are two ways to - accomplish this: - - **First way -** Do a `docker login` on your local machine: + If you don't need access to the registry from your computer, you + can do a `docker logout`: + + ```bash + docker logout registry.example.com:5000 + ``` - ```bash - docker login registry.example.com:5000 --username my_username --password my_password - ``` +- **Second way -** In some setups, it's possible that Docker client +will use the available system keystore to store the result of `docker +login`. In that case, it's impossible to read `~/.docker/config.json`, +so you will need to prepare the required base64-encoded version of +`${username}:${password}` manually. Open a terminal and execute the +following command: - Then copy the content of `~/.docker/config.json`. - - **Second way -** In some setups, it's possible that Docker client will use - the available system keystore to store the result of `docker login`. In - that case, it's impossible to read `~/.docker/config.json`, so you will - need to prepare the required base64-encoded version of - `${username}:${password}` manually. Open a terminal and execute the - following command: + ```bash + echo -n "my_username:my_password" | base64 - ```bash - echo -n "my_username:my_password" | base64 + # Example output to copy + bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= + ``` - # Example output to copy - bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= - ``` +#### Configuring a job + +To configure a single job with access for `registry.example.com:5000`, +follow these steps: 1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: @@ -538,14 +561,6 @@ To configure access for `registry.example.com:5000`, follow these steps: } ``` -1. Optionally,if you followed the first way of finding the `DOCKER_AUTH_CONFIG` - value, do a `docker logout` on your computer if you don't need access to the - registry from it: - - ```bash - docker logout registry.example.com:5000 - ``` - 1. You can now use any private image from `registry.example.com:5000` defined in `image` and/or `services` in your `.gitlab-ci.yml` file: @@ -566,6 +581,37 @@ for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. Specifying only `registry.example.com` will not work. +### Configuring a Runner + +If you have many pipelines that access the same registry, it'll +probably be better to setup registry access at the runner level. This +allows pipeline authors to have access to a private registry just by +running a job on the appropriate runner. It also makes registry +changes and credential rotations much simpler. + +Of course this means that any job on that runner can access the +registry with the same privilege, even across projects. If you need to +control access to the registry, you'll need to be sure to control +access to the runner. + +To add `DOCKER_AUTH_CONFIG` to a Runner: + +1. Modify the Runner's `config.toml` file as follows: + + ```toml + [[runners]] + environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] + ``` + +1. Restart the Runner service. + +NOTE: **Note:** The double quotes included in the `DOCKER_AUTH_CONFIG` +data must be escaped with backslashes. This prevents them from being +interpreted as TOML. + +NOTE: **Note:** The `environment` option is a list. So your Runner may +have existing entries and you should add this to the list, not replace +it. ### Using Credentials Store @@ -574,10 +620,10 @@ Specifying only `registry.example.com` will not work. To configure credentials store, follow these steps: 1. To use a credentials store, you need an external helper program to interact with a specific keychain or external store. -Make sure helper program is available in GitLab Runner `$PATH`. + Make sure helper program is available in GitLab Runner `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: - - Create a + - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: @@ -741,7 +787,6 @@ creation. [tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/ [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ -[runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry [entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint [cmd]: https://docs.docker.com/engine/reference/builder/#cmd [register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/environments.md b/doc/ci/environments.md index bd419965a9c..a32dbc11a33 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -734,7 +734,7 @@ Below are some links you may find interesting: - [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) -- [Deploy Boards for your applications running on Kubernetes](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** +- [Deploy Boards for your applications running on Kubernetes](../user/project/deploy_boards.md) **[PREMIUM]** <!-- ## Troubleshooting diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index d2ea30839a4..2b4fe321cb3 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -47,7 +47,7 @@ to get paid for writing complete articles for GitLab. ## Adding templates to your GitLab installation **[PREMIUM ONLY]** -If you want to have customized examples and templates for your own self-managed GitLab instance available to your team, your GitLab administrator can [designate an instance template repository](https://docs.gitlab.com/ee/user/admin_area/settings/instance_template_repository.html) that contains examples and templates specific to your enterprise. +If you want to have customized examples and templates for your own self-managed GitLab instance available to your team, your GitLab administrator can [designate an instance template repository](../../user/admin_area/settings/instance_template_repository.md) that contains examples and templates specific to your enterprise. ## Other resources @@ -62,8 +62,8 @@ For examples of setting up GitLab CI/CD for cloud-based environments, see: - [Automating Kubernetes Deployments with GitLab CI/CD](https://www.youtube.com/watch?v=wEDRfAz6_Uw) - [How to autoscale continuous deployment with GitLab Runner on DigitalOcean](https://about.gitlab.com/2018/06/19/autoscale-continuous-deployment-gitlab-runner-digital-ocean/) - [How to create a CI/CD pipeline with Auto Deploy to Kubernetes using GitLab and Helm](https://about.gitlab.com/2017/09/21/how-to-create-ci-cd-pipeline-with-autodeploy-to-kubernetes-using-gitlab-and-helm/) +- [Demo - Deploying from GitLab to OpenShift Container Cluster](https://youtu.be/EwbhA53Jpp4) -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> See also the following video overviews: - [Containers, Schedulers, and GitLab CI](https://www.youtube.com/watch?v=d-9awBxEbvQ). @@ -100,6 +100,10 @@ For examples of others who have implemented GitLab CI/CD, see: - [Fast and natural continuous integration with GitLab CI](https://about.gitlab.com/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/) - [Demo: CI/CD with GitLab in action](https://about.gitlab.com/2017/03/13/ci-cd-demo/) +### Migrating to GitLab from third-party CI tools + +- [Migrating from Jenkins to GitLab](https://youtu.be/RlEVGOpYF5Y) + ### Integrating GitLab CI/CD with other systems To see how you can integrate GitLab CI/CD with third-party systems, see: diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index e85a13f2187..2117b342903 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -106,7 +106,7 @@ parameter in `.gitlab-ci.yml` to use the custom location instead of the default Now it's time we set up [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/) to automatically build, test and deploy the dependency! GitLab CI/CD uses a file in the root of the repo, named `.gitlab-ci.yml`, to read the definitions for jobs -that will be executed by the configured GitLab Runners. You can read more about this file in the [GitLab Documentation](https://docs.gitlab.com/ee/ci/yaml/). +that will be executed by the configured GitLab Runners. You can read more about this file in the [GitLab Documentation](../../yaml/README.md). First of all, remember to set up variables for your deployment. Navigate to your project's **Settings > CI/CD > Environment variables** page and add the following ones (replace them with your current values, of course): diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index 4c42811edf4..8ecac4a5a4f 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -60,7 +60,7 @@ provide a list of URLs to test, please see the TIP: **Tip:** For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically extracted and shown right in the merge request widget. -[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). +[Learn more on Browser Performance Testing in merge requests](../../user/project/merge_requests/browser_performance_testing.md). ## Performance testing on Review Apps diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md index b34637efc8d..0aa108a2e73 100644 --- a/doc/ci/examples/code_climate.md +++ b/doc/ci/examples/code_climate.md @@ -1,6 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/ci/examples/code_climate.html' -redirect_to: code_quality.md +redirect_to: 'code_quality.md' --- This document was moved to [another location](code_quality.md). diff --git a/doc/ci/examples/code_quality.md b/doc/ci/examples/code_quality.md index 4a9bfa51528..43f773dab7c 100644 --- a/doc/ci/examples/code_quality.md +++ b/doc/ci/examples/code_quality.md @@ -1,4 +1,5 @@ --- +redirect_from: 'https://docs.gitlab.com/ee/ci/examples/code_climate.html' type: reference, howto --- @@ -31,7 +32,7 @@ Due to implementation limitations we always take the latest Code Quality artifac TIP: **Tip:** For [GitLab Starter][ee] users, this information will be automatically extracted and shown right in the merge request widget. -[Learn more on Code Quality in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html). +[Learn more on Code Quality in merge requests](../../user/project/merge_requests/code_quality.md). ## Previous job definitions diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index 4d41e424f4a..6570f6b2d98 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +redirect_to: '../../user/application_security/container_scanning/index.md' --- -This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). +This document was moved to [another location](../../user/application_security/container_scanning/index.md). diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index b676c661267..9591abfc276 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dast/index.html' +redirect_to: '../../user/application_security/dast/index.md' --- -This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dast/index.html). +This document was moved to [another location](../../user/application_security/dast/index.md). diff --git a/doc/ci/examples/dependency_scanning.md b/doc/ci/examples/dependency_scanning.md index 3a8b53b425c..dc234a3489f 100644 --- a/doc/ci/examples/dependency_scanning.md +++ b/doc/ci/examples/dependency_scanning.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html' +redirect_to: '../../user/application_security/dependency_scanning/index.md' --- -This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). +This document was moved to [another location](../../user/application_security/dependency_scanning/index.md). diff --git a/doc/ci/examples/license_management.md b/doc/ci/examples/license_management.md index 08704425a75..53e38111bf3 100644 --- a/doc/ci/examples/license_management.md +++ b/doc/ci/examples/license_management.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://docs.gitlab.com/ee/user/application_security/license_management/index.html' +redirect_to: '../../user/application_security/license_management/index.md' --- -This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/license_management/index.html). +This document was moved to [another location](../../user/application_security/license_management/index.md). diff --git a/doc/ci/examples/sast.md b/doc/ci/examples/sast.md index 688cc79d0f6..7c644ac833d 100644 --- a/doc/ci/examples/sast.md +++ b/doc/ci/examples/sast.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://docs.gitlab.com/ee/user/application_security/sast/index.html' +redirect_to: '../../user/application_security/sast/index.md' --- -This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/sast/index.html). +This document was moved to [another location](../../user/application_security/sast/index.md). diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md index 570898b2d23..6570f6b2d98 100644 --- a/doc/ci/examples/sast_docker.md +++ b/doc/ci/examples/sast_docker.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +redirect_to: '../../user/application_security/container_scanning/index.md' --- -This document was moved to [another location](../../user/application_security/container_scanning/index.html). +This document was moved to [another location](../../user/application_security/container_scanning/index.md). diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index 0e33a1ba060..bd899240307 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -47,10 +47,10 @@ deploy: In the above configuration: - The `before_script` installs [SBT](http://www.scala-sbt.org/) and -displays the version that is being used. + displays the version that is being used. - The `test` stage executes SBT to compile and test the project. - - [sbt-scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT -plugin to measure test coverage. + - [sbt-scoverage](https://github.com/scoverage/sbt-scoverage) is used as an SBT + plugin to measure test coverage. - The `deploy` stage automatically deploys the project to Heroku using dpl. You can use other versions of Scala and SBT by defining them in diff --git a/doc/ci/img/collapsible_log.png b/doc/ci/img/collapsible_log.png Binary files differnew file mode 100644 index 00000000000..2785033b349 --- /dev/null +++ b/doc/ci/img/collapsible_log.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 1687716df2e..ef9f9a9973c 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -177,22 +177,22 @@ according to each stage (Verify, Package, Release). 1. **Verify**: - Automatically build and test your application with Continuous Integration. - - Analyze your source code quality with [GitLab Code Quality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html). **[STARTER]** - - Determine the performance impact of code changes with [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). **[PREMIUM]** - - Perform a series of tests, such as [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]**, [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]**, and [JUnit tests](../junit_test_reports.md). + - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). **[STARTER]** + - Determine the performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **[PREMIUM]** + - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **[ULTIMATE]**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **[ULTIMATE]**, and [JUnit tests](../junit_test_reports.md). - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. 1. **Package**: - Store Docker images with [Container Registry](../../user/project/container_registry.md). - - Store NPM packages with [NPM Registry](https://docs.gitlab.com/ee/user/project/packages/npm_registry.html). **[PREMIUM]** - - Store Maven artifacts with [Maven Repository](https://docs.gitlab.com/ee/user/project/packages/maven_repository.html). **[PREMIUM]** + - Store NPM packages with [NPM Registry](../../user/project/packages/npm_registry.md). **[PREMIUM]** + - Store Maven artifacts with [Maven Repository](../../user/project/packages/maven_repository.md). **[PREMIUM]** 1. **Release**: - Continuous Deployment, automatically deploying your app to production. - Continuous Delivery, manually click to deploy your app to production. - Deploy static websites with [GitLab Pages](../../user/project/pages/index.md). - - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html). **[PREMIUM]** - - Deploy your features behind [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html). **[PREMIUM]** + - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](../../user/project/canary_deployments.md). **[PREMIUM]** + - Deploy your features behind [Feature Flags](../../user/project/operations/feature_flags.md). **[PREMIUM]** - Add release notes to any Git tag with [GitLab Releases](../../user/project/releases/index.md). - - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html). **[PREMIUM]** + - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](../../user/project/deploy_boards.md). **[PREMIUM]** - Deploy your application to a production environment in a Kubernetes cluster with [Auto Deploy](../../topics/autodevops/index.md#auto-deploy). With GitLab CI/CD you can also: @@ -201,7 +201,7 @@ With GitLab CI/CD you can also: - Deploy your app to different [environments](../environments.md). - Install your own [GitLab Runner](https://docs.gitlab.com/runner/). - [Schedule pipelines](../../user/project/pipelines/schedules.md). -- Check for app vulnerabilities with [Security Test reports](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate). **[ULTIMATE]** +- Check for app vulnerabilities with [Security Test reports](../../user/project/merge_requests/index.md#security-reports-ultimate). **[ULTIMATE]** To see all CI/CD features, navigate back to the [CI/CD index](../README.md). diff --git a/doc/ci/merge_request_pipelines/img/merge_train_cancel.png b/doc/ci/merge_request_pipelines/img/merge_train_cancel.png Binary files differnew file mode 100644 index 00000000000..1561fdcc7a5 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_train_cancel.png diff --git a/doc/ci/merge_request_pipelines/img/merge_train_config.png b/doc/ci/merge_request_pipelines/img/merge_train_config.png Binary files differnew file mode 100644 index 00000000000..fb0af43556e --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_train_config.png diff --git a/doc/ci/merge_request_pipelines/img/merge_train_start.png b/doc/ci/merge_request_pipelines/img/merge_train_start.png Binary files differnew file mode 100644 index 00000000000..f20108157d2 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_train_start.png diff --git a/doc/ci/merge_request_pipelines/img/merge_train_start_when_pipeline_succeeds.png b/doc/ci/merge_request_pipelines/img/merge_train_start_when_pipeline_succeeds.png Binary files differnew file mode 100644 index 00000000000..62c2f2f5ff5 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_train_start_when_pipeline_succeeds.png diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index fe2fc790505..c3dbcf6a19f 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -4,12 +4,6 @@ type: reference # Pipelines for merge requests -NOTE: **Note**: -As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 -or higher due to the [recent refspecs -changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). -Anything lower will cause the pipeline to fail. - > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) in GitLab 11.6. Usually, when you create a new merge request, a pipeline runs with the @@ -23,6 +17,16 @@ for when you are running a pipeline in a merge request. This could be either adding or removing steps in the pipeline, to make sure that your pipelines are as efficient as possible. +## Requirements and limitations + +Pipelines for merge requests have the following requirements and limitations: + +- As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 + or higher due to the + [recent refspecs changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). +- Pipelines for merge requests are incompatible with + [CI/CD for external repositories](../ci_cd_for_external_repos/index.md). + ## Configuring pipelines for merge requests To configure pipelines for merge requests, add the `only: merge_requests` parameter to @@ -71,10 +75,10 @@ when a merge request was created or updated. For example:  -## Pipelines for Merged Results **[PREMIUM]** +## Pipelines for merged results **[PREMIUM]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. -> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186), but [can be enabled manually](#enabling-pipelines-for-merged-results). +> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222), but [can be enabled manually](#enabling-pipelines-for-merged-results). It's possible for your source and target branches to diverge, which can result in the scenario that source branch's pipeline was green, the target's pipeline was green, @@ -100,7 +104,22 @@ The detached state serves to warn you that you are working in a situation subjected to merge problems, and helps to highlight that you should get out of WIP status or resolve merge conflicts as soon as possible. -### Enabling Pipelines for Merged Results +### Requirements and limitations + +Pipelines for merged results require: + +- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. +- [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. + +In addition, pipelines for merged results have the following limitations: + +- Forking/cross-repo workflows are not currently supported. To follow progress, + see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). +- This feature is not available for + [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. + To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). + +### Enabling Pipelines for merged results To enable pipelines on merged results at the project level: @@ -114,13 +133,77 @@ CAUTION: **Warning:** Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](#configuring-pipelines-for-merge-requests), otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. -### Pipelines for Merged Result's limitations +## Merge Trains **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. +> This feature is disabled by default, but [can be enabled manually](#enabling-merge-trains). + +[Pipelines for merged results](#pipelines-for-merged-results-premium) introduces +running a build on the result of the merged code prior to merging, as a way to keep master green. + +There's a scenario, however, for teams with a high number of changes in the target branch (typically master) where in many or even all cases, +by the time the merged code is validated another commit has made it to master, invalidating the merged result. +You'd need some kind of queuing, cancellation or retry mechanism for these scenarios +in order to ensure an orderly flow of changes into the target branch. + +Each MR that joins a merge train joins as the last item in the train, +just as it works in the current state. However, instead of queuing and waiting, +each item takes the completed state of the previous (pending) merge ref, adds its own changes, +and starts the pipeline immediately in parallel under the assumption that everything is going to pass. + +In this way, if all the pipelines in the train merge successfully, no pipeline time is wasted either queuing or retrying. +If the button is subsequently pressed in a different MR, instead of creating a new pipeline for the target branch, +it creates a new pipeline targeting the merge result of the previous MR plus the target branch. +Pipelines invalidated through failures are immediately canceled and requeued. + +### Requirements and limitations + +Merge trains have the following requirements and limitations: + +- This feature requires that + [pipelines for merged results](#pipelines-for-merged-results-premium) are + **configured properly**. +- Each merge train can generate a merge ref and run a pipeline **one at a time**. + We plan to make the pipelines for merged results + [run in parallel](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222) in a + future release. + +### Enabling Merge Trains + +To enable merge trains at the project level: + +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Allow merge trains**. +1. Click **Save changes** button. + + + +### How to add a merge request to a merge train + +To add a merge request to a merge train: + +1. Click "Start/Add merge train" button in a merge request + + + +### How to remove a merge request from a merge train + +1. Click "Remove from merge train" button in the merge request widget. + + + +### Tips: Start/Add to merge train when pipeline succeeds + +You can add a merge request to a merge train only when the latest pipeline in the +merge request finished. While the pipeline is running or pending, you cannot add +the merge request to a train because the current change of the merge request may +be broken thus it could affect the following merge requests. + +In this case, you can schedule to add the merge request to a merge train **when the latest +pipeline succeeds**. You can see the following button instead of the regular "Start/Add merge train" +button while the latest pipeline is running. -- This feature requires [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- This feature requires [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. -- After the merge request pipeline succeeds, if the target branch has moved forward, the result of the pipeline is stale and must be retried. In busy repos, this can become a problem as it is highly probable that the target branch will have moved ahead. Improvements are [planned](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) for future versions of GitLab. -- Forking/cross-repo workflows are not currently supported. To follow progress, see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). -- This feature is not available for [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). + ## Excluding certain jobs diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 3ded6673149..f9cfc0892a7 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -9,11 +9,11 @@ Requires GitLab Runner 11.10 and above. ## Overview -GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](./junit_test_reports.md), [codequality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](junit_test_reports.md), [codequality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. - + ## Use cases diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 1ad3d516df9..4a07aa31f8a 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -139,6 +139,20 @@ The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time (4 - 1) + (7 - 6) => 4 ``` +### Expanding and collapsing job log sections + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/14664) in GitLab +> 12.0. + +Job logs are divided into sections that can be collapsed or expanded. + +In the following example: + +- Two sections are expanded and can be collapsed. +- One section is collapsed and can be expanded. + + + ## Configuring pipelines Pipelines, and their component jobs and stages, are defined in the [`.gitlab-ci.yml`](yaml/README.md) file for each project. diff --git a/doc/ci/review_apps/img/review_button.png b/doc/ci/review_apps/img/review_button.png Binary files differnew file mode 100644 index 00000000000..0b231c50858 --- /dev/null +++ b/doc/ci/review_apps/img/review_button.png diff --git a/doc/ci/review_apps/img/toolbar_feeback_form.png b/doc/ci/review_apps/img/toolbar_feeback_form.png Binary files differnew file mode 100644 index 00000000000..d147981a387 --- /dev/null +++ b/doc/ci/review_apps/img/toolbar_feeback_form.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 7b039fe6654..70934e074a0 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -154,6 +154,44 @@ After adding Review Apps to your workflow, you follow the branched Git flow. Tha 1. Wait for the Runner to build and deploy your web application. 1. Click on the link that provided in the merge request related to the branch to see the changes live. +### Visual Reviews **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10761) in GitLab Starter 12.0. + +The Visual Reviews feedback form can be added to a Review App to enable reviewers to post comments +directly from the app back to the merge request that spawned the Review App. + +For example, a form like the following can be configured to post the contents of the +text field into the discussion thread of a merge request: + + + +#### Using Visual Reviews + +If Visual Reviews has been [enabled](#configuring-visual-reviews) for the Review App, the Visual Reviews feedback form is overlaid on the app's pages at the bottom-right corner. + +To use the feedback form, you will need to create a [personal access token](../../user/profile/personal_access_tokens.md) with the API scope selected. + +Paste the token into the feedback box, when prompted. If you select **Remember me**, your browser stores the token so that future visits to Review Apps at the same URL will not require you to re-enter the token. To clear the token, click **Log out**. + +Because tokens must be entered on a per-domain basis and they can only be accessed once, you can save the token to your password manager specifically for the purpose of Visual Reviews. This way, you will not need to create additional tokens for each merge request. + +Comments can make use of all the [Markdown annotations](../../user/markdown.md) +available in merge request comment boxes. + +#### Configuring Visual Reviews + +The feedback form is served through a script you add to pages in your Review App. +To access the code to include the script, click the **Review** button in the **Pipeline** section of the merge request. + + + +The provided script hardcodes the project and merge request IDs. You may want to consider +using features of your programming language to use environment variables or other +means to inject these at runtime. + +Future enhancements [are planned](https://gitlab.com/gitlab-org/gitlab-ee/issues/11322) to make this process even easier. + ## Limitations Review App limitations are the same as [environments limitations](../environments.md#limitations). diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index b089229ab58..03a219e03ca 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -418,9 +418,9 @@ You can find the IP address of a Runner for a specific project by:  -[install]: http://docs.gitlab.com/runner/install/ +[install]: https://docs.gitlab.com/runner/install/ [fifo]: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics) -[register]: http://docs.gitlab.com/runner/register/ +[register]: https://docs.gitlab.com/runner/register/ [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md [project defined timeout]: ../../user/project/pipelines/settings.html#timeout diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index df455857dee..1b50273eca2 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -27,8 +27,7 @@ CI/CD's pipelines. Using variables means no hardcoded values. ### Predefined environment variables -GitLab CI/CD has a default set of -[predefined variables](predefined_variables.md) +GitLab CI/CD has a [default set of predefined variables](predefined_variables.md) which can be used without any specification needed. You can call issues numbers, user names, branch names, pipeline and commit IDs, and much more. @@ -36,7 +35,7 @@ pipeline and commit IDs, and much more. Predefined environment variables are the ones that GitLab provides out of the box for the local environment of the Runner. -GitLab reads the .gitlab-ci.yml file, sends the information +GitLab reads the `.gitlab-ci.yml` file, sends the information to the Runner (which runs the script commands), under which the variables are exposed. @@ -44,6 +43,9 @@ For example, two jobs under the same pipeline can share the same `CI_PIPELINE_ID` variable, but each one has its own `CI_JOB_ID` variable. +NOTE: **Note:** +Find here the full [**predefined variables reference table**](predefined_variables.md). + ### Custom environment variables When your use case requires a specific variable, you can @@ -337,7 +339,7 @@ Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. 1. Inputing variable types, keys, and values in the **Variables** section. -Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. + Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. Once you set them, they will be available for all subsequent pipelines. @@ -480,7 +482,7 @@ Below you can find supported syntax reference: > Example: `$VARIABLE == "some value"` - > Example: `$VARIABLE != "some value"` _(added in 11.11)_ + > Example: `$VARIABLE != "some value"` (introduced in GitLab 11.11) You can use equality operator `==` or `!=` to compare a variable content to a string. We support both, double quotes and single quotes to define a string @@ -491,7 +493,7 @@ Below you can find supported syntax reference: > Example: `$VARIABLE == null` - > Example: `$VARIABLE != null` _(added in 11.11)_ + > Example: `$VARIABLE != null` (introduced in GitLab 11.11) It sometimes happens that you want to check whether a variable is defined or not. To do that, you can compare a variable to `null` keyword, like @@ -502,7 +504,7 @@ Below you can find supported syntax reference: > Example: `$VARIABLE == ""` - > Example: `$VARIABLE != ""` _(added in 11.11)_ + > Example: `$VARIABLE != ""` (introduced in GitLab 11.11) If you want to check whether a variable is defined, but is empty, you can simply compare it against an empty string, like `$VAR == ''` or non-empty @@ -512,7 +514,7 @@ Below you can find supported syntax reference: > Example: `$VARIABLE_1 == $VARIABLE_2` - > Example: `$VARIABLE_1 != $VARIABLE_2` _(added in 11.11)_ + > Example: `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) It is possible to compare two variables. This is going to compare values of these variables. @@ -528,11 +530,11 @@ Below you can find supported syntax reference: `$STAGING` value needs to a string, with length higher than zero. Variable that contains only whitespace characters is not an empty variable. -1. Pattern matching _(added in 11.0)_ +1. Pattern matching (introduced in GitLab 11.0) > Example: `$VARIABLE =~ /^content.*/` - > Example: `$VARIABLE_1 !~ /^content.*/` _(added in 11.11)_ + > Example: `$VARIABLE_1 !~ /^content.*/` (introduced in GitLab 11.11) It is possible perform pattern matching against a variable and regular expression. Expression like this evaluates to truth if matches are found @@ -541,7 +543,7 @@ Below you can find supported syntax reference: Pattern matching is case-sensitive by default. Use `i` flag modifier, like `/pattern/i` to make a pattern case-insensitive. -1. Conjunction / Disjunction +1. Conjunction / Disjunction ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27925) in GitLab 12.0) > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 4655eec51de..7dbb9af2869 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -30,7 +30,7 @@ future GitLab releases.** | `CI_BUILDS_DIR` | all | 11.10 | Top-level directory where builds are executed. | | `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | -| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. Only populated when there is a merge request associated with the pipeline. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | | `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | | `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 8009b1d5e8a..b5296f26269 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -89,6 +89,15 @@ Supported: - In `script`, it will work in the following lines of `script`. - In `after_script`, it will work in following lines of `after_script`. +In the case of `after_script` scripts, they can: + +- Only use variables defined before the script within the same `after_script` + section. +- Not use variables defined in `before_script` and `script`. + +These restrictions are because `after_script` scripts are executed in a +[separated shell context](../yaml/README.md#before_script-and-after_script). + ## Persisted variables NOTE: **Note:** diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index fa4b0378f61..2759f1c5160 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -215,10 +215,25 @@ This can be an array or a multi-line string. `after_script` is used to define the command that will be run after all jobs, including failed ones. This has to be an array or a multi-line string. -The `before_script` and the main `script` are concatenated and run in a single context/container. -The `after_script` is run separately. The current working directory is set back to -default. Depending on the executor, changes done outside of the working tree might -not be visible, e.g. software installed in the `before_script`. +Scripts specified in `before_script` are: + +- Concatenated with scripts specified in the main `script`. Job-level + `before_script` definition override global-level `before_script` definition + when concatenated with `script` definition. +- Executed together with main `script` script as one script in a single shell + context. + +Scripts specified in `after_script`: + +- Have a current working directory set back to the default. +- Are executed in a shell context separated from `before_script` and `script` + scripts. +- Because of separated context, cannot see changes done by scripts defined + in `before_script` or `script` scripts, either: + - In shell. For example, command aliases and variables exported in `script` + scripts. + - Outside of the working tree (depending on the Runner executor). For example, + software installed by a `before_script` or `script` scripts. It's possible to overwrite the globally defined `before_script` and `after_script` if you set it per-job: @@ -1458,7 +1473,7 @@ combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `codequality` report collects [CodeQuality issues](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) +The `codequality` report collects [CodeQuality issues](../../user/project/merge_requests/code_quality.md) as artifacts. The collected Code Quality report will be uploaded to GitLab as an artifact and will @@ -1468,7 +1483,7 @@ be automatically shown in merge requests. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `sast` report collects [SAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/sast/index.html) +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) as artifacts. The collected SAST report will be uploaded to GitLab as an artifact and will @@ -1479,7 +1494,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html) +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) as artifacts. The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will @@ -1490,7 +1505,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html) +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) as artifacts. The collected Container Scanning report will be uploaded to GitLab as an artifact and will @@ -1501,7 +1516,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dast/index.html) +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) as artifacts. The collected DAST report will be uploaded to GitLab as an artifact and will @@ -1512,7 +1527,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `license_management` report collects [Licenses](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) +The `license_management` report collects [Licenses](../../user/project/merge_requests/license_management.md) as artifacts. The collected License Management report will be uploaded to GitLab as an artifact and will @@ -1523,7 +1538,7 @@ dashboards. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `performance` report collects [Performance metrics](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html) +The `performance` report collects [Performance metrics](../../user/project/merge_requests/browser_performance_testing.md) as artifacts. The collected Performance report will be uploaded to GitLab as an artifact and will @@ -1716,7 +1731,7 @@ parallel. This value has to be greater than or equal to two (2) and less than or This creates N instances of the same job that run in parallel. They're named sequentially from `job_name 1/N` to `job_name N/N`. -For every job, `CI_NODE_INDEX` and `CI_NODE_TOTAL` [environment variables](../variables/README.html#predefined-environment-variables) are set. +For every job, `CI_NODE_INDEX` and `CI_NODE_TOTAL` [environment variables](../variables/README.md#predefined-environment-variables) are set. A simple example: @@ -2605,6 +2620,24 @@ test: - pwd ``` +### Nested paths + +The value of `GIT_CLONE_PATH` is expanded once and nesting variables +within it is not supported. + +For example, you define both the variables below in your +`.gitlab-ci.yml` file: + +```yml +variables: + GOPATH: $CI_BUILDS_DIR/go + GIT_CLONE_PATH: $GOPATH/src/namespace/project +``` + +The value of `GIT_CLONE_PATH` is expanded once into +`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure +because `$CI_BUILDS_DIR` is not expanded. + ## Special YAML features It's possible to use special YAML features like anchors (`&`), aliases (`*`) diff --git a/doc/development/README.md b/doc/development/README.md index d2f09fc01de..5df6ec5fd56 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -16,7 +16,7 @@ description: 'Learn how to contribute to GitLab.' - [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/PROCESS.md) - [Generate a changelog entry with `bin/changelog`](changelog.md) -- [Code review guidelines](code_review.md) for reviewing code and having code reviewed. +- [Code review guidelines](code_review.md) for reviewing code and having code reviewed - [Automatic CE->EE merge](automatic_ce_ee_merge.md) - [Guidelines for implementing Enterprise Edition features](ee_features.md) - [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) @@ -33,9 +33,9 @@ description: 'Learn how to contribute to GitLab.' - [GitLab utilities](utilities.md) - [Logging](logging.md) - [API styleguide](api_styleguide.md) Use this styleguide if you are - contributing to the API. + contributing to the API - [GraphQL API styleguide](api_graphql_styleguide.md) Use this - styleguide if you are contribution to the [GraphQL API](../api/graphql/index.md) + styleguide if you are contributing to the [GraphQL API](../api/graphql/index.md) - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Manage feature flags](feature_flags.md) @@ -61,6 +61,8 @@ description: 'Learn how to contribute to GitLab.' - [How Git object deduplication works in GitLab](git_object_deduplication.md) - [Geo development](geo.md) - [Routing](routing.md) +- [Repository mirroring](repository_mirroring.md) +- [Git LFS](lfs.md) ## Performance guides @@ -106,6 +108,7 @@ description: 'Learn how to contribute to GitLab.' - [Database Debugging and Troubleshooting](database_debugging.md) - [Query Count Limits](query_count_limits.md) - [Database helper modules](database_helpers.md) +- [Code comments](code_comments.md) ## Integration guides diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 38270af682e..aeddad14995 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -1,5 +1,15 @@ # GraphQL API +## Deep Dive + +In March 2019, Nick Thomas hosted a [Deep Dive] on GitLab's [GraphQL API] to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.9, and while specific details may have changed since then, it should still serve as a good introduction. + +[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 +[Pull Repository Mirroring functionality]: ../api/graphql/ +[recording on YouTube]: https://www.youtube.com/watch?v=-9L_1MWrjkg +[Google Slides]: https://docs.google.com/presentation/d/1qOTxpkTdHIp1CRjuTvO-aXg0_rUtzE3ETfLUdnBB5uQ/edit +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8e78ea7f326b2ef649e7d7d569c26d56/GraphQL_Deep_Dive__Create_.pdf + ## Authentication Authentication happens through the `GraphqlController`, right now this @@ -188,9 +198,9 @@ abilities as in the Rails app. If the: - Currently authenticated user fails the authorization, the authorized -resource will be returned as `null`. + resource will be returned as `null`. - Resource is part of a collection, the collection will be filtered to -exclude the objects that the user's authorization checks failed against. + exclude the objects that the user's authorization checks failed against. TIP: **Tip:** Try to load only what the currently authenticated user is allowed to @@ -437,7 +447,7 @@ want to validate the abilities for. Alternatively, we can add a `find_object` method that will load the object on the mutation. This would allow you to use the -`authorized_find!` and `authorized_find!` helper methods. +`authorized_find!` helper method. When a user is not allowed to perform the action, or an object is not found, we should raise a diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 5f32cd7eba2..ee6d00331e3 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -40,7 +40,7 @@ A complete architecture diagram is available in our  -<!-- +<!-- To update this diagram, GitLab team members can edit this source file: https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit. --> @@ -93,8 +93,8 @@ graph TB Prometheus --> Alertmanager Migrations --> PostgreSQL Runner -- TCP 443 --> NGINX - Unicorn -- TCP 9200 --> ElasticSearch - Sidekiq -- TCP 9200 --> ElasticSearch + Unicorn -- TCP 9200 --> Elasticsearch + Sidekiq -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 80, 443 --> Sentry Unicorn -- TCP 80, 443 --> Sentry Sidekiq -- UDP 6831 --> Jaeger @@ -116,10 +116,10 @@ graph TB ### Component legend -* ✅ - Installed by default -* ⚙ - Requires additional configuration, or GitLab Managed Apps -* ⤓ - Manual installation required -* ❌ - Not supported or no instructions available +- ✅ - Installed by default +- ⚙ - Requires additional configuration, or GitLab Managed Apps +- ⤓ - Manual installation required +- ❌ - Not supported or no instructions available Component statuses are linked to configuration documentation for each component. @@ -141,15 +141,15 @@ Component statuses are linked to configuration documentation for each component. | [Consul](#consul) | Database node discovery, failover | [⚙][consul-omnibus] | [❌][consul-charts] | [❌][consul-charts] | [✅](../user/gitlab_com/index.md#consul) | ❌ | ❌ | EE Only | | [GitLab self-monitoring: Prometheus](#prometheus) | Time-series database, metrics collection, and query service | [✅][prometheus-omnibus] | [✅][prometheus-charts] | [⚙][prometheus-charts] | [✅](../user/gitlab_com/index.md#prometheus) | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | [⚙][alertmanager-omnibus] | [✅][alertmanager-charts] | [⚙][alertmanager-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | -| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [⚙][grafana-omnibus] | [⤓][grafana-charts] | [⤓][grafana-charts] | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [✅][grafana-omnibus] | [⤓][grafana-charts] | [⤓][grafana-charts] | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [⤓][sentry-omnibus] | [❌][sentry-charts] | [❌][sentry-charts] | [✅](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | | [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | [❌][jaeger-omnibus] | [❌][jaeger-charts] | [❌][jaeger-charts] | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | [⤓][jaeger-source] | [⚙][jaeger-gdk] | CE & EE | | [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [✅][redis-exporter-omnibus] | [✅][redis-exporter-charts] | [✅][redis-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [✅][postgres-exporter-omnibus] | [✅][postgres-exporter-charts] | [✅][postgres-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | [⚙][pgbouncer-exporter-omnibus] | [❌][pgbouncer-exporter-charts] | [❌][pgbouncer-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | -| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [❌][gitab-monitor-charts] | [❌][gitab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [✅][gitab-monitor-charts] | [✅][gitab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | [✅][node-exporter-omnibus] | [❌][node-exporter-charts] | [❌][node-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | -| [Mattermost](#mattermost) | Open-source Slack alternative | [⚙][mattermost-omnibus] | [⤓][mattermost-charts] | [⤓][mattermost-charts] | [⤓](../user/project/integrations/mattermost_slash_commands.md#manual-configuration), [⤓](../user/project/integrations/mattermost.html) | ❌ | ❌ | CE & EE | +| [Mattermost](#mattermost) | Open-source Slack alternative | [⚙][mattermost-omnibus] | [⤓][mattermost-charts] | [⤓][mattermost-charts] | [⤓](../user/project/integrations/mattermost.md) | ❌ | ❌ | CE & EE | | [MinIO](#minio) | Object storage service | [⤓][minio-omnibus] | [✅][minio-charts] | [✅][minio-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | ❌ | [⚙][minio-gdk] | CE & EE | | [Runner](#gitlab-runner) | Executes GitLab CI jobs | [⤓][runner-omnibus] | [✅][runner-charts] | [⚙][runner-charts] | [✅](../user/gitlab_com/index.md#shared-runners) | [⚙][runner-source] | [⚙][runner-gdk] | CE & EE | | [Database Migrations](#database-migrations) | Database migrations | [✅][database-migrations-omnibus] | [✅][database-migrations-charts] | [✅][database-migrations-charts] | ✅ | [⚙][database-migrations-source] | ✅ | CE & EE | @@ -158,7 +158,7 @@ Component statuses are linked to configuration documentation for each component. | [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | [⤓][gitlab-yml] | [⤓][ldap-gdk] | CE & EE | | [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | | [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | -| [ElasticSearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | [⤓][elasticsearch-source] | [⤓][elasticsearch-gdk] | EE Only | +| [Elasticsearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | [⤓][elasticsearch-source] | [⤓][elasticsearch-gdk] | EE Only | | [Sentry integration](#sentry) | Error tracking for deployed apps | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | CE & EE | | [Jaeger integration](#jaeger) | Distributed tracing for deployed apps | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | EE Only | | [GitLab Managed Apps](#gitlab-managed-apps) | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | CE & EE | @@ -304,7 +304,7 @@ GitLab is comprised of a large number of services that all log. We started bundl - Configuration: [Omnibus][mattermost-omnibus], [Charts][mattermost-charts] - Layer: Core Service (Processor) -Mattermost is an open source, private cloud, Slack-alternative from https://mattermost.com. +Mattermost is an open source, private cloud, Slack-alternative from <https://mattermost.com>. #### MinIO @@ -321,7 +321,7 @@ MinIO is an object storage server released under Apache License v2.0. It is comp - Layer: Core Service (Processor) - Process: `nginx` -Nginx as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. +Nginx as an ingress port for all HTTP requests and routes them to the appropriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. #### Node Exporter @@ -681,7 +681,7 @@ We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/ha [pgbouncer-exporter-omnibus]: ../administration/monitoring/prometheus/pgbouncer_exporter.md [pgbouncer-exporter-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql [gitlab-monitor-omnibus]: ../administration/monitoring/prometheus/gitlab_monitor_exporter.md -[gitab-monitor-charts]: https://gitlab.com/charts/gitlab/issues/319 +[gitab-monitor-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitlab-monitor/index.html [node-exporter-omnibus]: ../administration/monitoring/prometheus/node_exporter.md [node-exporter-charts]: https://gitlab.com/charts/gitlab/issues/1332 [mattermost-omnibus]: https://docs.gitlab.com/omnibus/gitlab-mattermost/ @@ -695,7 +695,7 @@ We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/ha [runner-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md [database-migrations-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html#disabling-automatic-database-migration [database-migrations-charts]: https://docs.gitlab.com/charts/charts/gitlab/migrations/ -[database-migrations-source]: ../update/upgrading_from_source.md#14-install-libs-migrations-etc +[database-migrations-source]: ../update/upgrading_from_source.md#13-install-libs-migrations-etc [certificate-management-omnibus]: https://docs.gitlab.com/omnibus/settings/ssl.html [certificate-management-charts]: https://docs.gitlab.com/charts/installation/tls.html [certificate-management-source]: ../install/installation.md#using-https diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index c63ec53414c..a7b402c3fb0 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -1,12 +1,13 @@ # Chatops on GitLab.com -Chatops on GitLab.com allows GitLabbers to run various automation tasks on GitLab.com using Slack. +ChatOps on GitLab.com allows GitLab team members to run various automation tasks on GitLab.com using Slack. ## Requesting access -GitLabbers may need access to Chatops on GitLab.com for administration tasks such as: +GitLab team-members may need access to Chatops on GitLab.com for administration +tasks such as: -- Configuring feature flags on staging. +- Configuring feature flags. - Running `EXPLAIN` queries against the GitLab.com production replica. To request access to Chatops on GitLab.com: @@ -18,4 +19,4 @@ To request access to Chatops on GitLab.com: - [Chatops Usage](https://docs.gitlab.com/ee/ci/chatops/README.html) - [Understanding EXPLAIN plans](understanding_explain_plans.md) - - [Feature Groups](feature_flags.md#feature-groups) + - [Feature Groups](feature_flags/development.md#feature-groups) diff --git a/doc/development/code_comments.md b/doc/development/code_comments.md new file mode 100644 index 00000000000..36962eb46d4 --- /dev/null +++ b/doc/development/code_comments.md @@ -0,0 +1,14 @@ +# Code comments + +Whenever you add comment to the code that is expected to be addressed at any time +in future, please create a technical debt issue for it. Then put a link to it +to the code comment you've created. This will allow other developers to quickly +check if a comment is still relevant and what needs to be done to address it. + +Examples: + +```rb +# Deprecated scope until code_owner column has been migrated to rule_type. +# To be removed with https://gitlab.com/gitlab-org/gitlab-ee/issues/11834. +scope :code_owner, -> { where(code_owner: true).or(where(rule_type: :code_owner)) } +``` diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 29e2aa1a581..6123f9f845a 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -60,6 +60,8 @@ from teams other than your own. 1. If your merge request includes backend changes [^1], it must be **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_backend)**. + 1. If your merge request includes database migrations or changes to expensive queries [^2], it must be + **approved by a [database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_database)**. 1. If your merge request includes frontend changes [^1], it must be **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_frontend)**. 1. If your merge request includes UX changes [^1], it must be @@ -377,3 +379,4 @@ Largely based on the [thoughtbot code review guide]. [team]: https://about.gitlab.com/team/ [build handbook]: https://about.gitlab.com/handbook/build/handbook/build#how-to-work-with-build [^1]: Please note that specs other than JavaScript specs are considered backend code. +[^2]: We encourage you to seek guidance from a database maintainer if your merge request is potentially introducing expensive queries. It is most efficient to comment on the line of code in question with the SQL queries so they can give their advice. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index e3a1dc711fd..d9595bd7bba 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -43,6 +43,10 @@ The descriptions on the [labels page][labels-page] explain what falls under each Subject labels are labels that define what area or feature of GitLab this issue hits. They are not always necessary, but very convenient. +Subject labels are now used to infer and apply relevant group and devops stage +labels. Please apply them whenever possible to facilitate accurate matching. +Please refer to [this merge request][inferred-labels] for more information. + Examples of subject labels are ~wiki, ~ldap, ~api, ~issues, ~"merge requests", ~labels, and ~"Container Registry". @@ -65,10 +69,12 @@ The current team labels are: - ~Defend - ~Distribution - ~Documentation +- ~Ecosystem - ~Geo - ~Gitaly - ~Growth - ~Manage +- ~Memory - ~Monitor - ~Plan - ~Quality @@ -163,44 +169,44 @@ or ~"Stretch". Any open issue for a previous milestone should be labeled Priority labels help us define the time a ~bug fix should be completed. Priority determines how quickly the defect turnaround time must be. If there are multiple defects, the priority decides which defect has to be fixed immediately versus later. -This label documents the planned timeline & urgency which is used to measure against our actual SLA on delivering ~bug fixes. +This label documents the planned timeline & urgency which is used to measure against our target SLO on delivering ~bug fixes. -| Label | Meaning | Defect SLA (applies only to ~bug and ~security defects) | +| Label | Meaning | Target SLO (applies only to ~bug and ~security defects) | |-------|-----------------|----------------------------------------------------------------------------| | ~P1 | Urgent Priority | The current release + potentially immediate hotfix to GitLab.com (30 days) | | ~P2 | High Priority | The next release (60 days) | | ~P3 | Medium Priority | Within the next 3 releases (approx one quarter or 90 days) | | ~P4 | Low Priority | Anything outside the next 3 releases (more than one quarter or 120 days) | -If an issue seems to fall between two priority labels, assign it to the higher- -priority label. - ## Severity labels Severity labels help us clearly communicate the impact of a ~bug on users. - -| Label | Meaning | Impact on Functionality | Example | -|-------|-------------------|-------------------------------------------------------|---------| -| ~S1 | Blocker | Outage, broken feature with no workaround | Unable to create an issue. Data corruption/loss. Security breach. | -| ~S2 | Critical Severity | Broken Feature, workaround too complex & unacceptable | Can push commits, but only via the command line. | -| ~S3 | Major Severity | Broken Feature, workaround acceptable | Can create merge requests only from the Merge Requests page, not through the Issue. | -| ~S4 | Low Severity | Functionality inconvenience or cosmetic issue | Label colors are incorrect / not being displayed. | - -If an issue seems to fall between two severity labels, even taking the -[severity impact guidance](#severity-impact-guidance) into account, assign -it to the higher-severity label. - -### Severity impact guidance - -Severity levels can be applied further depending on the facet of the impact; e.g. Affected customers, GitLab.com availability, performance and etc. The below is a guideline. - -| Severity | Affected Customers/Users | GitLab.com Availability | Performance Degradation | -|----------|---------------------------------------------------------------------|----------------------------------------------------|------------------------------| -| ~S1 | >50% users affected (possible company extinction level event) | Significant impact on all of GitLab.com | | -| ~S2 | Many users or multiple paid customers affected (but not apocalyptic)| Significant impact on large portions of GitLab.com | Degradation is guaranteed to occur in the near future | -| ~S3 | A few users or a single paid customer affected | Limited impact on important portions of GitLab.com | Degradation is likely to occur in the near future | -| ~S4 | No paid users/customer affected, or expected to in the near future | Minor impact on GitLab.com | Degradation _may_ occur but it's not likely | - +There can be multiple facets of the impact. The below is a guideline. + +| Label | Meaning | Functionality | Affected Users | GitLab.com Availability | Performance Degradation | +|-------|-------------------|-------------------------------------------------------|----------------------------------|----------------------------------------------------|------------------------------| +| ~S1 | Blocker | Unusable feature with no workaround, user is blocked | Impacts 50% or more of users | Outage, Significant impact on all of GitLab.com | | +| ~S2 | Critical Severity | Broken Feature, workaround too complex & unacceptable | Impacts between 25%-50% of users | Significant impact on large portions of GitLab.com | Degradation is guaranteed to occur in the near future | +| ~S3 | Major Severity | Broken feature with an acceptable workaround | Impacts up to 25% of users | Limited impact on important portions of GitLab.com | Degradation is likely to occur in the near future | +| ~S4 | Low Severity | Functionality inconvenience or cosmetic issue | Impacts less than 5% of users | Minor impact on GitLab.com | Degradation _may_ occur but it's not likely | + +If a bug seems to fall between two severity labels, assign it to the higher-severity label. + +- Example(s) of ~S1 + - Data corruption/loss. + - Security breach. + - Unable to create an issue or merge request. + - Unable to add a comment or discussion to the issue or merge request. +- Example(s) of ~S2 + - Cannot submit changes through the web IDE but the commandline works. + - A status widget on the merge request page is not working but information can be seen in the test pipeline page. +- Example(s) of ~S3 + - Can create merge requests only from the Merge Requests list view, not from an Issue page. + - Status is not updated in real time and needs a page refresh. +- Example(s) of ~S4 + - Label colors are incorrect. + - UI elements are not fully aligned. + ## Label for community contributors Issues that are beneficial to our users, 'nice to haves', that we currently do @@ -442,3 +448,4 @@ A recent example of this was the issue for [labels-page]: https://gitlab.com/gitlab-org/gitlab-ce/labels [ce-tracker]: https://gitlab.com/gitlab-org/gitlab-ce/issues [ee-tracker]: https://gitlab.com/gitlab-org/gitlab-ee/issues +[inferred-labels]: https://gitlab.com/gitlab-org/quality/triage-ops/merge_requests/155 diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 8a4aa5dfa7f..6064f59ed10 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -126,16 +126,16 @@ When writing commit messages, please follow the guidelines below: - The commit subject must contain at least 3 words. - The commit subject should ideally contain up to 50 characters, -and must not be longer than 72 characters. + and must not be longer than 72 characters. - The commit subject must start with a capital letter. - The commit subject must not end with a period. - The commit subject and body must be separated by a blank line. - The commit body must not contain more than 72 characters per line. - Commits that change 30 or more lines across at least 3 files must -describe these changes in the commit body. + describe these changes in the commit body. - The commit subject or body must not contain Emojis. - Use issues and merge requests' full URLs instead of short references, -as they are displayed as plain text outside of GitLab. + as they are displayed as plain text outside of GitLab. - The merge request must not contain more than 10 commit messages. If the guidelines are not met, the MR will not pass the diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index f319d00d7fe..87e61a7476f 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -31,8 +31,8 @@ This is also the style used by linting tools such as [Return to Contributing documentation](index.md) -[rss-source]: https://github.com/bbatsov/ruby-style-guide/blob/master/README.md#source-code-layout -[rss-naming]: https://github.com/bbatsov/ruby-style-guide/blob/master/README.md#naming +[rss-source]: https://github.com/rubocop-hq/ruby-style-guide/blob/master/README.adoc#source-code-layout +[rss-naming]: https://github.com/rubocop-hq/ruby-style-guide/blob/master/README.adoc#naming-conventions [doc-guidelines]: ../documentation/index.md "Documentation guidelines" [js-styleguide]: ../fe_guide/style_guide_js.md "JavaScript styleguide" [scss-styleguide]: ../fe_guide/style_guide_scss.md "SCSS styleguide" diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 68d33a9d8e0..de2c5b43411 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -85,3 +85,21 @@ eric 37709 0.0 0.0 2518640 7524 s006 S Wed11AM 0:00.79 s $ kill 87304 $ kill 37709 ``` + +### db:migrate `database version is too old to be migrated` error + +Users receive this error when `db:migrate` detects that the current schema version +is older than the `MIN_SCHEMA_VERSION` defined in the `Gitlab::Database` library +module. + +Over time we cleanup/combine old migrations in the codebase, so it is not always +possible to migrate GitLab from every previous version. + +In some cases you may want to bypass this check. For example, if you were on a version +of GitLab schema later than the `MIN_SCHEMA_VERSION`, and then rolled back the +to an older migration, from before. In this case, in order to migrate forward again, +you should set the `SKIP_SCHEMA_VERSION_CHECK` environment variable. + +```sh +bundle exec rake db:migrate SKIP_SCHEMA_VERSION_CHECK=true +``` diff --git a/doc/development/diffs.md b/doc/development/diffs.md index 56e869c21f8..5655398c886 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -6,6 +6,15 @@ Currently we rely on different sources to present diffs, these include: - Database (through `merge_request_diff_files`) - Redis (cached highlighted diffs) +## Deep Dive + +In Jaunary 2019, Oswaldo Ferreira hosted a [Deep Dive] on GitLab's Diffs and Commenting on Diffs functionality to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.7, and while specific details may have changed since then, it should still serve as a good introduction. + +[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 +[recording on YouTube]: https://www.youtube.com/watch?v=K6G3gMcFyek +[Google Slides]: https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/Create_Deep_Dive__Diffs_and_commenting_on_diffs.pdf + ## Architecture overview ### Merge request diffs @@ -124,4 +133,4 @@ File diff will be suppressed (technically different from collapsed, but behaves Diff Viewers, which can be found on `models/diff_viewer/*` are classes used to map metadata about each type of Diff File. It has information whether it's a binary, which partial should be used to render it or which File extensions this class accounts for. -`DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type in order to check if it can be rendered. +`DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type in order to check if it can be rendered.
\ No newline at end of file diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index c7fa40af930..418e58b22d5 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -76,10 +76,10 @@ After a given documentation path is aligned across CE and EE, all merge requests affecting that path must be submitted to CE, regardless of the content it has. This means that: -* For **EE-only docs changes**, you only have to submit a CE MR. -* For **EE-only features** that touch both the code and the docs, you have to submit -an EE MR containing all changes, and a CE MR containing only the docs changes -and without a changelog entry. +- For **EE-only docs changes**, you only have to submit a CE MR. +- For **EE-only features** that touch both the code and the docs, you have to submit + an EE MR containing all changes, and a CE MR containing only the docs changes + and without a changelog entry. This might seem like a duplicate effort, but it's only for the short term. A list of the already aligned docs can be found in diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index f2f4f5f0e1c..20eeebf444f 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -4,8 +4,9 @@ description: "Learn how GitLab docs' global navigation works and how to add new # Global navigation -> [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/362) -in November 2018 for GitLab 11.6. +> - [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/362) +in GitLab 11.6. +> - [Updated](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/482) in GitLab 12.1. The global nav adds to the left sidebar the ability to navigate and explore the contents of GitLab's documentation. @@ -217,13 +218,13 @@ and the following syntax rules. - Always use relative paths against the home of CE and EE. Examples: - For `https://docs.gitlab.com/ee/README.html`, the relative URL is `README.html`. - For `https://docs.gitlab.com/ee/user/project/cycle_analytics.html`, the relative - URL is `user/project/cycle_analytics.html` + URL is `user/project/cycle_analytics.html`. - For `README.html` files, add the complete path `path/to/README.html`. - For `index.html` files, use the clean (canonical) URL: `path/to/`. - For EE-only docs, use the same relative path, but add the attribute `ee_only: true` below - the `doc_url` or `category_url`, as explained above. This will guarantee that when - the user is looking at the CE docs, it will link to the EE docs. It also displays - an "info" icon on the CE nav to make the user aware that it's a different link. + the `doc_url` or `category_url`, as explained above. This displays + an "info" icon on the nav to make the user aware that the feature is + EE-only. DANGER: **Important!** All links present on the data file must end in `.html`, not `.md`. Do not @@ -283,7 +284,7 @@ For instance, for `https://docs.gitlab.com/ce/user/index.html`, #### Default URL The default and canonical URL for GitLab documentation is -`http://docs.gitlab.com/ee/`, thus, all links +`https://docs.gitlab.com/ee/`, thus, all links in the docs site should link to `/ee/` except when linking among `/ce/` docs themselves. @@ -293,7 +294,7 @@ point to `/ee/` docs. On the other hand, if the user is looking at `/ce/` docs, all the links in the CE nav should link internally to `/ce/` -files, except for [`ee-only` docs](#ee-only-docs). +files. ```html <% if dir != 'ce' %> @@ -314,21 +315,12 @@ categories (`cat[:category_url]`), and docs (`doc[:doc_url]`) URLs. #### `ee-only` docs -If the user is looking at the CE nav, a given doc is present only -in `/ee/`, it's tagged in the data file by `ee-only`, linking it -directly to `/ee/`. +Docs for features present only in GitLab EE are tagged +in the data file by `ee-only` and an icon is displayed on the nav +link indicating that the `ee-only` feature is not available in CE. -```html -<% if dir == 'ce' && cat[:ee_only] %> - <a href="/ee/<%= cat[:category_url] %>">...</a> -<% end %> -``` - -To make it clear that it it's a different link, an icon is displayed -on the nav link indicating that the `ee-only` doc is not available in CE. - -The `ee-only` attribute is available for `categories` (`<% if dir == 'ce' && cat[:ee_only] %>`) -and `docs` (`<% if dir == 'ce' && doc[:ee_only] %>`), but not for `sections`. +The `ee-only` attribute is available for `categories` (`<% if cat[:ee_only] %>`) +and `docs` (`<% if doc[:ee_only] %>`), but not for `sections`. ### CSS classes diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index ee3a9caf9a0..6dd12b5efa7 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -11,8 +11,40 @@ and deploy it to <https://docs.gitlab.com>. While the source of the documentation content is stored in GitLab's respective product repositories, the source that is used to build the documentation site _from that content_ -is located at <https://gitlab.com/gitlab-com/gitlab-docs>. See the README there for -detailed information. +is located at <https://gitlab.com/gitlab-com/gitlab-docs>. + +The following diagram illustrates the relationship between the repositories +from where content is sourced, the `gitlab-docs` project, and the published output. + +```mermaid + graph LR + A[gitlab-ce/doc] + B[gitlab-ee/doc] + C[gitlab-runner/docs] + D[omnibus-gitlab/doc] + E[charts/doc] + F[gitlab-docs] + A --> F + B --> F + C --> F + D --> F + E --> F + F -- Build pipeline --> G + G[docs.gitlab.com] + H[/ce/] + I[/ee/] + J[/runner/] + K[/omnibus/] + L[/charts/] + G --> H + G --> I + G --> J + G --> K + G --> L +``` + +See the [README there](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md) +for detailed information. ## Assets @@ -22,9 +54,9 @@ the GitLab Documentation website. ### Libraries -- [Bootstrap 3.3 components](https://getbootstrap.com/docs/3.3/components/) -- [Bootstrap 3.3 JS](https://getbootstrap.com/docs/3.3/javascript/) -- [jQuery](https://jquery.com/) 3.2.1 +- [Bootstrap 4.3.1 components](https://getbootstrap.com/docs/4.3/components/) +- [Bootstrap 4.3.1 JS](https://getbootstrap.com/docs/4.3/getting-started/javascript/) +- [jQuery](https://jquery.com/) 3.3.1 - [Clipboard JS](https://clipboardjs.com/) - [Font Awesome 4.7.0](https://fontawesome.com/v4.7.0/icons/) diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 5caca846cc9..6bfedcb1047 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -76,8 +76,8 @@ and cross-link between any related content. We employ a **docs-first methodology** to help ensure that the docs remain a complete and trusted resource, and to make communicating about the use of GitLab more efficient. -* If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. -* When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request to add this information to the docs. You can then share the MR in order to communicate this information. +- If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. +- When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request to add this information to the docs. You can then share the MR in order to communicate this information. New information that would be useful toward the future usage or troubleshooting of GitLab should not be written directly in a forum or other messaging system, but added to a docs MR and then referenced, as described above. Note that among any other doc changes, you can always add a Troubleshooting section to a doc if none exists, or un-comment and use the placeholder Troubleshooting section included as part of our [doc template](structure.md#template-for-new-docs), if present. @@ -100,6 +100,13 @@ use regular Markdown markup, following the rules in the linked style guide. Note that Kramdown-specific markup (e.g., `{:.class}`) will not render properly on GitLab instances under [`/help`](index.md#gitlab-help). +Hard-coded HTML is valid, although it's discouraged to be used while we have `/help`. HTML is permitted as long as: + +- There's no equivalent markup in markdown. +- Advanced tables are necessary. +- Special styling is required. +- Reviewed and approved by a technical writer. + ## Structure ### Organize by topic, not by type @@ -143,7 +150,8 @@ The table below shows what kind of documentation goes where. a proper naming would be `import_projects_from_github.md`. The same rule applies to images. 1. For image files, do not exceed 100KB. -1. We do not yet support embedded videos. Please link out. +1. Do not upload video files to the product repositories. + [Link or embed videos](#videos) instead. 1. There are four main directories, `user`, `administration`, `api` and `development`. 1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, `profile/`, `dashboard/` and `admin_area/`. @@ -165,13 +173,13 @@ The table below shows what kind of documentation goes where. `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. General user- and admin- related documentation, should be placed accordingly. 1. The directories `/workflow/`, `/university/`, and `/articles/` have -been **deprecated** and the majority their docs have been moved to their correct location -in small iterations. + been **deprecated** and the majority their docs have been moved to their correct location + in small iterations. If you are unsure where a document or a content addition should live, this should not stop you from authoring and contributing. You can use your best judgment and then ask the reviewer of your MR to confirm your decision, and/or ask a technical writer -at any stage in the process. The techncial writing team will review all documentation +at any stage in the process. The technical writing team will review all documentation changes, regardless, and can move content if there is a better place for it. ### Avoid duplication @@ -207,6 +215,7 @@ Do not include the same information in multiple places. [Link to a SSOT instead. ## Text +- [Write in markdown](#markdown). - Splitting long lines (preferably up to 100 characters) can make it easier to provide feedback on small chunks of text. - Insert an empty line for new paragraphs. - Use sentence case for titles, headings, labels, menu items, and buttons. @@ -409,11 +418,20 @@ To indicate the steps of navigation through the UI: ## Images - Place images in a separate directory named `img/` in the same directory where - the `.md` document that you're working on is located. Always prepend their - names with the name of the document that they will be included in. For - example, if there is a document called `twitter.md`, then a valid image name - could be `twitter_login_screen.png`. -- Images should have a specific, non-generic name that will differentiate and describe them properly. + the `.md` document that you're working on is located. +- Images should have a specific, non-generic name that will + differentiate and describe them properly. +- Always add to the end of the file name the GitLab release version + number corresponding to the release milestone the image was added to, + or corresponding to the release the screenshot was taken from, using the + format `image_name_vX_Y.png`. + ([Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/61027) in GitLab 12.1.) +- For example, for a screenshot taken from the pipelines page of + GitLab 11.1, a valid name is `pipelines_v11_1.png`. If you're + adding an illustration that does not include parts of the UI, + add the release number corresponding to the release the image + was added to. Example, for an MR added to 11.1's milestone, + a valid name for an illustration is `devops_diagram_v11_1.png`. - Keep all file names in lower case. - Consider using PNG images instead of JPEG. - Compress all images with <https://tinypng.com/> or similar tool. @@ -421,12 +439,12 @@ To indicate the steps of navigation through the UI: - Images should be used (only when necessary) to _illustrate_ the description of a process, not to _replace_ it. - Max image size: 100KB (gifs included). -- The GitLab docs do not support videos yet. +- See also how to link and embed [videos](#videos) to illustrate the docs. Inside the document: - The Markdown way of using an image inside a document is: - `` + `` - Always use a proper description for what the image is about. That way, when a browser fails to show the image, this text will be used as an alternative description. @@ -446,6 +464,85 @@ directly to an HTML `img` tag: <img src="path/to/image.jpg" alt="Alt text (required)" class="image-noshadow"> ``` +## Videos + +Adding GitLab's existing YouTube video tutorials to the documentation is +highly encouraged, unless the video is outdated. Videos should not +replace documentation, but complement or illustrate it. If content in a video is +fundamental to a feature and its key use cases, but this is not adequately covered in the documentation, +add this detail to the documentation text or create an issue to review the video and do so. + +Do not upload videos to the product repositories. [Link](#link-to-video) or [embed](#embed-videos) them instead. + +### Link to video + +To link out to a video, include a YouTube icon so that readers can +quickly and easily scan the page for videos before reading: + +```md +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Video Title](link-to-video). +``` + +You can link any up-to-date video that is useful to the GitLab user. + +### Embed videos + +> [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/472) in GitLab 12.1. + +GitLab docs (docs.gitlab.com) support embedded videos. + +You can only embed videos from +[GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). +For videos from other sources, [link](#link-to-video) them instead. + +In most cases, it is better to [link to video](#link-to-video) instead, +because an embed takes up a lot of space on the page and can be distracting +to readers. + +To embed a video, follow the instructions below and make sure +you have your MR reviewed and approved by a technical writer. + +1. Copy the code below and paste it into your markdown file. + Leave a blank line above and below it. Do NOT edit the code + (don't remove or add any spaces, etc). +1. On YouTube, visit the video URL you want to display. Copy + the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) + and replace the video title and link in the line under `<div class="video-fallback">`. +1. On YouTube, click **Share**, then **Embed**. +1. Copy the `<iframe>` source (`src`) **URL only** + (`https://www.youtube.com/embed/VIDEO-ID`), + and paste it, replacing the content of the `src` field in the + `iframe` tag. + +```html +leave a blank line here +<div class="video-fallback"> + See the video: [Video title](https://www.youtube.com/watch?v=MqL6BMOySIQ). +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> +leave a blank line here +``` + +This is how it renders on docs.gitlab.com: + +<div class="video-fallback"> + See the video: [What is GitLab](https://www.youtube.com/watch?v=enMumwvLAug). +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + +> Notes: +> +> - The `figure` tag is required for semantic SEO and the `video_container` +class is necessary to make sure the video is responsive and displays +nicely on different mobile devices. +> - The `<div class="video-fallback">` is a fallback necessary for GitLab's +`/help`, as GitLab's markdown processor does not support iframes. It's hidden on the docs site but will be displayed on GitLab's `/help`. + ## Code blocks - Always wrap code added to a sentence in inline code blocks (``` ` ```). @@ -480,7 +577,7 @@ directly to an HTML `img` tag: ## Alert boxes -Whenever you want to call the attention to a particular sentence, +Whenever you need to call special attention to particular sentences, use the following markup for highlighting. _Note that the alert boxes only work for one paragraph only. Multiple paragraphs, @@ -488,6 +585,23 @@ lists, headers, etc will not render correctly. For multiple lines, use blockquot ### Note +Notes catch the eye of most readers, and therefore should be used very sparingly. +In most cases, content considered for a note should be included: + +- As just another sentence in the previous paragraph or the most-relevant paragraph. +- As its own standalone paragraph. +- As content under a new subheading that introduces the topic, making it more visible/findable. + +#### When to use + +Use a note when there is a reason that most or all readers who browse the +section should see the content. That is, if missed, it’s likely to cause +major trouble for a minority of users or significant trouble for a majority +of users. + +Weigh the costs of distracting users to whom the content is not relevant against +the cost of users missing the content if it were not expressed as a note. + ```md NOTE: **Note:** This is something to note. @@ -710,7 +824,7 @@ for the changes to take effect. If the document you are editing resides in a place other than the GitLab CE/EE `doc/` directory, instead of the relative link, use the full path: -`http://docs.gitlab.com/ce/administration/restart_gitlab.html`. +`https://docs.gitlab.com/ce/administration/restart_gitlab.html`. Replace `reconfigure` with `restart` where appropriate. ### Installation guide @@ -962,6 +1076,6 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "domain_ [cURL]: http://curl.haxx.se/ "cURL website" [single spaces]: http://www.slate.com/articles/technology/technology/2011/01/space_invaders.html -[gfm]: http://docs.gitlab.com/ce/user/markdown.html#newlines "GitLab flavored markdown documentation" +[gfm]: https://docs.gitlab.com/ce/user/markdown.html#newlines "GitLab flavored markdown documentation" [ce-1242]: https://gitlab.com/gitlab-org/gitlab-ce/issues/1242 [doc-restart]: ../../administration/restart_gitlab.md "GitLab restart documentation" diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index cca52706ddc..34d41cf4958 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -909,11 +909,12 @@ import bundle from 'ee_else_ce/protected_branches/protected_branches_bundle.js'; See the frontend guide [performance section](fe_guide/performance.md) for information on managing page-specific javascript within EE. - ## Vue code in `assets/javascript` + ### script tag #### Child Component only used in EE + To separate Vue template differences we should [async import the components](https://vuejs.org/v2/guide/components-dynamic-async.html#Async-Components). Doing this allows for us to load the correct component in EE whilst in CE @@ -937,10 +938,12 @@ export default { ``` #### For JS code that is EE only, like props, computed properties, methods, etc, we will keep the current approach - - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. + +- Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin ##### Example: + ```javascript import mixin from 'ee_else_ce/path/mixin'; @@ -955,18 +958,20 @@ import mixin from 'ee_else_ce/path/mixin'; - You can see an MR with an example [here](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9762) #### `template` tag -* **EE Child components** + +- **EE Child components** - Since we are using the async loading to check which component to load, we'd still use the component's name, check [this example](#child-component-only-used-in-ee). -* **EE extra HTML** +- **EE extra HTML** - For the templates that have extra HTML in EE we should move it into a new component and use the `ee_else_ce` dynamic import ### Non Vue Files + For regular JS files, the approach is similar. 1. We will keep using the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) helper, this means that EE only code should be inside the `ee/` folder. - 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. - 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: + 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. + 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: ##### Example: @@ -996,6 +1001,7 @@ to isolate such ruleset from rest of CE rules (along with adding comment describ to avoid conflicts during CE to EE merge. #### Bad + ```scss .section-body { .section-title { @@ -1011,6 +1017,7 @@ to avoid conflicts during CE to EE merge. ``` #### Good + ```scss .section-body { .section-title { diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 8b0f4f02d19..603a756ff56 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -2,7 +2,17 @@ This area is to maintain a compendium of useful information when working with elasticsearch. -Information on how to enable ElasticSearch and perform the initial indexing is kept in ../integration/elasticsearch.md#enabling-elasticsearch +Information on how to enable Elasticsearch and perform the initial indexing is kept in ../integration/elasticsearch.md#enabling-elasticsearch + +## Deep Dive + +In June 2019, Mario de la Ossa hosted a [Deep Dive] on GitLab's [Elasticsearch integration] to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. + +[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 +[Elasticsearch integration]: ../integration/elasticsearch.md +[recording on YouTube]: https://www.youtube.com/watch?v=vrvl-tN2EaA +[Google Slides]: https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf ## Initial installation on OS X @@ -47,27 +57,32 @@ Additionally, if you need large repos or multiple forks for testing, please cons ## How does it work? -The ElasticSearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). +The Elasticsearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs). Search queries are generated by the concerns found in [ee/app/models/concerns/elastic](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! ## Existing Analyzers/Tokenizers/Filters -These are all defined in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb + +These are all defined in <https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb> ### Analyzers + #### `path_analyzer` + Used when indexing blobs' paths. Uses the `path_tokenizer` and the `lowercase` and `asciifolding` filters. Please see the `path_tokenizer` explanation below for an example. #### `sha_analyzer` + Used in blobs and commits. Uses the `sha_tokenizer` and the `lowercase` and `asciifolding` filters. Please see the `sha_tokenizer` explanation later below for an example. #### `code_analyzer` + Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: `code`, `edgeNGram_filter`, `lowercase`, and `asciifolding` The `whitespace` tokenizer was selected in order to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` in order to be properly searched. @@ -75,15 +90,19 @@ The `whitespace` tokenizer was selected in order to have more control over how t Please see the `code` filter for an explanation on how tokens are split. #### `code_search_analyzer` + Not directly used for indexing, but rather used to transform a search input. Uses the `whitespace` tokenizer and the `lowercase` and `asciifolding` filters. ### Tokenizers + #### `sha_tokenizer` + This is a custom tokenizer that uses the [`edgeNGram` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenizer.html) to allow SHAs to be searcheable by any sub-set of it (minimum of 5 chars). -example: +Example: `240c29dc7e` becomes: + - `240c2` - `240c29` - `240c29d` @@ -92,21 +111,26 @@ example: - `240c29dc7e` #### `path_tokenizer` + This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` in order to allow searches to find paths no matter how much or how little of the path is given as input. -example: +Example: `'/some/path/application.js'` becomes: + - `'/some/path/application.js'` - `'some/path/application.js'` - `'path/application.js'` - `'application.js'` ### Filters + #### `code` -Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. + +Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. Patterns: + - `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCased and lowedCameCased strings as separate tokens - `"(\\d+)"`: extracts digits - `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCased strings recursively. Ex: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` @@ -116,6 +140,7 @@ Patterns: - `'\/?([^\/]+)(?=\/|\b)'`: separate path terms `like/this/one` #### `edgeNGram_filter` + Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenfilter.html) to allow inputs with only parts of a token to find the token. For example it would turn `glasses` into permutations starting with `gl` and ending with `glasses`, which would allow a search for "`glass`" to find the original token `glasses` ## Gotchas @@ -130,13 +155,13 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/ You might get an error such as ``` -[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] - flood stage disk watermark [95%] exceeded on - [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], +[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] + flood stage disk watermark [95%] exceeded on + [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], all indices on this node will be marked read-only ``` -This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. +This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. In addition, the `read_only_allow_delete` setting will be set to `true`. It will block indexing, `forcemerge`, etc @@ -148,19 +173,19 @@ Add this to your `elasticsearch.yml` file: ``` # turn off the disk allocator -cluster.routing.allocation.disk.threshold_enabled: false +cluster.routing.allocation.disk.threshold_enabled: false ``` _or_ ``` # set your own limits -cluster.routing.allocation.disk.threshold_enabled: true +cluster.routing.allocation.disk.threshold_enabled: true cluster.routing.allocation.disk.watermark.flood_stage: 5gb # ES 6.x only -cluster.routing.allocation.disk.watermark.low: 15gb +cluster.routing.allocation.disk.watermark.low: 15gb cluster.routing.allocation.disk.watermark.high: 10gb ``` -Restart ElasticSearch, and the `read_only_allow_delete` will clear on it's own. +Restart Elasticsearch, and the `read_only_allow_delete` will clear on it's own. _from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator) and [6.x](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html)_ diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index df32242a522..64c793cfd64 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -5,8 +5,16 @@ [Chrome Accessibility Developer Tools][chrome-accessibility-developer-tools] are useful for testing for potential accessibility problems in GitLab. -Accessibility best-practices and more in-depth information is available on -[the Audit Rules page][audit-rules] for the Chrome Accessibility Developer Tools. +The [axe][axe-website] browser extension (available for [Firefox][axe-firefox-extension] and [Chrome][axe-chrome-extension]) is +also a handy tool for running audits and getting feedback on markup, CSS and even potentially problematic color usages. + +Accessibility best-practices and more in-depth information are available on +[the Audit Rules page][audit-rules] for the Chrome Accessibility Developer Tools. The "[awesome a11y][awesome-a11y]" list is also a +useful compilation of accessibility-related material. [chrome-accessibility-developer-tools]: https://github.com/GoogleChrome/accessibility-developer-tools [audit-rules]: https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules +[axe-website]: https://www.deque.com/axe/ +[axe-firefox-extension]: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/ +[axe-chrome-extension]: https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd +[awesome-a11y]: https://github.com/brunopulis/awesome-a11y diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 8e06aa5d173..9fcd32fddfa 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -18,6 +18,12 @@ To save query compilation at runtime, webpack can directly import `.graphql` files. This allows webpack to preprocess the query at compile time instead of the client doing compilation of queries. +To distinguish queries from mutations and fragments, the following naming convention is recommended: + +- `allUsers.query.graphql` for queries; +- `addUser.mutation.graphql` for mutations; +- `basicUser.fragment.graphql` for fragments. + ## Usage in Vue To use Vue Apollo, import the [Vue Apollo][vue-apollo] plugin as well diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index b25dce65ffe..5220c9eeea3 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -6,6 +6,7 @@ easy to maintain, and performant for the end-user. ## Rules ### Utility Classes + As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950) led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes. @@ -31,12 +32,12 @@ New utility classes should be added to [`utilities.scss`](https://gitlab.com/git #### When should I create component classes? -We recommend a "utility-first" approach. +We recommend a "utility-first" approach. 1. Start with utility classes. 2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it. -This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). +This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). Examples of component classes that were created using "utility-first" include: @@ -45,8 +46,8 @@ Examples of component classes that were created using "utility-first" include: Inspiration: -- https://tailwindcss.com/docs/utility-first -- https://tailwindcss.com/docs/extracting-components +- <https://tailwindcss.com/docs/utility-first> +- <https://tailwindcss.com/docs/extracting-components> ### Naming diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 8c6a73c6824..6c7572352ec 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -49,7 +49,9 @@ provided as a prop to the main component. Be sure to read about [page-specific JavaScript][page_specific_javascript]. ### Bootstrapping Gotchas + #### Providing data from HAML to JavaScript + While mounting a Vue application may be a need to provide data from Rails to JavaScript. To do that, provide the data through `data` attributes in the HTML element and query them while mounting the application. @@ -83,7 +85,8 @@ document.addEventListener('DOMContentLoaded', () => new Vue({ ``` #### Accessing the `gl` object -When we need to query the `gl` object for data that won't change during the application's life cyle, we should do it in the same place where we query the DOM. + +When we need to query the `gl` object for data that won't change during the application's life cycle, we should do it in the same place where we query the DOM. By following this practice, we can avoid the need to mock the `gl` object, which will make tests easier. It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component: @@ -118,12 +121,13 @@ You can read more about components in Vue.js site, [Component System][component- ### A folder for the Store #### Vuex + Check this [page](vuex.md) for more details. ### Mixing Vue and jQuery - Mixing Vue and jQuery is not recommended. -- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it][https://vuejs.org/v2/examples/select2.html]. +- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it](https://vuejs.org/v2/examples/select2.html). - It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. - It is not recommended to add new jQuery events for Vue to interact with jQuery. @@ -212,6 +216,7 @@ describe('Todos App', () => { ``` ### `mountComponent` helper + There is a helper in `spec/javascripts/helpers/vue_mount_component_helper.js` that allows you to mount a component with the given props: ```javascript @@ -225,10 +230,12 @@ const vm = mountComponent(Component, data); ``` ### Test the component's output + The main return value of a Vue component is the rendered output. In order to test the component we need to test the rendered output. [Vue][vue-test] guide's to unit test show us exactly that: ## Vue.js Expert Role + One should apply to be a Vue.js expert by opening an MR when the Merge Request's they create and review show: - Deep understanding of Vue and Vuex reactivy diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 13f0c5cc33e..6bad91d6287 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -1,127 +1 @@ -# Manage feature flags - -Starting from GitLab 9.3 we support feature flags for features in GitLab via -[Flipper](https://github.com/jnunemaker/flipper/). You should use the `Feature` -class (defined in `lib/feature.rb`) in your code to get, set and list feature -flags. - -During runtime you can set the values for the gates via the -[features API](../api/features.md) (accessible to admins only). - -## Feature groups - -Starting from GitLab 9.4 we support feature groups via -[Flipper groups](https://github.com/jnunemaker/flipper/blob/v0.10.2/docs/Gates.md#2-group). - -Feature groups must be defined statically in `lib/feature.rb` (in the -`.register_feature_groups` method), but their implementation can obviously be -dynamic (querying the DB etc.). - -Once defined in `lib/feature.rb`, you will be able to activate a -feature for a given feature group via the [`feature_group` param of the features API](../api/features.md#set-or-create-a-feature) - -For GitLab.com, [team members have access to feature flags through Chatops](chatops_on_gitlabcom.md). Only -percentage gates are supported at this time. Setting a feature to be used 50% of -the time, you should execute `/chatops run feature set my_feature_flag 50`. - -## Feature flags for user applications - -This document only covers feature flags used in the development of GitLab -itself. Feature flags in deployed user applications can be found at -[Feature Flags](../user/project/operations/feature_flags.md) - -## Developing with feature flags - -In general, it's better to have a group- or user-based gate, and you should prefer -it over the use of percentage gates. This would make debugging easier, as you -filter for example logs and errors based on actors too. Furthermore, this allows -for enabling for the `gitlab-org` group first, while the rest of the users -aren't impacted. - -```ruby -# Good -Feature.enabled?(:feature_flag, project) - -# Avoid, if possible -Feature.enabled?(:feature_flag) -``` - -To use feature gates based on actors, the model needs to respond to -`flipper_id`. For example, to enable for the Foo model: - -```ruby -class Foo < ActiveRecord::Base - include FeatureGate -end -``` - -Features that are developed and are intended to be merged behind a feature flag -should not include a changelog entry. The entry should be added in the merge -request removing the feature flags. - -In the rare case that you need the feature flag to be on automatically, use -`default_enabled: true` when checking: - -```ruby -Feature.enabled?(:feature_flag, project, default_enabled: true) -``` - -For more information about rolling out changes using feature flags, refer to the -[Rolling out changes using feature flags](rolling_out_changes_using_feature_flags.md) -guide. - -### Frontend - -For frontend code you can use the method `push_frontend_feature_flag`, which is -available to all controllers that inherit from `ApplicationController`. Using -this method you can expose the state of a feature flag as follows: - -```ruby -before_action do - push_frontend_feature_flag(:vim_bindings) -end - -def index - # ... -end - -def edit - # ... -end -``` - -You can then check for the state of the feature flag in JavaScript as follows: - -```javascript -if ( gon.features.vimBindings ) { - // ... -} -``` - -The name of the feature flag in JavaScript will always be camelCased, meaning -that checking for `gon.features.vim_bindings` would not work. - -### Specs - -In the test environment `Feature.enabled?` is stubbed to always respond to `true`, -so we make sure behavior under feature flag doesn't go untested in some non-specific -contexts. - -Whenever a feature flag is present, make sure to test _both_ states of the -feature flag. - -See the -[testing guide](testing_guide/best_practices.md#feature-flags-in-tests) -for information and examples on how to stub feature flags in tests. - -## Enabling a feature flag (in development) - -In the rails console (`rails c`), enter the following command to enable your feature flag - -```ruby -Feature.enable(:feature_flag_name) -``` - -## Enabling a feature flag (in production) - -Check how to [roll out changes using feature flags](rolling_out_changes_using_feature_flags.md). +This document was moved to [another location](feature_flags/index.md). diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md new file mode 100644 index 00000000000..c67467b7c11 --- /dev/null +++ b/doc/development/feature_flags/controls.md @@ -0,0 +1,123 @@ +# Access for enabling a feature flag in production + +In order to be able to turn on/off features behind feature flags in any of the +GitLab Inc. provided environments such as staging and production, you need to +have access to the chatops bot. Chatops bot is currently running on the ops instance, +which is different from GitLab.com or dev.gitlab.org. + +Follow the Chatops document to [request access](https://docs.gitlab.com/ee/development/chatops_on_gitlabcom.html#requesting-access). + +Once you are added to the project test if your access propagated, +run: + +``` +/chatops run feature --help +``` + +## Rolling out changes + +When the changes are deployed to the environments it is time to start +rolling out the feature to our users. The exact procedure of rolling out a +change is unspecified, as this can vary from change to change. However, in +general we recommend rolling out changes incrementally, instead of enabling them +for everybody right away. We also recommend you to _not_ enable a feature +_before_ the code is being deployed. +This allows you to separate rolling out a feature from a deploy, making it +easier to measure the impact of both separately. + +GitLab's feature library (using +[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature +Flags process](process.md) guide) supports rolling out changes to a percentage of +users. This in turn can be controlled using [GitLab chatops](../../ci/chatops/README.md). + +For an up to date list of feature flag commands please see [the source +code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). +Note that all the examples in that file must be preceded by +`/chatops run`. + +If you get an error "Whoops! This action is not allowed. This incident +will be reported." that means your Slack account is not allowed to +change feature flags or you do not [have access](#access-for-enabling-a-feature-flag-in-production). + +### Enabling feature for staging and dev.gitlab.org + +As a first step in a feature rollout, you should enable the feature on <https://staging.gitlab.com> +and <https://dev.gitlab.org>. + +For example, to enable a feature for 25% of all users, run the following in +Slack: + +``` +/chatops run feature set new_navigation_bar 25 --dev +/chatops run feature set new_navigation_bar 25 --staging +``` + +These two environments have different scopes. +`dev.gitlab.org` is a production CE environment that has internal GitLab Inc. +traffic and is used for some development and other related work. +`staging.gitlab.com` has a smaller subset of GitLab.com database and repositories +and does not have regular traffic. Staging is an EE instance and can give you +a (very) rough estimate of how your feature will look/behave on GitLab.com. +Both of these instances are connected to Sentry so make sure you check the projects +there for any exceptions while testing your feature after enabling the feature flag. + +Once you are confident enough that these environments are in a good state with your +feature enabled, you can roll out the change to GitLab.com. + +## Enabling feature for GitLab.com + +Similar to above, to enable a feature for 25% of all users, run the following in +Slack: + +``` +/chatops run feature set new_navigation_bar 25 +``` + +This will enable the feature for GitLab.com, with `new_navigation_bar` being the +name of the feature. + +If you are not certain what percentages to use, simply use the following steps: + +1. 25% +1. 50% +1. 75% +1. 100% + +Between every step you'll want to wait a little while and monitor the +appropriate graphs on <https://dashboards.gitlab.net>. The exact time to wait +may differ. For some features a few minutes is enough, while for others you may +want to wait several hours or even days. This is entirely up to you, just make +sure it is clearly communicated to your team, and the Production team if you +anticipate any potential problems. + +Feature gates can also be actor based, for example a feature could first be +enabled for only the `gitlab-ce` project. The project is passed by supplying a +`--project` flag: + +``` +/chatops run feature set --project=gitlab-org/gitlab-ce some_feature true +``` + +For groups the `--group` flag is available: + +``` +/chatops run feature set --group=gitlab-org some_feature true +``` + +## Cleaning up + +Once the change is deemed stable, submit a new merge request to remove the +feature flag. This ensures the change is available to all users and self-hosted +instances. Make sure to add the ~"feature flag" label to this merge request so +release managers are aware the changes are hidden behind a feature flag. If the +merge request has to be picked into a stable branch, make sure to also add the +appropriate "Pick into X" label (e.g. "Pick into XX.X"). +See [the process document](https://docs.gitlab.com/ee/development/feature_flags/process.html#including-a-feature-behind-feature-flag-in-the-final-release) for further details. + +When a feature gate has been removed from the code base, the value still exists +in the database. +This can be removed through ChatOps: + +``` +/chatops run feature delete some_feature +``` diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md new file mode 100644 index 00000000000..238052529d9 --- /dev/null +++ b/doc/development/feature_flags/development.md @@ -0,0 +1,131 @@ +# Developing with feature flags + +In general, it's better to have a group- or user-based gate, and you should prefer +it over the use of percentage gates. This would make debugging easier, as you +filter for example logs and errors based on actors too. Furthermore, this allows +for enabling for the `gitlab-org` or `gitlab-com` group first, while the rest of +the users aren't impacted. + +```ruby +# Good +Feature.enabled?(:feature_flag, project) + +# Avoid, if possible +Feature.enabled?(:feature_flag) +``` + +To use feature gates based on actors, the model needs to respond to +`flipper_id`. For example, to enable for the Foo model: + +```ruby +class Foo < ActiveRecord::Base + include FeatureGate +end +``` + +Features that are developed and are intended to be merged behind a feature flag +should not include a changelog entry. The entry should be added in the merge +request removing the feature flags. + +In the rare case that you need the feature flag to be on automatically, use +`default_enabled: true` when checking: + +```ruby +Feature.enabled?(:feature_flag, project, default_enabled: true) +``` + +The [`Project#feature_available?`][project-fa], +[`Namespace#feature_available?`][namespace-fa] (EE), and +[`License.feature_available?`][license-fa] (EE) methods all implicitly check for +a feature flag by the same name as the provided argument. + +For example if a feature is license-gated, there's no need to add an additional +explicit feature flag check since the flag will be checked as part of the +`License.feature_available?` call. Similarly, there's no need to "clean up" a +feature flag once the feature has reached general availability. + +You'd still want to use an explicit `Feature.enabled?` check if your new feature +isn't gated by a License or Plan. + +[project-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/app/models/project_feature.rb#L63-68 +[namespace-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/ee/namespace.rb#L71-85 +[license-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/license.rb#L293-300 + +An important side-effect of the implicit feature flags mentioned above is that +unless the feature is explicitly disabled or limited to a percentage of users, +the feature flag check will default to `true`. + +As an example, if you were to ship the backend half of a feature behind a flag, +you'd want to explicitly disable that flag until the frontend half is also ready +to be shipped. [You can do this via Chatops](https://docs.gitlab.com/ee/development/feature_flags/controls.html): + +``` +/chatops run feature set some_feature 0 +``` + +Note that you can do this at any time, even before the merge request using the +flag has been merged! + +## Feature groups + +Starting from GitLab 9.4 we support feature groups via +[Flipper groups](https://github.com/jnunemaker/flipper/blob/v0.10.2/docs/Gates.md#2-group). + +Feature groups must be defined statically in `lib/feature.rb` (in the +`.register_feature_groups` method), but their implementation can obviously be +dynamic (querying the DB etc.). + +Once defined in `lib/feature.rb`, you will be able to activate a +feature for a given feature group via the [`feature_group` param of the features API](../../api/features.md#set-or-create-a-feature) + +### Frontend + +For frontend code you can use the method `push_frontend_feature_flag`, which is +available to all controllers that inherit from `ApplicationController`. Using +this method you can expose the state of a feature flag as follows: + +```ruby +before_action do + push_frontend_feature_flag(:vim_bindings) +end + +def index + # ... +end + +def edit + # ... +end +``` + +You can then check for the state of the feature flag in JavaScript as follows: + +```javascript +if ( gon.features.vimBindings ) { + // ... +} +``` + +The name of the feature flag in JavaScript will always be camelCased, meaning +that checking for `gon.features.vim_bindings` would not work. + +### Specs + +In the test environment `Feature.enabled?` is stubbed to always respond to `true`, +so we make sure behavior under feature flag doesn't go untested in some non-specific +contexts. + +Whenever a feature flag is present, make sure to test _both_ states of the +feature flag. + +See the +[testing guide](../testing_guide/best_practices.md#feature-flags-in-tests) +for information and examples on how to stub feature flags in tests. + +### Enabling a feature flag (in development) + +In the rails console (`rails c`), enter the following command to enable your feature flag + +```ruby +Feature.enable(:feature_flag_name) +``` diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md new file mode 100644 index 00000000000..56872f8c075 --- /dev/null +++ b/doc/development/feature_flags/index.md @@ -0,0 +1,12 @@ +# Feature flags in development of GitLab + +Feature flags can be used to gradually roll out changes, be +it a new feature, or a performance improvement. By using feature flags, we can +comfortably measure the impact of our changes, while still being able to easily +disable those changes, without having to revert an entire release. + +Before using feature flags for GitLab's development, read through the following: + +- [Process for using features flags](process.md). +- [Developing with feature flags documentation](development.md). +- [Controlling feature flags documentation](controls.md). diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md new file mode 100644 index 00000000000..28d6080ce87 --- /dev/null +++ b/doc/development/feature_flags/process.md @@ -0,0 +1,131 @@ +# Feature flags process + +## Feature flags for user applications + +This document only covers feature flags used in the development of GitLab +itself. Feature flags in deployed user applications can be found at +[Feature Flags feature documentation](../../user/project/operations/feature_flags.md). + +## Feature flags in GitLab development + +The following highlights should be considered when deciding if feature flags +should be leveraged: + +- By default, the feature flags should be **off**. +- Feature flags should remain in the codebase for as short period as possible + to reduce the need for feature flag accounting. +- The person operating with feature flags is responsible for clearly communicating + the status of a feature behind the feature flag with responsible stakeholders. +- Merge requests that make changes hidden behind a feature flag, or remove an + existing feature flag because a feature is deemed stable must have the + ~"feature flag" label assigned. + +One might be tempted to think that feature flags will delay the release of a +feature by at least one month (= one release). This is not the case. A feature +flag does not have to stick around for a specific amount of time +(e.g. at least one release), instead they should stick around until the feature +is deemed stable. Stable means it works on GitLab.com without causing any +problems, such as outages. + +### When to use feature flags + +Starting with GitLab 11.4, developers are required to use feature flags for +non-trivial changes. Such changes include: + +- New features (e.g. a new merge request widget, epics, etc). +- Complex performance improvements that may require additional testing in + production, such as rewriting complex queries. +- Invasive changes to the user interface, such as a new navigation bar or the + removal of a sidebar. +- Adding support for importing projects from a third-party service. + +In all cases, those working on the changes can best decide if a feature flag is +necessary. For example, changing the color of a button doesn't need a feature +flag, while changing the navigation bar definitely needs one. In case you are +uncertain if a feature flag is necessary, simply ask about this in the merge +request, and those reviewing the changes will likely provide you with an answer. + +When using a feature flag for UI elements, make sure to _also_ use a feature +flag for the underlying backend code, if there is any. This ensures there is +absolutely no way to use the feature until it is enabled. + +### Including a feature behind feature flag in the final release + +In order to build a final release and present the feature for self-hosted +users, the feature flag should be at least defaulted to **on**. If the feature +is deemed stable and there is confidence that removing the feature flag is safe, +consider removing the feature flag altogether. Take into consideration that such +action can make the feature available on GitLab.com shortly after the change to +the feature flag is merged. + +Changing the default state or removing the feature flag has to be done before +the 22nd of the month, _at least_ 2 working days before, in order for the change +to be included in the final self-managed release. + +In addition to this, the feature behind feature flag should: + +- Run in all GitLab.com environments for a sufficient period of time. This time + period depends on the feature behind the feature flag, but as a general rule of + thumb 2-4 working days should be sufficient to gather enough feedback. +- The feature should be exposed to all users within the GitLab.com plan during + the above mentioned period of time. Exposing the feature to a smaller percentage + or only a group of users might not expose a sufficient amount of information to aid in + making a decision on feature stability. + +While rare, release managers may decide to reject picking or revert a change in +a stable branch, even when feature flags are used. This might be necessary if +the changes are deemed problematic, too invasive, or there simply isn't enough +time to properly measure how the changes behave on GitLab.com. + +### The cost of feature flags + +When reading the above, one might be tempted to think this procedure is going to +add a lot of work. Fortunately, this is not the case, and we'll show why. For +this example we'll specify the cost of the work to do as a number, ranging from +0 to infinity. The greater the number, the more expensive the work is. The cost +does _not_ translate to time, it's just a way of measuring complexity of one +change relative to another. + +Let's say we are building a new feature, and we have determined that the cost of +this is 10. We have also determined that the cost of adding a feature flag check +in a variety of places is 1. If we do not use feature flags, and our feature +works as intended, our total cost is 10. This however is the best case scenario. +Optimizing for the best case scenario is guaranteed to lead to trouble, whereas +optimizing for the worst case scenario is almost always better. + +To illustrate this, let's say our feature causes an outage, and there's no +immediate way to resolve it. This means we'd have to take the following steps to +resolve the outage: + +1. Revert the release. +1. Perform any cleanups that might be necessary, depending on the changes that + were made. +1. Revert the commit, ensuring the "master" branch remains stable. This is + especially necessary if solving the problem can take days or even weeks. +1. Pick the revert commit into the appropriate stable branches, ensuring we + don't block any future releases until the problem is resolved. + +As history has shown, these steps are time consuming, complex, often involve +many developers, and worst of all: our users will have a bad experience using +GitLab.com until the problem is resolved. + +Now let's say that all of this has an associated cost of 10. This means that in +the worst case scenario, which we should optimize for, our total cost is now 20. + +If we had used a feature flag, things would have been very different. We don't +need to revert a release, and because feature flags are disabled by default we +don't need to revert and pick any Git commits. In fact, all we have to do is +disable the feature, and in the worst case, perform cleanup. Let's say that +the cost of this is 2. In this case, our best case cost is 11: 10 to build the +feature, and 1 to add the feature flag. The worst case cost is now 13: 10 to +build the feature, 1 to add the feature flag, and 2 to disable and clean up. + +Here we can see that in the best case scenario the work necessary is only a tiny +bit more compared to not using a feature flag. Meanwhile, the process of +reverting our changes has been made significantly and reliably cheaper. + +In other words, feature flags do not slow down the development process. Instead, +they speed up the process as managing incidents now becomes _much_ easier. Once +continuous deployments are easier to perform, the time to iterate on a feature +is reduced even further, as you no longer need to wait weeks before your changes +are available on GitLab.com. diff --git a/doc/development/geo.md b/doc/development/geo.md index 6e59fab34c7..a10f13b069f 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -14,10 +14,10 @@ Geo handles replication for different components: - [Database](#database-replication): includes the entire application, except cache and jobs. - [Git repositories](#repository-replication): includes both projects and wikis. - [Uploaded blobs](#uploads-replication): includes anything from images attached on issues -to raw logs and assets from CI. + to raw logs and assets from CI. With the exception of the Database replication, on a *secondary* node, everything is coordinated -by the [Geo Log Cursor](#geo-log-cursor). +by the [Geo Log Cursor](#geo-log-cursor). ### Geo Log Cursor daemon @@ -31,8 +31,8 @@ picks the event up and schedules a `Geo::ProjectSyncWorker` job which will use the `Geo::RepositorySyncService` and `Geo::WikiSyncService` classes to update the repository and the wiki respectively. -The Geo Log Cursor daemon can operate in High Availability mode automatically. -The daemon will try to acquire a lock from time to time and once acquired, it +The Geo Log Cursor daemon can operate in High Availability mode automatically. +The daemon will try to acquire a lock from time to time and once acquired, it will behave as the *active* daemon. Any additional running daemons on the same node, will be in standby @@ -164,20 +164,20 @@ The Git Push Proxy exists as a functionality built inside the `gitlab-shell` com It is active on a **secondary** node only. It allows the user that has cloned a repository from the secondary node to push to the same URL. -Git `push` requests directed to a **secondary** node will be sent over to the **primary** node, +Git `push` requests directed to a **secondary** node will be sent over to the **primary** node, while `pull` requests will continue to be served by the **secondary** node for maximum efficiency. HTTPS and SSH requests are handled differently: - With HTTPS, we will give the user a `HTTP 302 Redirect` pointing to the project on the **primary** node. -The git client is wise enough to understand that status code and process the redirection. + The git client is wise enough to understand that status code and process the redirection. - With SSH, because there is no equivalent way to perform a redirect, we have to proxy the request. -This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request -to the HTTP protocol, and then proxying it to the **primary** node. + This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request + to the HTTP protocol, and then proxying it to the **primary** node. -The [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) daemon knows when to proxy based on the response -from `/api/v4/allowed`. A special `HTTP 300` status code is returned and we execute a "custom action", -specified in the response body. The response contains additional data that allows the proxied `push` operation +The [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) daemon knows when to proxy based on the response +from `/api/v4/allowed`. A special `HTTP 300` status code is returned and we execute a "custom action", +specified in the response body. The response contains additional data that allows the proxied `push` operation to happen on the **primary** node. ## Using the Tracking Database @@ -229,17 +229,17 @@ named `gitlab_secondary`. This configuration exists within the database's user context only. To access the `gitlab_secondary`, GitLab needs to use the same database user that had previously been configured. -The Geo Tracking Database accesses the readonly database replica via FDW as a regular user, -limited by its own restrictions. The credentials are configured as a -`USER MAPPING` associated with the `SERVER` mapped previously +The Geo Tracking Database accesses the readonly database replica via FDW as a regular user, +limited by its own restrictions. The credentials are configured as a +`USER MAPPING` associated with the `SERVER` mapped previously (`gitlab_secondary`). FDW configuration and credentials definition are managed automatically by the -Omnibus GitLab `gitlab-ctl reconfigure` command. +Omnibus GitLab `gitlab-ctl reconfigure` command. #### Refeshing the Foreign Tables -Whenever a new Geo node is configured or the database schema changes on the +Whenever a new Geo node is configured or the database schema changes on the **primary** node, you must refresh the foreign tables on the **secondary** node by running the following: @@ -279,11 +279,11 @@ on the Tracking Database: SELECT project_registry.* FROM project_registry JOIN gitlab_secondary.projects - ON (project_registry.project_id = gitlab_secondary.projects.id + ON (project_registry.project_id = gitlab_secondary.projects.id AND gitlab_secondary.projects.archived IS FALSE) ``` -At the ActiveRecord level, we have additional Models that represent the +At the ActiveRecord level, we have additional Models that represent the foreign tables. They must be mapped in a slightly different way, and they are read-only. Check the existing FDW models in `ee/app/models/geo/fdw` for reference. diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index b512d7611d3..c103a4527ff 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -113,7 +113,7 @@ are as follows: (`pool.source_project`) > TODO Fix invalid SQL data for pools created prior to GitLab 11.11 -> https://gitlab.com/gitlab-org/gitaly/issues/1653. +> <https://gitlab.com/gitlab-org/gitaly/issues/1653>. ### Assumptions @@ -157,7 +157,7 @@ are as follows: repository. > TODO should forks of forks be deduplicated? -> https://gitlab.com/gitlab-org/gitaly/issues/1532 +> <https://gitlab.com/gitlab-org/gitaly/issues/1532> ### Consequences @@ -215,4 +215,4 @@ the secondary, at which stage Git objects will get deduplicated. > TODO How do we handle the edge case where at the time the Geo > secondary tries to create the pool repository, the source project does -> not exist? https://gitlab.com/gitlab-org/gitaly/issues/1533 +> not exist? <https://gitlab.com/gitlab-org/gitaly/issues/1533> diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 4ef20d5fd74..5552d5d37b4 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -3,6 +3,16 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) is a high-level Git RPC service used by GitLab CE/EE, Workhorse and GitLab-Shell. +## Deep Dive + +In May 2019, Bob Van Landuyt hosted a [Deep Dive] on GitLab's [Gitaly project] and how to contribute to it as a Ruby developer, to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.11, and while specific details may have changed since then, it should still serve as a good introduction. + +[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 +[Gitaly project]: https://gitlab.com/gitlab-org/gitaly +[recording on YouTube]: https://www.youtube.com/watch?v=BmlEWFS8ORo +[Google Slides]: https://docs.google.com/presentation/d/1VgRbiYih9ODhcPnL8dS0W98EwFYpJ7GXMPpX-1TM6YE/edit +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/a4fdb1026278bda5c1c5bb574379cf80/Create_Deep_Dive__Gitaly_for_Create_Ruby_Devs.pdf + ## Beginner's guide Start by reading the gitaly repository's @@ -78,12 +88,12 @@ Until GitLab has eliminated most of these inefficiencies or the use of NFS is discontinued for Git data, Rugged implementations of some of the most commonly-used RPCs can be enabled via feature flags: -* `rugged_find_commit` -* `rugged_get_tree_entries` -* `rugged_tree_entry` -* `rugged_commit_is_ancestor` -* `rugged_commit_tree_entry` -* `rugged_list_commits_by_oid` +- `rugged_find_commit` +- `rugged_get_tree_entries` +- `rugged_tree_entry` +- `rugged_commit_is_ancestor` +- `rugged_commit_tree_entry` +- `rugged_list_commits_by_oid` A convenience Rake task can be used to enable or disable these flags all together. To enable: @@ -245,7 +255,7 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. // go implementation } else { findAllTagsRequests.WithLabelValues("ruby").Inc() - // ruby impelmentation + // ruby implementation } ``` diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 9fb8ea542d9..17462887162 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -77,6 +77,9 @@ Or: hello = _("Hello world!") ``` +NOTE: **Note:** Messages in the API (`lib/api/` or `app/graphql`) do +not need to be externalised. + ### HAML files Given the following content in HAML: @@ -132,7 +135,7 @@ There is also and alternative method to [translate messages from validation erro ### Interpolation Placeholders in translated text should match the code style of the respective source file. -For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. +For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make sure to [avoid splitting sentences when adding links](#avoid-splitting-sentences-when-adding-links). - In Ruby/HAML: @@ -264,20 +267,41 @@ should be externalized as follows: This also applies when using links in between translated sentences, otherwise these texts are not translatable in certain languages. -Instead of: +- In Ruby/HAML, instead of: + + ```haml + - zones_link = link_to(s_('ClusterIntegration|zones'), 'https://cloud.google.com/compute/docs/regions-zones/regions-zones', target: '_blank', rel: 'noopener noreferrer') + = s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } + ``` + + Set the link starting and ending HTML fragments as variables like so: + + ```haml + - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' + - zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } + = s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } + ``` + +- In JavaScript, instead of: -```haml -- zones_link = link_to(s_('ClusterIntegration|zones'), 'https://cloud.google.com/compute/docs/regions-zones/regions-zones', target: '_blank', rel: 'noopener noreferrer') -= s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } -``` + ```js + {{ + sprintf(s__("ClusterIntegration|Learn more about %{link}"), { + link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' + }) + }} + ``` -Set the link starting and ending HTML fragments as variables like so: + Set the link starting and ending HTML fragments as variables like so: -```haml -- zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' -- zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } -= s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } -``` + ```js + {{ + sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { + linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">' + linkEnd: '</a>', + }) + }} + ``` The reasoning behind this is that in some languages words change depending on context. For example in Japanese は is added to the subject of a sentence and を to the object. This is impossible to translate correctly if we extract individual words from the sentence. diff --git a/doc/development/import_export.md b/doc/development/import_export.md index fd067b80c16..64c91f151c5 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -147,7 +147,6 @@ The `ModelConfigurationSpec` checks and confirms the addition of new models: If you think this model should be included in the export, please add it to `#{Gitlab::ImportExport.config_file}`. Definitely add it to `#{File.expand_path(ce_models_yml)}` - #{"or `#{File.expand_path(ee_models_yml)}` if the model/associations are EE-specific\n" if ee_models_hash.any?} to signal that you've handled this error and to prevent it from showing up in the future. MSG ``` @@ -253,7 +252,7 @@ Model relationships to be included in the project import/export: ```yaml project_tree: - labels: - :priorities + - :priorities - milestones: - events: - :push_event_payload diff --git a/doc/development/lfs.md b/doc/development/lfs.md new file mode 100644 index 00000000000..8c3408eb6e2 --- /dev/null +++ b/doc/development/lfs.md @@ -0,0 +1,11 @@ +# Git LFS + +## Deep Dive + +In April 2019, Francisco Javier López hosted a [Deep Dive] on GitLab's [Git LFS] implementation to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.10, and while specific details may have changed since then, it should still serve as a good introduction. + +[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 +[Git LFS]: ../workflow/lfs/manage_large_binaries_with_git_lfs.html +[recording on YouTube]: https://www.youtube.com/watch?v=Yyxwcksr0Qc +[Google Slides]: https://docs.google.com/presentation/d/1E-aw6-z0rYd0346YhIWE7E9A65zISL9iIMAOq2zaw9E/edit +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf
\ No newline at end of file diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index 4564f678ec0..879b54bd93c 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -10,16 +10,16 @@ yarn clean ## Creating feature flags in development -The process for creating a feature flag is the same as [enabling a feature flag in development](../feature_flags.md#enabling-a-feature-flag-in-development). +The process for creating a feature flag is the same as [enabling a feature flag in development](../feature_flags/development.md#enabling-a-feature-flag-in-development). Your feature flag can now be: -- [made available to the frontend](../feature_flags.md#frontend) via the `gon` -- queried in [tests](../feature_flags.md#specs) -- queried in HAML templates and ruby files via the `Feature.enabled?(:my_shiny_new_feature_flag)` method +- [Made available to the frontend](../feature_flags/development.md#frontend) via the `gon` +- Queried in [tests](../feature_flags/development.md#specs) +- Queried in HAML templates and ruby files via the `Feature.enabled?(:my_shiny_new_feature_flag)` method ### More on feature flags - [Deleting a feature flag](../../api/features.md#delete-a-feature) -- [Manage feature flags](../feature_flags.md) +- [Manage feature flags](../feature_flags/process.md) - [Feature flags API](../../api/features.md) diff --git a/doc/development/performance.md b/doc/development/performance.md index 0e21d45f57c..c034f4a344b 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -424,7 +424,7 @@ might find using these gems more convenient: ### Examples You may find some useful examples in this snippet: -https://gitlab.com/gitlab-org/gitlab-ce/snippets/33946 +<https://gitlab.com/gitlab-org/gitlab-ce/snippets/33946> [#15607]: https://gitlab.com/gitlab-org/gitlab-ce/issues/15607 [yorickpeterse]: https://gitlab.com/yorickpeterse diff --git a/doc/development/profiling.md b/doc/development/profiling.md index b2f3a105b23..795523b82aa 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -38,10 +38,6 @@ For routes that require authorization you will need to provide a user to Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first) ``` -The user you provide will need to have a [personal access -token](https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) in -the GitLab instance. - Passing a `logger:` keyword argument to `Gitlab::Profiler.profile` will send ActiveRecord and ActionController log output to that logger. Further options are documented with the method source. diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index de5e1323e6a..17d8428b0c0 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -73,7 +73,7 @@ Similar to source browsing, is [Documentation browsing](https://github.com/pry/p ### Command history -With <kdb>Ctrl+R</kbd> you can search your [command history](https://github.com/pry/pry/wiki/History). +With **Ctrl+R** you can search your [command history](https://github.com/pry/pry/wiki/History). ## Stepping diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 28a12572961..4fc10b6af5c 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -80,30 +80,6 @@ There are a few environment flags you can pass to change how projects are seeded - `LARGE_PROJECTS`: defaults to false. If set will clone 6 large projects to help with testing. - `FORK`: defaults to false. If set to `true` will fork `torvalds/linux` five times. Can also be set to an existing project full_path and it will fork that instead. -### Notes for MySQL - -Since the seeds would contain various UTF-8 characters, such as emojis or so, -we'll need to make sure that we're using `utf8mb4` for all the encoding -settings and `utf8mb4_unicode_ci` for collation. Please check -[MySQL utf8mb4 support](../install/database_mysql.md#mysql-utf8mb4-support) - -Make sure that `config/database.yml` has `encoding: utf8mb4`, too. - -Next, we'll need to update the schema to make the indices fit: - -``` shell -sed -i 's/limit: 255/limit: 191/g' db/schema.rb -``` - -Then run the setup script: - -``` shell -bundle exec rake setup -``` - -To make sure that indices still fit. You could find great details in: -[How to support full Unicode in MySQL databases](https://mathiasbynens.be/notes/mysql-utf8mb4) - ## Run tests In order to run the test you can use the following commands: diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md new file mode 100644 index 00000000000..f8c33ff2b85 --- /dev/null +++ b/doc/development/repository_mirroring.md @@ -0,0 +1,11 @@ +# Repository mirroring + +## Deep Dive + +In December 2018, Tiago Botelho hosted a [Deep Dive] on GitLab's [Pull Repository Mirroring functionality] to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.6, and while specific details may have changed since then, it should still serve as a good introduction. + +[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 +[Pull Repository Mirroring functionality]: ../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter +[recording on YouTube]: https://www.youtube.com/watch?v=sSZq0fpdY-Y +[Google Slides]: https://docs.google.com/presentation/d/17BTT6M6RyNRckV4wTt-dr07nIfBvD325_xVBoLtSoPM/edit?usp=sharing +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf
\ No newline at end of file diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 01cedf734fb..59da02ed6fd 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -127,6 +127,42 @@ Everything in `lib/api`. Everything that resides in `app/services`. +#### ServiceResponse + +Service classes usually have an `execute` method, which can return a +`ServiceResponse`. You can use `ServiceResponse.success` and +`ServiceResponse.error` to return a response in `execute` method. + +In a successful case: + +``` ruby +response = ServiceResponse.success(message: 'Branch was deleted') + +response.success? # => true +response.error? # => false +response.status # => :success +response.message # => 'Branch was deleted' +``` + +In a failed case: + +``` ruby +response = ServiceResponse.error(message: 'Unsupported operation') + +response.success? # => false +response.error? # => true +response.status # => :error +response.message # => 'Unsupported operation' +``` + +An additional payload can also be attached: + +``` ruby +response = ServiceResponse.success(payload: { issue: issue }) + +response.payload[:issue] # => issue +``` + ### Finders Everything in `app/finders`, typically used for retrieving data from a database. diff --git a/doc/development/rolling_out_changes_using_feature_flags.md b/doc/development/rolling_out_changes_using_feature_flags.md index 84028b1b342..6bad91d6287 100644 --- a/doc/development/rolling_out_changes_using_feature_flags.md +++ b/doc/development/rolling_out_changes_using_feature_flags.md @@ -1,225 +1 @@ -# Rolling out changes using feature flags - -[Feature flags](feature_flags.md) can be used to gradually roll out changes, be -it a new feature, or a performance improvement. By using feature flags, we can -comfortably measure the impact of our changes, while still being able to easily -disable those changes, without having to revert an entire release. - -## When to use feature flags - -Starting with GitLab 11.4, developers are required to use feature flags for -non-trivial changes. Such changes include: - -- New features (e.g. a new merge request widget, epics, etc). -- Complex performance improvements that may require additional testing in - production, such as rewriting complex queries. -- Invasive changes to the user interface, such as a new navigation bar or the - removal of a sidebar. -- Adding support for importing projects from a third-party service. - -In all cases, those working on the changes can best decide if a feature flag is -necessary. For example, changing the color of a button doesn't need a feature -flag, while changing the navigation bar definitely needs one. In case you are -uncertain if a feature flag is necessary, simply ask about this in the merge -request, and those reviewing the changes will likely provide you with an answer. - -When using a feature flag for UI elements, make sure to _also_ use a feature -flag for the underlying backend code, if there is any. This ensures there is -absolutely no way to use the feature until it is enabled. - -## The cost of feature flags - -When reading the above, one might be tempted to think this procedure is going to -add a lot of work. Fortunately, this is not the case, and we'll show why. For -this example we'll specify the cost of the work to do as a number, ranging from -0 to infinity. The greater the number, the more expensive the work is. The cost -does _not_ translate to time, it's just a way of measuring complexity of one -change relative to another. - -Let's say we are building a new feature, and we have determined that the cost of -this is 10. We have also determined that the cost of adding a feature flag check -in a variety of places is 1. If we do not use feature flags, and our feature -works as intended, our total cost is 10. This however is the best case scenario. -Optimising for the best case scenario is guaranteed to lead to trouble, whereas -optimising for the worst case scenario is almost always better. - -To illustrate this, let's say our feature causes an outage, and there's no -immediate way to resolve it. This means we'd have to take the following steps to -resolve the outage: - -1. Revert the release. -1. Perform any cleanups that might be necessary, depending on the changes that - were made. -1. Revert the commit, ensuring the "master" branch remains stable. This is - especially necessary if solving the problem can take days or even weeks. -1. Pick the revert commit into the appropriate stable branches, ensuring we - don't block any future releases until the problem is resolved. - -As history has shown, these steps are time consuming, complex, often involve -many developers, and worst of all: our users will have a bad experience using -GitLab.com until the problem is resolved. - -Now let's say that all of this has an associated cost of 10. This means that in -the worst case scenario, which we should optimise for, our total cost is now 20. - -If we had used a feature flag, things would have been very different. We don't -need to revert a release, and because feature flags are disabled by default we -don't need to revert and pick any Git commits. In fact, all we have to do is -disable the feature, and in the worst case, perform cleanup. Let's say that -the cost of this is 2. In this case, our best case cost is 11: 10 to build the -feature, and 1 to add the feature flag. The worst case cost is now 13: 10 to -build the feature, 1 to add the feature flag, and 2 to disable and clean up. - -Here we can see that in the best case scenario the work necessary is only a tiny -bit more compared to not using a feature flag. Meanwhile, the process of -reverting our changes has been made significantly and reliably cheaper. - -In other words, feature flags do not slow down the development process. Instead, -they speed up the process as managing incidents now becomes _much_ easier. Once -continuous deployments are easier to perform, the time to iterate on a feature -is reduced even further, as you no longer need to wait weeks before your changes -are available on GitLab.com. - -## Rolling out changes - -The procedure of using feature flags is straightforward, and similar to not -using them. You add the necessary tests (make sure to test both the on and off -states of your feature flag(s)), make sure they all pass, have the code -reviewed, etc. You then submit your merge request, and add the ~"feature flag" -label. This label is used to signal to release managers that your changes are -hidden behind a feature flag and that it is safe to pick the MR into a stable -branch, without the need for an exception request. - -When the changes are deployed it is time to start rolling out the feature to our -users. The exact procedure of rolling out a change is unspecified, as this can -vary from change to change. However, in general we recommend rolling out changes -incrementally, instead of enabling them for everybody right away. We also -recommend you to _not_ enable a feature _before_ the code is being deployed. -This allows you to separate rolling out a feature from a deploy, making it -easier to measure the impact of both separately. - -GitLab's feature library (using -[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags](feature_flags.md) guide) supports rolling out changes to a percentage of -users. This in turn can be controlled using [GitLab -chatops](../ci/chatops/README.md). - -For an up to date list of feature flag commands please see [the source -code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). -Note that all the examples in that file must be preceded by -`/chatops run`. - -If you get an error "Whoops! This action is not allowed. This incident -will be reported." that means your Slack account is not allowed to -change feature flags. To test if you are allowed to do anything at all, -run: - -``` -/chatops run feature --help -``` - -For example, to enable a feature for 25% of all users, run the following in -Slack: - -``` -/chatops run feature set new_navigation_bar 25 -``` - -This will enable the feature for GitLab.com, with `new_navigation_bar` being the -name of the feature. We can also enable the feature for <https://dev.gitlab.org> -or <https://staging.gitlab.com>: - -``` -/chatops run feature set new_navigation_bar 25 --dev -/chatops run feature set new_navigation_bar 25 --staging -``` - -If you are not certain what percentages to use, simply use the following steps: - -1. 25% -1. 50% -1. 75% -1. 100% - -Between every step you'll want to wait a little while and monitor the -appropriate graphs on <https://dashboards.gitlab.net>. The exact time to wait -may differ. For some features a few minutes is enough, while for others you may -want to wait several hours or even days. This is entirely up to you, just make -sure it is clearly communicated to your team, and the Production team if you -anticipate any potential problems. - -Feature gates can also be actor based, for example a feature could first be -enabled for only the `gitlab-ce` project. The project is passed by supplying a -`--project` flag: - -``` -/chatops run feature set --project=gitlab-org/gitlab-ce some_feature true -``` - -For groups the `--group` flag is available: - -``` -/chatops run feature set --group=gitlab-org some_feature true -``` - -Once a change is deemed stable, submit a new merge request to remove the -feature flag. This ensures the change is available to all users and self-hosted -instances. Make sure to add the ~"feature flag" label to this merge request so -release managers are aware the changes are hidden behind a feature flag. If the -merge request has to be picked into a stable branch (e.g. after the 7th), make -sure to also add the appropriate "Pick into X" label (e.g. "Pick into 11.4"). - -One might be tempted to think this will delay the release of a feature by at -least one month (= one release). This is not the case. A feature flag does not -have to stick around for a specific amount of time (e.g. at least one release), -instead they should stick around until the feature is deemed stable. Stable -means it works on GitLab.com without causing any problems, such as outages. In -most cases this will translate to a feature (with a feature flag) being shipped -in RC1, followed by the feature flag being removed in RC2. This in turn means -the feature will be stable by the time we publish a stable package around the -22nd of the month. - -## Implicit feature flags - -The [`Project#feature_available?`][project-fa], -[`Namespace#feature_available?`][namespace-fa] (EE), and -[`License.feature_available?`][license-fa] (EE) methods all implicitly check for -a feature flag by the same name as the provided argument. - -For example if a feature is license-gated, there's no need to add an additional -explicit feature flag check since the flag will be checked as part of the -`License.feature_available?` call. Similarly, there's no need to "clean up" a -feature flag once the feature has reached general availability. - -You'd still want to use an explicit `Feature.enabled?` check if your new feature -isn't gated by a License or Plan. - -[project-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/app/models/project_feature.rb#L63-68 -[namespace-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/ee/namespace.rb#L71-85 -[license-fa]: https://gitlab.com/gitlab-org/gitlab-ee/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/license.rb#L293-300 - -### Undefined feature flags default to "on" - -An important side-effect of the [implicit feature flags](#implicit-feature-flags) -mentioned above is that unless the feature is explicitly disabled or limited to a -percentage of users, the feature flag check will default to `true`. - -As an example, if you were to ship the backend half of a feature behind a flag, -you'd want to explicitly disable that flag until the frontend half is also ready -to be shipped. You can do this via ChatOps: - -``` -/chatops run feature set some_feature 0 -``` - -Note that you can do this at any time, even before the merge request using the -flag has been merged! - -### Cleaning up - -When a feature gate has been removed from the code base, the value still exists -in the database. This can be removed through ChatOps: - -``` -/chatops run feature delete some_feature -``` +This document was moved to [another location](feature_flags/index.md). diff --git a/doc/development/swapping_tables.md b/doc/development/swapping_tables.md index 29cd6a43aff..5c62900dbff 100644 --- a/doc/development/swapping_tables.md +++ b/doc/development/swapping_tables.md @@ -39,14 +39,6 @@ PostgreSQL you can use the `reset_pk_sequence!` method like so: reset_pk_sequence!('events') ``` -For MySQL however you need to do run the following: - -```ruby -amount = Event.pluck('COALESCE(MAX(id), 1)').first - -execute "ALTER TABLE events AUTO_INCREMENT = #{amount}" -``` - Failure to reset the primary keys will result in newly created rows starting with an ID value of 1. Depending on the existing data this can then lead to duplicate key constraints from popping up, preventing users from creating new diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 82439c94c5a..448d9fd01c4 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -17,7 +17,7 @@ our test design. We can find some helpful heuristics documented in the Handbook ## Run tests against MySQL -By default, tests are only run againts PostgreSQL, but you can run them on +By default, tests are only run against PostgreSQL, but you can run them on demand against MySQL by following one of the following conventions: | Convention | Valid example | @@ -327,7 +327,7 @@ However, if a spec makes direct Redis calls, it should mark itself with the `:clean_gitlab_redis_queues` traits as appropriate. Sidekiq jobs are typically not run in specs, but this behaviour can be altered -in each spec through the use of `Sidekiq::Testing.inline!` blocks. Any spec that +in each spec through the use of `perform_enqueued_jobs` blocks. Any spec that causes Sidekiq jobs to be pushed to Redis should use the `:sidekiq` trait, to ensure that they are removed once the spec completes. diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index f7b3ca8bc89..aec0a3ede5a 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -5,8 +5,8 @@ We devised a solution to solve common test automation problems such as the dread Other problems that dynamic element validations solve are... - When we perform an action with the mouse, we expect something to occur. -- When our test is navigating to (or from) a page, we ensure that we are on the page we expect before -test continuation. +- When our test is navigating to (or from) a page, we ensure that we are on the page we expect before + test continuation. ## How it works @@ -19,7 +19,7 @@ We interpret user actions on the page to have some sort of effect. These actions When a page is navigated to, there are elements that will always appear on the page unconditionally. -Dynamic element validation is instituted when using +Dynamic element validation is instituted when using ```ruby Runtime::Browser.visit(:gitlab, Some::Page) @@ -27,7 +27,7 @@ Runtime::Browser.visit(:gitlab, Some::Page) ### Clicks -When we perform a click within our tests, we expect something to occur. That something could be a component to now +When we perform a click within our tests, we expect something to occur. That something could be a component to now appear on the webpage, or the test to navigate away from the page entirely. Dynamic element validation is instituted when using @@ -71,7 +71,7 @@ class MyPage < Page::Base element :another_element, required: true element :conditional_element end - + def open_layer click_element :my_element, Layer::MyLayer end @@ -95,7 +95,7 @@ execute_stuff ``` will invoke GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to -`execute_stuff` +`execute_stuff` ### Clicking diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index afd81ff00b2..527cd350633 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -126,6 +126,18 @@ See [Review Apps][review-apps] for more details about Review Apps. [helm-chart]: https://gitlab.com/charts/gitlab/ [cng]: https://gitlab.com/gitlab-org/build/CNG +## How do I run the tests? + +There are two main options for running the tests. If you simply want to run the +existing tests against a live GitLab instance or against a pre-built docker image +you can use the [GitLab QA orchestrator][gitlab-qa-readme]. See also [examples +of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). + +On the other hand, if you would like to run against a local development GitLab +environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/). +Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md#how-can-i-use-it) +and the section below. + ## How do I write tests? In order to write new tests, you first need to learn more about GitLab QA diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 73e1fd862c1..05cb03eb4bd 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -82,7 +82,7 @@ module Page end # ... - end + end end end ``` @@ -134,7 +134,7 @@ for each element defined. In our case, `qa-login-field`, `qa-password-field` and `qa-sign-in-button` -**app/views/my/view.html.haml** +**app/views/my/view.html.haml** ```haml = f.text_field :login, class: "form-control top qa-login-field", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required." @@ -146,7 +146,7 @@ Things to note: - The CSS class must be `kebab-cased` (separated with hyphens "`-`") - If the element appears on the page unconditionally, add `required: true` to the element. See -[Dynamic element validation](dynamic_element_validation.md) + [Dynamic element validation](dynamic_element_validation.md) ## Running the test locally diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md index afe76acf9c9..041bdf716b3 100644 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -220,7 +220,7 @@ As the pre-conditions for our test suite, the things that needs to happen before - The user logging in; - A premium license already being set; - A project being created with an issue and labels already set; -- The issue page being opened with only one scoped label applied to the it. +- The issue page being opened with only one scoped label applied to it. > When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-remote-grid-environment-variables), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). @@ -242,12 +242,12 @@ module QA issue = Resource::Issue.fabricate_via_api! do |issue| issue.title = 'Issue to test the scoped labels' - issue.labels = @initial_label + issue.labels = [@initial_label] end [@new_label_same_scope, @new_label_different_scope].each do |label| Resource::Label.fabricate_via_api! do |l| - l.project = issue.project.id + l.project = issue.project l.title = label end end @@ -357,11 +357,21 @@ In the following we describe the changes needed in each of the resource files me Now, let's make it possible to create an issue resource through the API. -First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its labels attribute. +First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its id and labels attributes. -Add the following `attribute :labels` right below the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). +Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). -> This line is needed to allow for labels to be automatically added to an issue when fabricating it via API. +> This line is needed to allow for the issue fabrication, and for labels to be automatically added to the issue when fabricating it via API. + +> We add the attributes above the existing attribute to keep them alphabetically organized. + +Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: + +```ruby +def initialize + @labels = [] +end +``` Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. @@ -376,8 +386,8 @@ end def api_post_body { - title: title, - labels: [labels] + labels: labels, + title: title } end ``` @@ -392,7 +402,7 @@ By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://g By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. -> Notice that we pass both `title` and `labels` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](https://docs.gitlab.com/ee/api/issues.html#new-issue). +> Notice that we pass both `labels` and `title` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](https://docs.gitlab.com/ee/api/issues.html#new-issue). Also, notice that we keep them alphabetically organized. **Label resource** @@ -412,13 +422,13 @@ def api_get_path end def api_post_path - "/projects/#{project}/labels" + "/projects/#{project.id}/labels" end def api_post_body { - name: @title, - color: @color + color: @color, + name: @title } end ``` @@ -431,13 +441,13 @@ By defining the `api_post_path` method, we allow for the [`ApiFabricator `](http By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. -> Notice that we pass both `name` and `color` attributes in the `api_post_body` since [those are required](https://docs.gitlab.com/ee/api/labels.html#create-a-new-label). +> Notice that we pass both `color` and `name` attributes in the `api_post_body` since [those are required](https://docs.gitlab.com/ee/api/labels.html#create-a-new-label). Also, notice that we keep them alphabetically organized. ### 8. Page Objects Page Objects are used in end-to-end tests for maintenance reasons, where a page's elements and methods are defined to be reused in any test. -> Page Objects are auto-loaded in the `qa/qa.rb` file and available in all the test files (`*_spec.rb`). +> Page Objects are auto-loaded in the [`qa/qa.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa.rb) file and available in all the test files (`*_spec.rb`). Take a look at the [Page Objects] documentation. @@ -487,25 +497,25 @@ Let's now update the Issue Page Object. The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/page/project/issue/show.rb). -First, add the following code right below the definition of an already implemented view: +First, add the following code right below the definition of an already implemented view (keep in mind that view's definitions and their elements should be alphabetically ordered): ```ruby -view 'app/views/shared/issuable/_sidebar.html.haml' do - element :labels_block - element :edit_link_labels - element :dropdown_menu_labels -end - view 'app/helpers/dropdowns_helper.rb' do element :dropdown_input_field end + +view 'app/views/shared/issuable/_sidebar.html.haml' do + element :dropdown_menu_labels + element :edit_link_labels + element :labels_block +end ``` Similarly to what we did before, let's first change the Page Object even without the elements being defined in the view (`_sidebar.html.haml`) and the `dropdowns_helper.rb` files, and later we will update them by adding the appropriate CSS selectors. Now, let's implement the methods `select_labels_and_refresh` and `text_of_labels_block`. -Somewhere between the definition of the views and the private methods, add the following snippet of code: +Somewhere between the definition of the views and the private methods, add the following snippet of code (these should also be alphabetically ordered for organization reasons): ```ruby def select_labels_and_refresh(labels) @@ -530,6 +540,7 @@ end ##### Details of `select_labels_and_refresh` Notice that we have not only moved the `select_labels_and_refresh` method, but we have also changed its implementation to: + 1. Click the `:edit_link_labels` element previously defined, instead of using `find('.block.labels .edit-link').click` 2. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` 3. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` @@ -545,19 +556,31 @@ Now let's change the view and the `dropdowns_helper` files to add the selectors In the [app/views/shared/issuable/_sidebar.html.haml](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/views/shared/issuable/_sidebar.html.haml) file, on [line 105 ](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L105), add an extra class `qa-edit-link-labels`. -The code should look like this: `= link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link qa-edit-link-labels float-right'`. +The code should look like this: + +```haml += link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link float-right qa-edit-link-labels' +``` In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L121), add an extra class `.qa-dropdown-menu-labels`. -The code should look like this: `.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.qa-dropdown-menu-labels.dropdown-menu-selectable`. +The code should look like this: + +```haml +.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.dropdown-menu-selectable.qa-dropdown-menu-labels +``` In the [`dropdowns_helper.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/helpers/dropdowns_helper.rb) file, on [line 94](https://gitlab.com/gitlab-org/gitlab-ee/blob/99e51a374f2c20bee0989cac802e4b5621f72714/app/helpers/dropdowns_helper.rb#L94), add an extra class `qa-dropdown-input-field`. -The code should look like this: `filter_output = search_field_tag search_id, nil, class: "dropdown-input-field qa-dropdown-input-field", placeholder: placeholder, autocomplete: 'off'`. +The code should look like this: + +```ruby +filter_output = search_field_tag search_id, nil, class: "dropdown-input-field qa-dropdown-input-field", placeholder: placeholder, autocomplete: 'off' +``` > Classes starting with `qa-` are used for testing purposes only, and by defining such classes in the elements we add **testability** in the application. -> When defining a class like `qa-labels-block`, it is transformed into `:labels_block` for usage in the Page Objects. So, `qa-edit-link-labels` is tranformed into `:edit_link_labels`, `qa-dropdown-menu-labels` is transformed into `:dropdown_menu_labels`, and `qa-dropdown-input-field` is transformed into `:dropdown_input_field`. Also, we use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective `qa-` selectors in the specified views. +> When defining a class like `qa-labels-block`, it is transformed into `:labels_block` for usage in the Page Objects. So, `qa-edit-link-labels` is transformed into `:edit_link_labels`, `qa-dropdown-menu-labels` is transformed into `:dropdown_menu_labels`, and `qa-dropdown-input-field` is transformed into `:dropdown_input_field`. Also, we use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective `qa-` selectors in the specified views. > We did not define the `qa-labels-block` class in the `app/views/shared/issuable/_sidebar.html.haml` file because it was already there to be used. @@ -565,7 +588,7 @@ The code should look like this: `filter_output = search_field_tag search_id, nil The last thing that we have to do is to update `QA::Page::Base` class to add the `send_keys_to_element` method on it. -Add the following snippet of code somewhere where class methods are defined: +Add the following snippet of code somewhere where class methods are defined (remember to organize methods alphabetically, and if you see a place where this standard is not being followed, it would be helpful if you could rearrange it): ```ruby def send_keys_to_element(name, keys) diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 4c9d1684c00..28ebb6f0f64 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -27,14 +27,30 @@ we need to solve before being able to use Jest for all our needs. ### Differences to Karma - Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). -- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. +- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. +- Jest does not have access to Webpack loaders or aliases. + The aliases used by Jest are defined in its [own config](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/jest.config.js). - All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks). - `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/en/manual-mocks). +- No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. + This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. + Instead you should declare shared variables in the context that they are needed (via `const` / `let`). - The following will cause tests to fail in Jest: - Unmocked requests. - Unhandled Promise rejections. - Calls to `console.warn`, including warnings from libraries like Vue. +### Limitations of jsdom + +As mentioned [above](#differences-to-karma), Jest uses jsdom instead of a browser for running tests. +This comes with a number of limitations, namely: + +- [No scrolling support](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/browser/Window.js#L623-L625) +- [No element sizes or positions](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/living/nodes/Element-impl.js#L334-L371) +- [No layout engine](https://github.com/jsdom/jsdom/issues/1322) in general + +See also the issue for [support running Jest tests in browsers](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). + ### Debugging Jest tests Running `yarn jest-debug` will run Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index b5fde92a23d..ae40d628717 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -160,6 +160,78 @@ secure note named **gitlab-{ce,ee} Review App's root password**. `review-qa-raise-e-12chm0-migrations.1-nqwtx`. 1. Click on the `Container logs` link. +### Troubleshoot a pending `dns-gitlab-review-app-external-dns` Deployment + +#### Finding the problem + +[In the past](https://gitlab.com/gitlab-org/gitlab-ce/issues/62834), it happened +that the `dns-gitlab-review-app-external-dns` Deployment was in a pending state, +effectively preventing all the Review Apps from getting a DNS record assigned, +making them unreachable via domain name. + +This in turn prevented other components of the Review App to properly start +(e.g. `gitlab-runner`). + +After some digging, we found that new mounts were failing, when being performed +with transient scopes (e.g. pods) of `systemd-mount`: + +``` +MountVolume.SetUp failed for volume "dns-gitlab-review-app-external-dns-token-sj5jm" : mount failed: exit status 1 +Mounting command: systemd-run +Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/06add1c3-87b4-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/dns-gitlab-review-app-external-dns-token-sj5jm --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/06add1c3-87b4-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/dns-gitlab-review-app-external-dns-token-sj5jm +Output: Failed to start transient scope unit: Connection timed out +``` + +This probably happened because the GitLab chart creates 67 resources, leading to +a lot of mount points being created on the underlying GCP node. + +The [underlying issue seems to be a `systemd` bug](https://github.com/kubernetes/kubernetes/issues/57345#issuecomment-359068048) +that was fixed in `systemd` `v237`. Unfortunately, our GCP nodes are currently +using `v232`. + +For the record, the debugging steps to find out this issue were: + +1. Switch kubectl context to review-apps-ce (we recommend using [kubectx](https://kubectx.dev/)) +1. `kubectl get pods | grep dns` +1. `kubectl describe pod <pod name>` & confirm exact error message +1. Web search for exact error message, following rabbit hole to [a relevant kubernetes bug report](https://github.com/kubernetes/kubernetes/issues/57345) +1. Access the node over SSH via the GCP console (**Computer Engine > VM + instances** then click the "SSH" button for the node where the `dns-gitlab-review-app-external-dns` pod runs) +1. In the node: `systemctl --version` => systemd 232 +1. Gather some more information: + - `mount | grep kube | wc -l` => e.g. 290 + - `systemctl list-units --all | grep -i var-lib-kube | wc -l` => e.g. 142 +1. Check how many pods are in a bad state: + - Get all pods running a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME` + - Get all the `Running` pods on a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME | grep Running` + - Get all the pods in a bad state on a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME | grep -v 'Running' | grep -v 'Completed'` + +#### Solving the problem + +To resolve the problem, we needed to (forcibly) drain some nodes: + +1. Try a normal drain on the node where the `dns-gitlab-review-app-external-dns` + pod runs so that Kubernetes automatically move it to another node: `kubectl drain NODE_NAME` +1. If that doesn't work, you can also perform a forcible "drain" the node by removing all pods: `kubectl delete pods --field-selector=spec.nodeName=NODE_NAME` +1. In the node: + - Perform `systemctl daemon-reload` to remove the dead/inactive units + - If that doesn't solve the problem, perform a hard reboot: `sudo systemctl reboot` +1. Uncordon any cordoned nodes: `kubectl uncordon NODE_NAME` + +In parallel, since most Review Apps were in a broken state, we deleted them to +clean up the list of non-`Running` pods. +Following is a command to delete Review Apps based on their last deployment date +(current date was June 6th at the time) with + +``` +helm ls -d | grep "Jun 4" | cut -f1 | xargs helm delete --purge +``` + +#### Mitigation steps taken to avoid this problem in the future + +We've created a new node pool with smaller machines so that it's less likely +that a machine will hit the "too many mount points" problem in the future. + ## Frequently Asked Questions **Isn't it too much to trigger CNG image builds on every test run? This creates @@ -172,11 +244,11 @@ thousands of unused Docker images.** **How big are the Kubernetes clusters (`review-apps-ce` and `review-apps-ee`)?** > The clusters are currently set up with a single pool of preemptible nodes, - with a minimum of 1 node and a maximum of 50 nodes. + with a minimum of 1 node and a maximum of 500 nodes. **What are the machine running on the cluster?** - > We're currently using `n1-standard-16` (16 vCPUs, 60 GB memory) machines. + > We're currently using `n1-standard-1` (1 vCPU, 3.75 GB memory) machines. **How do we secure this from abuse? Apps are open to the world so we need to find a way to limit it to only us.** @@ -185,7 +257,7 @@ find a way to limit it to only us.** ## Other resources -* [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) +- [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) [charts-1068]: https://gitlab.com/charts/gitlab/issues/1068 [gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/44362587 diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index 00ac6d6b650..0c268eff9f1 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -24,6 +24,7 @@ The following are guides to basic GitLab functionality: - [Add an image](add-image.md), to add new images to a project's repository. - [Create an issue](../user/project/issues/create_new_issue.md), to start collaborating within a project. - [Create a merge request](add-merge-request.md), to request changes made in a branch be merged into a project's repository. +- See how these features come together in the [GitLab Flow introduction video](https://youtu.be/InKNIvky2KE) and [GitLab Flow page](../workflow/gitlab_flow.md). ## Git basics diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index 1cf883679d7..b7e6844f43a 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -29,7 +29,9 @@ git clone PASTE HTTPS OR SSH HERE A clone of the project will be created in your computer. ->**Note:** If you clone your project via a URL that contains special characters, make sure that characters are URL-encoded. +NOTE: **Note:** +If you clone your project via a URL that contains special characters, make sure +that characters are URL-encoded. ### Go into a project directory to work in it @@ -86,12 +88,18 @@ cat README.md ### Remove a file +DANGER: **Danger:** +This will permanently delete the file. + ``` rm NAME-OF-FILE ``` ### Remove a directory and all of its contents +DANGER: **Danger:** +This will permanently delete the directory and all of its contents. + ``` rm -r NAME-OF-DIRECTORY ``` @@ -113,9 +121,13 @@ history You will be asked for an administrator’s password. ``` -sudo +sudo COMMAND ``` +CAUTION: **Caution:** +Be careful of the commands you run with `sudo`. Certain commands may cause +damage to your data and system. + ### Show which directory I am in ``` diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md deleted file mode 100644 index cbb3b766b4e..00000000000 --- a/doc/install/database_mysql.md +++ /dev/null @@ -1,319 +0,0 @@ ---- -type: reference ---- - -# Database MySQL - -NOTE: **Note:** -We do not recommend using MySQL due to various issues. -For example, there have been bugs with case -[(in)sensitivity](https://dev.mysql.com/doc/refman/5.7/en/case-sensitivity.html). - -Bugs relating to case sensitivity: - -- <https://bugs.mysql.com/bug.php?id=65830> -- <https://bugs.mysql.com/bug.php?id=50909> -- <https://bugs.mysql.com/bug.php?id=65830> -- <https://bugs.mysql.com/bug.php?id=63164> - -## Initial database setup - -```sh -# Install the database packages -sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev - -# Ensure you have MySQL version 5.6 or later -mysql --version - -# Pick a MySQL root password (can be anything), type it and press enter -# Retype the MySQL root password and press enter - -# Secure your installation -sudo mysql_secure_installation - -# Login to MySQL -mysql -u root -p - -# Type the MySQL root password - -# Create a user for GitLab -# do not type the 'mysql>', this is part of the prompt -# change $password in the command below to a real password you pick -mysql> CREATE USER 'git'@'localhost' IDENTIFIED BY '$password'; - -# Ensure you can use the InnoDB engine which is necessary to support long indexes -# If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off" -mysql> SET storage_engine=INNODB; - -# If you have MySQL < 5.7.7 and want to enable utf8mb4 character set support with your GitLab install, you must set the following NOW: -mysql> SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1; - -# If you use MySQL with replication, or just have MySQL configured with binary logging, you need to run the following to allow the use of `TRIGGER`: -mysql> SET GLOBAL log_bin_trust_function_creators = 1; - -# Create the GitLab production database -mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`; - -# Grant the GitLab user necessary permissions on the database -mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES, TRIGGER ON `gitlabhq_production`.* TO 'git'@'localhost'; - -# Quit the database session -mysql> \q - -# Try connecting to the new database with the new user -sudo -u git -H mysql -u git -p -D gitlabhq_production - -# Type the password you replaced $password with earlier - -# You should now see a 'mysql>' prompt - -# Quit the database session -mysql> \q -``` - -You are done installing the database for now and can go back to the rest of the installation. -Please proceed to the rest of the installation **before** running through the steps below. - -### `log_bin_trust_function_creators` - -If you use MySQL with replication, or just have MySQL configured with binary logging, all of your MySQL servers will need to have `log_bin_trust_function_creators` enabled to allow the use of `TRIGGER` in migrations. You have already set this global variable in the steps above, but to make it persistent, add the following to your `my.cnf` file: - -``` -log_bin_trust_function_creators=1 -``` - -### MySQL utf8mb4 support - -After installation or upgrade, remember to [convert any new tables](#tables-and-data-conversion-to-utf8mb4) to `utf8mb4`/`utf8mb4_general_ci`. - ---- - -GitLab 8.14 has introduced [a feature](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7420) requiring `utf8mb4` encoding to be supported in your GitLab MySQL Database, which is not the case if you have set up your database before GitLab 8.16. - -Follow the below instructions to ensure you use the most up to date requirements for your GitLab MySQL Database. - -**We are about to do the following:** - -- Ensure you can enable `utf8mb4` encoding and `utf8mb4_general_ci` collation for your GitLab DB, tables and data. -- Convert your GitLab tables and data from `utf8`/`utf8_general_ci` to `utf8mb4`/`utf8mb4_general_ci`. - -### Check for utf8mb4 support - -#### Check for InnoDB File-Per-Table Tablespaces - -We need to check, enable and maybe convert your existing GitLab DB tables to the [InnoDB File-Per-Table Tablespaces](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) as a prerequisite for supporting **utfb8mb4 with long indexes** required by recent GitLab databases. - - # Login to MySQL - mysql -u root -p - - # Type the MySQL root password - mysql > use gitlabhq_production; - - # Check your MySQL version is >= 5.5.3 (GitLab requires 5.5.14+) - mysql > SHOW VARIABLES LIKE 'version'; - +---------------+-----------------+ - | Variable_name | Value | - +---------------+-----------------+ - | version | 5.5.53-0+deb8u1 | - +---------------+-----------------+ - - # Note where is your MySQL data dir for later: - mysql > select @@datadir; - +----------------+ - | @@datadir | - +----------------+ - | /var/lib/mysql | - +----------------+ - - # Note whether your MySQL server runs with innodb_file_per_table ON or OFF: - mysql> SELECT @@innodb_file_per_table; - +-------------------------+ - | @@innodb_file_per_table | - +-------------------------+ - | 1 | - +-------------------------+ - - # You can now quit the database session - mysql> \q - -> You need **MySQL 5.5.3 or later** to perform this update. - -Whatever the results of your checks above, we now need to check if your GitLab database has been created using [InnoDB File-Per-Table Tablespaces](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) (i.e. `innodb_file_per_table` was set to **1** at initial setup time). - -NOTE: **Note:** -This setting is [enabled by default](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_file_per_table) since MySQL 5.6.6. - - # Run this command with root privileges, replace the data dir if different: - sudo ls -lh /var/lib/mysql/gitlabhq_production/*.ibd | wc -l - - # Run this command with root privileges, replace the data dir if different: - sudo ls -lh /var/lib/mysql/gitlabhq_production/*.frm | wc -l - -- **Case 1: a result > 0 for both commands** - -Congratulations, your GitLab database uses the right InnoDB tablespace format. - -However, you must still ensure that any **future tables** created by GitLab will still use the right format: - -- If `SELECT @@innodb_file_per_table` returned **1** previously, your server is running correctly. - - > It's however a requirement to check *now* that this setting is indeed persisted in your [`my.cnf`](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) file! - -- If `SELECT @@innodb_file_per_table` returned **0** previously, your server is not running correctly. - - > [Enable innodb_file_per_table](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) by running in a MySQL session as root the command `SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda;` and persist the two settings in your [`my.cnf`](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) file. - -Now, if you have a **different result** returned by the 2 commands above, it means you have a **mix of tables format** uses in your GitLab database. This can happen if your MySQL server had different values for `innodb_file_per_table` in its life and you updated GitLab at different moments with those inconsistent values. So keep reading. - -- **Case 2: a result equals to "0" OR not the same result for both commands** - -Unfortunately, none or only some of your GitLab database tables use the GitLab requirement of [InnoDB File-Per-Table Tablespaces](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html). - -Let's enable what we need on the running server: - - # Login to MySQL - mysql -u root -p - - # Type the MySQL root password - - # Enable innodb_file_per_table and set innodb_file_format on the running server: - mysql > SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda; - - # You can now quit the database session - mysql> \q - -> Now, **persist** [innodb_file_per_table](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) and [innodb_file_format](https://dev.mysql.com/doc/refman/5.7/en/innodb-file-format-enabling.html) in your `my.cnf` file. - -Ensure at this stage that your GitLab instance is indeed **stopped**. - -Now, let's convert all the GitLab database tables to the new tablespace format: - - # Login to MySQL - mysql -u root -p - - # Type the MySQL root password - mysql > use gitlabhq_production; - - # Safety check: you should still have those values set as follows: - mysql> SELECT @@innodb_file_per_table, @@innodb_file_format; - +-------------------------+----------------------+ - | @@innodb_file_per_table | @@innodb_file_format | - +-------------------------+----------------------+ - | 1 | Barracuda | - +-------------------------+----------------------+ - - mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE"; - - # If previous query returned results, copy & run all shown SQL statements - - # You can now quit the database session - mysql> \q - ---- - -#### Check for proper InnoDB File Format, Row Format, Large Prefix and tables conversion - -We need to check, enable and probably convert your existing GitLab DB tables to use the [Barracuda InnoDB file format](https://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html), the [DYNAMIC row format](https://dev.mysql.com/doc/refman/5.7/en/glossary.html#glos_dynamic_row_format) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) as a second prerequisite for supporting **utfb8mb4 with long indexes** used by recent GitLab databases. - - # Login to MySQL - mysql -u root -p - - # Type the MySQL root password - mysql > use gitlabhq_production; - - # Set innodb_file_format and innodb_large_prefix on the running server: - # Note: These are the default settings only for MySQL 5.7.7 and later. - - mysql > SET GLOBAL innodb_file_format=Barracuda, innodb_large_prefix=1; - - # Your DB must be (still) using utf8/utf8_general_ci as default encoding and collation. - # We will NOT change the default encoding and collation on the DB in order to support future GitLab migrations creating tables - # that require "long indexes support" on installations using MySQL <= 5.7.9. - # However, when such migrations occur, you will have to follow this guide again to convert the newly created tables to the proper encoding/collation. - - # This should return the following: - mysql> SELECT @@character_set_database, @@collation_database; - +--------------------------+----------------------+ - | @@character_set_database | @@collation_database | - +--------------------------+----------------------+ - | utf8 | utf8_general_ci | - +--------------------------+----------------------+ - -> Now, ensure that [innodb_file_format](https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) are **persisted** in your `my.cnf` file. - -#### Tables and data conversion to utf8mb4 - -Now that you have a persistent MySQL setup, you can safely upgrade tables after setup or upgrade time: - - # Convert tables not using ROW_FORMAT DYNAMIC: - - mysql> SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ROW_FORMAT=DYNAMIC;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE" AND ROW_FORMAT!="Dynamic"; - - # !! If previous query returned results, copy & run all shown SQL statements - - # Convert tables/columns not using utf8mb4/utf8mb4_general_ci as encoding/collation: - - mysql > SET foreign_key_checks = 0; - - mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_COLLATION != "utf8mb4_general_ci" AND TABLE_TYPE="BASE TABLE"; - - # !! If previous query returned results, copy & run all shown SQL statements - - # Turn foreign key checks back on - mysql > SET foreign_key_checks = 1; - - # You can now quit the database session - mysql> \q - -Ensure your GitLab database configuration file uses a proper connection encoding and collation: - -`sudo -u git -H editor config/database.yml` - - production: - adapter: mysql2 - encoding: utf8mb4 - collation: utf8mb4_general_ci - -[Restart your GitLab instance](../administration/restart_gitlab.md). - -## MySQL strings limits - -After installation or upgrade, remember to run the `add_limits_mysql` Rake task: - -**Omnibus GitLab installations** - -```sh -sudo gitlab-rake add_limits_mysql -``` - -**Installations from source** - -```sh -bundle exec rake add_limits_mysql RAILS_ENV=production -``` - -The `text` type in MySQL has a different size limit than the `text` type in -PostgreSQL. In MySQL `text` columns are limited to ~65kB, whereas in PostgreSQL -`text` columns are limited up to ~1GB! - -The `add_limits_mysql` Rake task converts some important `text` columns in the -GitLab database to `longtext` columns, which can persist values of up to 4GB -(sometimes less if the value contains multibyte characters). - -Details can be found in the [PostgreSQL][postgres-text-type] and -[MySQL](https://dev.mysql.com/doc/refman/5.7/en/string-type-overview.html) manuals. - -[postgres-text-type]: http://www.postgresql.org/docs/9.2/static/datatype-character.html -[ce-38152]: https://gitlab.com/gitlab-org/gitlab-ce/issues/38152 - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/installation.md b/doc/install/installation.md index eb484dde545..4c1021d097f 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -263,7 +263,7 @@ Since GitLab 8.17, GitLab requires the use of Node to compile JavaScript assets, and Yarn to manage JavaScript dependencies. The current minimum requirements for these are: -- `node` >= v8.10.0. +- `node` >= v8.10.0. (We recommend node 12.x as it is faster) - `yarn` >= v1.10.0. In many distros, @@ -271,8 +271,8 @@ the versions provided by the official package repositories are out of date, so we'll need to install through the following commands: ```sh -# install node v8.x -curl --location https://deb.nodesource.com/setup_8.x | sudo bash - +# install node v12.x +curl --location https://deb.nodesource.com/setup_12.x | sudo bash - sudo apt-get install -y nodejs curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - @@ -293,10 +293,9 @@ sudo adduser --disabled-login --gecos 'GitLab' git ## 6. Database -We recommend using a PostgreSQL database. For MySQL, see the [MySQL setup guide](database_mysql.md). - NOTE: **Note:** -Because we need to make use of extensions and concurrent index removal, you need at least PostgreSQL 9.2. +Starting from GitLab 12.1, only PostgreSQL is supported. Because we need to make +use of extensions and concurrent index removal, you need at least PostgreSQL 9.2. 1. Install the database packages: @@ -493,7 +492,7 @@ sudo -u git -H editor config/resque.yml ``` CAUTION: **Caution:** -Make sure to edit both `gitlab.yml` and `unicorn.rb` to match your setup. +Make sure to edit both `gitlab.yml` and `unicorn.rb` to match your setup. If you want to use Puma web server, see [Using Puma](#using-puma) for the additional steps. NOTE: **Note:** @@ -502,13 +501,8 @@ If you want to use HTTPS, see [Using HTTPS](#using-https) for the additional ste ### Configure GitLab DB Settings ```sh -# PostgreSQL only: sudo -u git cp config/database.yml.postgresql config/database.yml -# MySQL only: -sudo -u git cp config/database.yml.mysql config/database.yml - -# PostgreSQL only: # Remove host, username, and password lines from config/database.yml. # Once modified, the `production` settings will be as follows: # @@ -520,7 +514,7 @@ sudo -u git cp config/database.yml.mysql config/database.yml # sudo -u git -H editor config/database.yml -# MySQL and remote PostgreSQL only: +# Remote PostgreSQL only: # Update username/password in config/database.yml. # You only need to adapt the production settings (first part). # If you followed the database guide then please do as follows: @@ -528,7 +522,6 @@ sudo -u git -H editor config/database.yml # You can keep the double quotes around the password sudo -u git -H editor config/database.yml -# PostgreSQL and MySQL: # Make config/database.yml readable to git only sudo -u git -H chmod o-rwx config/database.yml ``` @@ -544,11 +537,7 @@ Make sure you have `bundle` (run `bundle -v`): - `< 2.x`. ```sh -# For PostgreSQL (note, the option says "without ... mysql") sudo -u git -H bundle install --deployment --without development test mysql aws kerberos - -# Or if you use MySQL (note, the option says "without ... postgres") -sudo -u git -H bundle install --deployment --without development test postgres aws kerberos ``` NOTE: **Note:** diff --git a/doc/install/kubernetes/gitlab_chart.md b/doc/install/kubernetes/gitlab_chart.md index 43655767002..d067c341be8 100644 --- a/doc/install/kubernetes/gitlab_chart.md +++ b/doc/install/kubernetes/gitlab_chart.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/ +redirect_to: 'https://docs.gitlab.com/charts/' --- This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index 43655767002..d067c341be8 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/ +redirect_to: 'https://docs.gitlab.com/charts/' --- This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index 08ccf2cf9ad..be58c957166 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/runner/install/kubernetes.html +redirect_to: 'https://docs.gitlab.com/runner/install/kubernetes.html' --- This document was moved to [another location](https://docs.gitlab.com/runner/install/kubernetes.html). diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index 43655767002..d067c341be8 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/ +redirect_to: 'https://docs.gitlab.com/charts/' --- This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/preparation/connect.md b/doc/install/kubernetes/preparation/connect.md index db55e03d3d4..839461c982c 100644 --- a/doc/install/kubernetes/preparation/connect.md +++ b/doc/install/kubernetes/preparation/connect.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/cloud/ +redirect_to: 'https://docs.gitlab.com/charts/installation/cloud/' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/). diff --git a/doc/install/kubernetes/preparation/eks.md b/doc/install/kubernetes/preparation/eks.md index 975d35c11c6..c3f53c2f580 100644 --- a/doc/install/kubernetes/preparation/eks.md +++ b/doc/install/kubernetes/preparation/eks.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/cloud/eks.html +redirect_to: 'https://docs.gitlab.com/charts/installation/cloud/eks.html' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/eks.html). diff --git a/doc/install/kubernetes/preparation/networking.md b/doc/install/kubernetes/preparation/networking.md index 2af16a752dc..7e88bbd3cd1 100644 --- a/doc/install/kubernetes/preparation/networking.md +++ b/doc/install/kubernetes/preparation/networking.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns +redirect_to: 'https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns). diff --git a/doc/install/kubernetes/preparation/rbac.md b/doc/install/kubernetes/preparation/rbac.md index f94e7c24cdc..fc18b91641c 100644 --- a/doc/install/kubernetes/preparation/rbac.md +++ b/doc/install/kubernetes/preparation/rbac.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#rbac +redirect_to: 'https://docs.gitlab.com/charts/installation/deployment.html#rbac' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#rbac). diff --git a/doc/install/kubernetes/preparation/tiller.md b/doc/install/kubernetes/preparation/tiller.md index 66d6c8faece..c1c7910703e 100644 --- a/doc/install/kubernetes/preparation/tiller.md +++ b/doc/install/kubernetes/preparation/tiller.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/tools.html +redirect_to: 'https://docs.gitlab.com/charts/installation/tools.html' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/kubernetes/preparation/tools_installation.md b/doc/install/kubernetes/preparation/tools_installation.md index 66d6c8faece..c1c7910703e 100644 --- a/doc/install/kubernetes/preparation/tools_installation.md +++ b/doc/install/kubernetes/preparation/tools_installation.md @@ -1,5 +1,5 @@ --- -redirect_to: https://docs.gitlab.com/charts/installation/tools.html +redirect_to: 'https://docs.gitlab.com/charts/installation/tools.html' --- This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 18981c43464..45d07ec5d11 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -145,12 +145,12 @@ Login successful. You have access to the following projects and can switch between them with 'oc project <projectname>': - * cockpit - * default (current) - * delete - * openshift - * openshift-infra - * sample +- cockpit +- default (current) +- delete +- openshift +- openshift-infra +- sample Using project "default". ``` diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 96b7d0f3648..b53624a33bf 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -123,7 +123,7 @@ To disable the relative URL: 1. Follow the same as above starting from 2. and set up the GitLab URL to one that doesn't contain a relative path. -[omnibus-rel]: http://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab "How to set up relative URL in Omnibus GitLab" +[omnibus-rel]: https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab "How to set up relative URL in Omnibus GitLab" [restart gitlab]: ../administration/restart_gitlab.md#installations-from-source "How to restart GitLab" <!-- ## Troubleshooting diff --git a/doc/install/requirements.md b/doc/install/requirements.md index ee3d17704a2..68c1bcbc801 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -9,7 +9,7 @@ as the hardware requirements that are needed to install and use GitLab. ## Operating Systems -### Supported Unix distributions +### Supported Linux distributions - Ubuntu - Debian @@ -21,7 +21,7 @@ as the hardware requirements that are needed to install and use GitLab. For the installations options, see [the main installation page](README.md). -### Unsupported Unix distributions +### Unsupported Linux distributions and Unix-like operating systems - Arch Linux - Fedora @@ -29,13 +29,13 @@ For the installations options, see [the main installation page](README.md). - Gentoo - macOS -On the above unsupported distributions is still possible to install GitLab yourself. +Installation of GitLab on these operating systems is possible, but not supported. Please see the [installation from source guide](installation.md) and the [installation guides](https://about.gitlab.com/installation/) for more information. -### Non-Unix operating systems such as Windows +### Microsoft Windows -GitLab is developed for Unix operating systems. -It does **not** run on Windows, and we have no plans to support it in the near future. For the latest development status view this [issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46567). +GitLab is developed for Linux-based operating systems. +It does **not** run on Microsoft Windows, and we have no plans to support it in the near future. For the latest development status view this [issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46567). Please consider using a virtual machine to run GitLab. ## Ruby versions @@ -104,32 +104,10 @@ installation (e.g. the number of users, projects, etc). We currently support the following databases: -- PostgreSQL (highly recommended) -- MySQL/MariaDB (strongly discouraged, not all GitLab features are supported, no support for [MySQL/MariaDB GTID](https://mariadb.com/kb/en/mariadb/gtid/)) - -We highly recommend the use of PostgreSQL instead of MySQL/MariaDB as not all -features of GitLab work with MySQL/MariaDB: - -1. MySQL support for subgroups was [dropped with GitLab 9.3][post]. - See [issue #30472][30472] for more information. -1. Geo does [not support MySQL](../administration/geo/replication/database.md). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** -1. [Zero downtime migrations](../update/README.md#upgrading-without-downtime) do not work with MySQL. -1. [Database load balancing](../administration/database_load_balancing.md) is - supported only for PostgreSQL. **[PREMIUM ONLY]** -1. GitLab [optimizes the loading of dashboard events](https://gitlab.com/gitlab-org/gitlab-ce/issues/31806) using [PostgreSQL LATERAL JOINs](https://blog.heapanalytics.com/postgresqls-powerful-new-join-type-lateral/). -1. In general, SQL optimized for PostgreSQL may run much slower in MySQL due to - differences in query planners. For example, subqueries that work well in PostgreSQL - may not be [performant in MySQL](https://dev.mysql.com/doc/refman/5.7/en/optimizing-subqueries.html). -1. Binary column index length is limited to 20 bytes. This is accomplished with [a hack](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/initializers/mysql_set_length_for_binary_indexes.rb). -1. MySQL requires a variety of hacks to increase limits on various columns, [for example](https://gitlab.com/gitlab-org/gitlab-ce/issues/49583). -1. [The milestone filter runs slower queries on MySQL](https://gitlab.com/gitlab-org/gitlab-ce/issues/51173#note_99391731). -1. We expect this list to grow over time. - -Existing users using GitLab with MySQL/MariaDB are advised to -[migrate to PostgreSQL](../update/mysql_to_postgresql.md) instead. - -[30472]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30472 -[post]: https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#dropping-support-for-subgroups-in-mysql +- PostgreSQL + +Support for MySQL was removed in GitLab 12.1. Existing users using GitLab with +MySQL/MariaDB are advised to [migrate to PostgreSQL](../update/mysql_to_postgresql.md) before upgrading. ### PostgreSQL Requirements diff --git a/doc/integration/README.md b/doc/integration/README.md index 1fea6a32c28..135952a1b08 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -13,10 +13,10 @@ See the documentation below for details on how to configure these services. - [Auth0 OmniAuth](auth0.md) Enable the Auth0 OmniAuth provider - [Bitbucket](bitbucket.md) Import projects from Bitbucket.org and login to your GitLab instance with your Bitbucket.org account - [CAS](cas.md) Configure GitLab to sign in using CAS -- [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. +- [External issue tracker](external-issue-tracker.md) Redmine, Jira, etc. - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages - [Jenkins](jenkins.md) Integrate with the Jenkins CI -- [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Jira](../user/project/integrations/jira.md) Integrate with the Jira issue tracker - [Kerberos](kerberos.md) Integrate with Kerberos - [LDAP](ldap.md) Set up sign in via LDAP - [OAuth2 provider](oauth_provider.md) OAuth2 application creation @@ -55,7 +55,7 @@ at Super User also has relevant information. **Omnibus Trusted Chain** -[Install the self signed certificate or custom certificate authorities](http://docs.gitlab.com/omnibus/common_installation_problems/README.html#using-self-signed-certificate-or-custom-certificate-authorities) +[Install the self signed certificate or custom certificate authorities](https://docs.gitlab.com/omnibus/common_installation_problems/README.html#using-self-signed-certificate-or-custom-certificate-authorities) in to GitLab Omnibus. It is enough to concatenate the certificate to the main trusted certificate @@ -71,4 +71,4 @@ After that restart GitLab with: sudo gitlab-ctl restart ``` -[jenkins]: http://docs.gitlab.com/ee/integration/jenkins.html +[jenkins]: https://docs.gitlab.com/ee/integration/jenkins.html diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 6064c417900..f64fdb1e28b 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -130,7 +130,7 @@ The following Elasticsearch settings are available: | `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). | | `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). | | `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | -| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., `http://host1, https://host2:9200`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | | `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) | | `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. | | `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects). @@ -156,7 +156,7 @@ If no namespaces or projects are selected, no Elasticsearch indexing will take p CAUTION: **Warning**: If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data for filtering to work correctly. To do this run the rake tasks `gitlab:elastic:create_empty_index` and -`gitlab:elastic:clear_index_status` Afterwards, removing a namespace or a projeect from the list will delete the data +`gitlab:elastic:clear_index_status`. Afterwards, removing a namespace or a project from the list will delete the data from the Elasticsearch index as expected. ## Disabling Elasticsearch @@ -275,19 +275,18 @@ You can also use the `gitlab:elastic:clear_index_status` Rake task to force the indexer to "forget" all progress, so retrying the indexing process from the start. -To index all wikis: +The `index_projects` command enqueues jobs to index all project and wiki +repositories, and most database content. However, snippets still need to be +indexed separately. To do so, run one of these commands: ```sh # Omnibus installations -sudo gitlab-rake gitlab:elastic:index_wikis +sudo gitlab-rake gitlab:elastic:index_snippets # Installations from source -bundle exec rake gitlab:elastic:index_wikis RAILS_ENV=production +bundle exec rake gitlab:elastic:index_snippets RAILS_ENV=production ``` -The wiki indexer also supports the `ID_FROM` and `ID_TO` parameters if you want -to limit a project set. - Enable replication and refreshing again after indexing (only if you previously disabled it): ```bash @@ -305,7 +304,7 @@ For Elasticsearch 6.x, before proceeding with the force merge, the index should ```bash curl --request PUT localhost:9200/gitlab-production/_settings --data '{ "settings": { - "index.blocks.write": true + "index.blocks.write": true } }' ``` @@ -320,7 +319,7 @@ After this, if your index is in read-only, switch back to read-write: ```bash curl --request PUT localhost:9200/gitlab-production/_settings --data '{ "settings": { - "index.blocks.write": false + "index.blocks.write": false } }' ``` @@ -335,14 +334,11 @@ There are several rake tasks available to you via the command line: - `sudo gitlab-rake gitlab:elastic:create_empty_index` - `sudo gitlab-rake gitlab:elastic:clear_index_status` - `sudo gitlab-rake gitlab:elastic:index_projects` - - `sudo gitlab-rake gitlab:elastic:index_wikis` - `sudo gitlab-rake gitlab:elastic:index_snippets` - [sudo gitlab-rake gitlab:elastic:index_projects](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This iterates over all projects and queues sidekiq jobs to index them in the background. - [sudo gitlab-rake gitlab:elastic:index_projects_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. -- [sudo gitlab-rake gitlab:elastic:index_wikis](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - - Iterates over every project, determines if said project contains wiki data, and then indexes the blobs (content) of said wiki data. - [sudo gitlab-rake gitlab:elastic:create_empty_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This generates an empty index on the Elasticsearch side. - [sudo gitlab-rake gitlab:elastic:clear_index_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) @@ -396,9 +392,9 @@ When performing a search, the GitLab index will use the following scopes: Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index. -In general, we recommend simply letting Elasticseach merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ +In general, we recommend simply letting Elasticsearch merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ -However, some larger installations may wish to tune the merge policy settings: +However, some larger installations may wish to tune the merge policy settings: - Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently. @@ -429,14 +425,14 @@ Here are some common pitfalls and how to overcome them: - **How can I verify my GitLab instance is using Elasticsearch?** The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: - + ```ruby u = User.find_by_username('your-username') s = SearchService.new(u, {:search => 'search_term'}) pp s.search_objects.class.name ``` - - If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. + + If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. - **I updated GitLab and now I can't find anything** @@ -447,23 +443,23 @@ Here are some common pitfalls and how to overcome them: - **I indexed all the repositories but I can't find anything** Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). - + Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. - + If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): - + ```ruby u = User.find_by_username('your-username') s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’}) pp s.search_objects.to_a ``` - + See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data. -- **I indexed all the repositories but then switched elastic search servers and now I can't find anything** +- **I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything** You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. - + - **The indexing process is taking a very long time** The more data present in your GitLab instance, the longer the indexing process takes. diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index afe1268555a..e6496ae3a2e 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -19,28 +19,28 @@ GitLab's Jenkins integration allows you to trigger a Jenkins build when you push code to a repository, or when a merge request is created. Additionally, it shows the pipeline status on merge requests widgets and on the project's home page. -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For a video overview, see [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y). +Videos are also available on [GitLab workflow with Jira issues and Jenkins pipelines](https://youtu.be/Jn-_fyra7xQ) +and [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF5Y). ## Use cases - Suppose you are new to GitLab, and want to keep using Jenkins until you prepare -your projects to build with [GitLab CI/CD](../ci/README.md). You set up the -integration between GitLab and Jenkins, then you migrate to GitLab CI later. While -you organize yourself and your team to onboard GitLab, you keep your pipelines -running with Jenkins, but view the results in your project's repository in GitLab. + your projects to build with [GitLab CI/CD](../ci/README.md). You set up the + integration between GitLab and Jenkins, then you migrate to GitLab CI later. While + you organize yourself and your team to onboard GitLab, you keep your pipelines + running with Jenkins, but view the results in your project's repository in GitLab. - Your team uses [Jenkins Plugins](https://plugins.jenkins.io/) for other proceedings, -therefore, you opt for keep using Jenkins to build your apps. Show the results of your -pipelines directly in GitLab. + therefore, you opt for keep using Jenkins to build your apps. Show the results of your + pipelines directly in GitLab. For a real use case, read the blog post [Continuous integration: From Jenkins to GitLab using Docker](https://about.gitlab.com/2017/07/27/docker-my-precious/). ## Requirements -* [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) -* [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) -* Git clone access for Jenkins from the GitLab repository -* GitLab API access to report build status +- [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +- [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) +- Git clone access for Jenkins from the GitLab repository +- GitLab API access to report build status ## Configure GitLab users @@ -65,7 +65,7 @@ Go to Manage Jenkins -> Configure System and scroll down to the 'GitLab' section Enter the GitLab server URL in the 'GitLab host URL' field and paste the API token copied earlier in the 'API Token' field. -For more information, see GitLab Plugin documentation about +For more information, see GitLab Plugin documentation about [Jenkins-to-GitLab authentication](https://github.com/jenkinsci/gitlab-plugin#jenkins-to-gitlab-authentication)  @@ -76,8 +76,8 @@ Follow the GitLab Plugin documentation about [Jenkins Job Configuration](https:/ NOTE: **Note:** Be sure to include the steps about [Build status configuration](https://github.com/jenkinsci/gitlab-plugin#build-status-configuration). -The 'Publish build status to GitLab' post-build step is required to view -Jenkins build status in GitLab Merge Requests. +The 'Publish build status to GitLab' post-build step is required to view +Jenkins build status in GitLab Merge Requests. ## Configure a GitLab project @@ -114,21 +114,21 @@ and storing build status for Commits and Merge Requests. All steps are implemented using AJAX requests on the merge request page. 1. In order to display the build status in a merge request you must create a project service in GitLab. -2. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. -3. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. -4. The response is parsed to give a response in GitLab (success/failed/pending). +1. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. +1. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. +1. The response is parsed to give a response in GitLab (success/failed/pending). ## Troubleshooting ### Error in merge requests - "Could not connect to the CI server" This integration relies on Jenkins reporting the build status back to GitLab via -the [Commit Status API](../api/commits.md#commit-status). +the [Commit Status API](../api/commits.md#commit-status). The error 'Could not connect to the CI server' usually means that GitLab did not receive a build status update via the API. Either Jenkins was not properly -configured or there was an error reporting the status via the API. +configured or there was an error reporting the status via the API. 1. [Configure the Jenkins server](#configure-the-jenkins-server) for GitLab API access -2. [Configure a Jenkins project](#configure-a-jenkins-project), including the - 'Publish build status to GitLab' post-build action. +1. [Configure a Jenkins project](#configure-a-jenkins-project), including the + 'Publish build status to GitLab' post-build action. diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md index 8001c5dbd83..eae705c9637 100644 --- a/doc/integration/jenkins_deprecated.md +++ b/doc/integration/jenkins_deprecated.md @@ -8,13 +8,13 @@ Please use documentation for the new [Jenkins CI service](jenkins.md). Integration includes: -* Trigger Jenkins build after push to repo -* Show build status on Merge Request page +- Trigger Jenkins build after push to repo +- Show build status on Merge Request page Requirements: -* [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) -* git clone access for Jenkins from GitLab repo (via ssh key) +- [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +- git clone access for Jenkins from GitLab repo (via ssh key) ## Jenkins diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index a13e9f73f48..bf5debc7694 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -194,11 +194,7 @@ from the Omniauth provider's documentation. gem "omniauth-your-auth-provider" -- If you're using MySQL, install the new Omniauth provider gem by running the following command: - - sudo -u git -H bundle install --without development test postgres --path vendor/bundle --no-deployment - -- If you're using PostgreSQL, install the new Omniauth provider gem by running the following command: +- Install the new Omniauth provider gem by running the following command: sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment diff --git a/doc/integration/ultra_auth.md b/doc/integration/ultra_auth.md index 139cca456aa..69b2a75050d 100644 --- a/doc/integration/ultra_auth.md +++ b/doc/integration/ultra_auth.md @@ -71,8 +71,8 @@ To get the credentials (a pair of Client ID and Client Secret), you must registe 1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. -On the sign in page, there should now be a UltraAuth icon below the regular sign in form. +On the sign in page, there should now be an UltraAuth icon below the regular sign in form. Click the icon to begin the authentication process. UltraAuth will ask the user to sign in and authorize the GitLab application. If everything goes well, the user will be returned to GitLab and will be signed in. -**Note:** GitLab requires the email address of each new user. Once the user is logged in using UltraAuth, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email. +GitLab requires the email address of each new user. Once the user is logged in using UltraAuth, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email. Password authentication will be disabled for UltraAuth users and two-factor authentication (2FA) will be enforced. diff --git a/doc/intro/README.md b/doc/intro/README.md index 9bc7c3d9ea4..d9c733d4285 100644 --- a/doc/intro/README.md +++ b/doc/intro/README.md @@ -43,4 +43,4 @@ Install and update your GitLab installation. - [Install GitLab](https://about.gitlab.com/installation/) - [Update GitLab](https://about.gitlab.com/update/) -- [Explore Omnibus GitLab configuration options](http://docs.gitlab.com/omnibus/settings/configuration.html) +- [Explore Omnibus GitLab configuration options](https://docs.gitlab.com/omnibus/settings/configuration.html) diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index b2d626a0a74..2142f5a5f69 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -26,11 +26,11 @@ Every push rule could have its own use case, but let's consider some examples. Let's assume you have the following requirements for your workflow: -- every commit should reference a JIRA issue, for example: `Refactored css. Fixes JIRA-123.` +- every commit should reference a Jira issue, for example: `Refactored css. Fixes JIRA-123.` - users should not be able to remove git tags with `git push` All you need to do is write a simple regular expression that requires the mention -of a JIRA issue in the commit message, like `JIRA\-\d+`. +of a Jira issue in the commit message, like `JIRA\-\d+`. Now when a user tries to push a commit with a message `Bugfix`, their push will be declined. Only pushing commits with messages like `Bugfix according to JIRA-123` diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 764916ca82d..092b4375208 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -18,16 +18,16 @@ installed on your system. If you installed GitLab: -- Using the Omnibus package, you're all set. -- From source, make sure `rsync` is installed: +- Using the Omnibus package, you're all set. +- From source, make sure `rsync` is installed: - ```sh - # Debian/Ubuntu - sudo apt-get install rsync + ```sh + # Debian/Ubuntu + sudo apt-get install rsync - # RHEL/CentOS - sudo yum install rsync - ``` + # RHEL/CentOS + sudo yum install rsync + ``` ### Tar @@ -269,17 +269,17 @@ For Omnibus GitLab packages: 1. Add the following to `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-west-1', - 'aws_access_key_id' => 'AKIAKIAKI', - 'aws_secret_access_key' => 'secret123' - # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key - # 'use_iam_profile' => true - } - gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' - ``` + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' + # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key + # 'use_iam_profile' => true + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` 1. [Reconfigure GitLab] for the changes to take effect @@ -289,16 +289,16 @@ This example can be used for a bucket in Amsterdam (AMS3). 1. Add the following to `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'ams3', - 'aws_access_key_id' => 'AKIAKIAKI', - 'aws_secret_access_key' => 'secret123', - 'endpoint' => 'https://ams3.digitaloceanspaces.com' - } - gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' - ``` + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'ams3', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123', + 'endpoint' => 'https://ams3.digitaloceanspaces.com' + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` 1. [Reconfigure GitLab] for the changes to take effect @@ -321,31 +321,31 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - # snip - upload: - # Fog storage connection settings, see http://fog.io/storage/ . - connection: - provider: AWS - region: eu-west-1 - aws_access_key_id: AKIAKIAKI - aws_secret_access_key: 'secret123' - # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty - # ie. aws_access_key_id: '' - # use_iam_profile: 'true' - # The remote 'directory' to store your backups. For S3, this would be the bucket name. - remote_directory: 'my.s3.bucket' - # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional - # encryption: 'AES256' - # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional - # This should be set to the base64-encoded encryption key for Amazon S3 to use to encrypt or decrypt your data. - # 'encryption' must also be set in order for this to have any effect. - # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. - # encryption_key: '<base64 key>' - # Specifies Amazon S3 storage class to use for backups, this is optional - # storage_class: 'STANDARD' - ``` + ```yaml + backup: + # snip + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: AWS + region: eu-west-1 + aws_access_key_id: AKIAKIAKI + aws_secret_access_key: 'secret123' + # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty + # ie. aws_access_key_id: '' + # use_iam_profile: 'true' + # The remote 'directory' to store your backups. For S3, this would be the bucket name. + remote_directory: 'my.s3.bucket' + # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional + # encryption: 'AES256' + # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional + # This should be set to the base64-encoded encryption key for Amazon S3 to use to encrypt or decrypt your data. + # 'encryption' must also be set in order for this to have any effect. + # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. + # encryption_key: '<base64 key>' + # Specifies Amazon S3 storage class to use for backups, this is optional + # storage_class: 'STANDARD' + ``` 1. [Restart GitLab] for the changes to take effect @@ -417,14 +417,14 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'Google', - 'google_storage_access_key_id' => 'Access Key', - 'google_storage_secret_access_key' => 'Secret' - } - gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' - ``` + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_storage_access_key_id' => 'Access Key', + 'google_storage_secret_access_key' => 'Secret' + } + gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' + ``` 1. [Reconfigure GitLab] for the changes to take effect @@ -434,15 +434,15 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - upload: - connection: - provider: 'Google' - google_storage_access_key_id: 'Access Key' - google_storage_secret_access_key: 'Secret' - remote_directory: 'my.google.bucket' - ``` + ```yaml + backup: + upload: + connection: + provider: 'Google' + google_storage_access_key_id: 'Access Key' + google_storage_secret_access_key: 'Secret' + remote_directory: 'my.google.bucket' + ``` 1. [Restart GitLab] for the changes to take effect @@ -477,16 +477,16 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_upload_connection'] = { - :provider => 'Local', - :local_root => '/mnt/backups' - } + ```ruby + gitlab_rails['backup_upload_connection'] = { + :provider => 'Local', + :local_root => '/mnt/backups' + } - # The directory inside the mounted folder to copy backups to - # Use '.' to store them in the root directory - gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' - ``` + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -496,17 +496,17 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - upload: - # Fog storage connection settings, see http://fog.io/storage/ . - connection: - provider: Local - local_root: '/mnt/backups' - # The directory inside the mounted folder to copy backups to - # Use '.' to store them in the root directory - remote_directory: 'gitlab_backups' - ``` + ```yaml + backup: + upload: + # Fog storage connection settings, see http://fog.io/storage/ . + connection: + provider: Local + local_root: '/mnt/backups' + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + remote_directory: 'gitlab_backups' + ``` 1. [Restart GitLab] for the changes to take effect. @@ -521,9 +521,9 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable - ``` + ```ruby + gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -533,10 +533,10 @@ For installations from source: 1. Edit `/home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - archive_permissions: 0644 # Makes the backup archives world-readable - ``` + ```yaml + backup: + archive_permissions: 0644 # Makes the backup archives world-readable + ``` 1. [Restart GitLab] for the changes to take effect. @@ -550,10 +550,10 @@ For Omnibus GitLab packages: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - ## Limit backup lifetime to 7 days - 604800 seconds - gitlab_rails['backup_keep_time'] = 604800 - ``` + ```ruby + ## Limit backup lifetime to 7 days - 604800 seconds + gitlab_rails['backup_keep_time'] = 604800 + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -586,11 +586,11 @@ For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: - ```yaml - backup: - ## Limit backup lifetime to 7 days - 604800 seconds - keep_time: 604800 - ``` + ```yaml + backup: + ## Limit backup lifetime to 7 days - 604800 seconds + keep_time: 604800 + ``` 1. [Restart GitLab] for the changes to take effect. @@ -840,13 +840,13 @@ columns containing sensitive information. If the key is lost, GitLab will be unable to decrypt those columns. This will break a wide range of functionality, including (but not restricted to): -* [CI/CD variables](../ci/variables/README.md) -* [Kubernetes / GCP integration](../user/project/clusters/index.md) -* [Custom Pages domains](../user/project/pages/getting_started_part_three.md) -* [Project error tracking](../user/project/operations/error_tracking.md) -* [Runner authentication](../ci/runners/README.md) -* [Project mirroring](../workflow/repository_mirroring.md) -* [Web hooks](../user/project/integrations/webhooks.md) +- [CI/CD variables](../ci/variables/README.md) +- [Kubernetes / GCP integration](../user/project/clusters/index.md) +- [Custom Pages domains](../user/project/pages/getting_started_part_three.md) +- [Project error tracking](../user/project/operations/error_tracking.md) +- [Runner authentication](../ci/runners/README.md) +- [Project mirroring](../workflow/repository_mirroring.md) +- [Web hooks](../user/project/integrations/webhooks.md) In cases like CI/CD variables and Runner authentication, you might experience some unexpected behavior such as: @@ -865,76 +865,106 @@ backup beforehand. #### Reset CI/CD variables -1. Enter the DB console: +1. Enter the DB console: - For Omnibus GitLab packages: + For Omnibus GitLab packages: - ```sh - sudo gitlab-rails dbconsole - ``` + ```sh + sudo gitlab-rails dbconsole + ``` - For installations from source: + For installations from source: - ```sh - sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production - ``` + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` -1. Check the `ci_group_variables` and `ci_variables` tables: +1. Check the `ci_group_variables` and `ci_variables` tables: - ```sql - SELECT * FROM public."ci_group_variables"; - SELECT * FROM public."ci_variables"; - ``` + ```sql + SELECT * FROM public."ci_group_variables"; + SELECT * FROM public."ci_variables"; + ``` - Those are the variables that you need to delete. + Those are the variables that you need to delete. -1. Drop the table: +1. Drop the table: - ```sql - DELETE FROM ci_group_variables; - DELETE FROM ci_variables; - ``` + ```sql + DELETE FROM ci_group_variables; + DELETE FROM ci_variables; + ``` 1. You may need to reconfigure or restart GitLab for the changes to take effect. - #### Reset Runner registration tokens -1. Enter the DB console: +1. Enter the DB console: - For Omnibus GitLab packages: + For Omnibus GitLab packages: - ```sh - sudo gitlab-rails dbconsole - ``` + ```sh + sudo gitlab-rails dbconsole + ``` - For installations from source: + For installations from source: - ```sh - sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production - ``` + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` 1. Clear all the tokens for projects, groups, and the whole instance: - CAUTION: **Caution:** - The last UPDATE operation will stop the runners being able to pick up - new jobs. You must register new runners. - - ```sql - -- Clear project tokens - UPDATE projects SET runners_token = null, runners_token_encrypted = null; - -- Clear group tokens - UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; - -- Clear instance tokens - UPDATE application_settings SET runners_registration_token_encrypted = null; - -- Clear runner tokens - UPDATE ci_runners SET token = null, token_encrypted = null; - ``` + CAUTION: **Caution:** + The last UPDATE operation will stop the runners being able to pick up + new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted, GitLab can be brought back into working order, and the lost data can be manually replaced. +### Container Registry push failures after restoring from a backup + +If you use the [Container Registry](../user/project/container_registry.md), you +may see pushes to the registry fail after restoring your backup on an Omnibus +GitLab instance after restoring the registry data. + +These failures will mention permission issues in the registry logs, like: + +``` +level=error +msg="response completed with error" +err.code=unknown +err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/...: permission denied" +err.message="unknown error" +``` + +This is caused by the restore being run as the unprivileged user `git` which was +unable to assign the correct ownership to the registry files during the restore +([issue 62759](https://gitlab.com/gitlab-org/gitlab-ce/issues/62759 "Incorrect permissions on registry filesystem after restore")). + +To get your registry working again: + +```bash +sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker +``` + +NOTE: **Note:** +If you have changed the default filesystem location for the registry, you will +want to run the chown against your custom location instead of +`/var/opt/gitlab/gitlab-rails/shared/registry/docker`. + [reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure [restart GitLab]: ../administration/restart_gitlab.md#installations-from-source diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index f5c788af578..f880f31c39e 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -92,3 +92,48 @@ I, [2018-08-02T10:26:47.598424 #45087] INFO -- : Looking for orphaned remote up I, [2018-08-02T10:26:47.753131 #45087] INFO -- : Moved to lost and found: @hashed/6b/DSC_6152.JPG -> lost_and_found/@hashed/6b/DSC_6152.JPG I, [2018-08-02T10:26:47.764356 #45087] INFO -- : Moved to lost and found: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg -> lost_and_found/@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg ``` + +## Remove orphan artifact files + +When you notice there are more job artifacts files on disk than there +should be, you can run: + +```shell +gitlab-rake gitlab:cleanup:orphan_job_artifact_files +``` + +This command: + +- Scans through the entire artifacts folder. +- Checks which files still have a record in the database. +- If no database record is found, the file is deleted from disk. + +By default, this task does not delete anything but shows what it can +delete. Run the command with `DRY_RUN=false` if you actually want to +delete the files: + +```shell +gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false +``` + +You can also limit the number of files to delete with `LIMIT`: + +```shell +gitlab-rake gitlab:cleanup:orphan_job_artifact_files LIMIT=100` +``` + +This will only delete up to 100 files from disk. You can use this to +delete a small set for testing purposes. + +If you provide `DEBUG=1`, you'll see the full path of every file that +is detected as being an orphan. + +If `ionice` is installed, the tasks uses it to ensure the command is +not causing too much load on the disk. You can configure the niceness +level with `NICENESS`. Below are the valid levels, but consult +`man 1 ionice` to be sure. + +- `0` or `None` +- `1` or `Realtime` +- `2` or `Best-effort` (default) +- `3` or `Idle` diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index df3dab118b2..2c6ae0749dd 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -2,42 +2,54 @@ ## Add a webhook for **ALL** projects: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" - # source installations - bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" +# source installations +bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production +``` ## Add a webhook for projects in a given **NAMESPACE**: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme - # source installations - bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme +# source installations +bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +``` ## Remove a webhook from **ALL** projects using: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" - # source installations - bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" +# source installations +bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production +``` ## Remove a webhook from projects in a given **NAMESPACE**: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme - # source installations - bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme +# source installations +bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production +``` ## List **ALL** webhooks: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:list - # source installations - bundle exec rake gitlab:web_hook:list RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:list +# source installations +bundle exec rake gitlab:web_hook:list RAILS_ENV=production +``` ## List the webhooks from projects in a given **NAMESPACE**: - # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme - # source installations - bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production +```sh +# omnibus-gitlab +sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme +# source installations +bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production +``` diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index fa4b0d1fb09..8695b5d2194 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -53,8 +53,9 @@ For more information on how to use these options check out The following settings can be configured: - `enabled`: By default this is set to `false`. Set this to `true` to enable Rack Attack. -- `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a ruby array. - For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3"]`. +- `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a Ruby array. + CIDR notation is supported in GitLab v12.1 and up. + For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`. - `maxretry`: The maximum amount of times a request can be made in the specified time. - `findtime`: The maximum amount of time that failed requests can count against an IP diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index ad5daef805a..49dadd5abc2 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -39,8 +39,26 @@ If you want to enforce 2FA only for certain groups, you can: To change this setting, you need to be administrator or owner of the group. -If there are multiple 2FA requirements (i.e. group + all users, or multiple -groups) the shortest grace period will be used. +> [From](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24965) GitLab 12.0, 2FA settings for a group are also applied to subgroups. + +If you want to enforce 2FA only for certain groups, you can enable it in the +group settings and specify a grace period as above. To change this setting you +need to be administrator or owner of the group. + +The following are important notes about 2FA: + +- Projects belonging to a 2FA-enabled group that + [is shared](../user/project/members/share_project_with_groups.md) + with a 2FA-disabled group will *not* require members of the 2FA-disabled group to use + 2FA for the project. For example, if project *P* belongs to 2FA-enabled group *A* and + is shared with 2FA-disabled group *B*, members of group *B* can access project *P* + without 2FA. To ensure this scenario doesn't occur, + [prevent sharing of projects](../user/group/index.md#share-with-group-lock) + for the 2FA-enabled group. +- If you add additional members to a project within a group or subgroup that has + 2FA enabled, 2FA is **not** required for those individually added members. +- If there are multiple 2FA requirements (for example, group + all users, or multiple + groups) the shortest grace period will be used. ## Disabling 2FA for everyone diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 09b5fbd9260..dbd9bcee935 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -64,9 +64,14 @@ Following [best practices](https://linux-audit.com/using-ed25519-openssh-keys-in you should always favor [ED25519](https://ed25519.cr.yp.to/) SSH keys, since they are more secure and have better performance over the other types. -They were introduced in OpenSSH 6.5, so any modern OS should include the -option to create them. If for any reason your OS or the GitLab instance you -interact with doesn't support this, you can fallback to RSA. +ED25519 SSH keys were introduced in OpenSSH 6.5, +so any modern OS should include the option to create them. +If for any reason your OS or the GitLab instance you interact with doesn't +support ED25519, you can fallback to RSA. + +NOTE: **Note:** +Omnibus does not ship with OpenSSH, so it uses the version on your GitLab server. If using +Omnibus, ensure the version of OpenSSH installed is version 6.5 or newer if you want to use ED25519 SSH keys. ### RSA SSH keys diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 37051f6b10f..745a2253e84 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -14,7 +14,7 @@ Learn how GitLab helps you in the stages of the DevOps lifecycle by learning mor ### Self-managed: Install GitLab -Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. +Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. ### GitLab.com: Create a user and group @@ -74,11 +74,11 @@ Please note that you need to be a group owner to associate a group to your subsc To see the status of your GitLab.com subscription, you can click on the Billings section of the relevant namespace: -* For individuals, this is located at https://gitlab.com/profile/billings under -in your Settings, -* For groups, this is located under the group's Settings dropdown, under Billing. +- For individuals, this is located at <https://gitlab.com/profile/billings> under + in your Settings, +- For groups, this is located under the group's Settings dropdown, under Billing. -For groups, you can see details of your subscription - including your current +For groups, you can see details of your subscription - including your current plan - in the included table:  @@ -86,11 +86,11 @@ plan - in the included table: | Field | Description | | ------ | ------ | | Seats in subscription | If this is a paid plan, this represents the number of seats you've paid to support in your group. | -| Seats currently in use | The number of active seats currently in use. | -| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | -| Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | -| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | -| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | +| Seats currently in use | The number of active seats currently in use. | +| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | +| Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | +| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | +| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | ### Subscription changes and your data diff --git a/doc/topics/application_development_platform/index.md b/doc/topics/application_development_platform/index.md new file mode 100644 index 00000000000..8742606479d --- /dev/null +++ b/doc/topics/application_development_platform/index.md @@ -0,0 +1,62 @@ +# Application Development Platform + +The GitLab Application Development Platform refers to the set of GitLab features used to create, configure, and manage +a complete software development environment. It provides development, operations, and security teams with a robust feature set aimed at supporting best practices out of the box. + +## Overview + +The GitLab Application Development Platform aims to: + +- Reduce and even eliminate the time it takes for an Operations team + to provide a full environment for software developers. +- Get developers up and running fast so they can focus on writing + great applications with a robust development feature set. +- Provide best-of-breed security features so that applications developed + with GitLab are not affected by vulnerabilities that may lead to security + problems and unintended use. + +It is comprised of the following high-level elements: + +1. Compute +1. Build, test, and deploy a wide range of applications +1. Security +1. Observability + +We believe the use of these common building blocks equate to big gains for teams of all sizes, resulting from the adoption +of newer, more efficient, more profitable, and less error-prone techniques for shipping software applications. + +### Compute + +Because at GitLab we are [cloud-native first](https://about.gitlab.com/handbook/product/#cloud-native-first) our +Application Development Platform initially focuses on providing robust support for Kubernetes, with other platforms +to follow. Teams can bring their own clusters and we will additionally make it easy to create new infrastructure +with various cloud providers. + +### Build, test, deploy + +In order to provide modern DevOps workflows, our Application Development Platform will rely on +[Auto DevOps](https://docs.gitlab.com/ee/topics/autodevops/) to provide those workflows. Auto DevOps works with +any Kubernetes cluster; you're not limited to running on GitLab's infrastructure. Additionally, Auto DevOps offers +an incremental consumption path. Because it is [composable](https://docs.gitlab.com/ee/topics/autodevops/#using-components-of-auto-devops), +you can use as much or as little of the default pipeline as you'd like, and deeply customize without having to integrate a completely different platform. + +### Security + +The Application Development Platform helps you ensure that the applications you create are not affected by vulnerabilities +that may lead to security problems and unintended use. This can be achieved by making use of the embedded security features of Auto DevOps, +which inform security teams and developers if there is something to consider changing in their apps +before it is too late to create a preventative fix. The following features are included: + +- [Auto SAST (Static Application Security Testing)](https://docs.gitlab.com/ee/topics/autodevops/#auto-sast-ultimate) +- [Auto Dependency Scanning](https://docs.gitlab.com/ee/topics/autodevops/#auto-dependency-scanning-ultimate) +- [Auto Container Scanning](https://docs.gitlab.com/ee/topics/autodevops/#auto-container-scanning-ultimate) +- [Auto DAST (Dynamic Application Security Testing)](https://docs.gitlab.com/ee/topics/autodevops/#auto-dast-ultimate) + +### Observability + +Performance is a critical aspect of the user experience, and ensuring your application is responsive and available is everyone's +responsibility. The Application Development Platform integrates key performance analytics and feedback +into GitLab, automatically. The following features are included: + +- [Auto Monitoring](https://docs.gitlab.com/ee/topics/autodevops/#auto-monitoring) +- [In-app Kubernetes Pod Logs](https://docs.gitlab.com/ee/user/project/clusters/kubernetes_pod_logs.html)
\ No newline at end of file diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 2fd26bb4899..41d12128e51 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -14,7 +14,7 @@ with minimal configuration. Just push your code and GitLab takes care of everything else. This makes it easier to start new projects and brings consistency to how applications are set up throughout a company. -For a demonstration of Auto DevOps, watch the video [GitLab Auto DevOps demo](https://youtu.be/4Uo_QP9rSGM). +For an introduction to Auto DevOps, watch [AutoDevOps in GitLab 11.0](https://youtu.be/0Tc0YYBxqi4). ## Enabled by default @@ -315,7 +315,8 @@ If a project's repository contains a `Dockerfile`, Auto Build will use If you are also using Auto Review Apps and Auto Deploy and choose to provide your own `Dockerfile`, make sure you expose your application to port -`5000` as this is the port assumed by the default Helm chart. +`5000` as this is the port assumed by the +[default Helm chart](https://gitlab.com/gitlab-org/charts/auto-deploy-app). #### Auto Build using Heroku buildpacks @@ -722,7 +723,7 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From Gitlab 11.11, this variable can be used to set a username to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD) | | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From Gitlab 11.11, this variable can be used to set a password to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_USERNAME) | | `REPLICAS` | The number of replicas to deploy; defaults to 1. | -| `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. This takes precedence over `REPLICAS`; defaults to 1. | +| `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | | `CANARY_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md); defaults to 1 | | `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1 | | `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. | diff --git a/doc/university/README.md b/doc/university/README.md index c116e54ad48..9d861460618 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -149,7 +149,6 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres 1. [How to Install GitLab with Omnibus - Video](https://www.youtube.com/watch?v=Q69YaOjqNhg) 1. [Installing GitLab - Online Course](https://courses.platzi.com/classes/git-gitlab/concepto/part-1/part-3/material/) 1. [Using a Non-Packaged PostgreSQL Database](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#using-a-non-packaged-postgresql-database-management-server) -1. [Using a MySQL Database](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#using-a-mysql-database-management-server-enterprise-edition-only) 1. [Installing GitLab on Microsoft Azure](https://about.gitlab.com/2016/07/13/how-to-setup-a-gitlab-instance-on-microsoft-azure/) 1. [Installing GitLab on Digital Ocean](https://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/) @@ -182,7 +181,7 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres ### 3.9. Integrations -1. [How to Integrate JIRA and Jenkins with GitLab - Video](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) +1. [How to Integrate Jira and Jenkins with GitLab - Video](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) 1. [How to Integrate Jira with GitLab](../user/project/integrations/jira.md) 1. [How to Integrate Jenkins with GitLab](../integration/jenkins.md) 1. [How to Integrate Bamboo with GitLab](../user/project/integrations/bamboo.md) diff --git a/doc/university/process/README.md b/doc/university/process/README.md index fdf6224f7f6..b278e02ccd5 100644 --- a/doc/university/process/README.md +++ b/doc/university/process/README.md @@ -9,11 +9,11 @@ title: University | Process # Suggesting improvements If you would like to teach a class or participate or help in any way please -submit a merge request and assign it to [Job](https://gitlab.com/u/JobV). +submit a merge request and assign it to [Job](https://gitlab.com/JobV). If you have suggestions for additional courses you would like to see, please submit a merge request to add an upcoming class, assign to -[Chad](https://gitlab.com/u/chadmalchow) and /cc [Job](https://gitlab.com/u/JobV). +[Chad](https://gitlab.com/chadmalchow) and /cc [Job](https://gitlab.com/JobV). ## Adding classes @@ -31,4 +31,4 @@ please submit a merge request to add an upcoming class, assign to 1. Please upload any video recordings to our Youtube channel. We prefer them to be public, if needed they can be unlisted but if so they should be linked from this page. -1. Please create a merge request and assign to [Erica](https://gitlab.com/u/Erica). +1. Please create a merge request and assign to [Erica](https://gitlab.com/Erica). diff --git a/doc/university/support/README.md b/doc/university/support/README.md index 9563492c137..2c6e52acfde 100644 --- a/doc/university/support/README.md +++ b/doc/university/support/README.md @@ -80,7 +80,7 @@ Our integrations add great value to GitLab. User questions often relate to integ - Learn about our Integrations (specially, not only): - [LDAP](../../integration/ldap.md) - - [JIRA](../../project_services/jira.md) + - [Jira](../../project_services/jira.md) - [Jenkins](../../integration/jenkins.md) - [SAML](../../integration/saml.md) @@ -145,7 +145,7 @@ Zendesk is our Support Centre and our main communication line with our Customers Some tickets need specific knowledge or a deep understanding of a particular component and will need to be escalated to a Senior Service Engineer or Developer -- Read about [Escalation](https://about.gitlab.com/handbook/support/onboarding/#create-issuesa-namecreate-issuea) +- Read about [Escalation](https://about.gitlab.com/handbook/support/workflows/shared/support_workflows/issue_escalations.html) - Find the macros in Zendesk for ticket escalations - Take a look at the [GitLab.com Team page](https://about.gitlab.com/team/) to find the resident experts in their fields diff --git a/doc/university/training/index.md b/doc/university/training/index.md index ddfc662123d..4c8ae0d9ce8 100644 --- a/doc/university/training/index.md +++ b/doc/university/training/index.md @@ -34,7 +34,7 @@ This section contains the following topics: ## Additional Resources -1. [GitLab Documentation](http://docs.gitlab.com) +1. [GitLab Documentation](https://docs.gitlab.com) 1. [GUI Clients](http://git-scm.com/downloads/guis) 1. [Pro Git book](http://git-scm.com/book) 1. [Platzi Course](https://courses.platzi.com/courses/git-gitlab/) diff --git a/doc/update/README.md b/doc/update/README.md index e2fffadb1ea..974982da5d0 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -49,8 +49,7 @@ However, for this to work there are the following requirements: - You have to use [post-deployment migrations](../development/post_deployment_migrations.md) (included in zero downtime update steps below). -- You are using PostgreSQL. If you are using MySQL please look at the release - post to see if downtime is required. +- You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported. Most of the time you can safely upgrade from a patch release to the next minor release if the patch release is not the latest. For example, upgrading from @@ -140,13 +139,11 @@ possible. - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL. -- [MySQL installation guide](../install/database_mysql.md) contains additional - information about configuring GitLab to work with a MySQL database. - [Restoring from backup after a failed upgrade](restore_after_failure.md) - [Upgrading PostgreSQL Using Slony](upgrading_postgresql_using_slony.md), for upgrading a PostgreSQL database with minimal downtime. -[omnidocker]: http://docs.gitlab.com/omnibus/docker/README.html +[omnidocker]: https://docs.gitlab.com/omnibus/docker/README.html [old-ee-upgrade-docs]: https://gitlab.com/gitlab-org/gitlab-ee/tree/11-8-stable-ee/doc/update [old-ce-upgrade-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/11-8-stable/doc/update [source-ce-to-ee]: upgrading_from_ce_to_ee.md diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index b83abcd36f7..4b13e41ab53 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,5 +1,5 @@ --- -last_updated: 2019-03-27 +last_updated: 2019-06-18 --- # Migrating from MySQL to PostgreSQL @@ -9,6 +9,10 @@ migrate it to a PostgreSQL database. ## Requirements +NOTE: **Note:** +Support for MySQL was removed in GitLab 12.1. This procedure should be performed +**before** installing GitLab 12.1. + [pgloader](http://pgloader.io) 3.4.1+ is required. You can install it directly from your distribution, for example in diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index f2df4277ca8..4300d6d56c7 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -13,8 +13,6 @@ You can select the tag in the version dropdown in the top left corner of GitLab ### 0. Backup It's useful to make a backup just in case things go south: -(With MySQL, this may require granting "LOCK TABLES" privileges to the GitLab -user on the database version) ```bash cd /home/git/gitlab @@ -48,12 +46,8 @@ sudo -u git -H git checkout LATEST_TAG -b LATEST_TAG ```bash cd /home/git/gitlab -# PostgreSQL sudo -u git -H bundle install --without development test mysql --deployment -# MySQL -sudo -u git -H bundle install --without development test postgres --deployment - # Optional: clean up old gems sudo -u git -H bundle clean diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index 428377adb19..7ae716d2cb3 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -25,14 +25,14 @@ Branch names use the format `major-minor-stable-ee` for Enterprise Edition, and `major-minor-stable` for Community Edition. For example, for 11.8.0 you would use the following branches: -* Enterprise Edition: `11-8-stable-ee` -* Community Edition: `11-8-stable` +- Enterprise Edition: `11-8-stable-ee` +- Community Edition: `11-8-stable` ### 0. Backup Make a backup just in case something goes wrong: -```bash +```sh cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` @@ -42,13 +42,13 @@ privileges to the GitLab user on the database version. ### 1. Stop server -```bash +```sh sudo service gitlab stop ``` ### 2. Get the EE code -```bash +```sh cd /home/git/gitlab sudo -u git -H git remote add -f ee https://gitlab.com/gitlab-org/gitlab-ee.git sudo -u git -H git checkout EE_BRANCH @@ -56,7 +56,7 @@ sudo -u git -H git checkout EE_BRANCH ### 3. Install libs, migrations, etc. -```bash +```sh cd /home/git/gitlab # MySQL installations (note: the line below states '--without postgres') @@ -80,7 +80,7 @@ document linked above and enable the indexer usage in the GitLab admin settings. ### 5. Start application -```bash +```sh sudo service gitlab start sudo service nginx restart ``` @@ -89,13 +89,13 @@ sudo service nginx restart Check if GitLab and its environment are configured correctly: -```bash +```sh sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production ``` To make sure you didn't miss anything run a more thorough check with: -```bash +```sh sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production ``` @@ -105,14 +105,14 @@ If all items are green, then congratulations upgrade complete! ### 1. Revert the code to the previous version -```bash +```sh cd /home/git/gitlab sudo -u git -H git checkout CE_BRANCH ``` ### 2. Restore from the backup -```bash +```sh cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production ``` diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index bc8e5fed774..023dc7d6de3 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -4,6 +4,10 @@ comments: false # Upgrading Community Edition and Enterprise Edition from source +NOTE: **Note:** +Users wishing to upgrade to 12.0.0 will have to take some extra steps. See the +version specific upgrade instructions for 12.0.0 for more details. + Make sure you view this update guide from the branch (version) of GitLab you would like to install (e.g., `11.8`. You can select the version in the version dropdown at the top left corner of GitLab (below the menu bar). @@ -235,29 +239,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) sudo -u git -H make ``` -### 12. Update MySQL permissions - -If you are using MySQL you need to grant the GitLab user the necessary -permissions on the database: - -```bash -mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" -``` - -If you use MySQL with replication, or just have MySQL configured with binary logging, -you will need to also run the following on all of your MySQL servers: - -```bash -mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" -``` - -You can make this setting permanent by adding it to your `my.cnf`: - -``` -log_bin_trust_function_creators=1 -``` - -### 13. Update configuration files +### 12. Update configuration files #### New configuration options for `gitlab.yml` @@ -331,18 +313,13 @@ For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload ``` -### 14. Install libs, migrations, etc. +### 13. Install libs, migrations, etc. ```bash cd /home/git/gitlab -# PostgreSQL installations (note: the line below states '--without mysql') sudo -u git -H bundle install --deployment --without development test mysql aws kerberos -# MySQL installations (note: the line below states '--without postgres') -sudo -u git -H bundle install --deployment --without development test postgres aws kerberos - - # Optional: clean up old gems sudo -u git -H bundle clean @@ -360,17 +337,14 @@ sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:c sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production ``` -**MySQL installations**: Run through the `MySQL strings limits` and `Tables and -data conversion to utf8mb4` [tasks](../install/database_mysql.md). - -### 15. Start application +### 14. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 16. Check application status +### 15. Check application status Check if GitLab and its environment are configured correctly: @@ -404,6 +378,20 @@ Example: Additional instructions here. --> +### 12.0.0 + +In 12.0.0 we made various database related changes. These changes require that +users first upgrade to the latest 11.11 patch release. Once upgraded to 11.11.x, +users can upgrade to 12.x. Failure to do so may result in database migrations +not being applied, which could lead to application errors. + +Example 1: you are currently using GitLab 11.11.3, which is the latest patch +release for 11.11.x. You can upgrade as usual to 12.0.0, 12.1.0, etc. + +Example 2: you are currently using a version of GitLab 10.x. To upgrade, first +upgrade to 11.11.3. Once upgraded to 11.11.3 you can safely upgrade to 12.0.0 +or future versions. + ## Things went south? Revert to previous version ### 1. Revert the code to the previous version diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 91c771764f8..d2947ae3371 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -18,22 +18,22 @@ Only admin users can access the Admin Area. The Admin Area is made up of the following sections: -| Section | Description | -|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| -| Overview | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administer-projects), [users](#administer-users), [groups](#administer-groups), [jobs](#administer-jobs), [Runners](#administer-runners), and [Gitaly servers](#administer-gitaly-servers). | -| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | -| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | -| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | -| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | -| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | -| License **[STARTER ONLY]** | Upload, display, and remove [licenses](license.md). | -| Push Rules **[STARTER]** | Configure pre-defined git [push rules](../../push_rules/push_rules.md) for projects. | -| Geo **[PREMIUM ONLY]** | Configure and maintain [Geo nodes](geo_nodes.md). | -| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | -| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | -| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | -| Appearance | Customize [GitLab's appearance](../../customization/index.md). | -| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | +| Section | Description | +|:------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Overview](#overview-section) | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administering-projects), [users](#administering-users), [groups](#administering-groups), [jobs](#administering-jobs), [Runners](#administering-runners), and [Gitaly servers](#administering-gitaly-servers). | +| Monitoring | View GitLab [system information](#system-info), and information on [background jobs](#background-jobs), [logs](#logs), [health checks](monitoring/health_check.md), [requests profiles](#requests-profiles), and [audit logs](#audit-log-premium-only). | +| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | +| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | +| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | +| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | +| License **[STARTER ONLY]** | Upload, display, and remove [licenses](license.md). | +| Push Rules **[STARTER]** | Configure pre-defined git [push rules](../../push_rules/push_rules.md) for projects. | +| Geo **[PREMIUM ONLY]** | Configure and maintain [Geo nodes](geo_nodes.md). | +| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | +| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | +| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | +| Appearance | Customize [GitLab's appearance](../../customization/index.md). | +| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | ## Admin Dashboard @@ -46,16 +46,20 @@ To access the Dashboard, either: The Dashboard is the default view of the Admin Area, and is made up of the following sections: -| Section | Description | -|------------|---------------| -| Projects | The total number of projects, up to 10 of the latest projects, and the option of creating a new project. | -| Users | The total number of users, up to 10 of the latest users, and the option of creating a new user. | -| Groups | The total number of groups, up to 10 of the latest groups, and the option of creating a new group. | -| Statistics | Totals of all elements of the GitLab instance. | +| Section | Description | +|:-----------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Projects | The total number of projects, up to 10 of the latest projects, and the option of creating a new project. | +| Users | The total number of users, up to 10 of the latest users, and the option of creating a new user. | +| Groups | The total number of groups, up to 10 of the latest groups, and the option of creating a new group. | +| Statistics | Totals of all elements of the GitLab instance. | | Features | All features available on the GitLab instance. Enabled features are marked with a green circle icon, and disabled features are marked with a power icon. | -| Components | The major components of GitLab and the version number of each. A link to the Gitaly Servers is also included. | +| Components | The major components of GitLab and the version number of each. A link to the Gitaly Servers is also included. | -## Administer Projects +## Overview section + +The following topics document the **Overview** section of the Admin Area. + +### Administering Projects You can administer all projects in the GitLab instance from the Admin Area's Projects page. @@ -95,7 +99,7 @@ You can combine the filter options. For example, to list only public projects wi 1. Click the **Public** tab. 1. Enter `score` in the **Filter by name...** input box. -## Administer Users +### Administering Users You can administer all users in the GitLab instance from the Admin Area's Users page. @@ -120,7 +124,7 @@ To search for users, enter your criteria in the search field. The user search is insensitive, and applies partial matching to name and username. To search for an email address, you must provide the complete email address. -## Administer Groups +### Administering Groups You can administer all groups in the GitLab instance from the Admin Area's Groups page. @@ -139,7 +143,7 @@ insensitive, and applies partial matching. To [Create a new group](../group/index.md#create-a-new-group) click **New group**. -## Administer Jobs +### Administering Jobs You can administer all jobs in the GitLab instance from the Admin Area's Jobs page. @@ -163,9 +167,9 @@ For each job, the following details are listed: | Timing | Duration of the job, and how long ago the job completed. | | Coverage | Percentage of tests coverage. | -## Administer Runners +### Administering Runners -You can adminster all Runners in the GitLab instance from the Admin Area's **Runners** page. See +You can administer all Runners in the GitLab instance from the Admin Area's **Runners** page. See [GitLab Runner](https://docs.gitlab.com/runner/) for more information on Runner itself. To access the **Runners** page, go to **Admin Area > Overview > Runners**. @@ -182,7 +186,7 @@ the sort order to *Last Contacted* from the dropdown beside the search field. To search Runners' descriptions: 1. In the **Search or filter results...** field, type the description of the Runner you want to -find. + find. 1. Press Enter. You can also filter Runners by status, type, and tag. To filter: @@ -209,7 +213,7 @@ For each Runner, the following attributes are listed: You can also edit, pause, or remove each Runner. -## Administer Gitaly servers +### Administering Gitaly servers You can list all Gitaly servers in the GitLab instance from the Admin Area's **Gitaly Servers** page. For more details, see [Gitaly](../../administration/gitaly/index.md). @@ -225,3 +229,66 @@ For each Gitaly server, the following details are listed: | Server version | Gitaly version | | Git version | Version of Git installed on the Gitaly server | | Up to date | Indicates if the Gitaly server version is the latest version available. A green dot indicates the server is up to date. | + +## Monitoring section + +The following topics document the **Monitoring** section of the Admin Area. + +### System Info + +The **System Info** page provides the following statistics: + +| Field | Description | +| :----------- | :---------- | +| CPU | Number of CPU cores available | +| Memory Usage | Memory in use, and total memory available | +| Disk Usage | Disk space in use, and total disk space available | +| Uptime | Approximate uptime of the GitLab instance | + +These statistics are updated only when you navigate to the **System Info** page, or you refresh the page in your browser. + +### Background Jobs + +The **Background Jobs** page displays the Sidekiq dashboard. Sidekiq is used by GitLab to +perform processing in the background. + +The Sidekiq dashboard consists of the following elements: + +- A tab per jobs' status. +- A breakdown of background job statistics. +- A live graph of **Processed** and **Failed** jobs, with a selectable polling interval. +- An historical graph of **Processed** and **Failed** jobs, with a selectable time span. +- Redis statistics, including: + - Version number + - Uptime, measured in days + - Number of connections + - Current memory usage, measured in MB + - Peak memory usage, measured in MB + +### Logs + +The **Logs** page provides access to the following log files: + +| Log file | Contents | +| :---------------------- | :------- | +| `application.log` | GitLab user activity | +| `githost.log` | Failed GitLab interaction with Git repositories | +| `production.log` | Requests received from Unicorn, and the actions taken to serve those requests | +| `sidekiq.log` | Background jobs | +| `repocheck.log` | Repository activity | +| `integrations_json.log` | Activity between GitLab and integrated systems | +| `kubernetes.log` | Kubernetes activity | + +The contents of these log files can be useful when troubleshooting a problem. Access is available to GitLab admins, without requiring direct access to the log files. + +For details of these log files and their contents, see [Log system](../../administration/logs.md). + +The content of each log file is listed in chronological order. To minimize performance issues, a maximum 2000 lines of each log file are shown. + +### Requests Profiles + +The **Requests Profiles** page contains the token required for profiling. For more details, see [Request Profiling](../../administration/monitoring/performance/request_profiling.md). + +### Audit Log **[PREMIUM ONLY]** + +The **Audit Log** page lists changes made within the GitLab server. With this information you can control, analyze, and track every change. diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index f80d4e9d2aa..35e7b6fb541 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -41,42 +41,51 @@ The readiness and liveness probes will provide a report of system health in JSON ```json { - "queues_check" : { - "status" : "ok" + "db_check":{ + "status":"ok" }, - "redis_check" : { - "status" : "ok" + "redis_check":{ + "status":"ok" }, - "shared_state_check" : { - "status" : "ok" + "cache_check":{ + "status":"ok" }, - "db_check" : { - "status" : "ok" + "queues_check":{ + "status":"ok" }, - "cache_check" : { - "status" : "ok" + "shared_state_check":{ + "status":"ok" + }, + "gitaly_check":{ + "status":"ok", + "labels":{ + "shard":"default" + } + } } -} ``` `liveness` probe example output: ```json { - "cache_check" : { - "status" : "ok" + "db_check":{ + "status":"ok" + }, + "redis_check":{ + "status":"ok" }, - "db_check" : { - "status" : "ok" + "cache_check":{ + "status":"ok" }, - "redis_check" : { - "status" : "ok" + "queues_check":{ + "status":"ok" }, - "queues_check" : { - "status" : "ok" + "shared_state_check":{ + "status":"ok" }, - "shared_state_check" : { - "status" : "ok" + "gitaly_check":{ + "status":"ok" } } ``` diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 6c4abce83c2..84596ff6a2c 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -89,21 +89,16 @@ are enabled.  -When the pipeline minutes quota for a group is set to a value different than 0, -the **Pipelines quota** page is available to the group page settings list. -You can see there an overview of the pipeline minutes quota of all projects of -the group. +You can see an overview of the pipeline minutes quota of all projects of +a group in the **Usage Quotas** page available to the group page settings list.  +## Extra Shared Runners pipeline minutes quota **[FREE ONLY]** -## Extra Shared Runners pipeline minutes quota - -NOTE: **Note:** -Only available on GitLab.com. - -You can purchase additional CI minutes so your pipelines will not be blocked after you have -used all your CI minutes from your main quota. +If you're using GitLab.com, you can purchase additional CI minutes so your +pipelines will not be blocked after you have used all your CI minutes from your +main quota. In order to purchase additional minutes, you should follow these steps: @@ -112,27 +107,35 @@ In order to purchase additional minutes, you should follow these steps:  1. Locate the subscription card that is linked to your group on GitLab.com, -click on **Buy more CI minutes**, and complete the details about the transaction. + click on **Buy more CI minutes**, and complete the details about the transaction.  1. Once we have processed your payment, the extra CI minutes -will be synced to your Group and you can visualize it from the -**Group > Settings > Pipelines quota** page: + will be synced to your Group and you can visualize it from the + **Group > Settings > Pipelines quota** page:  Be aware that: 1. If you have purchased extra CI minutes before the purchase of a paid plan, -we will calculate a pro-rated charge for your paid plan. That means you may -be charged for less than one year since your subscription was previously -created with the extra CI minutes. + we will calculate a pro-rated charge for your paid plan. That means you may + be charged for less than one year since your subscription was previously + created with the extra CI minutes. 1. Once the extra CI minutes has been assigned to a Group they cannot be transferred -to a different Group. + to a different Group. 1. If you have some minutes used over your default quota, these minutes will -be deducted from your Additional Minutes quota immediately after your purchase of additional -minutes. + be deducted from your Additional Minutes quota immediately after your purchase of additional + minutes. + +## What happens when my CI minutes quota run out + +When the CI minutes quota run out, an email is sent automatically to notifies the owner(s) of the group/namespace which +includes a link to [purchase more minutes](https://customers.gitlab.com/plans). + +If you are not the owner of the group, you will need to contact them to let them know they need to +[purchase more minutes](https://customers.gitlab.com/plans). ## Archive jobs **[CORE ONLY]** @@ -160,4 +163,4 @@ questions that you know someone might ask. Each scenario can be a third-level heading, e.g. `### Getting error message X`. If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. -->
\ No newline at end of file +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 11c0867da17..c1aa04f7bc2 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -76,13 +76,19 @@ service with this body: { "user_identifier": "jane@acme.org", "project_classification_label": "project-label", - "user_ldap_dn": "CN=Jane Doe,CN=admin,DC=acme" + "user_ldap_dn": "CN=Jane Doe,CN=admin,DC=acme", + "identities": [ + { "provider": "ldap", "extern_uid": "CN=Jane Doe,CN=admin,DC=acme" }, + { "provider": "bitbucket", "extern_uid": "2435223452345" } + ] } ``` The `user_ldap_dn` is optional and is only sent when the user is logged in through LDAP. +`identities` will contain the details of all the identities associated with the user. This will be an empty array if there are no identities associated with the user. + When the external authorization service responds with a status code 200, the user is granted access. When the external service responds with a status code 401 or 403, the user is denied access. In any case, the request is cached for 6 hours. diff --git a/doc/user/admin_area/settings/img/group_pipelines_quota.png b/doc/user/admin_area/settings/img/group_pipelines_quota.png Binary files differindex d94b609ad6f..318527426bd 100644 --- a/doc/user/admin_area/settings/img/group_pipelines_quota.png +++ b/doc/user/admin_area/settings/img/group_pipelines_quota.png diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index a5f8d05f662..a1bce5a6c69 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -17,7 +17,7 @@ To enforce acceptance of a Terms of Service and Privacy Policy: 1. Go to **Admin Area > Settings > General**. 1. Expand the **Terms of Service and Privacy Policy** section. 1. Check the **Require all users to accept Terms of Service and Privacy Policy when they access -GitLab.** checkbox. + GitLab.** checkbox. 1. Input the text of the **Terms of Service and Privacy Policy**. Markdown formatting can be used in this input box. 1. Click **Save changes**. 1. When you are presented with the **Terms of Service** statement, click **Accept terms**. diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index a1229484388..63879935fd8 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -4,12 +4,15 @@ type: reference # Visibility and access controls -GitLab allows admins to: +GitLab allows administrators to: - Control access and visibility to GitLab resources including branches and projects. - Select from which hosting sites code can be imported into GitLab. - Select the protocols permitted to access GitLab. - Enable or disable repository mirroring. +- Prevent non-administrators from deleting projects + ([introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5615) in GitLab 12.0). + **[PREMIUM ONLY]** To access the visibility and access control options: diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index a24374dff1d..9dfbe326f1d 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -40,6 +40,9 @@ To enable Container Scanning in your pipeline, you need: [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) executor running in privileged mode. If you're using the shared Runners on GitLab.com, this is enabled by default. +- Docker `18.09.03` or higher installed on the machine where the Runners are + running. If you're using the shared Runners on GitLab.com, this is already + the case. - To [build and push](../../../ci/docker/using_docker_build.md#container-registry-examples) your Docker image to your project's [Container Registry](../../project/container_registry.md). The name of the Docker image should match the following scheme: @@ -202,3 +205,25 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). + +## Troubleshooting + +### docker: Error response from daemon: failed to copy xattrs + +When the GitLab Runner uses the Docker executor and NFS is used +(e.g., `/var/lib/docker` is on an NFS mount), Container Scanning might fail with +an error like the following: + +``` +docker: Error response from daemon: failed to copy xattrs: failed to set xattr "security.selinux" on /path/to/file: operation not supported. +``` + +This is a result of a bug in Docker which is now [fixed](https://github.com/containerd/continuity/pull/138 "fs: add WithAllowXAttrErrors CopyOpt"). +To prevent the error, ensure the Docker version that the Runner is using is +`18.09.03` or higher. For more information, see +[issue #10241](https://gitlab.com/gitlab-org/gitlab-ee/issues/10241 "Investigate why Container Scanning is not working with NFS mounts"). diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 028ff72a160..2283efe3a44 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -58,7 +58,7 @@ To enable DAST in your project, define a job in your `.gitlab-ci.yml` file that This can be done in two ways: -- For GitLab 11.9 and later, including the provided DAST `.gitlab-ci.yml` template (recommended). +- For GitLab 11.9 and later, including the provided `DAST.gitlab-ci.yml` template (recommended). - Manually specifying the job definition. Not recommended unless using GitLab 11.8 and earlier. @@ -259,3 +259,8 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md new file mode 100644 index 00000000000..937ded287e5 --- /dev/null +++ b/doc/user/application_security/dependency_scanning/analyzers.md @@ -0,0 +1,133 @@ +# Dependency Scanning Analyzers **[ULTIMATE]** + +Dependency Scanning relies on underlying third party tools that are wrapped into +what we call "Analyzers". An analyzer is a +[dedicated project](https://gitlab.com/gitlab-org/security-products/analyzers) +that wraps a particular tool to: + +- Expose its detection logic. +- Handle its execution. +- Convert its output to the common format. + +This is achieved by implementing the [common API](https://gitlab.com/gitlab-org/security-products/analyzers/common). + +Dependency Scanning supports the following official analyzers: + +- [`bundler-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit) +- [`gemnasium`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) +- [`gemnasium-maven`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven) +- [`gemnasium-python`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python) +- [`retire.js`](https://gitlab.com/gitlab-org/security-products/analyzers/retire.js) + +The analyzers are published as Docker images that Dependency Scanning will use +to launch dedicated containers for each analysis. + +Dependency Scanning is pre-configured with a set of **default images** that are +maintained by GitLab, but users can also integrate their own **custom images**. + +## Official default analyzers + +Any custom change to the official analyzers can be achieved by using an +[environment variable in your `.gitlab-ci.yml`](index.md#customizing-the-dependency-scanning-settings). + +### Using a custom Docker mirror + +You can switch to a custom Docker registry that provides the official analyzer +images under a different prefix. For instance, the following instructs Dependency +Scanning to pull `my-docker-registry/gl-images/gemnasium` +instead of `registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium`. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_ANALYZER_IMAGE_PREFIX: my-docker-registry/gl-images +``` + +This configuration requires that your custom registry provides images for all +the official analyzers. + +### Selecting specific analyzers + +You can select the official analyzers you want to run. Here's how to enable +`bundler-audit` and `gemnasium` while disabling all the other default ones. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_DEFAULT_ANALYZERS: "bundler-audit,gemnasium" +``` + +`bundler-audit` runs first. When merging the reports, Dependency Scanning will +remove the duplicates and will keep the `bundler-audit` entries. + +### Disabling default analyzers + +Setting `DS_DEFAULT_ANALYZERS` to an empty string will disable all the official +default analyzers. In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_DEFAULT_ANALYZERS: "" +``` + +That's needed when one totally relies on [custom analyzers](#custom-analyzers). + +## Custom analyzers + +You can provide your own analyzers as a comma separated list of Docker images. +Here's how to add `analyzers/nugget` and `analyzers/perl` to the default images. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_ANALYZER_IMAGES: "my-docker-registry/analyzers/nugget,amy-docker-registry/nalyzers/perl" +``` + +The values must be the full path to the container registry images, +like what you would feed to the `docker pull` command. + +NOTE: **Note:** +This configuration doesn't benefit from the integrated detection step. Dependency +Scanning has to fetch and spawn each Docker image to establish whether the +custom analyzer can scan the source code. + +## Analyzers data + +The following table lists the data available for each official analyzer. + +| Property \ Tool | Gemnasium | bundler-audit | Retire.js | +|---------------------------------------|:------------------:|:------------------:|:------------------:| +| Severity | 𐄂 | ✓ | ✓ | +| Title | ✓ | ✓ | ✓ | +| File | ✓ | ⚠ | ✓ | +| Start line | 𐄂 | 𐄂 | 𐄂 | +| End line | 𐄂 | 𐄂 | 𐄂 | +| External ID (e.g., CVE) | ✓ | ✓ | ⚠ | +| URLs | ✓ | ✓ | ✓ | +| Internal doc/explanation | ✓ | 𐄂 | 𐄂 | +| Solution | ✓ | ✓ | 𐄂 | +| Confidence | 𐄂 | 𐄂 | 𐄂 | +| Affected item (e.g. class or package) | ✓ | ✓ | ✓ | +| Source code extract | 𐄂 | 𐄂 | 𐄂 | +| Internal ID | ✓ | 𐄂 | 𐄂 | +| Date | ✓ | 𐄂 | 𐄂 | +| Credits | ✓ | 𐄂 | 𐄂 | + +- ✓ => we have that data +- ⚠ => we have that data but it's partially reliable, or we need to extract that data from unstructured content +- 𐄂 => we don't have that data or it would need to develop specific or inefficient/unreliable logic to obtain it. + +The values provided by these tools are heterogeneous so they are sometimes +normalized into common values (e.g., `severity`, `confidence`, etc). diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index c65f369244a..9145e034dcb 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -46,17 +46,33 @@ this is enabled by default. The following languages and dependency managers are supported. -| Language (package managers) | Scan tool | -|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| -| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [Retire.js](https://retirejs.github.io/retire.js) | -| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | -| Ruby ([gem](https://rubygems.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [bundler-audit](https://github.com/rubysec/bundler-audit) | -| Java ([Maven](https://maven.apache.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | -| PHP ([Composer](https://getcomposer.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | - -Some scanners require to send a list of project dependencies to GitLab's central -servers to check for vulnerabilities. To learn more about this or to disable it, -refer to the [GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). +| Language (package managers) | Supported | Scan tool(s) | +|----------------------------- | --------- | ------------ | +| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js) | +| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| Ruby ([gem](https://rubygems.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) | +| Java ([Maven](https://maven.apache.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| PHP ([Composer](https://getcomposer.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| Python ([poetry](https://poetry.eustace.io/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7006 "Support Poetry in Dependency Scanning")) | not available | +| Python ([Pipfile](https://docs.pipenv.org/en/latest/basics/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/11756 "Pipfile.lock support for Dependency Scanning"))| not available | +| Go ([Golang](https://golang.org/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7132 "Dependency Scanning for Go")) | not available | + +## Remote checks + +While some tools pull a local database to check vulnerabilities, some others +like Gemnasium require sending data to GitLab central servers to analyze them: + +1. Gemnasium scans the dependencies of your project locally and sends a list of + packages to GitLab central servers. +1. The servers return the list of known vulnerabilities for all versions of + these packages. +1. The client picks up the relevant vulnerabilities by comparing with the versions + of the packages that are used by the project. + +The Gemnasium client does **NOT** send the exact package versions your project relies on. + +You can disable the remote checks by [using](#customizing-the-dependency-scanning-settings) +the `DS_DISABLE_REMOTE_CHECKS` environment variable and setting it to `true`. ## Configuring Dependency Scanning @@ -66,8 +82,7 @@ file that generates the This can be done in two ways: -- For GitLab 11.9 and later, including the provided Dependency Scanning - `.gitlab-ci.yml` template (recommended). +- For GitLab 11.9 and later, including the provided `Dependency-Scanning.gitlab-ci.yml` template (recommended). - Manually specifying the job definition. Not recommended unless using GitLab 11.8 and earlier. @@ -98,17 +113,10 @@ The report will be saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest Dependency Scanning artifact available. -Some security scanners require to send a list of project dependencies to GitLab -central servers to check for vulnerabilities. To learn more about this or to -disable it, check the -[GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). - #### Customizing the Dependency Scanning settings -The Dependency Scanning settings can be changed through environment variables by using the +The Dependency Scanning settings can be changed through [environment variables](#available-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. -These variables are documented in the -[Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings). For example: @@ -117,7 +125,7 @@ include: template: Dependency-Scanning.gitlab-ci.yml variables: - DEP_SCAN_DISABLE_REMOTE_CHECKS: true + DS_DISABLE_REMOTE_CHECKS: true ``` Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline @@ -138,6 +146,24 @@ dependency_scanning: CI_DEBUG_TRACE: "true" ``` +#### Available variables + +Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings) +using environment variables. + +| Environment variable | Function | +|-------------------------------- |----------| +| `DS_ANALYZER_IMAGES` | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). | +| `DS_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | +| `DS_ANALYZER_IMAGE_TAG` | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). | +| `DS_DEFAULT_ANALYZERS` | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). | +| `DS_DISABLE_REMOTE_CHECKS` | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). | +| `DS_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to `0` to disable). | +| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | +| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | +| `DS_PULL_ANALYZER_IMAGE_TIMEOUT` | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | +| `DS_RUN_ANALYZER_TIMEOUT` | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | + ### Manual job definition for GitLab 11.5 and later For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning` @@ -172,7 +198,7 @@ dependency_scanning: dependency_scanning: gl-dependency-scanning-report.json ``` -You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings) +You can supply many other [settings variables](#available-variables) via `docker run --env` to customize your job execution. ### Manual job definition for GitLab 11.4 and earlier (deprecated) @@ -219,7 +245,7 @@ dependency_scanning: CAUTION: **Caution:** The JSON report artifacts are not a public API of Dependency Scanning and their format may change in future. -The Dependency Scanning tool emits a JSON report file. Here is an example of a structure for a report will all important parts of +The Dependency Scanning tool emits a JSON report file. Here is an example of the report structure with all important parts of it highlighted: ```json-doc @@ -344,10 +370,10 @@ the report JSON unless stated otherwise. Presence of optional fields depends on | `vulnerabilities[].severity` | How much the vulnerability impacts the software. Possible values: `Undefined` (an analyzer has not provided this info), `Info`, `Unknown`, `Low`, `Medium`, `High`, `Critical`. | | `vulnerabilities[].confidence` | How reliable the vulnerability's assessment is. Possible values: `Undefined` (an analyzer has not provided this info), `Ignore`, `Unknown`, `Experimental`, `Low`, `Medium`, `High`, `Confirmed`. | | `vulnerabilities[].solution` | Explanation of how to fix the vulnerability. Optional. | -| `vulnerabilities[].scanner` | A node that describes the analyzer used find this vulnerability. | +| `vulnerabilities[].scanner` | A node that describes the analyzer used to find this vulnerability. | | `vulnerabilities[].scanner.id` | Id of the scanner as a snake_case string. | | `vulnerabilities[].scanner.name` | Name of the scanner, for display purposes. | -| `vulnerabilities[].location` | A node that tells which class and/or method is affected by the vulnerability. | +| `vulnerabilities[].location` | A node that tells where the vulnerability is located. | | `vulnerabilities[].location.file` | Path to the dependencies file (e.g., `yarn.lock`). Optional. | | `vulnerabilities[].location.dependency` | A node that describes the dependency of a project where the vulnerability is located. | | `vulnerabilities[].location.dependency.package` | A node that provides the information on the package where the vulnerability is located. | @@ -361,7 +387,7 @@ the report JSON unless stated otherwise. Presence of optional fields depends on | `vulnerabilities[].links` | An array of references to external documentation pieces or articles that describe the vulnerability further. Optional. | | `vulnerabilities[].links[].name` | Name of the vulnerability details link. Optional. | | `vulnerabilities[].links[].url` | URL of the vulnerability details document. Optional. | -| `remediations` | An array of objects containing information on cured vulnerabilities along with patch diffs to apply. | +| `remediations` | An array of objects containing information on cured vulnerabilities along with patch diffs to apply. Empty if no remediations provided by an underlying analyzer. | | `remediations[].fixes` | An array of strings that represent references to vulnerabilities fixed by this particular remediation. | | `remediations[].fixes[].cve` | A string value that describes a fixed vulnerability occurrence in the same format as `vulnerabilities[].cve`. | | `remediations[].summary` | Overview of how the vulnerabilities have been fixed. | @@ -378,19 +404,28 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). + ## Dependency List -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) -in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + +An additional benefit of Dependency Scanning is the ability to get a list of your +project's dependencies with their versions. This list can be generated only for +[languages and package managers](#supported-languages-and-package-managers) +supported by Gemnasium. -An additional benefit of Dependency Scanning is the ability to get a list of your project's dependencies with their versions. +To see the generated dependency list, navigate to your project's **Project > Dependency List**. -This list can be generated only for [supported languages and package managers](#supported-languages-and-package-managers). +## Versioning and release process -To see the generated dependency list, navigate to the Dependency List page under your project's left sidebar menu **Project > Dependency List**. +Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md). ## Contributing to the vulnerability database You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project to find a vulnerability in the Gemnasium database. -You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md).
\ No newline at end of file +You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md). diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 679847b76d7..69fa1ec5da6 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -10,7 +10,7 @@ high-level view on projects and groups, and start remediation processes when nee GitLab can scan and report any vulnerabilities found in your project. -| Secure scanning tools | Description | +| Secure scanning tool | Description | |:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------| | [Container Scanning](container_scanning/index.md) **[ULTIMATE]** | Scan Docker containers for known vulnerabilities. | | [Dependency Scanning](dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | @@ -19,6 +19,29 @@ GitLab can scan and report any vulnerabilities found in your project. | [Security Dashboard](security_dashboard/index.md) **[ULTIMATE]** | View vulnerabilities in all your projects and groups. | | [Static Application Security Testing (SAST)](sast/index.md) **[ULTIMATE]** | Analyze source code for known vulnerabilities. | +## Maintenance and update of the vulnerabilities database + +The various scanning tools and the vulnerabilities database are updated regularly. + +| Secure scanning tool | Vulnerabilities database updates | +|:-------------------------------------------------------------|-------------------------------------------| +| [Container Scanning](container_scanning/index.md) | Uses `clair` underneath and the latest `clair-db` version is used for each job run by running the [`latest` docker image tag](https://gitlab.com/gitlab-org/gitlab-ee/blob/438a0a56dc0882f22bdd82e700554525f552d91b/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml#L37). The `clair-db` database [is updated daily according to the author](https://github.com/arminc/clair-local-scan#clair-server-or-local). | +| [Dependency Scanning](dependency_scanning/index.md) | Relies on `bundler-audit` (for Rubygems), `retire.js` (for NPM packages) and `gemnasium` (GitLab's own tool for all libraries). `bundler-audit` and `retire.js` both fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated at least once a week. | +| [Dynamic Application Security Testing (DAST)](dast/index.md) | Updated weekly on Sundays. The underlying tool, `zaproxy`, downloads fresh rules at startup. | +| [Static Application Security Testing (SAST)](sast/index.md) | Relies exclusively on [the tools GitLab is wrapping](sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. | + +You don't have to update GitLab to benefit from the latest vulnerabilities definitions, +but you may have to in the future. + +The security tools are released as Docker images, and the vendored job definitions +to enable them are using the `x-y-stable` image tags that get overridden each time a new +release of the tools is pushed. The Docker images are updated to match the +previous GitLab releases, so they automatically get the latest versions of the +scanning tools without the user having to do anything. + +This workflow comes with some drawbacks and there's a +[plan to change this](https://gitlab.com/gitlab-org/gitlab-ee/issues/9725). + ## Interacting with the vulnerabilities > Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.8. diff --git a/doc/user/application_security/license_management/img/license_management_add_license.png b/doc/user/application_security/license_management/img/license_management_add_license.png Binary files differnew file mode 100644 index 00000000000..1e1a698515b --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_add_license.png diff --git a/doc/user/application_security/license_management/img/license_management_search.png b/doc/user/application_security/license_management/img/license_management_search.png Binary files differnew file mode 100644 index 00000000000..7b6006cef9d --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_search.png diff --git a/doc/user/application_security/license_management/img/license_management_settings.png b/doc/user/application_security/license_management/img/license_management_settings.png Binary files differindex b5490e59074..1a2bfa78a03 100644 --- a/doc/user/application_security/license_management/img/license_management_settings.png +++ b/doc/user/application_security/license_management/img/license_management_settings.png diff --git a/doc/user/application_security/license_management/index.md b/doc/user/application_security/license_management/index.md index 8b75995c377..957c4ede981 100644 --- a/doc/user/application_security/license_management/index.md +++ b/doc/user/application_security/license_management/index.md @@ -44,14 +44,14 @@ library whose license is incompatible with yours. The following languages and package managers are supported. -| Language | Package managers | -|------------|-------------------------------------------------------------------| -| JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/) | -| Go | [Godep](https://github.com/tools/godep), go get | -| Java | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) | -| .NET | [Nuget](https://www.nuget.org/) | -| Python | [pip](https://pip.pypa.io/en/stable/) | -| Ruby | [gem](https://rubygems.org/) | +| Language | Package managers | Scan Tool | +|------------|-------------------------------------------------------------------|----------------------------------------------------------| +| JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/) |[License Finder](https://github.com/pivotal/LicenseFinder)| +| Go | [Godep](https://github.com/tools/godep), go get |[License Finder](https://github.com/pivotal/LicenseFinder)| +| Java | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) |[License Finder](https://github.com/pivotal/LicenseFinder)| +| .NET | [Nuget](https://www.nuget.org/) |[License Finder](https://github.com/pivotal/LicenseFinder)| +| Python | [pip](https://pip.pypa.io/en/stable/) |[License Finder](https://github.com/pivotal/LicenseFinder)| +| Ruby | [gem](https://rubygems.org/) |[License Finder](https://github.com/pivotal/LicenseFinder)| ## Requirements @@ -65,10 +65,13 @@ file that generates the [License Management report artifact](../../../ci/yaml/RE This can be done in two ways: -- For GitLab 11.9 and later, including the provided License Management `.gitlab-ci.yml` template (recommended). +- For GitLab 11.9 and later, including the provided `License-Management.gitlab-ci.yml` template (recommended). - Manually specifying the job definition. Not recommended unless using GitLab 11.8 and earlier. +The License Management settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. These variables are documented in the [License Management documentation](https://gitlab.com/gitlab-org/security-products/license-management#settings). + ### Including the provided template NOTE: **Note:** @@ -165,6 +168,23 @@ to explicitly add `-DskipTests` to your options. If you still need to run tests during `mvn install`, add `-DskipTests=false` to `MAVEN_CLI_OPTS`. +### Selecting the version of Python + +> [Introduced](https://gitlab.com/gitlab-org/security-products/license-management/merge_requests/36) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + +License Management uses Python 2.7 and pip 10.0 by default. +If your project requires Python 3, you can switch to Python 3.5 and pip 19.1 +by setting the `LM_PYTHON_VERSION` environment variable to `3`. + +```yaml +include: + template: License-Management.gitlab-ci.yml + +license_management: + variables: + LM_PYTHON_VERSION: 3 +``` + ### Manual job definition for GitLab 11.5 and later For GitLab 11.5 and GitLab Runner 11.5 and later, the following `license_management` @@ -242,6 +262,8 @@ To approve or blacklist a license: navigate to the project's **Settings > CI/CD** and expand the **License Management** section. 1. Click the **Add a license** button. + +  1. In the **License name** dropdown, either: - Select one of the available licenses. You can search for licenses in the field at the top of the list. @@ -250,8 +272,22 @@ To approve or blacklist a license: 1. Select the **Approve** or **Blacklist** radio button to approve or blacklist respectively the selected license. + + +To modify an existing license: + +1. In the **License Management** list, click the **Approved/Declined** dropdown to change it to the desired status. +  +Searching for Licenses: + +1. Use the **Search** box to search for a specific license. + +  + + + ## License Management report under pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5491) diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 19befb04b76..9074ac3f4a1 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -80,7 +80,7 @@ To enable SAST in your project, define a job in your `.gitlab-ci.yml` file that This can be done in two ways: -- For GitLab 11.9 and later, including the provided SAST `.gitlab-ci.yml` template (recommended). +- For GitLab 11.9 and later, including the provided `SAST.gitlab-ci.yml` template (recommended). - Manually specifying the job definition. Not recommended unless using GitLab 11.8 and earlier. @@ -226,7 +226,7 @@ sast: CAUTION: **Caution:** The JSON report artifacts are not a public API of SAST and their format may change in the future. -The SAST tool emits a JSON report report file. Here is an example of a structure for a report will all important parts of +The SAST tool emits a JSON report report file. Here is an example of the report structure with all important parts of it highlighted: ```json-doc @@ -269,10 +269,9 @@ it highlighted: "url": "https://cwe.mitre.org/data/definitions/330.html" } ] - }, + }, { "category": "sast", - // "name" may be omitted because it could be not reported by a particular analyzer "message": "Probable insecure usage of temp file/directory.", "cve": "python/hardcoded/hardcoded-tmp.py:4ad6d4c40a8c263fc265f3384724014e0a4f8dd6200af83e51ff120420038031:B108", "severity": "Medium", @@ -297,7 +296,7 @@ it highlighted: "url": "https://docs.openstack.org/bandit/latest/plugins/b108_hardcoded_tmp_directory.html" } ] - }, + }, ], "remediations": [] } @@ -318,10 +317,10 @@ the report JSON unless stated otherwise. Presence of optional fields depends on | `vulnerabilities[].severity` | How much the vulnerability impacts the software. Possible values: `Undefined` (an analyzer has not provided this info), `Info`, `Unknown`, `Low`, `Medium`, `High`, `Critical`. | | `vulnerabilities[].confidence` | How reliable the vulnerability's assessment is. Possible values: `Undefined` (an analyzer has not provided this info), `Ignore`, `Unknown`, `Experimental`, `Low`, `Medium`, `High`, `Confirmed`. | | `vulnerabilities[].solution` | Explanation of how to fix the vulnerability. Optional. | -| `vulnerabilities[].scanner` | A node that describes the analyzer used find this vulnerability. | +| `vulnerabilities[].scanner` | A node that describes the analyzer used to find this vulnerability. | | `vulnerabilities[].scanner.id` | Id of the scanner as a snake_case string. | | `vulnerabilities[].scanner.name` | Name of the scanner, for display purposes. | -| `vulnerabilities[].location` | A node that tells which class and/or method is affected by the vulnerability. | +| `vulnerabilities[].location` | A node that tells where the vulnerability is located. | | `vulnerabilities[].location.file` | Path to the file where the vulnerability is located. Optional. | | `vulnerabilities[].location.start_line` | The first line of the code affected by the vulnerability. Optional. | | `vulnerabilities[].location.end_line` | The last line of the code affected by the vulnerability. Optional. | @@ -331,7 +330,7 @@ the report JSON unless stated otherwise. Presence of optional fields depends on | `vulnerabilities[].identifiers[].type` | Type of the identifier. Possible values: common identifier types (among `cve`, `cwe`, `osvdb`, and `usn`) or analyzer-dependent ones (e.g., `bandit_test_id` for [Bandit analyzer](https://wiki.openstack.org/wiki/Security/Projects/Bandit)). | | `vulnerabilities[].identifiers[].name` | Name of the identifier for display purposes. | | `vulnerabilities[].identifiers[].value` | Value of the identifier for matching purposes. | -| `vulnerabilities[].identifiers[].url` | URL to identifier's documentation. Optional. | +| `vulnerabilities[].identifiers[].url` | URL to identifier's documentation. Optional. | ## Secret detection @@ -364,3 +363,8 @@ vulnerabilities in your groups and projects. Read more about the Once a vulnerability is found, you can interact with it. Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Vulnerabilities database update + +For more information about the vulnerabilities database update, check the +[maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md new file mode 100644 index 00000000000..0ed9bf3f518 --- /dev/null +++ b/doc/user/asciidoc.md @@ -0,0 +1,372 @@ +# AsciiDoc + +GitLab uses the [Asciidoctor](https://asciidoctor.org) gem to convert AsciiDoc content to HTML5. +Consult the [Asciidoctor User Manual](https://asciidoctor.org/docs/user-manual) for a complete Asciidoctor reference. + +## Syntax + +Here's a brief reference of the most commonly used AsciiDoc syntax. +You can find the full documentation for the AsciiDoc syntax at <https://asciidoctor.org/docs>. + +### Paragraphs + +```asciidoc +A normal paragraph. +Line breaks are not preserved. +``` + +Line comments, which are lines that start with `//`, are skipped: + +``` +// this is a comment +``` + +A blank line separates paragraphs. + +A paragraph with the `[%hardbreaks]` option will preserve line breaks: + +```asciidoc +[%hardbreaks] +This paragraph carries the `hardbreaks` option. +Notice how line breaks are now preserved. +``` + +An indented (literal) paragraph disables text formatting, +preserves spaces and line breaks, and is displayed in a +monospaced font: + +```asciidoc + This literal paragraph is indented with one space. + As a consequence, *text formatting*, spaces, + and lines breaks will be preserved. +``` + +An admonition paragraph grabs the reader's attention: + +```asciidoc +NOTE: This is a brief reference, please read the full documentation at https://asciidoctor.org/docs. + +TIP: Lists can be indented. Leading whitespace is not significant. +``` + +### Text Formatting + +**Constrained (applied at word boundaries)** + +```asciidoc +*strong importance* (aka bold) +_stress emphasis_ (aka italic) +`monospaced` (aka typewriter text) +"`double`" and '`single`' typographic quotes ++passthrough text+ (substitutions disabled) +`+literal text+` (monospaced with substitutions disabled) +``` + +**Unconstrained (applied anywhere)** + +```asciidoc +**C**reate+**R**ead+**U**pdate+**D**elete +fan__freakin__tastic +``mono``culture +``` + +**Replacements** + +```asciidoc +A long time ago in a galaxy far, far away... +(C) 1976 Arty Artisan +I believe I shall--no, actually I won't. +``` + +**Macros** + +```asciidoc +// where c=specialchars, q=quotes, a=attributes, r=replacements, m=macros, p=post_replacements, etc. +The European icon:flag[role=blue] is blue & contains pass:[************] arranged in a icon:circle-o[role=yellow]. +The pass:c[->] operator is often referred to as the stabby lambda. +Since `pass:[++]` has strong priority in AsciiDoc, you can rewrite pass:c,a,r[C++ => C{pp}]. +// activate stem support by adding `:stem:` to the document header +stem:[sqrt(4) = 2] +``` + +### Attributes + +```asciidoc +// define attributes in the document header +:name: value +``` + +```asciidoc +:url-gem: https://rubygems.org/gems/asciidoctor + +You can download and install Asciidoctor {asciidoctor-version} from {url-gem}. +C{pp} is not required, only Ruby. +Use a leading backslash to output a word enclosed in curly braces, like \{name}. +``` + +### Links + +```asciidoc +https://example.org/page[A webpage] +link:../path/to/file.txt[A local file] +xref:document.adoc[A sibling document] +mailto:hello@example.org[Email to say hello!] +``` + +### Anchors + +```asciidoc +[[idname,reference text]] +// or written using normal block attributes as `[#idname,reftext=reference text]` +A paragraph (or any block) with an anchor (aka ID) and reftext. + +See <<idname>> or <<idname,optional text of internal link>>. + +xref:document.adoc#idname[Jumps to anchor in another document]. + +This paragraph has a footnote.footnote:[This is the text of the footnote.] +``` + +### Lists + +#### Unordered + +```asciidoc +* level 1 +** level 2 +*** level 3 +**** level 4 +***** etc. +* back at level 1 ++ +Attach a block or paragraph to a list item using a list continuation (which you can enclose in an open block). + +.Some Authors +[circle] +- Edgar Allen Poe +- Sheri S. Tepper +- Bill Bryson +``` + +#### Ordered + +```asciidoc +. Step 1 +. Step 2 +.. Step 2a +.. Step 2b +. Step 3 + +.Remember your Roman numerals? +[upperroman] +. is one +. is two +. is three +``` + +#### Checklist + +```asciidoc +* [x] checked +* [ ] not checked +``` +#### Callout + +```asciidoc +// enable callout bubbles by adding `:icons: font` to the document header +[,ruby] +---- +puts 'Hello, World!' # <1> +---- +<1> Prints `Hello, World!` to the console. +``` + +#### Description + +```asciidoc +first term:: description of first term +second term:: +description of second term +``` +### Document Structure + +#### Header + +```asciidoc += Document Title +Author Name <author@example.org> +v1.0, 2019-01-01 +``` +#### Sections + +```asciidoc += Document Title (Level 0) +== Level 1 +=== Level 2 +==== Level 3 +===== Level 4 +====== Level 5 +== Back at Level 1 +``` + +#### Includes + +```asciidoc +include::basics.adoc[] + +// define -a allow-uri-read to allow content to be read from URI +include::https://example.org/installation.adoc[] +``` +### Blocks + +```asciidoc +-- +open - a general-purpose content wrapper; useful for enclosing content to attach to a list item +-- +``` + +```asciidoc +// recognized types include CAUTION, IMPORTANT, NOTE, TIP, and WARNING +// enable admonition icons by setting `:icons: font` in the document header +[NOTE] +==== +admonition - a notice for the reader, ranging in severity from a tip to an alert +==== +``` + +```asciidoc +==== +example - a demonstration of the concept being documented +==== +``` + +```asciidoc +.Toggle Me +[%collapsible] +==== +collapsible - these details are revealed by clicking the title +==== +``` + +```asciidoc +**** +sidebar - auxiliary content that can be read independently of the main content +**** +``` + +```asciidoc +.... +literal - an exhibit that features program output +.... +``` + +```asciidoc +---- +listing - an exhibit that features program input, source code, or the contents of a file +---- +``` + +```asciidoc +[,language] +---- +source - a listing that is embellished with (colorized) syntax highlighting +---- +``` + +```asciidoc +\```language +fenced code - a shorthand syntax for the source block +\``` +``` + +```asciidoc +[,attribution,citetitle] +____ +quote - a quotation or excerpt; attribution with title of source are optional +____ +``` + +```asciidoc +[verse,attribution,citetitle] +____ +verse - a literary excerpt, often a poem; attribution with title of source are optional +____ +``` + +```asciidoc +++++ +pass - content passed directly to the output document; often raw HTML +++++ +``` + +```asciidoc +// activate stem support by adding `:stem:` to the document header +[stem] +++++ +x = y^2 +++++ +``` + +```asciidoc +//// +comment - content which is not included in the output document +//// +``` + +### Tables + +```asciidoc +.Table Attributes +[cols=>1h;2d,width=50%,frame=topbot] +|=== +| Attribute Name | Values + +| options +| header,footer,autowidth + +| cols +| colspec[;colspec;...] + +| grid +| all \| cols \| rows \| none + +| frame +| all \| sides \| topbot \| none + +| stripes +| all \| even \| odd \| none + +| width +| (0%..100%) + +| format +| psv {vbar} csv {vbar} dsv +|=== +``` + +### Multimedia + +```asciidoc +image::screenshot.png[block image,800,450] + +Press image:reload.svg[reload,16,opts=interactive] to reload the page. + +video::movie.mp4[width=640,start=60,end=140,options=autoplay] + +video::aHjpOzsQ9YI[youtube] + +video::300817511[vimeo] +``` + +### Breaks + +```asciidoc +// thematic break (aka horizontal rule) +--- +``` + +```asciidoc +// page break +<<< +``` + diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index b520c4fb579..f2516c6db0c 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -251,6 +251,7 @@ The applications below can be uninstalled. | Application | GitLab version | Notes | | ----------- | -------------- | ----- | +| JupyterHub | 12.1+ | All data not committed to GitLab will be deleted and cannot be restored. | | Prometheus | 11.11+ | All data will be deleted and cannot be restored. | To uninstall an application: @@ -287,4 +288,3 @@ To avoid installation errors: kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem diff a.pem b.pem ``` - diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 5d69efc3600..20eabdada79 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -53,6 +53,8 @@ persist through a commit ID change when: - force-pushing after a rebase - amending a commit +This functionality is also demonstrated in the video [How to use Merge Request Commit Discussions](https://www.youtube.com/watch?v=TviJH6oRboo). + To create a commit diff discussion: 1. Navigate to the merge request **Commits** tab. A list of commits that diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 3c5e820c1ca..8d4ffd93f59 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -57,10 +57,6 @@ differentiate the new cluster from the rest. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22011) in GitLab 11.5. > Became [optional](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26565) in GitLab 11.11. -NOTE: **Note:** -Only available when creating clusters. Existing clusters not managed by GitLab -cannot become GitLab-managed later. - You can choose to allow GitLab to manage your cluster for you. If your cluster is managed by GitLab, resources for your projects will be automatically created. See the [Access controls](../../project/clusters/index.md#access-controls) section for details on which resources will @@ -142,14 +138,6 @@ The result will then be: - The Staging cluster will be used for the `deploy to staging` job. - The Production cluster will be used for the `deploy to production` job. -## Unavailable features - -The following features are not currently available for group-level clusters: - -1. Terminals (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55487)). -1. Pod logs (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55488)). -1. Deployment boards (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55489)). - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 7e6cb24a51e..a555b7723df 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -21,9 +21,9 @@ page. ## Use cases - Analyze your team's contributions over a period of time, and offer a bonus for the top -contributors. + contributors. - Identify opportunities for improvement with group members who may benefit from additional -support. + support. ## Using Contribution Analytics @@ -54,13 +54,13 @@ Select the desired period from the calendar dropdown. Contributions per group member are also presented in tabular format. Click a column header to sort the table by that column: -* Member name -* Number of pushed events -* Number of opened issues -* Number of closed issues -* Number of opened MRs -* Number of accepted MRs -* Number of total contributions +- Member name +- Number of pushed events +- Number of opened issues +- Number of closed issues +- Number of opened MRs +- Number of accepted MRs +- Number of total contributions  diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 2e4106f55e5..f53c1dd95d7 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -202,8 +202,8 @@ You may also consult the [group permissions table][permissions]. ## Thread - Comments: collaborate on that epic by posting comments in its thread. -These text fields also fully support -[GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). + These text fields also fully support + [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). ## Comment, or start a discussion @@ -216,7 +216,7 @@ Once you wrote your comment, you can either: - You can [award an emoji](../../award_emojis.md) to that epic or its comments. -## Notifications +## Notifications - [Receive notifications](../../../workflow/notifications.md) for epic events. diff --git a/doc/user/group/img/group_storage_usage_quota.png b/doc/user/group/img/group_storage_usage_quota.png Binary files differnew file mode 100644 index 00000000000..c5d81ad7a8b --- /dev/null +++ b/doc/user/group/img/group_storage_usage_quota.png diff --git a/doc/user/group/index.md b/doc/user/group/index.md index b123a2b917b..7240b8e118b 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -41,12 +41,13 @@ You can create groups for numerous reasons. To name a couple: - Make it easier to `@mention` all of your team at once in issues and merge requests by creating a group and including the appropriate members. For example, you could create a group for your company members, and create a [subgroup](subgroups/index.md) for each individual team. Let's say you create a group called `company-team`, and you create subgroups in this group for the individual teams `backend-team`, `frontend-team`, and `production-team`. - - When you start a new implementation from an issue, you add a comment: - _"`@company-team`, let's do it! `@company-team/backend-team` you're good to go!"_ - - When your backend team needs help from frontend, they add a comment: - _"`@company-team/frontend-team` could you help us here please?"_ - - When the frontend team completes their implementation, they comment: - _"`@company-team/backend-team`, it's done! Let's ship it `@company-team/production-team`!"_ + +- When you start a new implementation from an issue, you add a comment: + _"`@company-team`, let's do it! `@company-team/backend-team` you're good to go!"_ +- When your backend team needs help from frontend, they add a comment: + _"`@company-team/frontend-team` could you help us here please?"_ +- When the frontend team completes their implementation, they comment: + _"`@company-team/backend-team`, it's done! Let's ship it `@company-team/production-team`!"_ ## Namespaces @@ -217,6 +218,8 @@ Get an overview of the vulnerabilities of all the projects in a group and its su ## Insights **[ULTIMATE]** +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + Configure the Insights that matter for your groups or projects, allowing users to explore data such as: @@ -232,8 +235,8 @@ to explore data such as: From GitLab 10.5, you can transfer groups in the following ways: - Transfer a subgroup to a new parent group. -- Convert a top-level group into a subgroup by transfering it to the desired group. -- Convert a subgroup into a top-level group by transfering it out of its current group. +- Convert a top-level group into a subgroup by transferring it to the desired group. +- Convert a subgroup into a top-level group by transferring it out of its current group. When transferring groups, note: @@ -268,9 +271,10 @@ be unique. To change your group path: -1. Navigate to your group's **Settings > General**. -1. Enter a new name under **Group path**. -1. Click **Save group**. +1. Navigate to your group's **Settings > General** page. +1. Expand the **Path, transfer, remove** section. +1. Enter a new name under **Change group path**. +1. Click **Change group path**. CAUTION: **Caution:** It is currently not possible to rename a namespace if it contains a @@ -323,6 +327,25 @@ This will disable the option for all users who previously had permissions to operate project memberships, so no new users can be added. Furthermore, any request to add a new user to a project through API will not be possible. +#### IP access restriction **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1985) in +[GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + +To make sure only people from within your organization can access particular +resources, you have the option to restrict access to groups and their +underlying projects, issues, etc, by IP address. This can help ensure that +particular content doesn't leave the premises, while not blocking off access to +the entire instance. + +Add whitelisted IP subnet using CIDR notation to the group settings and anyone +coming from a different IP address won't be able to access the restricted +content. + +Restriction currently applies to UI, API access is not restricted. +To avoid accidental lock-out, admins and group owners are are able to access +the group regardless of the IP restriction. + #### Group file templates **[PREMIUM]** Group file templates allow you to share a set of templates for common file @@ -362,6 +385,14 @@ Define project templates at a group level by setting a group as the template sou for the group. **[STARTER ONLY]** - **Pipelines quota**: Keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group. +#### Storage usage quota **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13294) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.0. + +A group owner can check the aggregated storage usage for all the project in a group, sub-groups included, in the **Storage** tab of the **Usage Quotas** page available to the group page settings list. + + + ## User contribution analysis **[STARTER]** With [GitLab Contribution Analytics](contribution_analytics/index.md), diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 1aba5d0986b..e6ba47939b3 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -4,8 +4,7 @@ type: reference, howto # Insights **[ULTIMATE]** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. -> **Generally Available** (GA) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. Configure the Insights that matter for your groups to explore data such as triage hygiene, issues created/closed per a given period, average time for merge diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index fcfd638f185..26893f7e31e 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -35,6 +35,8 @@ SSO enforcement was: With this option enabled, users must use your group's GitLab single sign on URL to be added to the group or be added via SCIM. Users cannot be added manually, and may only access project/group resources via the UI by signing in through the SSO URL. +However, users will not be prompted to log via SSO on each visit. GitLab will check whether a user has authenticated through the SSO link, and will only prompt the user to login via SSO if it has been longer than 7 days. + We intend to add a similar SSO requirement for [Git and API activity](https://gitlab.com/gitlab-org/gitlab-ee/issues/9152) in the future. ### NameID diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 96cc523f4ec..5aef463d782 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -24,27 +24,27 @@ The following identity providers are supported: ## Requirements -- [Group SSO](index.md) needs to be configured. +- [Group SSO](index.md) needs to be configured. - The `scim_group` feature flag must be enabled: Run the following commands in a Rails console: - + ```sh # Omnibus GitLab gitlab-rails console - + # Installation from source cd /home/git/gitlab sudo -u git -H bin/rails console RAILS_ENV=production ``` - + To enable SCIM for a group named `group_name`: - + ```ruby group = Group.find_by_full_path('group_name') Feature.enable(:group_scim, group) ``` - + ### GitLab configuration Once [Single sign-on](index.md) has been configured, we can: @@ -53,7 +53,7 @@ Once [Single sign-on](index.md) has been configured, we can: 1. Click on the **Generate a SCIM token** button. 1. Save the token and URL so they can be used in the next step. - + ## SCIM IdP configuration @@ -63,15 +63,15 @@ In the [Single sign-on](index.md) configuration for the group, make sure that the **Name identifier value** (NameID) points to a unique identifier, such as the `user.objectid`. This will match the `extern_uid` used on GitLab. -The GitLab app in Azure needs to be configured following +The GitLab app in Azure needs to be configured following [Azure's SCIM setup](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#getting-started). Note the following: - The `Tenant URL` and `secret token` are the ones retrieved in the -[previous step](#gitlab-configuration). + [previous step](#gitlab-configuration). - Should there be any problems with the availability of GitLab or similar -errors, the notification email set will get those. + errors, the notification email set will get those. - For mappings, we will only leave `Synchronize Azure Active Directory Users to AppName` enabled. You can then test the connection clicking on `Test Connection`. @@ -79,14 +79,14 @@ You can then test the connection clicking on `Test Connection`. ### Synchronize Azure Active Directory users 1. Click on `Synchronize Azure Active Directory Users to AppName`, to configure -the attribute mapping. + the attribute mapping. 1. Select the unique identifier (in the example `objectId`) as the `id` and `externalId`, -and enable the `Create`, `Update`, and `Delete` actions. + and enable the `Create`, `Update`, and `Delete` actions. 1. Map the `userPricipalName` to `emails[type eq "work"].value` and `mailNickname` to -`userName`. + `userName`. Example configuration: - +  1. Click on **Show advanced options > Edit attribute list for AppName**. @@ -95,11 +95,11 @@ and enable the `Create`, `Update`, and `Delete` actions. NOTE: **Note:** `username` should neither be primary nor required as we don't support that field on GitLab SCIM yet. - +  1. Save all the screens and, in the **Provisioning** step, set -the `Provisioning Status` to `ON`. + the `Provisioning Status` to `ON`. NOTE: **Note:** You can control what is actually synced by selecting the `Scope`. For example, diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 79ae94dd9ef..1c6cca049c5 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -5,8 +5,7 @@ type: reference, howto, concepts # Subgroups NOTE: **Note:** -[Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2772) in GitLab 9.0. Not available when using MySQL as external -database (support removed in GitLab 9.3 [due to performance reasons](https://gitlab.com/gitlab-org/gitlab-ce/issues/30472#note_27747600)). +[Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2772) in GitLab 9.0. Subgroups, also known as nested groups or hierarchical groups, allow you to have up to 20 levels of groups. @@ -21,16 +20,6 @@ By using subgroups you can do the following: - **Make it easier to manage people and control visibility.** Give people different [permissions](../../permissions.md#group-members-permissions) depending on their group [membership](#membership). -## Database Requirements - -Subgroups are only supported when you use PostgreSQL. Supporting subgroups on MySQL in an -efficient way is not possible due to MySQL's limitations. - -See the following links for more information: - -- <https://gitlab.com/gitlab-org/gitlab-ce/issues/30472> -- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10885> - ## Overview A group can have many subgroups inside it, and at the same time a group can have diff --git a/doc/user/img/color_inline_colorchip_render_gfm.png b/doc/user/img/color_inline_colorchip_render_gfm.png Binary files differdeleted file mode 100644 index 6f93dbeeb10..00000000000 --- a/doc/user/img/color_inline_colorchip_render_gfm.png +++ /dev/null diff --git a/doc/user/img/markdown_inline_diffs_tags_rendered.png b/doc/user/img/markdown_inline_diffs_tags_rendered.png Binary files differdeleted file mode 100644 index 4279a20b5a0..00000000000 --- a/doc/user/img/markdown_inline_diffs_tags_rendered.png +++ /dev/null diff --git a/doc/user/img/math_inline_sup_render_gfm.png b/doc/user/img/math_inline_sup_render_gfm.png Binary files differdeleted file mode 100644 index 3ee2abb14df..00000000000 --- a/doc/user/img/math_inline_sup_render_gfm.png +++ /dev/null diff --git a/doc/user/img/task_list_ordered_render_gfm.png b/doc/user/img/task_list_ordered_render_gfm.png Binary files differdeleted file mode 100644 index 98ec791e958..00000000000 --- a/doc/user/img/task_list_ordered_render_gfm.png +++ /dev/null diff --git a/doc/user/index.md b/doc/user/index.md index 1fc4e4c43cf..899026a801f 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -66,7 +66,7 @@ With GitLab Enterprise Edition, you can also: - Leverage continuous delivery method with [Canary Deployments](project/canary_deployments.md). - Scan your code for vulnerabilities and [display them in merge requests](application_security/sast/index.md). -You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, JIRA, and a lot more. +You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, Jira, and a lot more. ## Projects @@ -153,7 +153,7 @@ you have quick access to. You can also gather feedback on them through ## Integrations [Integrate GitLab](../integration/README.md) with your preferred tool, -such as Trello, JIRA, etc. +such as Trello, Jira, etc. ## Webhooks diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 31c8093ced7..a08b41aaecb 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -1,15 +1,20 @@ # GitLab Markdown -This markdown guide is **valid for GitLab's system markdown entries and files**. -It is not valid for the [GitLab documentation website](https://docs.gitlab.com) -nor [GitLab's main website](https://about.gitlab.com), as they both use -[Kramdown](https://kramdown.gettalong.org) as their markdown engine. -The documentation website uses an extended Kramdown gem, [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown). -Consult the [GitLab Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) for a complete Kramdown reference. +This markdown guide is **valid only for GitLab's internal markdown rendering system for entries and files**. +It is **not** valid for the [GitLab documentation website](https://docs.gitlab.com) +or [GitLab's main website](https://about.gitlab.com), as they both use +[Kramdown](https://kramdown.gettalong.org) as their markdown engine. The documentation +website uses an extended Kramdown gem, [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown). +Consult the [GitLab Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) +for a complete Kramdown reference. + +NOTE: **Note:** We encourage you to view this document as [rendered by GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md). ## GitLab Flavored Markdown (GFM) -GitLab uses "GitLab Flavored Markdown" (GFM). It extends the [CommonMark specification][commonmark-spec] (which is based on standard Markdown) in a few significant ways to add additional useful functionality. It was inspired by [GitHub Flavored Markdown](https://help.github.com/articles/basic-writing-and-formatting-syntax/). +GitLab uses "GitLab Flavored Markdown" (GFM). It extends the [CommonMark specification](https://spec.commonmark.org/current/) +(which is based on standard Markdown) in several ways to add additional useful functionality. +It was inspired by [GitHub Flavored Markdown](https://help.github.com/articles/basic-writing-and-formatting-syntax/). You can use GFM in the following areas: @@ -22,35 +27,29 @@ You can use GFM in the following areas: - Markdown documents inside repositories - Epics **[ULTIMATE]** -You can also use other rich text files in GitLab. You might have to install a -dependency to do so. Please see the [`github-markup` gem readme](https://github.com/gitlabhq/markup#markups) for more information. - -> **Notes:** -> -> We encourage you to view this document as [rendered by GitLab itself](markdown.md). -> -> As of 11.1, GitLab uses the [CommonMark Ruby Library][commonmarker] for Markdown -processing of all new issues, merge requests, comments, and other Markdown content -in the GitLab system. As of 11.3, wiki pages and Markdown files (`.md`) in the -repositories are also processed with CommonMark. As of 11.8, the [Redcarpet -Ruby library][redcarpet] has been removed and all issues/comments, including -those from pre-11.1, are now processed using [CommonMark Ruby -Library][commonmarker]. -> -> The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) -in October 2018. -> -> _Where there are significant differences, we will try to call them out in this document._ +You can also use other rich text files in GitLab. You might have to install a dependency +to do so. Please see the [`gitlab-markup` gem project](https://gitlab.com/gitlab-org/gitlab-markup) +for more information. + +### Transition from Redcarpet to CommonMark -### Transitioning to CommonMark +Since 11.1, GitLab uses the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker) +for Markdown processing of all new issues, merge requests, comments, and other Markdown +content in the GitLab system. Since 11.3, wiki pages and Markdown files (`*.md`) in +repositories are also processed with CommonMark. As of 11.8, the [Redcarpet Ruby library](https://github.com/vmg/redcarpet) +has been removed and all issues and comments, including those from pre-11.1, are now processed +using the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker). -You may have older issues/merge requests or Markdown documents in your -repository that were written using some of the nuances of RedCarpet's version +The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) +in October 2018. + +You may have older issues, merge requests, or Markdown documents in your +repository that were written using some of the nuances of GitLab's RedCarpet version of Markdown. Since CommonMark uses a slightly stricter syntax, these documents -may now display a little strangely since we've transitioned to CommonMark. -Numbered lists with nested lists in particular can be displayed incorrectly. +may now display a little differently since we've transitioned to CommonMark. -It is usually quite easy to fix. In the case of a nested list such as this: +It is usually quite easy to fix. For example, numbered lists with nested lists may +render incorrectly: ```markdown 1. Chocolate @@ -58,7 +57,14 @@ It is usually quite easy to fix. In the case of a nested list such as this: - milk ``` -simply add a space to each nested item: +1. Chocolate + - dark + - milk + +--- + +Simply add a space to each nested item to align the `-` with the first character of +the top list item (`C` in this case): ```markdown 1. Chocolate @@ -66,515 +72,674 @@ simply add a space to each nested item: - milk ``` -In the documentation below, we try to highlight some of the differences. +1. Chocolate + - dark + - milk + +NOTE: **Note:** We will flag any significant differences between Redcarpet and CommonMark + markdown in this document. If you have a large volume of Markdown files, it can be tedious to determine -if they will be displayed correctly or not. You can use the +if they will display correctly or not. You can use the [diff_redcarpet_cmark](https://gitlab.com/digitalmoksha/diff_redcarpet_cmark) -tool (not an officially supported product) to generate a list of files and +tool (not an officially supported product) to generate a list of files, and the differences between how RedCarpet and CommonMark render the files. It can give -you a great idea if anything needs to be changed - many times nothing will need -to changed. +an indication if anything needs to be changed - often nothing will need +to change. + +### GFM extends standard markdown + +GitLab makes full use of the standard (CommonMark) formatting, but also includes additional +functionality useful for GitLab users. + +It makes use of [new markdown features](#new-GFM-markdown-extensions), +not found in standard markdown: + +- [Color "chips" written in HEX, RGB or HSL](#colors) +- [Diagrams and flowcharts using Mermaid](#diagrams-and-flowcharts-using-mermaid) +- [Emoji](#emoji) +- [Front matter](#front-matter) +- [Inline diffs](#inline-diff) +- [Math equations and symbols written in LaTeX](#math) +- [Special GitLab references](#special-gitlab-references) +- [Task Lists](#task-lists) +- [Wiki specific markdown](#wiki-specific-markdown) + +It also has [extended markdown features](#standard-markdown-and-extensions-in-gitlab), without +changing how standard markdown is used: + +| Standard markdown | Extended markdown in GitLab | +| ------------------------------------- | ------------------------- | +| [blockquotes](#blockquotes) | [multiline blockquotes](#multiline-blockquote) | +| [code blocks](#code-spans-and-blocks) | [colored code and syntax highlighting](#colored-code-and-syntax-highlighting) | +| [emphasis](#emphasis) | [multiple underscores in words](#multiple-underscores-in-words-and-mid-word-emphasis) +| [headers](#headers) | [linkable Header IDs](#header-ids-and-links) | +| [images](#images) | [embedded videos](#videos) | +| [linebreaks](#line-breaks) | [more linebreak control](#newlines) | +| [links](#links) | [automatically linking URLs](#url-auto-linking) | + +## New GFM markdown extensions -### Newlines +### Colors -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#newlines +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colors). -GFM honors the markdown specification in how [paragraphs and line breaks are handled][commonmark-spec]. +It is possible to have color written in HEX, RGB or HSL format rendered with a color +indicator. -A paragraph is simply one or more consecutive lines of text, separated by one or -more blank lines. -Line-breaks, or soft returns, are rendered if you end a line with two or more spaces: +Supported formats (named colors are not supported): -<!-- (Do *NOT* remove the two ending whitespaces in the following line.) --> -<!-- (They are needed for the Markdown text to render correctly.) --> - Roses are red [followed by two or more spaces] - Violets are blue +- HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` +- RGB: `` `RGB[A](R, G, B[, A])` `` +- HSL: `` `HSL[A](H, S, L[, A])` `` - Sugar is sweet +Color written inside backticks will be followed by a color "chip": -<!-- (Do *NOT* remove the two ending whitespaces in the following line.) --> -<!-- (They are needed for the Markdown text to render correctly.) --> -Roses are red -Violets are blue +```markdown +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.3)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.3)` +``` -Sugar is sweet +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.3)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.3)` -### Multiple underscores in words +### Diagrams and flowcharts using Mermaid -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiple-underscores-in-words +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15107) in +GitLab 10.3. -It is not reasonable to italicize just _part_ of a word, especially when you're -dealing with code and names that often appear with multiple underscores. -Therefore, GFM ignores multiple underscores in words: +It is possible to generate diagrams and flowcharts from text using [Mermaid](https://mermaidjs.github.io/). +Visit the official page for more details. - perform_complicated_task +In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block: - do_this_and_do_that_and_another_thing +~~~ +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` +~~~ -perform_complicated_task +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` -do_this_and_do_that_and_another_thing +### Emoji -### URL auto-linking +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji). -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#url-auto-linking +```md +Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: -GFM will autolink almost any URL you copy and paste into your text: +:zap: You can use emoji anywhere GFM is supported. :v: - * https://www.google.com - * https://google.com/ - * ftp://ftp.us.debian.org/debian/ - * smb://foo/bar/baz - * irc://irc.freenode.net/gitlab - * http://localhost:3000 +You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People will :heart: you for that. -* https://www.google.com -* https://google.com/ -* ftp://ftp.us.debian.org/debian/ -* <a href="smb://foo/bar/baz">smb://foo/bar/baz</a> -* <a href="irc://irc.freenode.net/gitlab">irc://irc.freenode.net/gitlab</a> -* http://localhost:3000 +If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up one of the supported codes. -### Multiline blockquote +Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: +``` -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiline-blockquote +Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: -On top of standard Markdown [blockquotes](#blockquotes), which require prepending `>` to quoted lines, -GFM supports multiline blockquotes fenced by <code>>>></code>: +<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0"> -``` ->>> -If you paste a message from somewhere else +You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. -that +If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. -spans +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> -multiple lines, +> **Note:** The emoji example above uses hard-coded images for this documentation. The emoji, +when rendered within GitLab, may appear different depending on the OS and browser used. -you can quote that without having to manually prepend `>` to every line! ->>> -``` +Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. -<blockquote dir="auto"> -<p>If you paste a message from somewhere else</p> -<p>that</p> -<p>spans</p> -<p>multiple lines,</p> -<p>you can quote that without having to manually prepend <code>></code> to every line!</p> -</blockquote> +NOTE: **Note:** On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) +to get full native emoji support. Ubuntu 18.04 (like many modern Linux distros) has +this font installed by default. -### Code and syntax highlighting +### Front matter -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#code-and-syntax-highlighting +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) + in GitLab 11.6. -_GitLab uses the [Rouge Ruby library][rouge] for syntax highlighting. For a -list of supported languages visit the Rouge website._ +Front matter is metadata included at the beginning of a markdown document, preceding +its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/), +[Hugo](https://gohugo.io/content-management/front-matter/), and many other applications. -Blocks of code are either fenced by lines with three back-ticks <code>```</code>, -or are indented with four spaces. Only the fenced code blocks support syntax -highlighting: +When you view a Markdown file rendered by GitLab, any front matter is displayed as-is, +in a box at the top of the document, before the rendered HTML content. To view an example, +you can toggle between the source and rendered version of a [GitLab documentation file](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/README.md). -``` -Inline `code` has `back-ticks around` it. -``` +In GitLab, front matter is only used in Markdown files and wiki pages, not the other +places where Markdown formatting is supported. It must be at the very top of the document, +and must be between delimiters, as explained below. -Inline `code` has `back-ticks around` it. +The following delimeters are supported: -Example: +- YAML (`---`): - ```javascript - var s = "JavaScript syntax highlighting"; - alert(s); - ``` + ~~~yaml + --- + title: About Front Matter + example: + language: yaml + --- + ~~~ - ```python - def function(): - #indenting works just fine in the fenced code block - s = "Python syntax highlighting" - print s - ``` +- TOML (`+++`): - ```ruby - require 'redcarpet' - markdown = Redcarpet.new("Hello World!") - puts markdown.to_html - ``` + ~~~toml + +++ + title = "About Front Matter" + [example] + language = "toml" + +++ + ~~~ - ``` - No language indicated, so no syntax highlighting. - s = "There is no highlighting for this." - But let's throw in a <b>tag</b>. - ``` +- JSON (`;;;`): -becomes: + ~~~json + ;;; + { + "title": "About Front Matter" + "example": { + "language": "json" + } + } + ;;; + ~~~ -```javascript -var s = "JavaScript syntax highlighting"; -alert(s); -``` +Other languages are supported by adding a specifier to any of the existing +delimiters. For example: -```python -def function(): - #indenting works just fine in the fenced code block - s = "Python syntax highlighting" - print s +```php +---php +$title = "About Front Matter"; +$example = array( + 'language' => "php", +); +--- ``` -```ruby -require 'redcarpet' -markdown = Redcarpet.new("Hello World!") -puts markdown.to_html -``` +### Inline diff -``` -No language indicated, so no syntax highlighting. -s = "There is no highlighting for this." -But let's throw in a <b>tag</b>. -``` +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-diff). -### Inline diff +With inline diff tags you can display {+ additions +} or [- deletions -]. + +The wrapping tags can be either curly braces or square brackets: -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-diff +```markdown +- {+ addition 1 +} +- [+ addition 2 +] +- {- deletion 3 -} +- [- deletion 4 -] +``` -With inline diffs tags you can display {+ additions +} or [- deletions -]. +- {+ addition 1 +} +- [+ addition 2 +] +- {- deletion 3 -} +- [- deletion 4 -] -The wrapping tags can be either curly braces or square brackets. +--- -Examples: +However the wrapping tags cannot be mixed: +```markdown +- {+ addition +] +- [+ addition +} +- {- deletion -] +- [- deletion -} ``` -- {+ additions +} -- [+ additions +] -- {- deletions -} -- [- deletions -] -``` -becomes: +### Math + +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#math). + +It is possible to have math written with LaTeX syntax rendered using [KaTeX](https://github.com/Khan/KaTeX). + +Math written between dollar signs `$` will be rendered inline with the text. Math written +inside a [code block](#code-spans-and-blocks) with the language declared as `math`, will be rendered +on a separate line: - +~~~ +This math is inline $`a^2+b^2=c^2`$. -However the wrapping tags cannot be mixed as such: +This is on a separate line +```math +a^2+b^2=c^2 ``` -- {+ additions +] -- [+ additions +} -- {- deletions -] -- [- deletions -} +~~~ + +This math is inline $`a^2+b^2=c^2`$. + +This is on a separate line + +```math +a^2+b^2=c^2 ``` -### Emoji +_Be advised that KaTeX only supports a [subset](https://katex.org/docs/supported.html) of LaTeX._ -```md -Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: +NOTE: **Note:** This also works for the asciidoctor `:stem: latexmath`. For details see +the [asciidoctor user manual](http://asciidoctor.org/docs/user-manual/#activating-stem-support). -:zap: You can use emoji anywhere GFM is supported. :v: +### Special GitLab references -You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People will :heart: you for that. +GFM recognizes special GitLab related references. For example, you can easily reference +an issue, a commit, a team member or even the whole team within a project. GFM will turn +that reference into a link so you can navigate between them easily. -If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up one of the supported codes. +Additionally, GFM recognizes certain cross-project references, and also has a shorthand +version to reference other projects from the same namespace. -Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: +GFM will recognize the following: -Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. +| references | input | cross-project reference | shortcut within same namespace | +| :------------------------------ | :------------------------- | :-------------------------------------- | :----------------------------- | +| specific user | `@user_name` | | | +| specific group | `@group_name` | | | +| entire team | `@all` | | | +| project | `namespace/project>` | | | +| issue | ``#123`` | `namespace/project#123` | `project#123` | +| merge request | `!123` | `namespace/project!123` | `project!123` | +| snippet | `$123` | `namespace/project$123` | `project$123` | +| epic **[ULTIMATE]** | `&123` | `group1/subgroup&123` | | +| label by ID | `~123` | `namespace/project~123` | `project~123` | +| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | +| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | +| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | +| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | +| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | +| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | +| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | +| repository file references | `[README](doc/README)` | | | +| repository file line references | `[README](doc/README#L13)` | | | -On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) to get full native emoji support. +### Task lists + +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#task-lists). + +You can add task lists anywhere markdown is supported, but you can only "click" to +toggle the boxes if they are in issues, merge requests, or comments. In other places +you must edit the markdown manually to change the status by adding or removing the `x`. -Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. +To create a task list, add a specially-formatted Markdown list. You can use either +unordered or ordered lists: + +```markdown +- [x] Completed task +- [ ] Incomplete task + - [ ] Sub-task 1 + - [x] Sub-task 2 + - [ ] Sub-task 3 +1. [x] Completed task +1. [ ] Incomplete task + 1. [ ] Sub-task 1 + 1. [x] Sub-task 2 ``` -Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: +- [x] Completed task +- [ ] Incomplete task + - [ ] Sub-task 1 + - [x] Sub-task 2 + - [ ] Sub-task 3 +1. [x] Completed task +1. [ ] Incomplete task + 1. [ ] Sub-task 1 + 1. [x] Sub-task 2 -<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0"> +### Wiki-specific Markdown -You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. +The following examples show how links inside wikis behave. -If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. +#### Wiki - Direct page link -Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> +A link which just includes the slug for a page will point to that page, +_at the base level of the wiki_. -Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. +This snippet would link to a `documentation` page at the root of your wiki: -On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) to get full native emoji support. +```markdown +[Link to Documentation](documentation) +``` -Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. +#### Wiki - Direct file link -### Special GitLab references +Links with a file extension point to that file, _relative to the current page_. -GFM recognizes special references. +If the snippet below was placed on a page at `<your_wiki>/documentation/related`, +it would link to `<your_wiki>/documentation/file.md`: -You can easily reference e.g. an issue, a commit, a team member or even the whole team within a project. +```markdown +[Link to File](file.md) +``` -GFM will turn that reference into a link so you can navigate between them easily. +#### Wiki - Hierarchical link -GFM will recognize the following: +A link can be constructed relative to the current wiki page using `./<page>`, +`../<page>`, etc. -| input | references | -|:---------------------------|:--------------------------------| -| `@user_name` | specific user | -| `@group_name` | specific group | -| `@all` | entire team | -| `namespace/project>` | project | -| `#12345` | issue | -| `!123` | merge request | -| `$123` | snippet | -| `&123` | epic **[ULTIMATE]** | -| `~123` | label by ID | -| `~bug` | one-word label by name | -| `~"feature request"` | multi-word label by name | -| `%123` | project milestone by ID | -| `%v1.23` | one-word milestone by name | -| `%"release candidate"` | multi-word milestone by name | -| `9ba12248` | specific commit | -| `9ba12248...b19a04f5` | commit range comparison | -| `[README](doc/README)` | repository file references | -| `[README](doc/README#L13)` | repository file line references | - -GFM also recognizes certain cross-project references: - -| input | references | -|:----------------------------------------|:------------------------| -| `namespace/project#123` | issue | -| `namespace/project!123` | merge request | -| `namespace/project%123` | project milestone | -| `namespace/project$123` | snippet | -| `namespace/project@9ba12248` | specific commit | -| `group1/subgroup&123` | epic **[ULTIMATE]** | -| `namespace/project@9ba12248...b19a04f5` | commit range comparison | -| `namespace/project~"Some label"` | issues with given label | - -It also has a shorthand version to reference other projects from the same namespace: - -| input | references | -|:------------------------------|:------------------------| -| `project#123` | issue | -| `project!123` | merge request | -| `project%123` | project milestone | -| `project$123` | snippet | -| `project@9ba12248` | specific commit | -| `project@9ba12248...b19a04f5` | commit range comparison | -| `project~"Some label"` | issues with given label | +If this snippet was placed on a page at `<your_wiki>/documentation/main`, +it would link to `<your_wiki>/documentation/related`: -### Task lists +```markdown +[Link to Related Page](./related) +``` + +If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, +it would link to `<your_wiki>/documentation/main`: -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#task-lists +```markdown +[Link to Related Page](../main) +``` -You can add task lists to issues, merge requests and comments. To create a task list, add a specially-formatted Markdown list, like so: +If this snippet was placed on a page at `<your_wiki>/documentation/main`, +it would link to `<your_wiki>/documentation/related.md`: +```markdown +[Link to Related Page](./related.md) ``` -- [x] Completed task -- [ ] Incomplete task - - [ ] Sub-task 1 - - [x] Sub-task 2 - - [ ] Sub-task 3 + +If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, +it would link to `<your_wiki>/documentation/main.md`: + +```markdown +[Link to Related Page](../main.md) ``` - +#### Wiki - Root link + +A link starting with a `/` is relative to the wiki root. -Tasks formatted as ordered lists are supported as well: +This snippet links to `<wiki_root>/documentation`: +```markdown +[Link to Related Page](/documentation) ``` -1. [x] Completed task -1. [ ] Incomplete task - 1. [ ] Sub-task 1 - 1. [x] Sub-task 2 + +This snippet links to `<wiki_root>/miscellaneous.md`: + +```markdown +[Link to Related Page](/miscellaneous.md) ``` - +## Standard markdown and extensions in GitLab -Task lists can only be created in descriptions, not in titles. Task item state can be managed by editing the description's Markdown or by toggling the rendered check boxes. +All standard markdown formatting should work as expected within GitLab. Some standard +functionality is extended with additional features, without affecting the standard usage. +If a functionality is extended, the new option will be listed as a sub-section. -### Videos +### Blockquotes -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#videos +Blockquotes are an easy way to highlight information, such as a side-note. It is generated +by starting the lines of the blockquote with `>`: -Image tags with a video extension are automatically converted to a video player. +```markdown +> Blockquotes are very handy to emulate reply text. +> This line is part of the same quote. -The valid video extensions are `.mp4`, `.m4v`, `.mov`, `.webm`, and `.ogv`. +Quote break. - Here's a sample video: +> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. +``` -  +> Blockquotes are very handy to emulate reply text. +> This line is part of the same quote. -Here's a sample video: +Quote break. -<div class="video-container"> - <video src="img/markdown_video.mp4" width="400" controls="true" data-setup="{}" data-title="Sample Video"></video> - <p><a href="img/markdown_video.mp4" target="_blank" rel="noopener noreferrer" title="Download 'Sample Video'">Sample Video</a></p> -</div> +> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. -### Math +#### Multiline blockquote -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#math +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiline-blockquote). -It is possible to have math written with the LaTeX syntax rendered using [KaTeX][katex]. +GFM extends the standard markdown standard by also supporting multiline blockquotes +fenced by `>>>`: -Math written inside ```$``$``` will be rendered inline with the text. +``` +>>> +If you paste a message from somewhere else -Math written inside triple back quotes, with the language declared as `math`, will be rendered on a separate line. +that spans multiple lines, -Example: +you can quote that without having to manually prepend `>` to every line! +>>> +``` + +>>> +If you paste a message from somewhere else - This math is inline $`a^2+b^2=c^2`$. +that spans multiple lines, - This is on a separate line - ```math - a^2+b^2=c^2 - ``` +you can quote that without having to manually prepend `>` to every line! +>>> -Becomes: +### Code spans and blocks -This math is inline . +You can easily highlight anything that should be viewed as code and not simple text. -This is on a separate line +Simple inline code is easily highlighted with single backticks `` ` ``: -<img src="./img/math_inline_sup_render_gfm.png" > +```markdown +Inline `code` has `back-ticks around` it. +``` -_Be advised that KaTeX only supports a [subset][katex-subset] of LaTeX._ +Inline `code` has `back-ticks around` it. ->**Note:** -This also works for the asciidoctor `:stem: latexmath`. For details see the [asciidoctor user manual][asciidoctor-manual]. +--- -### Colors +Similarly, a whole block of code can be fenced with triple backticks ```` ``` ````, +triple tildes (`~~~`), or indended 4 or more spaces to achieve a similar effect for +a larger body of code. test. -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colors +~~~ +``` +def function(): + #indenting works just fine in the fenced code block + s = "Python code" + print s +``` -It is possible to have color written in HEX, RGB or HSL format rendered with a color indicator. + Using 4 spaces + is like using + 3-backtick fences. +~~~ -Color written inside backticks will be followed by a color "chip". +``` +~~~ +Tildes are OK too. +~~~ +``` -Examples: +The three examples above render as: - `#F00` - `#F00A` - `#FF0000` - `#FF0000AA` - `RGB(0,255,0)` - `RGB(0%,100%,0%)` - `RGBA(0,255,0,0.7)` - `HSL(540,70%,50%)` - `HSLA(540,70%,50%,0.7)` +``` +def function(): + #indenting works just fine in the fenced code block + s = "Python code" + print s +``` -Becomes: + Using 4 spaces + is like using + 3-backtick fences. - +~~~ +Tildes are OK too. +~~~ -#### Supported formats: +#### Colored code and syntax highlighting -* HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` -* RGB: `` `RGB[A](R, G, B[, A])` `` -* HSL: `` `HSL[A](H, S, L[, A])` `` +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colored-code-and-syntax-highlighting). -### Mermaid +GitLab uses the [Rouge Ruby library](http://rouge.jneen.net/) for more colorful syntax +highlighting in code blocks. For a list of supported languages visit the +[Rouge project wiki](https://github.com/jneen/rouge/wiki/List-of-supported-languages-and-lexers). +Syntax highlighting is only supported in code blocks, it is not possible to highlight +code when it is inline. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15107) in -GitLab 10.3. -> -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#mermaid +Blocks of code are fenced by lines with three back-ticks ```` ``` ```` or three tildes `~~~`, and have +the language identified at the end of the first fence: -It is possible to generate diagrams and flowcharts from text using [Mermaid](https://mermaidjs.github.io/). +~~~ +```javascript +var s = "JavaScript syntax highlighting"; +alert(s); +``` -In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block. +```python +def function(): + #indenting works just fine in the fenced code block + s = "Python syntax highlighting" + print s +``` -Example: +```ruby +require 'redcarpet' +markdown = Redcarpet.new("Hello World!") +puts markdown.to_html +``` - ```mermaid - graph TD; - A-->B; - A-->C; - B-->D; - C-->D; - ``` +``` +No language indicated, so no syntax highlighting. +s = "There is no highlighting for this." +But let's throw in a <b>tag</b>. +``` +~~~ -Becomes: +The four examples above render as: -```mermaid -graph TD; - A-->B; - A-->C; - B-->D; - C-->D; +```javascript +var s = "JavaScript syntax highlighting"; +alert(s); ``` -For details see the [Mermaid official page](https://mermaidjs.github.io/). +```python +def function(): + #indenting works just fine in the fenced code block + s = "Python syntax highlighting" + print s +``` -### Front matter +```ruby +require 'redcarpet' +markdown = Redcarpet.new("Hello World!") +puts markdown.to_html +``` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) - in GitLab 11.6. +``` +No language indicated, so no syntax highlighting. +s = "There is no highlighting for this." +But let's throw in a <b>tag</b>. +``` -Front matter is metadata included at the beginning of a markdown document, preceding -its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/) and [Hugo](https://gohugo.io/content-management/front-matter/), -and many other applications. +### Emphasis -In GitLab, front matter is only used in Markdown files and wiki pages, not the other places where Markdown formatting is supported. -When you view a Markdown file rendered by GitLab, any front matter is displayed as-is, in a box at the top of the document, before the rendered HTML content. -To view an example, you can toggle between the source and rendered version of a [GitLab documentation file](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/README.md). +There are multiple ways to emphasize text in markdown. You can italicize, bold, strikethrough, +as well as combine these emphasis styles together. -The following delimeters are supported: +Examples: -- YAML (`---`): +```markdown +Emphasis, aka italics, with *asterisks* or _underscores_. - ``` - --- - title: About Front Matter - example: - language: yaml - --- - ``` +Strong emphasis, aka bold, with double **asterisks** or __underscores__. -- TOML (`+++`): +Combined emphasis with **asterisks and _underscores_**. - ``` - +++ - title = "About Front Matter" - [example] - language = "toml" - +++ - ``` +Strikethrough uses two tildes. ~~Scratch this.~~ +``` -- JSON (`;;;`): +Emphasis, aka italics, with *asterisks* or _underscores_. - ``` - ;;; - { - "title": "About Front Matter" - "example": { - "language": "json" - } - } - ;;; - ``` +Strong emphasis, aka bold, with double **asterisks** or __underscores__. -Other languages are supported by adding a specifier to any of the existing -delimiters. For example: +Combined emphasis with **asterisks and _underscores_**. + +Strikethrough uses two tildes. ~~Scratch this.~~ + +NOTE: **Note:** Strikethrough is not part of the core Markdown standard, but is part of GFM. + +#### Multiple underscores in words and mid-word emphasis + +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiple-underscores-in-words). +It is not usually useful to italicize just _part_ of a word, especially when you're +dealing with code and names that often appear with multiple underscores. As a result, +GFM extends the standard markdown standard by ignoring multiple underlines in words, +to allow better rendering of markdown documents discussing code: + +```md +perform_complicated_task + +do_this_and_do_that_and_another_thing + +but_emphasis is_desired _here_ ``` ----php -$title = "About Front Matter"; -$example = array( - 'language' => "php", -); + +perform_complicated_task + +do_this_and_do_that_and_another_thing + +but_emphasis is_desired _here_ + --- + +If you wish to emphasize only a part of a word, it can still be done with asterisks: + +```md +perform*complicated*task + +do*this*and*do*that*and*another thing ``` -## Standard Markdown +perform*complicated*task -### Headers +do*this*and*do*that*and*another thing + +### Footnotes + +Footnotes add a link to a note rendered at the end of a markdown file: +```markdown +You can add footnotes to your text as follows.[^1] + +[^1]: This is my awesome footnote (later in file). ``` + +You can add footnotes to your text as follows.[^1] + +[^1]: This is my awesome footnote (later in file). + +### Headers + +```markdown # H1 ## H2 ### H3 @@ -593,9 +758,11 @@ Alt-H2 #### Header IDs and links -All Markdown-rendered headers automatically get IDs, which can be linked to, except in comments. +GFM extends the standard markdown standard so that all Markdown-rendered headers automatically +get IDs, which can be linked to, except in comments. -On hover, a link to those IDs becomes visible to make it easier to copy the link to the header to use it somewhere else. +On hover, a link to those IDs becomes visible to make it easier to copy the link to +the header to use it somewhere else. The IDs are generated from the content of the header according to the following rules: @@ -606,7 +773,7 @@ The IDs are generated from the content of the header according to the following 1. If a header with the same ID has already been generated, a unique incrementing number is appended, starting at 1. -For example: +Example: ``` # This header has spaces in it @@ -626,216 +793,165 @@ Would generate the following link IDs: 1. `this-header-has-spaces-in-it-2` 1. `this-header-has-3-5-in-it-and-parentheses` -Note that the Emoji processing happens before the header IDs are generated, so the Emoji is converted to an image which then gets removed from the ID. +Note that the Emoji processing happens before the header IDs are generated, so the +Emoji is converted to an image which is then removed from the ID. -### Emphasis - -Examples: - -``` -Emphasis, aka italics, with *asterisks* or _underscores_. - -Strong emphasis, aka bold, with **asterisks** or __underscores__. - -Combined emphasis with **asterisks and _underscores_**. - -Strikethrough uses two tildes. ~~Scratch this.~~ -``` - -Becomes: - -Emphasis, aka italics, with *asterisks* or _underscores_. +### Horizontal Rule -Strong emphasis, aka bold, with **asterisks** or __underscores__. +It's very simple to create a horizontal rule, by using three or more hyphens, asterisks, +or underscores: -Combined emphasis with **asterisks and _underscores_**. +```markdown +Three or more hyphens, -Strikethrough uses two tildes. ~~Scratch this.~~ +--- -### Lists +asterisks, -Examples: +*** -``` -1. First ordered list item -2. Another item - * Unordered sub-list. -1. Actual numbers don't matter, just that it's a number - 1. Ordered sub-list -4. And another item. +or underscores -* Unordered list can use asterisks -- Or minuses -+ Or pluses +___ ``` -Becomes: - -1. First ordered list item -2. Another item - * Unordered sub-list. -1. Actual numbers don't matter, just that it's a number - 1. Ordered sub-list -4. And another item. - -* Unordered list can use asterisks -- Or minuses -+ Or pluses - -If a list item contains multiple paragraphs, -each subsequent paragraph should be indented to the same level as the start of the list item text +Three or more hyphens, -Example: +--- -``` -1. First ordered list item +asterisks, - Second paragraph of first item. - -2. Another item -``` +*** -Becomes: +or underscores -1. First ordered list item +___ - Paragraph of first item. +### Images -2. Another item +Examples: -If the paragraph of the first item is not indented with the proper number of spaces, -the paragraph will appear outside the list, instead of properly indented under the list item. +```markdown +Inline-style (hover to see title text): -Example: + -``` -1. First ordered list item +Reference-style (hover to see title text): - Paragraph of first item. +![alt text1][logo] -2. Another item +[logo]: img/markdown_logo.png "Title Text" ``` -Becomes: +Inline-style (hover to see title text): -1. First ordered list item + - Paragraph of first item. +Reference-style (hover to see title text): -2. Another item - -### Links +![alt text][logo] -There are two ways to create links, inline-style and reference-style. +[logo]: img/markdown_logo.png "Title Text" -```markdown -[I'm an inline-style link](https://www.google.com) -[I'm a link to a repository file in the same directory](index.md) -[I am an absolute reference within the repository](/doc/user/index.md) -[I'm a relative link to the Milestones page](../README.md) +#### Videos -[I link to a section on a different markdown page, using a header ID](index.md#overview) -[I link to a different section on the same page, using the header ID](#header-ids-and-links) +> If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#videos). -[I'm a reference-style link][Arbitrary case-insensitive reference text] -[You can use numbers for reference-style link definitions][1] -Or leave it empty and use the [link text itself][] +Image tags that link to files with a video extension are automatically converted to +a video player. The valid video extensions are `.mp4`, `.m4v`, `.mov`, `.webm`, and `.ogv`: -Some text to show that the reference links can follow later. +```md +Here's a sample video: -[arbitrary case-insensitive reference text]: https://www.mozilla.org -[1]: http://slashdot.org -[link text itself]: https://www.reddit.com + ``` ->**Note:** -Relative links do not allow referencing project files in a wiki page or wiki -page in a project file. The reason for this is that, in GitLab, wiki is always -a separate Git repository. For example, `[I'm a reference-style link](style)` -will point the link to `wikis/style` when the link is inside of a wiki markdown file. - -### Images +Here's a sample video: -Examples: + - Here's our logo (hover to see the title text): +### Inline HTML - Inline-style: -  +> To see the markdown rendered within HTML in the second example, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-html). - Reference-style: - ![alt text1][logo] +You can also use raw HTML in your Markdown, and it'll usually work pretty well. - [logo]: img/markdown_logo.png +See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubydoc.info/gems/html-pipeline/1.11.0/HTML/Pipeline/SanitizationFilter#WHITELIST-constant) +class for the list of allowed HTML tags and attributes. In addition to the default +`SanitizationFilter` whitelist, GitLab allows `span`, `abbr`, `details` and `summary` elements. -Becomes: +```html +<dl> + <dt>Definition list</dt> + <dd>Is something people use sometimes.</dd> -Here's our logo: + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML <em>tags</em> will <b>always</b> work.</dd> +</dl> +``` -Inline-style: +<dl> + <dt>Definition list</dt> + <dd>Is something people use sometimes.</dd> - + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML <em>tags</em> will <b>always</b> work.</dd> +</dl> -Reference-style: +--- -![alt text][logo] +It is still possible to use markdown inside HTML tags, but only if the lines containing markdown +are separated into their own lines: -[logo]: img/markdown_logo.png +```html +<dl> + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML tags will always work.</dd> -### Blockquotes + <dt>Markdown in HTML</dt> + <dd> -Examples: + Does *not* work **very** well. HTML tags will always work. + </dd> +</dl> ``` -> Blockquotes are very handy in email to emulate reply text. -> This line is part of the same quote. -Quote break. +<!-- Note: The example below uses HTML to force correct rendering on docs.gitlab.com, markdown will be fine in GitLab --> -> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. -``` +<dl> + <dt>Markdown in HTML</dt> + <dd>Does *not* work **very** well. HTML tags will always work.</dd> -Becomes: + <dt>Markdown in HTML</dt> + <dd> -> Blockquotes are very handy in email to emulate reply text. -> This line is part of the same quote. + Does <em>not</em> work <b>very</b> well. HTML tags will always work. -Quote break. + </dd> +</dl> -> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. +#### Details and Summary -### Inline HTML +> To see the markdown rendered within HTML in the second example, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#details-and-summary). -You can also use raw HTML in your Markdown, and it'll mostly work pretty well. +Content can be collapsed using HTML's [`<details>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details) +and [`<summary>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/summary) +tags. This is especially useful for collapsing long logs so they take up less screen space. -See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubydoc.info/gems/html-pipeline/1.11.0/HTML/Pipeline/SanitizationFilter#WHITELIST-constant) class for the list of allowed HTML tags and attributes. In addition to the default `SanitizationFilter` whitelist, GitLab allows `span`, `abbr`, `details` and `summary` elements. +```html +<p> +<details> +<summary>Click me to collapse/fold.</summary> -Examples: +These details <em>will</em> remain <strong>hidden</strong> until expanded. -``` -<dl> - <dt>Definition list</dt> - <dd>Is something people use sometimes.</dd> +<pre><code>PASTE LOGS HERE</code></pre> - <dt>Markdown in HTML</dt> - <dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd> -</dl> +</details> +</p> ``` -Becomes: - -<dl> - <dt>Definition list</dt> - <dd>Is something people use sometimes.</dd> - - <dt>Markdown in HTML</dt> - <dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd> -</dl> - -#### Details and Summary - -Content can be collapsed using HTML's [`<details>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details) and [`<summary>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/summary) tags. This is especially useful for collapsing long logs so they take up less screen space. - <p> <details> <summary>Click me to collapse/fold.</summary> @@ -847,7 +963,10 @@ These details <em>will</em> remain <strong>hidden</strong> until expanded. </details> </p> -**Note:** Markdown inside these tags is supported, as long as you have a blank line after the `</summary>` tag and before the `</details>` tag, as shown in the example. +--- + +Markdown inside these tags is supported as well, as long as you have a blank line +after the `</summary>` tag and before the `</details>` tag, as shown in the example: ```html <details> @@ -860,232 +979,302 @@ These details _will_ remain **hidden** until expanded. </details> ``` -### Horizontal Rule +<!-- Note: The example below uses HTML to force correct rendering on docs.gitlab.com, markdown will be fine in GitLab --> -Examples: +<details> +<summary>Click me to collapse/fold.</summary> -``` -Three or more... +These details <em>will</em> remain <b>hidden</b> until expanded. ---- + PASTE LOGS HERE -Hyphens +</details> -*** +### Line Breaks -Asterisks +A line break will be inserted (a new paragraph will start) if the previous text is +ended with two newlines, i.e. you hit <kbd>Enter</kbd> twice in a row. If you only +use one newline (hit <kbd>Enter</kbd> once), the next sentence will be part of the +same paragraph. This is useful if you want to keep long lines from wrapping, and keep +them easily editable: -___ +```markdown +Here's a line for us to start with. -Underscores -``` +This longer line is separated from the one above by two newlines, so it will be a *separate paragraph*. -Becomes: +This line is also a separate paragraph, but... +These lines are only separated by single newlines, +so they *do not break* and just follow the previous lines +in the *same paragraph*. +``` -Three or more... +Here's a line for us to start with. ---- +This longer line is separated from the one above by two newlines, so it will be a *separate paragraph*. -Hyphens +This line is also a separate paragraph, but... +These lines are only separated by single newlines, +so they *do not break* and just follow the previous lines +in the *same paragraph*. -*** +#### Newlines -Asterisks +GFM adheres to the markdown specification in how [paragraphs and line breaks are handled](https://spec.commonmark.org/current/). -___ +A paragraph is simply one or more consecutive lines of text, separated by one or +more blank lines (i.e. two newlines at the end of the first paragraph), as [explained above](#line-breaks). -Underscores +If you need more control over line-breaks or soft returns, you can add a single line-break +by ending a line with a backslash, or two or more spaces. Two newlines in a row will create a new +paragraph, with a blank line in between: -### Line Breaks +```markdown +First paragraph. +Another line in the same paragraph. +A third line in the same paragraph, but this time ending with two spaces.{space}{space} +A new line directly under the first paragraph. + +Second paragraph. +Another line, this time ending with a backslash.\ +A new line due to the previous backslash. +``` -A good way to learn how line breaks work is to experiment and discover -- hit <kbd>Enter</kbd> once (i.e., insert one newline), then hit it twice (i.e., insert two newlines), see what happens. You'll soon learn to get what you want. The "Preview" tab is your friend. +<!-- (Do *NOT* remove the two ending whitespaces in the third line) --> +<!-- (They are needed for the Markdown text to render correctly) --> -Here are some things to try out: +First paragraph. +Another line in the same paragraph. +A third line in the same paragraph, but this time ending with two spaces. +A new line directly under the first paragraph. -Examples: +<!-- (Do *NOT* remove the two ending whitespaces in the second line) --> +<!-- (They are needed for the Markdown text to render correctly on docs.gitlab.com, the backslash works fine inside GitLab itself) --> -``` -Here's a line for us to start with. +Second paragraph. +Another line, this time ending with a backslash. +A new line due to the previous backslash. -This line is separated from the one above by two newlines, so it will be a *separate paragraph*. +### Links -This line is also a separate paragraph, but... -This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. +There are two ways to create links, inline-style and reference-style: -This line is also a separate paragraph, and... -This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) +```md +- This is an [inline-style link](https://www.google.com) +- This is a [link to a repository file in the same directory](index.md) +- This is a [relative link to a readme one directory higher](../README.md) +- This is a [link that also has title text](https://www.google.com "This link takes you to Google!") -spaces. -``` +Using header ID anchors: -Becomes: +- This links to [a section on a different markdown page, using a "#" and the header ID](index.md#overview) +- This links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) -Here's a line for us to start with. +Using references: -This line is separated from the one above by two newlines, so it will be a *separate paragraph*. +- This is a [reference-style link, see below][Arbitrary case-insensitive reference text] +- You can [use numbers for reference-style link definitions, see below][1] +- Or leave it empty and use the [link text itself][], see below. -This line is also a separate paragraph, but... -This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. +Some text to show that the reference links can follow later. -This line is also a separate paragraph, and... -This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) +[arbitrary case-insensitive reference text]: https://www.mozilla.org +[1]: http://slashdot.org +[link text itself]: https://www.reddit.com +``` -spaces. +- This is an [inline-style link](https://www.google.com) +- This is a [link to a repository file in the same directory](index.md) +- This is a [relative link to a readme one directory higher](../README.md) +- This is a [link that also has title text](https://www.google.com "This link takes you to Google!") -### Tables +Using header ID anchors: -Tables aren't part of the core Markdown spec, but they are part of GFM. +- This links to [a section on a different markdown page, using a "#" and the header ID](index.md#overview) +- This links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) -Example: +Using references: -``` -| header 1 | header 2 | -| -------- | -------- | -| cell 1 | cell 2 | -| cell 3 | cell 4 | -``` +- This is a [reference-style link, see below][Arbitrary case-insensitive reference text] +- You can [use numbers for reference-style link definitions, see below][1] +- Or leave it empty and use the [link text itself][], see below. -Becomes: +Some text to show that the reference links can follow later. -| header 1 | header 2 | -| -------- | -------- | -| cell 1 | cell 2 | -| cell 3 | cell 4 | +[arbitrary case-insensitive reference text]: https://www.mozilla.org +[1]: http://slashdot.org +[link text itself]: https://www.reddit.com -**Note:** The row of dashes between the table header and body must have at least three dashes in each column. +NOTE: **Note:** Relative links do not allow the referencing of project files in a wiki +page, or a wiki page in a project file. The reason for this is that a wiki is always +in a separate Git repository in GitLab. For example, `[I'm a reference-style link](style)` +will point the link to `wikis/style` only when the link is inside of a wiki markdown file. -By including colons in the header row, you can align the text within that column. +#### URL auto-linking -Example: +GFM will autolink almost any URL you put into your text: -``` -| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | -| :----------- | :------: | ------------: | :----------- | :------: | ------------: | -| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | -| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +```markdown +- https://www.google.com +- https://google.com/ +- ftp://ftp.us.debian.org/debian/ +- smb://foo/bar/baz +- irc://irc.freenode.net/gitlab +- http://localhost:3000 ``` -Becomes: +- <https://www.google.com> +- <https://google.com/> +- <ftp://ftp.us.debian.org/debian/> +- <smb://foo/bar/baz> +- <irc://irc.freenode.net/gitlab> +- <http://localhost:3000> -| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | -| :----------- | :------: | ------------: | :----------- | :------: | ------------: | -| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | -| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +### Lists -### Footnotes +Ordered and unordered lists can be easily created. Add the number you want the list +to start with, like `1. ` (with a space) at the start of each line for ordered lists. +After the first number, it does not matter what number you use, ordered lists will be +numbered automatically by vertical order, so repeating `1. ` for all items in the +same list is common. If you start with a number other than `1. `, it will use that as the first +number, and count up from there. -Example: +Add a `* `, `- ` or `+ ` (with a space) at the start of each line for unordered lists, but +you should not use a mix of them. -``` -You can add footnotes to your text as follows.[^2] -[^2]: This is my awesome footnote. +Examples: + +```md +1. First ordered list item +2. Another item + - Unordered sub-list. +1. Actual numbers don't matter, just that it's a number + 1. Ordered sub-list + 1. Next ordered sub-list item +4. And another item. + +* Unordered lists can use asterisks +- Or minuses ++ Or pluses ``` -Becomes: +1. First ordered list item +2. Another item + - Unordered sub-list. +1. Actual numbers don't matter, just that it's a number + 1. Ordered sub-list + 1. Next ordered sub-list item +4. And another item. -You can add footnotes to your text as follows.[^2] +* Unordered lists can use asterisks +- Or minuses ++ Or pluses -### Superscripts / Subscripts +--- -CommonMark and GFM currently do not support the superscript syntax ( `x^2` ) that Redcarpet does. You can use the standard HTML syntax for superscripts and subscripts. +If a list item contains multiple paragraphs, each subsequent paragraph should be indented +to the same level as the start of the list item text. -``` -The formula for water is H<sub>2</sub>O -while the equation for the theory of relativity is E = mc<sup>2</sup>. -``` +Example: -The formula for water is H<sub>2</sub>O while the equation for the theory of relativity is E = mc<sup>2</sup>. +```markdown +1. First ordered list item -## Wiki-specific Markdown + Second paragraph of first item. -The following examples show how links inside wikis behave. +2. Another item +``` -### Wiki - Direct page link +1. First ordered list item -A link which just includes the slug for a page will point to that page, -_at the base level of the wiki_. + Second paragraph of first item. -This snippet would link to a `documentation` page at the root of your wiki: +2. Another item -```markdown -[Link to Documentation](documentation) -``` +--- -### Wiki - Direct file link +If the paragraph of the first item is not indented with the proper number of spaces, +the paragraph will appear outside the list, instead of properly indented under the list item. -Links with a file extension point to that file, _relative to the current page_. +Example: -If this snippet was placed on a page at `<your_wiki>/documentation/related`, -it would link to `<your_wiki>/documentation/file.md`: +``` +1. First ordered list item -```markdown -[Link to File](file.md) + Paragraph of first item. + +2. Another item ``` -### Wiki - Hierarchical link +1. First ordered list item -A link can be constructed relative to the current wiki page using `./<page>`, -`../<page>`, etc. + Paragraph of first item. -- If this snippet was placed on a page at `<your_wiki>/documentation/main`, - it would link to `<your_wiki>/documentation/related`: +2. Another item - ```markdown - [Link to Related Page](related) - ``` +### Superscripts / Subscripts -- If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, - it would link to `<your_wiki>/documentation/main`: +CommonMark and GFM currently do not support the superscript syntax ( `x^2` ) that +Redcarpet does. You can use the standard HTML syntax for superscripts and subscripts: + +```html +The formula for water is H<sub>2</sub>O +while the equation for the theory of relativity is E = mc<sup>2</sup>. +``` - ```markdown - [Link to Related Page](../main) - ``` +The formula for water is H<sub>2</sub>O +while the equation for the theory of relativity is E = mc<sup>2</sup>. -- If this snippet was placed on a page at `<your_wiki>/documentation/main`, - it would link to `<your_wiki>/documentation/related.md`: +### Tables - ```markdown - [Link to Related Page](related.md) - ``` +Tables aren't part of the core Markdown spec, but they are part of GFM. -- If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, - it would link to `<your_wiki>/documentation/main.md`: +1. The first line contains the headers, separated by "pipes" (`|`). +1. The second line separates the headers from the cells, and must contain three or more dashes. +1. The third, and any following lines, contain the cell values. + - You **can't** have cells separated over many lines in the markdown, they must be kept to single lines, + but they can be very long. You can also include HTML `<br>` tags to force newlines if needed. + - The cell sizes **don't** have to match each other. They are flexible, but must be separated + by pipes (`|`). + - You **can** have blank cells. - ```markdown - [Link to Related Page](../main.md) - ``` +Example: -### Wiki - Root link +```markdown +| header 1 | header 2 | header 3 | +| --- | ------ |----------| +| cell 1 | cell 2 | cell 3 | +| cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It will eventually wrap the text when the cell is too large for the display size. | +| cell 7 | | cell <br> 9 | +``` -A link starting with a `/` is relative to the wiki root. +| header 1 | header 2 | header 3 | +| --- | ------ |---------:| +| cell 1 | cell 2 | cell 3 | +| cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It will eventually wrap the text when the cell is too large for the display size. | +| cell 7 | | cell <br> 9 | -- This snippet links to `<wiki_root>/documentation`: +Additionally, you can choose the alignment of text within columns by adding colons (`:`) +to the sides of the "dash" lines in the second row. This will affect every cell in the column. - ```markdown - [Link to Related Page](/documentation) - ``` +> Note that the headers are always right aligned [within GitLab itself itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#tables). -- This snippet links to `<wiki_root>/miscellaneous.md`: +```markdown +| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | +| :--- | :---: | ---: | :----------- | :------: | ------------: | +| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | +| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +``` - ```markdown - [Link to Related Page](/miscellaneous.md) - ``` +| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | +| :--- | :---: | ---: | :----------- | :------: | ------------: | +| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | +| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | ## References - This document leveraged heavily from the [Markdown-Cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet). -- The original [Markdown Syntax Guide](https://daringfireball.net/projects/markdown/syntax) at Daring Fireball is an excellent resource for a detailed explanation of standard markdown. -- The detailed specification for CommonMark can be found in the [CommonMark Spec][commonmark-spec] +- The original [Markdown Syntax Guide](https://daringfireball.net/projects/markdown/syntax) + at Daring Fireball is an excellent resource for a detailed explanation of standard markdown. +- The detailed specification for CommonMark can be found in the [CommonMark Spec](https://spec.commonmark.org/current/) - The [CommonMark Dingus](http://try.commonmark.org) is a handy tool for testing CommonMark syntax. - -[^1]: This link will be broken if you see this document from the Help page or docs.gitlab.com -[^2]: This is my awesome footnote. - -[markdown.md]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md -[rouge]: http://rouge.jneen.net/ "Rouge website" -[redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" -[katex]: https://github.com/Khan/KaTeX "KaTeX website" -[katex-subset]: https://katex.org/docs/supported.html "Macros supported by KaTeX" -[asciidoctor-manual]: http://asciidoctor.org/docs/user-manual/#activating-stem-support "Asciidoctor user manual" -[commonmarker]: https://github.com/gjtorikian/commonmarker -[commonmark-spec]: https://spec.commonmark.org/current/ diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index 66362f27299..54bf3ff8a40 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -1,9 +1,6 @@ # Operations Dashboard **[PREMIUM]** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) -in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. -[Moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/9218) to -[GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. [Moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/9218) to [GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. The Operations Dashboard provides a summary of each project's operational health, including pipeline and alert status. @@ -16,9 +13,9 @@ dashboard icon: ## Adding a project to the dashboard NOTE: **Note:** -For GitLab.com, the Operations Dashboard is available for free for public projects. -If your project is private, the group it belongs to must have a -[Gold](https://about.gitlab.com/pricing/) plan. +For GitLab.com, you can add your project to the Operations Dashboard for free if +your project is public. If your project is private, the group it belongs to must +have a [Silver](https://about.gitlab.com/pricing/) plan. To add a project to the dashboard: diff --git a/doc/user/permissions.md b/doc/user/permissions.md index a6e2f187090..03abef9fc62 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -17,7 +17,10 @@ will be unassigned automatically. GitLab [administrators](../administration/index.md) receive all permissions. To add or import a user, you can follow the -[project members documentation](../user/project/members/index.md). +[project members documentation](project/members/index.md). + +For information on eligible approvers for Merge Requests, see +[Eligible approvers](project/merge_requests/merge_request_approvals.md#eligible-approvers). ## Principles behind permissions @@ -92,6 +95,7 @@ The following table depicts the various user permission levels in a project. | Dismiss vulnerability **[ULTIMATE]** | | | ✓ | ✓ | ✓ | | Apply code change suggestions | | | ✓ | ✓ | ✓ | | Create and edit wiki pages | | | ✓ | ✓ | ✓ | +| Rewrite/remove Git tags | | | ✓ | ✓ | ✓ | | Use environment terminals | | | | ✓ | ✓ | | Run Web IDE's Interactive Web Terminals **[ULTIMATE ONLY]** | | | | ✓ | ✓ | | Add new team members | | | | ✓ | ✓ | @@ -99,7 +103,6 @@ The following table depicts the various user permission levels in a project. | Push to protected branches | | | | ✓ | ✓ | | Turn on/off protected branch push for devs | | | | ✓ | ✓ | | Enable/disable tag protections | | | | ✓ | ✓ | -| Rewrite/remove Git tags | | | | ✓ | ✓ | | Edit project | | | | ✓ | ✓ | | Add deploy keys to project | | | | ✓ | ✓ | | Configure project hooks | | | | ✓ | ✓ | @@ -196,21 +199,22 @@ Any user can remove themselves from a group, unless they are the last Owner of the group. The following table depicts the various user permission levels in a group. -| Action | Guest | Reporter | Developer | Maintainer | Owner | -|---------------------------------------|-------|----------|-----------|------------|-------| -| Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | -| View Insights charts **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View group epic **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | -| Create/edit group epic **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | -| Manage group labels | | ✓ | ✓ | ✓ | ✓ | -| Create project in group | | | ✓ | ✓ | ✓ | -| Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | -| Edit group | | | | | ✓ | -| Create subgroup | | | | | ✓ | -| Manage group members | | | | | ✓ | -| Remove group | | | | | ✓ | -| Delete group epic **[ULTIMATE]** | | | | | ✓ | -| View group Audit Events | | | | | ✓ | +| Action | Guest | Reporter | Developer | Maintainer | Owner | +|-------------------------------------------------|-------|----------|-----------|------------|-------| +| Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Insights charts **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View group epic **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | +| Create/edit group epic **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | +| Manage group labels | | ✓ | ✓ | ✓ | ✓ | +| Create project in group | | | ✓ | ✓ | ✓ | +| Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | +| Enable/disable a dependency proxy **[PREMIUM]** | | | ✓ | ✓ | ✓ | +| Edit group | | | | | ✓ | +| Create subgroup | | | | | ✓ | +| Manage group members | | | | | ✓ | +| Remove group | | | | | ✓ | +| Delete group epic **[ULTIMATE]** | | | | | ✓ | +| View group Audit Events | | | | | ✓ | ### Subgroup permissions @@ -361,3 +365,8 @@ for details about the pipelines security model. Since GitLab 8.15, LDAP user permissions can now be manually overridden by an admin user. Read through the documentation on [LDAP users permissions](../administration/auth/how_to_configure_ldap_gitlab_ee/index.html) to learn more. + +## Project aliases + +Project aliases can only be read, created and deleted by a GitLab administrator. +Read through the documentation on [Project aliases](../user/project/index.md#project-aliases-premium-only) to learn more. diff --git a/doc/user/profile/account/img/2fa.png b/doc/user/profile/account/img/2fa.png Binary files differdeleted file mode 100644 index bb464efa685..00000000000 --- a/doc/user/profile/account/img/2fa.png +++ /dev/null diff --git a/doc/user/profile/account/img/2fa_auth.png b/doc/user/profile/account/img/2fa_auth.png Binary files differdeleted file mode 100644 index 0caaed10805..00000000000 --- a/doc/user/profile/account/img/2fa_auth.png +++ /dev/null diff --git a/doc/user/profile/account/img/2fa_u2f_authenticate.png b/doc/user/profile/account/img/2fa_u2f_authenticate.png Binary files differdeleted file mode 100644 index ff2e936764d..00000000000 --- a/doc/user/profile/account/img/2fa_u2f_authenticate.png +++ /dev/null diff --git a/doc/user/profile/account/img/2fa_u2f_register.png b/doc/user/profile/account/img/2fa_u2f_register.png Binary files differdeleted file mode 100644 index 1cc142aa851..00000000000 --- a/doc/user/profile/account/img/2fa_u2f_register.png +++ /dev/null diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index df413a11af0..e3e8c9a0d6d 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -10,17 +10,18 @@ is to know your username and password *and* have access to your one time passwor ## Overview -> **Note:** -When you enable 2FA, don't forget to back up your recovery codes. - -In addition to one time authenticators (TOTP), GitLab supports U2F (universal 2nd factor) devices as -the second factor of authentication. Once enabled, in addition to supplying your username and -password to login, you'll be prompted to activate your U2F device (usually by pressing -a button on it), and it will perform secure authentication on your behalf. - -The U2F workflow is [supported by](https://caniuse.com/#search=U2F) Google Chrome, Opera, and Firefox. - -We recommend that you set up 2FA with both a [one-time password authenticator](#enable-2fa-via-one-time-password-authenticator) and a [U2F device](#enable-2fa-via-u2f-device), so you can still access your account +TIP: **Tip:** +When you enable 2FA, don't forget to back up your [recovery codes](#recovery-codes)! + +In addition to time-based one time passwords (TOTP), GitLab supports U2F +(universal 2nd factor) devices as the second factor of authentication. Once +enabled, in addition to supplying your username and password to login, you'll +be prompted to activate your U2F device (usually by pressing a button on it), +and it will perform secure authentication on your behalf. + +It is highly recommended that you set up 2FA with both a +[one-time password authenticator](#enable-2fa-via-one-time-password-authenticator) +and a [U2F device](#enable-2fa-via-u2f-device), so you can still access your account if you lose your U2F device. ## Enabling 2FA @@ -30,41 +31,52 @@ or a U2F device. ### Enable 2FA via one time password authenticator -**In GitLab:** - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Account**. -1. Click **Enable Two-factor Authentication**. - - - -**On your phone:** - -1. Install a compatible application. We recommend [Google Authenticator] - \(proprietary\) or [FreeOTP] \(open source\). -1. In the application, add a new entry in one of two ways: - - Scan the code with your phone's camera to add the entry automatically. - - Enter the details provided to add the entry manually. - -**In GitLab:** - -1. Enter the six-digit pin number from the entry on your phone into the **Pin - code** field. -1. Click **Submit**. +To enable 2FA: + +1. **In GitLab:** + 1. Log in to your GitLab account. + 1. Go to your **Profile Settings**. + 1. Go to **Account**. + 1. Click **Enable Two-factor Authentication**. +1. **On your device (usually your phone):** + 1. Install a compatible application, like: + - [Authenticator](https://mattrubin.me/authenticator/): open source app for iOS devices. + - [andOTP](https://github.com/andOTP/andOTP): feature rich open source app for Android which supports PGP encrypted backups. + - [FreeOTP](https://freeotp.github.io/): open source app for Android. + - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en): proprietary app for iOS and Android. + 1. In the application, add a new entry in one of two ways: + - Scan the code presented in GitLab with your device's camera to add the + entry automatically. + - Enter the details provided to add the entry manually. +1. **In GitLab:** + 1. Enter the six-digit pin number from the entry on your device into the **Pin + code** field. + 1. Click **Submit**. If the pin you entered was correct, you'll see a message indicating that Two-Factor Authentication has been enabled, and you'll be presented with a list -of recovery codes. +of [recovery codes](#recovery-codes). Make sure you download them and keep them +in a safe place. ### Enable 2FA via U2F device -> **Notes:** -> -> - GitLab officially only supports [Yubikey] U2F devices. -> - Support for U2F devices was added in GitLab 8.8. +GitLab officially only supports [YubiKey](https://www.yubico.com/products/yubikey-hardware/) +U2F devices, but users have successfully used [SoloKeys](https://solokeys.com/). + +The U2F workflow is [supported by](https://caniuse.com/#search=U2F) the +following desktop browsers: -**In GitLab:** +- Chrome +- Edge +- Firefox (disabled by default) +- Opera + +NOTE: **Note:** +For Firefox, you can enable the FIDO U2F API in +[about:config](https://support.mozilla.org/en-US/kb/about-config-editor-firefox). +Search for `security.webauth.u2f` and double click on it to toggle to `true`. + +To set up 2FA with a U2F device: 1. Log in to your GitLab account. 1. Go to your **Profile Settings**. @@ -77,20 +89,21 @@ of recovery codes. You will see a message indicating that your device was successfully set up. Click on **Register U2F Device** to complete the process. - +## Recovery codes -## Recovery Codes - -> **Note:** +NOTE: **Note:** Recovery codes are not generated for U2F devices. -Should you ever lose access to your one time password authenticator, you can use one of the ten provided -backup codes to login to your account. We suggest copying them, printing them, or downloading them using -the **Download codes** button for storage in a safe place. - CAUTION: **Caution:** Each code can be used only once to log in to your account. +Immediately after successfully enabling two-factor authentication, you'll be +prompted to download a set of set recovery codes. Should you ever lose access +to your one time password authenticator, you can use one of them to log in to +your account. We suggest copying them, printing them, or downloading them using +the **Download codes** button for storage in a safe place. If you choose to +download them, the file will be called **gitlab-recovery-codes.txt**. + If you lose the recovery codes or just want to generate new ones, you can do so [using SSH](#generate-new-recovery-codes-using-ssh). @@ -100,24 +113,26 @@ Logging in with 2FA enabled is only slightly different than a normal login. Enter your username and password credentials as you normally would, and you'll be presented with a second prompt, depending on which type of 2FA you've enabled. -### Log in via mobile application - -Enter the pin from your one time password authenticator's application or a recovery code to log in. +### Log in via a one-time password - +When asked, enter the pin from your one time password authenticator's application or a +recovery code to log in. ### Log in via U2F device -1. Click **Login via U2F Device**. -1. A light will start blinking on your device. Activate it by pressing its button. +To log in via a U2F device: -You will see a message indicating that your device responded to the authentication request. -Click on **Authenticate via U2F Device** to complete the process. +1. Click **Login via U2F Device**. +1. A light will start blinking on your device. Activate it by touching/pressing + its button. - +You will see a message indicating that your device responded to the authentication +request and you will be automatically logged in. ## Disabling 2FA +If you ever need to disable 2FA: + 1. Log in to your GitLab account. 1. Go to your **Profile Settings**. 1. Go to **Account**. @@ -130,7 +145,8 @@ applications and U2F devices. When 2FA is enabled, you can no longer use your normal account password to authenticate with Git over HTTPS on the command line or when using -[GitLab's API][api], you must use a [personal access token][pat] instead. +[GitLab's API](../../../api/README.md). You must use a +[personal access token](../personal_access_tokens.md) instead. ## Recovery options @@ -149,7 +165,6 @@ codes. If you saved these codes, you can use one of them to sign in. To use a recovery code, enter your username/email and password on the GitLab sign-in page. When prompted for a two-factor code, enter the recovery code. -> **Note:** Once you use a recovery code, you cannot re-use it. You can still use the other recovery codes you saved. @@ -157,13 +172,18 @@ recovery codes you saved. Users often forget to save their recovery codes when enabling two-factor authentication. If an SSH key is added to your GitLab account, you can generate -a new set of recovery codes with SSH. +a new set of recovery codes with SSH: -1. Run `ssh git@gitlab.example.com 2fa_recovery_codes`. -1. You are prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes. +1. Run: + + ```sh + ssh git@gitlab.example.com 2fa_recovery_codes + ``` + +1. You will then be prompted to confirm that you want to generate new codes. + Continuing this process invalidates previously saved codes: ```sh - $ ssh git@gitlab.example.com 2fa_recovery_codes Are you sure you want to generate new two-factor recovery codes? Any existing recovery codes you saved will be invalidated. (yes/no) @@ -191,7 +211,6 @@ a new set of recovery codes with SSH. When prompted for a two-factor code, enter one of the recovery codes obtained from the command-line output. -> **Note:** After signing in, visit your **Profile settings > Account** immediately to set up two-factor authentication with a new device. @@ -219,9 +238,3 @@ Sign in and re-enable two-factor authentication as soon as possible. - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because the U2F key has only been registered on `first.host.xyz`. - -[Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en -[FreeOTP]: https://freeotp.github.io/ -[YubiKey]: https://www.yubico.com/products/yubikey-hardware/ -[api]: ../../../api/README.md -[pat]: ../personal_access_tokens.md diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index dc21db603d6..c6ee168bad0 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -71,7 +71,6 @@ new Kubernetes cluster to your project: - **Number of nodes** - Enter the number of nodes you wish the cluster to have. - **Machine type** - The [machine type](https://cloud.google.com/compute/docs/machine-types) of the Virtual Machine instance that the cluster will be based on. - - **RBAC-enabled cluster** - Leave this checked if using default GKE creation options, see the [RBAC section](#rbac-cluster-resources) for more information. - **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. See the [Managed clusters section](#gitlab-managed-clusters) for more information. 1. Finally, click the **Create Kubernetes cluster** button. @@ -86,6 +85,9 @@ account](#access-controls). Starting from [GitLab creation process will explicitly request that basic authentication and client certificate is enabled. +NOTE: **Note:** +Starting from [GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/55902), all GKE clusters created by GitLab are RBAC enabled. Take a look at the [RBAC section](#rbac-cluster-resources) for more information. + ## Adding an existing Kubernetes cluster To add an existing Kubernetes cluster to your project: @@ -225,10 +227,6 @@ applications running on the cluster. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22011) in GitLab 11.5. > Became [optional](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26565) in GitLab 11.11. -NOTE: **Note:** -Only available when creating clusters. Existing clusters not managed by GitLab -cannot become GitLab-managed later. - You can choose to allow GitLab to manage your cluster for you. If your cluster is managed by GitLab, resources for your projects will be automatically created. See the [Access controls](#access-controls) section for details on which resources will @@ -521,10 +519,11 @@ service account of the cluster integration. ### Troubleshooting failed deployment jobs -GitLab will create a namespace and service account specifically for your -deployment jobs. On project level clusters, this happens when the cluster -is created. On group level clusters, resources are created immediately -before the deployment job starts. +Before the deployment jobs starts, GitLab creates the following specifically for +the deployment job: + +- A namespace. +- A service account. However, sometimes GitLab can not create them. In such instances, your job will fail with the message: @@ -534,14 +533,20 @@ This job failed because the necessary resources were not successfully created. To find the cause of this error when creating a namespace and service account, check the [logs](../../../administration/logs.md#kuberneteslog). -Common reasons for failure include: +Reasons for failure include: -- The token you gave GitLab did not have [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) +- The token you gave GitLab does not have [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) privileges required by GitLab. - Missing `KUBECONFIG` or `KUBE_TOKEN` variables. To be passed to your job, they must have a matching [`environment:name`](../../../ci/environments.md#defining-environments). If your job has no `environment:name` set, it will not be passed the Kubernetes credentials. +NOTE: **NOTE:** +Project-level clusters upgraded from GitLab 12.0 or older may be configured +in a way that causes this error. Ensure you deselect the +[GitLab-managed cluster](#gitlab-managed-clusters) option if you want to manage +namespaces and service accounts yourself. + ## Monitoring your Kubernetes cluster **[ULTIMATE]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4701) in [GitLab Ultimate][ee] 10.6. diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index 368031070c1..25d8abebf07 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -12,9 +12,9 @@ By displaying the logs directly in GitLab, developers can avoid having to manage 1. Go to **Operations > Environments** and find the environment which contains the desired pod, like `production`. 1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). 1. When mousing over the list of pods, a tooltip will appear with the exact pod name and status. - +  1. Click on the desired pod to bring up the logs view, which will contain the last 500 lines for that pod. Support for pods with multiple containers is coming [in a future release](https://gitlab.com/gitlab-org/gitlab-ee/issues/6502). - +  ## Requirements diff --git a/doc/user/project/clusters/serverless/img/function-endpoint.png b/doc/user/project/clusters/serverless/img/function-endpoint.png Binary files differnew file mode 100644 index 00000000000..f3c7ae7a00d --- /dev/null +++ b/doc/user/project/clusters/serverless/img/function-endpoint.png diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index 2cda850f24e..a06c3d3c662 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -83,6 +83,70 @@ NOTE: **Note:** You can deploy either [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) on a given project but not both. The current implementation makes use of a `serverless.yml` file to signal a FaaS project. +## Using an existing installation of Knative + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/58941) in GitLab 12.0. + +NOTE: **Note:** +The "invocations" monitoring feature of GitLab serverless will not work when +adding an existing installation of Knative. + +It is also possible to use GitLab Serverless with an existing Kubernetes +cluster which already has Knative installed. + +You must do the following: + +1. Follow the steps to + [add an existing Kubernetes cluster](../index.md#adding-an-existing-kubernetes-cluster). + +1. Ensure GitLab can manage Knative: + - For a non-GitLab managed cluster, ensure that the service account for the token + provided can manage resources in the `serving.knative.dev` API group. + - For a GitLab managed cluster, + GitLab uses a service account with the `edit` cluster role. This account needs + the ability to manage resources in the `serving.knative.dev` API group. + We suggest you do this with an [aggregated ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) + adding rules to the default `edit` cluster role: + First, save the following YAML as `knative-serving-only-role.yaml`: + + ```yaml + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: knative-serving-only-role + labels: + rbac.authorization.k8s.io/aggregate-to-edit: "true" + rules: + - apiGroups: + - serving.knative.dev + resources: + - configurations + - configurationgenerations + - routes + - revisions + - revisionuids + - autoscalers + - services + verbs: + - get + - list + - create + - update + - delete + - patch + - watch + ``` + + Then run the following command: + + ```bash + kubectl apply -f knative-serving-only-role.yaml + ``` + +1. Follow the steps to deploy [functions](#deploying-functions) + or [serverless applications](#deploying-serverless-applications) onto your + cluster. + ## Deploying functions > Introduced in GitLab 11.6. @@ -306,3 +370,275 @@ loading or is not available at this time._ It will appear upon the first access page, but should go away after a few seconds. If the message does not disappear, then it is possible that GitLab is unable to connect to the Prometheus instance running on the cluster. + +## Enabling TLS for Knative services + +By default, a GitLab serverless deployment will be served over `http`. In order to serve over `https` you +must manually obtain and install TLS certificates. + +The simplest way to accomplish this is to +use [Certbot to manually obtain Let's Encrypt certificates](https://knative.dev/docs/serving/using-a-tls-cert/#using-certbot-to-manually-obtain-let-s-encrypt-certificates). Certbot is a free, open source software tool for automatically using Let’s Encrypt certificates on manually-administrated websites to enable HTTPS. + +NOTE: **Note:** +The instructions below relate to installing and running Certbot on a Linux server and may not work on other operating systems. + +1. Install Certbot by running the + [`certbot-auto` wrapper script](https://certbot.eff.org/docs/install.html#certbot-auto). + On the command line of your server, run the following commands: + + ```sh + wget https://dl.eff.org/certbot-auto + sudo mv certbot-auto /usr/local/bin/certbot-auto + sudo chown root /usr/local/bin/certbot-auto + chmod 0755 /usr/local/bin/certbot-auto + /usr/local/bin/certbot-auto --help + ``` + + To check the integrity of the `certbot-auto` script, run: + + ```sh + wget -N https://dl.eff.org/certbot-auto.asc + gpg2 --keyserver ipv4.pool.sks-keyservers.net --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc /usr/local/bin/certbot-auto + ``` + + The output of the last command should look something like: + + ```sh + gpg: Signature made Mon 10 Jun 2019 06:24:40 PM EDT + gpg: using RSA key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + gpg: key 4D17C995CD9775F2 marked as ultimately trusted + gpg: checking the trustdb + gpg: marginals needed: 3 completes needed: 1 trust model: pgp + gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u + gpg: next trustdb check due at 2027-11-22 + gpg: Good signature from "Let's Encrypt Client Team <letsencrypt-client@eff.org>" [ultimate] + ``` + +1. Run the following command to use Certbot to request a certificate + using DNS challenge during authorization: + + + ```sh + ./certbot-auto certonly --manual --preferred-challenges dns -d '*.<namespace>.example.com' + ``` + + Where `<namespace>` is the namespace created by GitLab for your serverless project (composed of `<projectname+id>`) and + `example.com` is the domain being used for your project. If you are unsure what the namespace of your project is, navigate + to the **Operations > Serverless** page of your project and inspect + the endpoint provided for your function/app. + +  + + In the above image, the namespace for the project is `node-function-11909507` and the domain is `knative.info`, thus + certificate request line would look like this: + + ```sh + ./certbot-auto certonly --manual --preferred-challenges dns -d '*.node-function-11909507.knative.info' + ``` + + The Certbot tool walks you through the steps of validating that you own each domain that you specify by creating TXT records in those domains. + After this process is complete, the output should look something like this: + + ```sh + IMPORTANT NOTES: + - Congratulations! Your certificate and chain have been saved at: + /etc/letsencrypt/live/namespace.example.com/fullchain.pem + Your key file has been saved at: + /etc/letsencrypt/live/namespace.example/privkey.pem + Your cert will expire on 2019-09-19. To obtain a new or tweaked + version of this certificate in the future, simply run certbot-auto + again. To non-interactively renew *all* of your certificates, run + "certbot-auto renew" + -----BEGIN PRIVATE KEY----- + - Your account credentials have been saved in your Certbot + configuration directory at /etc/letsencrypt. You should make a + secure backup of this folder now. This configuration directory will + also contain certificates and private keys obtained by Certbot so + making regular backups of this folder is ideal. + ``` + +1. Create certificate and private key files. Using the contents of the files + returned by Certbot, we'll create two files in order to create the + Kubernetes secret: + + Run the following command to see the contents of `fullchain.pem`: + + ```sh + sudo cat /etc/letsencrypt/live/node-function-11909507.knative.info/fullchain.pem + ``` + + Output should look like this: + + ```sh + -----BEGIN CERTIFICATE----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b4ag== + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + K2fcb195768c39e9a94cec2c2e30Qg== + -----END CERTIFICATE----- + ``` + + Create a file with the name `cert.pem` with the contents of the entire output. + + Once `cert.pem` is created, run the following command to see the contents of `privkey.pem`: + + ```sh + sudo cat /etc/letsencrypt/live/namespace.example/privkey.pem + ``` + + Output should look like this: + + ```sh + -----BEGIN PRIVATE KEY----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + -----BEGIN CERTIFICATE----- + fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 4f294d1eaca42b8692017b4262== + -----END PRIVATE KEY----- + ``` + + Create a new file with the name `cert.pk` with the contents of the entire output. + +1. Create a Kubernetes secret to hold your TLS certificate, `cert.pem`, and + the private key `cert.pk`: + + NOTE: **Note:** + Running `kubectl` commands on your cluster requires setting up access to the cluster first. + For clusters created on GKE, see + [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl). + For other platforms, [install `kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + + ```sh + kubectl create --namespace istio-system secret tls istio-ingressgateway-certs \ + --key cert.pk \ + --cert cert.pem + ``` + + Where `cert.pem` and `cert.pk` are your certificate and private key files. Note that the `istio-ingressgateway-certs` secret name is required. + +1. Configure Knative to use the new secret that you created for HTTPS + connections. Run the + following command to open the Knative shared `gateway` in edit mode: + + ```sh + kubectl edit gateway knative-ingress-gateway --namespace knative-serving + ``` + + Update the gateway to include the following tls: section and configuration: + + ```sh + tls: + mode: SIMPLE + privateKey: /etc/istio/ingressgateway-certs/tls.key + serverCertificate: /etc/istio/ingressgateway-certs/tls.crt + ``` + + Example: + + ```sh + apiVersion: networking.istio.io/v1alpha3 + kind: Gateway + metadata: + # ... skipped ... + spec: + selector: + istio: ingressgateway + servers: + - hosts: + - "*" + port: + name: http + number: 80 + protocol: HTTP + - hosts: + - "*" + port: + name: https + number: 443 + protocol: HTTPS + tls: + mode: SIMPLE + privateKey: /etc/istio/ingressgateway-certs/tls.key + serverCertificate: /etc/istio/ingressgateway-certs/tls.crt + ``` + + After your changes are running on your Knative cluster, you can begin using the HTTPS protocol for secure access your deployed Knative services. + In the event a mistake is made during this process and you need to update the cert, you will need to edit the gateway `knative-ingress-gateway` + to switch back to `PASSTHROUGH` mode. Once corrections are made, edit the file again so the gateway will use the new certificates.
\ No newline at end of file diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md index 58b7fe33906..fdf9ce3e225 100644 --- a/doc/user/project/container_registry.md +++ b/doc/user/project/container_registry.md @@ -113,6 +113,7 @@ This feature requires GitLab 8.8 and GitLab Runner 1.2. Make sure that your GitLab Runner is configured to allow building Docker images by following the [Using Docker Build](../../ci/docker/using_docker_build.md) and [Using the GitLab Container Registry documentation](../../ci/docker/using_docker_build.md#using-the-gitlab-container-registry). +Alternatively, you can [build images with Kaniko](../../ci/docker/using_kaniko.md) if the Docker builds are not an option for you. ## Using with private projects diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index 2f2a9c5eec3..175384bc985 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -59,6 +59,8 @@ specific environment, there are lot of uses cases. To name a few: To display the Deploy Boards for a specific [environment] you should: +1. Have [defined an environment](../../ci/environments.md#defining-environments) with a deploy stage. + 1. Have a Kubernetes cluster up and running. NOTE: **Running on OpenShift:** @@ -86,8 +88,10 @@ To display the Deploy Boards for a specific [environment] you should: Kubernetes. NOTE: **Note:** - The Kubernetes label of `app` is deprecated and may be removed in next major - release, GitLab 12.0. + Matching based on the Kubernetes `app` label was removed in [GitLab + 12.1](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14020). + To migrate, please apply the required annotations (see above) and + re-deploy your application.  @@ -96,12 +100,6 @@ navigate to the environments page under **Operations > Environments**. Deploy Boards are visible by default. You can explicitly click the triangle next to their respective environment name in order to hide them. -GitLab will then query Kubernetes for the state of each pod (e.g., waiting, -deploying, finished, unknown), and the Deploy Board status will finally appear. - -GitLab will only display a Deploy Board for top-level environments. Foldered -environments like `review/*` (usually used for [Review Apps]) won't have a -Deploy Board attached to them. ## Canary Deployments diff --git a/doc/user/project/deploy_tokens/img/deploy_tokens.png b/doc/user/project/deploy_tokens/img/deploy_tokens.png Binary files differindex 55c537fd1d3..421aa1ab3e5 100644 --- a/doc/user/project/deploy_tokens/img/deploy_tokens.png +++ b/doc/user/project/deploy_tokens/img/deploy_tokens.png diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 92a29b68a22..5e11e7c0203 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -15,7 +15,7 @@ You can create as many deploy tokens as you like from the settings of your proje 1. Go to the project you want to create Deploy Tokens for. 1. Go to **Settings** > **Repository**. 1. Click on "Expand" on **Deploy Tokens** section. -1. Choose a name and optionally an expiry date for the token. +1. Choose a name, expiry date (optional), and username (optional) for the token. 1. Choose the [desired scopes](#limiting-scopes-of-a-deploy-token). 1. Click on **Create deploy token**. 1. Save the deploy token somewhere safe. Once you leave or refresh @@ -39,6 +39,13 @@ the following table. | `read_repository` | Allows read-access to the repository through `git clone` | | `read_registry` | Allows read-access to [container registry] images if a project is private and authorization is required. | +## Deploy token custom username + +> [Introduced][https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29639] in GitLab 12.1. + +The default username format is `gitlab+deploy-token-#{n}`. Some tools or platforms may not support this format, +in such case you can specify custom username to be used when creating the deploy token. + ## Usage ### Git clone a repository diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 05ad15476ab..7520237251a 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -25,14 +25,14 @@ templates of the default branch will be taken into account. ## Use-cases - Add a template to be used in every issue for a specific project, -giving instructions and guidelines, requiring for information specific to that subject. -For example, if you have a project for tracking new blog posts, you can require the -title, outlines, author name, author social media information, etc. + giving instructions and guidelines, requiring for information specific to that subject. + For example, if you have a project for tracking new blog posts, you can require the + title, outlines, author name, author social media information, etc. - Following the previous example, you can make a template for every MR submitted -with a new blog post, requiring information about the post date, frontmatter data, -images guidelines, link to the related issue, reviewer name, etc. + with a new blog post, requiring information about the post date, frontmatter data, + images guidelines, link to the related issue, reviewer name, etc. - You can also create issues and merge request templates for different -stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. + stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. ## Creating issue templates @@ -58,6 +58,7 @@ changes you made after picking the template and return it to its initial status. ## Setting a default template for issues and merge requests **[STARTER]** > **Notes:** +> > - This feature was introduced before [description templates](#overview) and is > available in [GitLab Starter][products]. It can be enabled > in the project's settings. @@ -65,7 +66,7 @@ changes you made after picking the template and return it to its initial status. > - Templates for merge requests were [introduced][ee-7478ece] in GitLab EE 6.9. The visibility of issues and/or merge requests should be set to either "Everyone -with access" or "Only team members" in your project's **Settings** otherwise the +with access" or "Only Project Members" in your project's **Settings** otherwise the template text areas won't show. This is the default behavior so in most cases you should be fine. diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 3386eb9d0d4..40603790c12 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -1,8 +1,6 @@ # File Locking **[PREMIUM]** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/440) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.9. -> - This feature needs to have a license with the "File Lock" option enabled. -> - If you are using Premium but you don't see the "Lock" button, ask your GitLab administrator. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/440) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.9. File Locking helps you avoid merge conflicts and better manage your binary files. Lock any file or directory, make your changes, and then unlock it so another @@ -30,12 +28,51 @@ The file locking feature is useful in situations when: Locked directories are locked recursively, which means that everything that lies under them is also locked. +## Locking a file or a directory + +NOTE: **Note:** +Locking only works for the default branch you have set in the project's settings +(usually `master`). + +To lock a file: + +1. Navigate to your project's **Repository > Files**. +1. Pick the file you want to lock. +1. Click the "Lock" button. + +  + +To lock an entire directory, look for the "Lock" link next to "History". + +After you lock a file or directory, it will appear as locked in the repository +view. + + + +Once locked, any merge request to the default branch will fail +to merge until the file becomes unlocked. + +## Unlocking a file or a directory + +To unlock a file or a directory, follow the same procedure as when you locked +them. For a detailed view of every existing lock, see the next section on +"Viewing and managing existing locks". + +You can unlock a file that yourself or someone else previously locked as long +as you have Maintainer or above [permissions](../permissions.md) to the project. + +## Viewing and managing existing locks + +To view or manage every existing lock, navigate to the +**Project > Repository > Locked Files** area. There, you can view all existing +locks and [remove the ones you have permission for](#permissions-on-file-locking). + ## Permissions on file locking The user that locks a file or directory **is the only one** that can edit and push their changes back to the repository where the locked objects are located. -Locks can be created by any person who has [push access](../../user/permissions.md) to the repository; i.e., +Locks can be created by any person who has [push access](../permissions.md) to the repository; i.e., Developer and higher level, and can be removed solely by their author and any user with Maintainer permissions and above. @@ -61,41 +98,3 @@ accepts a merge request, an error message will appear stating that the file is locked.  - -## Locking a file or a directory - ->**Note:** -Locking only works for the default branch you have set in the project's settings -(usually `master`). - -To lock a file, navigate to the repository tree under the **Repository > Files** tab, -pick the file you want to lock and hit the "Lock" button. - - - ---- - -To lock an entire directory, look for the "Lock" link next to "History". - - - ---- - -After you lock a file or directory, it will appear as locked in the repository -view. - - - -## Unlocking a file or a directory - -To unlock a file or a directory, follow the same procedure as when you locked -them. For a detailed view of every existing lock, see the next section on -"Viewing and managing existing locks". - -## Viewing and managing existing locks - -To view or manage every existing lock, navigate to the -**Project > Repository > Locked Files** area. There, you can view all existing -locks and [remove the ones you have permission for](#permissions-on-file-locking). - - diff --git a/doc/user/project/img/file_lock.png b/doc/user/project/img/file_lock.png Binary files differindex 33f96f20e91..82699a12ffd 100644 --- a/doc/user/project/img/file_lock.png +++ b/doc/user/project/img/file_lock.png diff --git a/doc/user/project/img/file_lock_folders.png b/doc/user/project/img/file_lock_folders.png Binary files differdeleted file mode 100644 index 5ff3d4d9dbd..00000000000 --- a/doc/user/project/img/file_lock_folders.png +++ /dev/null diff --git a/doc/user/project/img/file_lock_list.png b/doc/user/project/img/file_lock_list.png Binary files differdeleted file mode 100644 index 2c276335c83..00000000000 --- a/doc/user/project/img/file_lock_list.png +++ /dev/null diff --git a/doc/user/project/img/file_lock_merge_request_error_message.png b/doc/user/project/img/file_lock_merge_request_error_message.png Binary files differindex 65bc3692da0..4ef04b15bef 100644 --- a/doc/user/project/img/file_lock_merge_request_error_message.png +++ b/doc/user/project/img/file_lock_merge_request_error_message.png diff --git a/doc/user/project/img/file_lock_repository_view.png b/doc/user/project/img/file_lock_repository_view.png Binary files differindex 8f4ef52aacd..a2cab0decab 100644 --- a/doc/user/project/img/file_lock_repository_view.png +++ b/doc/user/project/img/file_lock_repository_view.png diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md index 7f79ebf6353..3b071ff590f 100644 --- a/doc/user/project/import/gemnasium.md +++ b/doc/user/project/import/gemnasium.md @@ -40,14 +40,14 @@ some steps to migrate your projects. There is no automatic import since GitLab doesn't know anything about any projects which existed on Gemnasium.com. Security features are free for public (open-source) projects hosted on GitLab.com. -### If your project is hosted on GitLab (https://gitlab.com / self-hosted) +### If your project is hosted on GitLab (`https://gitlab.com` / self-hosted) You're almost set! If you're already using [Auto DevOps](../../../topics/autodevops/), you are already covered. Otherwise, you must configure your `.gitlab-ci.yml` according to the [dependency scanning page](../../application_security/dependency_scanning/index.md). -### If your project is hosted on GitHub (https://github.com / GitHub Enterprise) +### If your project is hosted on GitHub (`https://github.com` / GitHub Enterprise) Since [GitLab 10.6 comes with GitHub integration](https://about.gitlab.com/features/github/), GitLab users can now create a CI/CD project in GitLab connected to an external diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 587b4121e4e..06286951e20 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -193,6 +193,28 @@ password <personal_access_token> To quickly access a project from the GitLab UI using the project ID, visit the `/projects/:id` URL in your browser or other tool accessing the project. +## Project aliases **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3264) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.1. + +When migrating repositories to GitLab and they are being accessed by other systems, +it's very useful to be able to access them using the same name especially when +they are a lot. It reduces the risk of changing significant number of Git URLs in +a large number of systems. + +GitLab provides a functionality to help with this. In GitLab, repositories are +usually accessed with a namespace and project name. It is also possible to access +them via a project alias. This feature is only available on Git over SSH. + +A project alias can be only created via API and only by GitLab administrators. +Follow the [Project Aliases API documentation](../../api/project_aliases.md) for +more details. + +Once an alias has been created for a project (e.g., an alias `gitlab-ce` for the +project `https://gitlab.com/gitlab-org/gitlab-ce`), the repository can be cloned +using the alias (e.g `git clone git@gitlab.com:gitlab-ce.git` instead of +`git clone git@gitlab.com:gitlab-org/gitlab-ce.git`). + ## Project APIs There are numerous [APIs](../../api/README.md) to use with your projects: @@ -212,3 +234,4 @@ There are numerous [APIs](../../api/README.md) to use with your projects: - [Templates](../../api/project_templates.md) - [Traffic](../../api/project_statistics.md) - [Variables](../../api/project_level_variables.md) +- [Aliases](../../api/project_aliases.md) diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md index 2e2a27f112e..1c6ad0b8b2b 100644 --- a/doc/user/project/insights/index.md +++ b/doc/user/project/insights/index.md @@ -1,7 +1,6 @@ # Insights **[ULTIMATE]** -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. -> **Generally Available** (GA) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. Configure the Insights that matter for your projects to explore data such as triage hygiene, issues created/closed per a given period, average time for merge diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index cdb0e34fdf6..680fcdb78bb 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -17,7 +17,7 @@ and is automatically configured on [GitHub import](../../../integration/github.m This integration requires a [GitHub API token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) with `repo:status` access granted: -1. Go to your "Personal access tokens" page at https://github.com/settings/tokens +1. Go to your "Personal access tokens" page at <https://github.com/settings/tokens> 1. Click "Generate New Token" 1. Ensure that `repo:status` is checked and click "Generate token" 1. Copy the generated token to use on GitLab diff --git a/doc/user/project/integrations/hipchat.md b/doc/user/project/integrations/hipchat.md index 0fd847d415f..7a0540aa9e3 100644 --- a/doc/user/project/integrations/hipchat.md +++ b/doc/user/project/integrations/hipchat.md @@ -23,7 +23,7 @@ allow GitLab to send messages only to *one* room. 1. Find "Build Your Own!" and click "Create". 1. Select the desired room, name the integration "GitLab", and click "Create". 1. In the "Send messages to this room by posting this URL" column, you should -see a URL in the format: + see a URL in the format: ``` https://api.hipchat.com/v2/room/<room>/notification?auth_token=<token> diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index c652149052e..8f2e5a55b5f 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -8,6 +8,8 @@ While you can always migrate content and process from Jira to GitLab Issues, you can also opt to continue using Jira and use it together with GitLab through our integration. +For a video demonstration of integration with Jira, watch [GitLab workflow with Jira issues and Jenkins pipelines](https://youtu.be/Jn-_fyra7xQ). + Once you integrate your GitLab project with your Jira instance, you can automatically detect and cross-reference activity between the GitLab project and any of your projects in Jira. This includes the ability to close or transition Jira issues when the work @@ -37,7 +39,7 @@ a GitLab project with any single Jira project. If you have one Jira instance, you can pre-fill the settings page with a default template. See the [Services Templates][services-templates] docs. -Configuration happens via user name and password. Connecting to a Jira server +Configuration happens via user name and password. Connecting to a Jira Server via CAS is not possible. In order to enable the Jira service in GitLab, you need to first configure the @@ -45,13 +47,13 @@ project in Jira and then enter the correct values in GitLab. ### Configuring Jira -When connecting to **JIRA Server**, which supports basic authentication, a **username and password** are required. Check the link below and proceed to the next step: +When connecting to **Jira Server**, which supports basic authentication, a **username and password** are required. Check the link below and proceed to the next step: -- [Setting up a user in JIRA server](jira_server_configuration.md) +- [Setting up a user in Jira Server](jira_server_configuration.md) -When connecting to **JIRA Cloud**, which supports authentication via API token, an **email and API token**, are required. Check the link below and proceed to the next step: +When connecting to **Jira Cloud**, which supports authentication via API token, an **email and API token**, are required. Check the link below and proceed to the next step: -- [Setting up a user in JIRA cloud](jira_cloud_configuration.md) +- [Setting up a user in Jira Cloud](jira_cloud_configuration.md) ### Configuring GitLab @@ -75,8 +77,8 @@ in the table below. | ----- | ----------- | | `Web URL` | The base URL to the Jira instance web interface which is being linked to this GitLab project. E.g., `https://Jira.example.com`. | | `Jira API URL` | The base URL to the Jira instance API. Web URL value will be used if not set. E.g., `https://jira-api.example.com`. | -| `Username/Email` | Created when [configuring Jira step](#configuring-jira). Use `username` for **JIRA server** or `email` for **JIRA cloud**. | -| `Password/API token` |Created in [configuring Jira step](#configuring-jira). Use `password` for **JIRA server** or `API token` for **JIRA cloud**. | +| `Username/Email` | Created when [configuring Jira step](#configuring-jira). Use `username` for **Jira Server** or `email` for **Jira Cloud**. | +| `Password/API token` |Created in [configuring Jira step](#configuring-jira). Use `password` for **Jira Server** or `API token` for **Jira Cloud**. | | `Transition ID` | This is the ID of a transition that moves issues to the desired state. It is possible to insert transition ids separated by `,` or `;` which means the issue will be moved to each state after another using the given order. **Closing Jira issues via commits or Merge Requests won't work if you don't set the ID correctly.** | ### Obtaining a transition ID diff --git a/doc/user/project/integrations/jira_cloud_configuration.md b/doc/user/project/integrations/jira_cloud_configuration.md index 849df707521..614f05d5b7e 100644 --- a/doc/user/project/integrations/jira_cloud_configuration.md +++ b/doc/user/project/integrations/jira_cloud_configuration.md @@ -1,18 +1,18 @@ -# Creating an API token in JIRA cloud +# Creating an API token in Jira Cloud -An API token is needed when integrating with JIRA Cloud, follow the steps +An API token is needed when integrating with Jira Cloud, follow the steps below to create one: 1. Log in to <https://id.atlassian.com> with your email. 1. **Click API tokens**, then **Create API token**. - + - + 1. Make sure to write down your new API token as you will need it in the next [steps](jira.md#configuring-gitlab). NOTE: **Note** -It is important that the user associated with this email has 'write' access to projects in JIRA. +It is important that the user associated with this email has 'write' access to projects in Jira. -The JIRA configuration is complete. You are going to need this new created token and the email you used to log in when [configuring GitLab in the next section](jira.md#configuring-gitlab). +The Jira configuration is complete. You are going to need this newly created token and the email you used to log in, when [configuring GitLab in the next section](jira.md#configuring-gitlab). diff --git a/doc/user/project/integrations/jira_server_configuration.md b/doc/user/project/integrations/jira_server_configuration.md index 13d65c4d8e4..32991714973 100644 --- a/doc/user/project/integrations/jira_server_configuration.md +++ b/doc/user/project/integrations/jira_server_configuration.md @@ -1,4 +1,4 @@ -# Creating a username and password for JIRA server +# Creating a username and password for Jira Server We need to create a user in Jira which will have access to all projects that need to integrate with GitLab. diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index d7fd75fd728..ea58a08e127 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -1,5 +1,9 @@ # Mattermost Notifications Service +The Mattermost Notifications Service allows your GitLab project to send events (e.g., `issue created`) to your existing Mattermost team as notifications. This requires configurations in both Mattermost and GitLab. + +You can also use Mattermost slash commands to control GitLab inside Mattermost. This is the separately configured [Mattermost slash commands](mattermost_slash_commands.md). + ## On Mattermost To enable Mattermost integration you must create an incoming webhook integration: diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index 9c69437537a..41be26c1d30 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -6,6 +6,9 @@ Mattermost commands give users an extra interface to perform common operations from the chat environment. This allows one to, for example, create an issue as soon as the idea was discussed in Mattermost. +GitLab can also send events (e.g., `issue created`) to Mattermost as notifications. +This is the separately configured [Mattermost Notifications Service](mattermost.md). + ## Prerequisites Mattermost 3.4 and up is required. diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index 0bfee3bac99..0e4c71a9d3e 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -38,7 +38,7 @@ Click on the service links to see further configuration instructions and details | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | | [HipChat](hipchat.md) | Private group chat and IM | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | -| [JIRA](jira.md) | JIRA issue tracker | +| [Jira](jira.md) | Jira issue tracker | | [Jenkins](../../../integration/jenkins.md) **[STARTER]** | An extendable open source continuous integration server | | JetBrains TeamCity CI | A continuous integration and build server | | [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 751e8e44e60..aab7131e353 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -160,7 +160,7 @@ receivers: > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. -Alerts can be used to trigger actions, like open an issue automatically. To configure the actions: +Alerts can be used to trigger actions, like open an issue automatically (enabled by default since `12.1`). To configure the actions: 1. Navigate to your project's **Settings > Operations > Incidents**. 1. Enable the option to create issues. diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index d5f28ddabc1..04a9d9568ca 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -24,7 +24,7 @@ to the webhook URL. In most cases, you'll need to set up your own [webhook receiver](#example-webhook-receiver) to receive information from GitLab, and send it to another app, according to your needs. -We already have a [built-in receiver](http://docs.gitlab.com/ce/project_services/slack.html) +We already have a [built-in receiver](https://docs.gitlab.com/ce/project_services/slack.html) for sending [Slack](https://api.slack.com/incoming-webhooks) notifications _per project_. ## Overview @@ -326,7 +326,7 @@ X-Gitlab-Event: Issue Hook "current": 1 }, "updated_at": { - "previous": "2017-09-15 16:50:55 UTC", + "previous": "2017-09-15 16:50:55 UTC", "current": "2017-09-15 16:52:00 UTC" }, "labels": { @@ -888,7 +888,7 @@ X-Gitlab-Event: Merge Request Hook "current": 1 }, "updated_at": { - "previous": "2017-09-15 16:50:55 UTC", + "previous": "2017-09-15 16:50:55 UTC", "current":"2017-09-15 16:52:00 UTC" }, "labels": { diff --git a/doc/user/project/issues/img/link_zoom_call_in_issue.png b/doc/user/project/issues/img/link_zoom_call_in_issue.png Binary files differnew file mode 100644 index 00000000000..3153a0a9b07 --- /dev/null +++ b/doc/user/project/issues/img/link_zoom_call_in_issue.png diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 76dc6e49bce..4acbb4cc3f6 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -134,4 +134,4 @@ For more information, see [Crosslinking issues](crosslinking_issues.md). - [Export issues](csv_export.md) **[STARTER]** - [Issues API](../../../api/issues.md) - Configure an [external issue tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, -or Bugzilla. + or Bugzilla. diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index da585022263..2103f331aa2 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -149,6 +149,19 @@ The plain text title and description of the issue fill the top center of the iss The description fully supports [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm), allowing many formatting options. +##### Zoom call links + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/62966) in GitLab 12.0. + +Including a link to a [Zoom](https://zoom.us) call in the description of an issue +results in a **Join Zoom meeting** button at the top of the issue, just under the header. + +For example: + + + +To remove the button, edit the description and remove the Zoom call link. + #### 17. Mentions You can mention a user or a group present in your GitLab instance with `@username` or diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index e5f62a3bb8d..3eca1313a18 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -24,14 +24,12 @@ in the label’s title, using the format `key::value`. For example:  -Two scoped labels with the same key but a different value cannot simultaneously -apply to an issue, epic, or merge request. For example, if an issue already has `priority::3` -and you apply `priority::2` to it, `priority::3` is automatically removed from the issue. - An issue, epic, or merge request cannot have two scoped labels with the same key. For example, if an issue is already labeled `priority::3` and you apply the label `priority::2` to it, `priority::3` is automatically removed. +This functionality is demonstrated in a video titled [Use scoped labels in GitLab 11.10 for custom fields and custom workflows](https://www.youtube.com/watch?v=4BCBby6du3c). + ### Labels with multiple colon pairs If labels have multiple instances of `::`, the longest path from left to right, until the last `::`, is considered the "key" or the "scope". diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index fd151a6df45..8e8ec26daf2 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -12,9 +12,9 @@ to approve a merge request before it can be unblocked for merging. ## Use cases 1. Enforcing review of all code that gets merged into a repository. -2. Specifying code maintainers for an entire repository. -3. Specifying reviewers for a given proposed code change. -4. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. +1. Specifying code maintainers for an entire repository. +1. Specifying reviewers for a given proposed code change. +1. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. ## Enabling the new approvals interface @@ -246,7 +246,7 @@ restrictions (compared to [GitLab Starter](#overriding-the-merge-request-approva - Approval rules can be added to an MR with no restriction. - For project sourced approval rules, editing and removing approvers is not allowed. - The approvals required of all approval rules is configurable, but if a rule is backed by a project rule, then it is restricted -to the minimum approvals required set in the project's corresponding rule. + to the minimum approvals required set in the project's corresponding rule. ## Resetting approvals on push diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index c93c7a5fe08..0dd60d84c42 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -42,6 +42,8 @@ Navigate to your project's settings page and expand the **Merge requests** secti In the **Merge checks** subsection, select the **Pipelines must succeed** check box and hit **Save** for the changes to take effect. +NOTE: **Note:** This setting also prevents merge requests from being merged if there is no pipeline. +  From now on, every time the pipeline fails you will not be able to merge the @@ -49,6 +51,21 @@ merge request from the UI, until you make all relevant jobs pass.  +### Limitations + +When this setting is enabled, a merge request is prevented from being merged if there is no pipeline. This may conflict with some use cases where [`only/except`](../../../ci/yaml/README.md#onlyexcept-advanced) rules are used and they don't generate any pipelines. + +Users that expect to be able to merge a merge request in this scenario should ensure that [there is always a pipeline](https://gitlab.com/gitlab-org/gitlab-ce/issues/54226) and that it's succesful. + +For example, to that on merge requests there is always a passing job even though `only/except` rules may not generate any other jobs: + +```yaml +enable_merge: + only: merge_requests + script: + - echo true +``` + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index d585c19fc5c..bc9a11504cd 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1,5 +1,5 @@ --- -last_updated: 2019-06-04 +last_updated: 2019-06-25 type: concepts, reference, howto --- @@ -138,9 +138,9 @@ verify your domain's ownership with a TXT record: > - **Do not** add any special chars after the default Pages domain. E.g., **do not** point your `subdomain.domain.com` to `namespace.gitlab.io.` or `namespace.gitlab.io/`. -> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017 +> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017. > - GitLab Pages IP on GitLab.com [has been changed](https://about.gitlab.com/2018/07/19/gcp-move-update/#gitlab-pages-and-custom-domains) - from `52.167.214.135` to `35.185.44.232` in 2018 + from `52.167.214.135` to `35.185.44.232` in 2018. ### Add your custom domain to GitLab Pages settings @@ -199,7 +199,7 @@ Certificates are NOT required to add to your custom highly recommendable. Let's start with an introduction to the importance of HTTPS. -Alternatively, jump ahead to [adding certificates to your project](#adding-certificates-to-your-project). +Alternatively, jump ahead to [adding certificates to your project](#adding-certificates-to-pages). ### Why should I care about HTTPS? @@ -255,12 +255,12 @@ which also offers a [free CDN service](https://blog.cloudflare.com/cloudflares-f Their certs are valid up to 15 years. See the tutorial on [how to add a CloudFlare Certificate to your GitLab Pages website](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/). -### Adding certificates to your project +### Adding certificates to Pages Regardless the CA you choose, the steps to add your certificate to your Pages project are the same. -### What do you need +#### Requirements 1. A PEM certificate 1. An intermediate certificate @@ -270,7 +270,7 @@ your Pages project are the same. These fields are found under your **Project**'s **Settings** > **Pages** > **New Domain**. -### What's what? +#### Certificate types - A PEM certificate is the certificate generated by the CA, which needs to be added to the field **Certificate (PEM)**. @@ -283,21 +283,32 @@ These fields are found under your **Project**'s **Settings** > **Pages** > **New - A private key is an encrypted key which validates your PEM against your domain. -### Now what? +#### Add the certificate to your project -Now that you hopefully understand why you need all -of this, it's simple: +Once you've met the requirements: -- Your PEM certificate needs to be added to the first field +- Your PEM certificate needs to be added to the first field. - If your certificate is missing its intermediate, copy and paste the root certificate (usually available from your CA website) and paste it in the [same field as your PEM certificate](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/), just jumping a line between them. -- Copy your private key and paste it in the last field +- Copy your private key and paste it in the last field. ->**Note:** +NOTE: **Note:** **Do not** open certificates or encryption keys in regular text editors. Always use code editors (such as Sublime Text, Atom, Dreamweaver, Brackets, etc). -_Read on about [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md)_ +## Force HTTPS for GitLab Pages websites + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28857) in GitLab 10.7. + +To make your website's visitors even more secure, you can choose to +force HTTPS for GitLab Pages. By doing so, all attempts to visit your +website via HTTP will be automatically redirected to HTTPS via 301. + +It works with both GitLab's default domain and with your custom +domain (as long as you've set a valid certificate for it). + +To enable this setting, navigate to your project's **Settings > Pages** +and tick the checkbox **Force HTTPS (requires valid certificates)**. diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index 3e50cd4887c..fe92d19567d 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -77,10 +77,10 @@ containing the most popular SSGs templates to get you started. 1. [Fork](../../../gitlab-basics/fork-project.md) a sample project from the [GitLab Pages examples](https://gitlab.com/pages) group. 1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** -and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your -site to the server. + and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your + site to the server. 1. Once the pipeline has finished successfully, find the link to visit your -website from your project's **Settings > Pages**. + website from your project's **Settings > Pages**. You can also take some **optional** further steps: @@ -89,14 +89,14 @@ You can also take some **optional** further steps:  - _Make it a user or group website._ To turn a **project website** forked -from the Pages group into a **user/group** website, you'll need to: + from the Pages group into a **user/group** website, you'll need to: - Rename it to `namespace.gitlab.io`: go to your project's - **Settings > General** and expand **Advanced**. Scroll down to - **Rename repository** and change the path to `namespace.gitlab.io`. + **Settings > General** and expand **Advanced**. Scroll down to + **Rename repository** and change the path to `namespace.gitlab.io`. - Adjust your SSG's [base URL](#urls-and-baseurls) from `"project-name"` to - `""`. This setting will be at a different place for each SSG, as each of them - have their own structure and file tree. Most likely, it will be in the SSG's - config file. + `""`. This setting will be at a different place for each SSG, as each of them + have their own structure and file tree. Most likely, it will be in the SSG's + config file. ### Create a project from scratch diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 04bda212128..fa79c393b72 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -12,7 +12,6 @@ type: index, reference > - Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. > - Bundled project templates were [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47857) in GitLab 11.8. - **GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab.** @@ -105,10 +104,10 @@ To get started with GitLab Pages, you can either:  1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** -and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your -site to the server. + and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your + site to the server. 1. Once the pipeline has finished successfully, find the link to visit your -website from your project's **Settings > Pages**. + website from your project's **Settings > Pages**. Your website is then visible on your domain, and you can modify yourfiles as you wish. For every modification pushed to your repository, GitLab CI/CD diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 4fab7f79e0c..4ea3bd9be9b 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -13,17 +13,17 @@ To familiarize yourself with GitLab Pages first: - Read an [introduction to GitLab Pages](index.md#overview). - Learn [how to get started with Pages](index.md#getting-started). - Learn how to enable GitLab Pages -across your GitLab instance on the [administrator documentation](../../../administration/pages/index.md). + across your GitLab instance on the [administrator documentation](../../../administration/pages/index.md). ## GitLab Pages requirements In brief, this is what you need to upload your website in GitLab Pages: 1. Domain of the instance: domain name that is used for GitLab Pages -(ask your administrator). + (ask your administrator). 1. GitLab CI/CD: a `.gitlab-ci.yml` file with a specific job named [`pages`][pages] in the root directory of your repository. 1. A directory called `public` in your site's repo containing the content -to be published. + to be published. 1. GitLab Runner enabled for the project. ## GitLab Pages on GitLab.com diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 16f48c462eb..24e15a37a40 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -24,7 +24,8 @@ in `.gitlab-ci.yml`. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28919) in GitLab 12.0. -NOTE: **Note**: As of GitLab 12.0, newly created projects will automaticallyl have a default +NOTE: **Note**: +As of GitLab 12.0, newly created projects will automatically have a default `git depth` value of `50`. It is possible to limit the number of changes that GitLab CI/CD will fetch when cloning diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 56e8f1731ae..99cede557a0 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -180,6 +180,6 @@ for details about the pipelines security model. [ce-4892]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4892 "Allow developers to merge into a protected branch without having push access" [ce-5081]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5081 "Allow creating protected branches that can't be pushed to" [ce-21393]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21393 -[ee-restrict]: http://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users +[ee-restrict]: https://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users [perm]: ../permissions.md [ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 1d640966013..1281ba561b8 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -35,7 +35,7 @@ discussions, and descriptions: | `/label ~label1 ~label2` | Add label(s). Label names can also start without ~ but mixed syntax is not supported. | ✓ | ✓ | | `/unlabel ~label1 ~label2` | Remove all or specific label(s)| ✓ | ✓ | | `/relabel ~label1 ~label2` | Replace label | ✓ | ✓ | -| <code>/copy_metadata #issue | !merge_request</code> | Copy labels and milestone from other issue or merge request in the project | ✓ | ✓ | +| <code>/copy_metadata <#issue | !merge_request></code> | Copy labels and milestone from other issue or merge request in the project | ✓ | ✓ | | <code>/estimate <1w 3d 2h 14m></code> | Set time estimate | ✓ | ✓ | | `/remove_estimate` | Remove time estimate | ✓ | ✓ | | <code>/spend <time(1h 30m | -1h 5m)> <date(YYYY-MM-DD)></code> | Add or subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ | @@ -44,19 +44,20 @@ discussions, and descriptions: | `/unlock` | Unlock the discussion | ✓ | ✓ | | <code>/due <in 2 days | this Friday | December 31st></code>| Set due date | ✓ | | | `/remove_due_date` | Remove due date | ✓ | | -| `/weight 0,1,2, ...` | Set weight **[STARTER]** | ✓ | | +| <code>/weight <0 | 1 | 2 | ...></code> | Set weight **[STARTER]** | ✓ | | | `/clear_weight` | Clears weight **[STARTER]** | ✓ | | -| `/epic <&epic | group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | +| <code>/epic <&epic | group&epic | Epic URL></code> | Add to epic **[ULTIMATE]** | ✓ | | | `/remove_epic` | Removes from epic **[ULTIMATE]** | ✓ | | | `/promote` | Promote issue to epic **[ULTIMATE]** | ✓ | | | `/confidential` | Make confidential | ✓ | | -| `/duplicate #issue` | Mark this issue as a duplicate of another issue | ✓ | -| `/move path/to/project` | Move this issue to another project | ✓ | | +| `/duplicate <#issue>` | Mark this issue as a duplicate of another issue | ✓ | +| `/move <path/to/project>` | Move this issue to another project | ✓ | | | `/target_branch <Local branch Name>` | Set target branch | | ✓ | | `/wip` | Toggle the Work In Progress status | | ✓ | | `/approve` | Approve the merge request | | ✓ | | `/merge` | Merge (when pipeline succeeds) | | ✓ | | `/create_merge_request <branch name>` | Create a new merge request starting from the current issue | ✓ | | +| `/relate #issue1 #issue2` | Mark issues as related **[STARTER]** | ✓ | | ## Quick actions for commit messages @@ -85,3 +86,5 @@ The following quick actions are applicable for epics threads and description: | `/label ~label1 ~label2` | Add label(s) | | `/unlabel ~label1 ~label2` | Remove all or specific label(s) | | `/relabel ~label1 ~label2` | Replace label | +| <code>/child_epic <&epic | group&epic | Epic URL></code> | Adds child epic to epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) | +| <code>/remove_child_epic <&epic | group&epic | Epic URL></code> | Removes child epic from epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) | diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 6fccfd40987..165f4c15165 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -68,7 +68,7 @@ according to the markup language. | Plain text | `txt` | | [Markdown](../../markdown.md) | `mdown`, `mkd`, `mkdn`, `md`, `markdown` | | [reStructuredText](http://docutils.sourceforge.net/rst.html) | `rst` | -| [Asciidoc](https://asciidoctor.org/docs/what-is-asciidoc/) | `adoc`, `ad`, `asciidoc` | +| [AsciiDoc](../../asciidoc.md) | `adoc`, `ad`, `asciidoc` | | [Textile](https://txstyle.org/) | `textile` | | [rdoc](http://rdoc.sourceforge.net/doc/index.html) | `rdoc` | | [Orgmode](https://orgmode.org/) | `org` | diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index e3d771524ce..7765a3d7438 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -35,16 +35,16 @@ BFG Repo-Cleaner](#using-the-bfg-repo-cleaner). It's faster and simpler than `git filter-branch`, and GitLab can use its account of what has changed to clean up its own internal state, maximizing the space saved. -> **Warning:** -> Make sure to first make a copy of your repository since rewriting history will -> purge the files and information you are about to delete. Also make sure to -> inform any collaborators to not use `pull` after your changes, but use `rebase`. +CAUTION: **Caution:** +Make sure to first make a copy of your repository since rewriting history will +purge the files and information you are about to delete. Also make sure to +inform any collaborators to not use `pull` after your changes, but use `rebase`. -> **Warning:** -> This process is not suitable for removing sensitive data like password or keys -> from your repository. Information about commits, including file content, is -> cached in the database, and will remain visible even after they have been -> removed from the repository. +CAUTION: **Caution:** +This process is not suitable for removing sensitive data like password or keys +from your repository. Information about commits, including file content, is +cached in the database, and will remain visible even after they have been +removed from the repository. ## Using the BFG Repo-Cleaner @@ -99,11 +99,12 @@ up its own internal state, maximizing the space saved. `git gc` against the repository. You will receive an email once it has completed. +NOTE: **Note:** This process will remove some copies of the rewritten commits from GitLab's cache and database, but there are still numerous gaps in coverage - at present, -some of the copies may persist indefinitely. [Clearing the instance cache] -(../../../administration/raketasks/maintenance.md#clear-redis-cache) may help to -remove some of them, but it should not be depended on for security purposes! +some of the copies may persist indefinitely. [Clearing the instance cache](../../../administration/raketasks/maintenance.md#clear-redis-cache) +may help to remove some of them, but it should not be depended on for security +purposes! ## Using `git filter-branch` diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index ba890c5ac01..2bf8d4dfe7b 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -18,7 +18,7 @@ Adjust your project's name, description, avatar, [default branch](../repository/  -The project description also partially supports [standard markdown](../../markdown.md#standard-markdown). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. +The project description also partially supports [standard markdown](../../markdown.md#standard-markdown-and-extensions-in-gitlab). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. ### Sharing and permissions @@ -26,10 +26,10 @@ Set up your project's access, [visibility](../../../public_access/public_access.  -If Issues are disabled, or you can't access Issues because you're not a project member, then Lables and Milestones +If Issues are disabled, or you can't access Issues because you're not a project member, then Lables and Milestones links will be missing from the sidebar UI. -You can still access them with direct links if you can access Merge Requests. This is deliberate, if you can see +You can still access them with direct links if you can access Merge Requests. This is deliberate, if you can see Issues or Merge Requests, both of which use Labels and Milestones, then you shouldn't be denied access to Labels and Milestones pages. ### Issue settings @@ -109,8 +109,8 @@ You can transfer an existing project into a [group](../../group/index.md) if: 1. You have at least **Maintainer** [permissions] to that group. 1. The project is in a subgroup you own. 1. You are at least a **Maintainer** of the project under your personal namespace. -Similarly, if you are an owner of a group, you can transfer any of its projects -under your own user. + Similarly, if you are an owner of a group, you can transfer any of its projects + under your own user. To transfer a project: diff --git a/doc/user/project/web_ide/img/terminal_status.png b/doc/user/project/web_ide/img/terminal_status.png Binary files differnew file mode 100644 index 00000000000..c37aa02b07a --- /dev/null +++ b/doc/user/project/web_ide/img/terminal_status.png diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index a634a8b2f54..7d85f4adfed 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -67,8 +67,8 @@ shows you a preview of the merge request diff if you commit your changes. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19279) in [GitLab Core][ce] 11.0. -You can use the Web IDE to quickly fix failing tests by opening -the branch or merge request in the Web IDE and opening the logs of the failed +You can use the Web IDE to quickly fix failing tests by opening +the branch or merge request in the Web IDE and opening the logs of the failed job. You can access the status of all jobs for the most recent pipeline and job traces for the current commit by clicking the **Pipelines** button in the top right. @@ -80,8 +80,8 @@ left. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19318) in [GitLab Core][ce] 11.0. -To switch between your authored and assigned merge requests, click the -dropdown in the top of the sidebar to open a list of merge requests. You will +To switch between your authored and assigned merge requests, click the +dropdown in the top of the sidebar to open a list of merge requests. You will need to commit or discard all your changes before switching to a different merge request. @@ -89,9 +89,9 @@ request. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20850) in [GitLab Core][ce] 11.2. -To switch between branches of the current project repository, click the dropdown -in the top of the sidebar to open a list of branches. -You will need to commit or discard all your changes before switching to a +To switch between branches of the current project repository, click the dropdown +in the top of the sidebar to open a list of branches. +You will need to commit or discard all your changes before switching to a different branch. ## Client Side Evaluation @@ -117,7 +117,7 @@ GitLab.com  Once you have done that, you can preview projects with a `package.json` file and -a `main` entry point inside the Web IDE. An example `package.json` is shown +a `main` entry point inside the Web IDE. An example `package.json` is shown below. ```json @@ -135,24 +135,20 @@ below. CAUTION: **Warning:** Interactive Web Terminals for the Web IDE is currently in **Beta**. +Shared Runners [do not yet support Interactive Web Terminals](https://gitlab.com/gitlab-org/gitlab-ce/issues/52611), +so you would need to use your own private Runner(s) to make use of this feature. -[Interactive web terminals](../../../ci/interactive_web_terminal/index.md) -give the user access to a terminal to interact with the Runner directly from +[Interactive Web Terminals](../../../ci/interactive_web_terminal/index.md) +give the project [Maintainers](../../permissions.md#project-members-permissions) +user access to a terminal to interact with the Runner directly from GitLab, including through the Web IDE. -Only project [**maintainers**](../../permissions.md#project-members-permissions) -can run Interactive Web Terminals through the Web IDE. - -CAUTION: **Warning:** -GitLab.com [does not support Interactive Web Terminals yet](https://gitlab.com/gitlab-org/gitlab-ce/issues/52611). -Shared Runners in private instances are not supported either. - ### Runner configuration Some things need to be configured in the runner for the interactive web terminal to work: -- The Runner needs to have +- The Runner needs to have [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section). - If you are using a reverse proxy with your GitLab instance, web terminals need to be [enabled](../../../administration/integration/terminal.md#enabling-and-disabling-terminal-support). **[ULTIMATE ONLY]** @@ -175,13 +171,13 @@ syntax but with some restrictions: - No global blocks can be defined (ie: `before_script` or `after_script`) - Only one job named `terminal` can be added to this file. - Only the keywords `image`, `services`, `tags`, `before_script`, `script`, and -`variables` are allowed to be used to configure the job. + `variables` are allowed to be used to configure the job. - To connect to the interactive terminal, the `terminal` job must be still alive -and running, otherwise the terminal won't be able to connect to the job's session. -By default the `script` keyword has the value `sleep 60` to prevent -the job from ending and giving the Web IDE enough time to connect. This means -that, if you override the default `script` value, you'll have to add a command -which would keep the job running, like `sleep`. + and running, otherwise the terminal won't be able to connect to the job's session. + By default the `script` keyword has the value `sleep 60` to prevent + the job from ending and giving the Web IDE enough time to connect. This means + that, if you override the default `script` value, you'll have to add a command + which would keep the job running, like `sleep`. In the code below there is an example of this configuration file: @@ -204,7 +200,7 @@ the selected branch of the Web IDE. If there is no configuration file in a branch, an error message will be shown. -### Running Interactive Terminals in the Web IDE +### Running interactive terminals in the Web IDE If Interactive Terminals are available for the current user, the **Terminal** button will be visible in the right sidebar of the Web IDE. Click this button to open @@ -212,7 +208,7 @@ or close the terminal tab. Once open, the tab will show the **Start Web Terminal** button. This button may be disabled if the environment is not configured correctly. If so, a status -message will describe the issue. Here are some reasons why **Start Web Terminal** +message will describe the issue. Here are some reasons why **Start Web Terminal** may be disabled: - `.gitlab/.gitlab-webide.yml` does not exist or is set up incorrectly. @@ -231,9 +227,62 @@ While the terminal is running, it can be stopped by clicking **Stop Terminal**. This will disconnect the terminal and stop the runner's terminal job. From here, click **Restart Terminal** to start a new terminal session. +### File syncing to web terminal + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5276) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + +File changes in the Web IDE can be synced to a running web terminal. +This enables users to test their code changes in a preconfigured terminal +environment. + +NOTE: **Note:** +Only file changes in the Web IDE are synced to the terminal. +Changes made in the terminal are **not** synced to the Web IDE. +This feature is only available for Kubernetes Runners. + +To enable file syncing to the web terminal, the `.gitlab/.gitlab-webide.yml` +file needs to have a `webide-file-sync` service configured. Here is an example +configuration for a Node JS project which uses this service: + +```yaml +terminal: + # This can be any image that has the necessary runtime environment for your project. + image: + name: node:10-alpine + services: + - name: registry.gitlab.com/gitlab-org/webide-file-sync:latest + alias: webide-file-sync + entrypoint: ["/bin/sh"] + command: ["-c", "sleep 5 && ./webide-file-sync -project-dir $CI_PROJECT_DIR"] + ports: + # The `webide-file-sync` executable defaults to port 3000. + - number: 3000 +``` + +- The `webide-file-sync` executable must start **after** the project + directory is available. This is why we need to add `sleep 5` to the `command`. + See [this issue](https://gitlab.com/gitlab-org/webide-file-sync/issues/7) for + more info. +- `$CI_PROJECT_DIR` is a + [predefined environment variable](../../../ci/variables/predefined_variables.md) + for GitLab Runners. This is where your project's repository will be. + +Once you have configured the web terminal for file syncing, then when the web +terminal is started, a **Terminal** status will be visible in the status bar. + + + +Changes made to your files via the Web IDE will sync to the running terminal +when: + +- <kbd>Ctrl</kbd> + <kbd>S</kbd> (or <kbd>Cmd</kbd> + <kbd>S</kbd> on Mac) + is pressed while editing a file. +- Anything outside the file editor is clicked after editing a file. +- A file or folder is created, deleted, or renamed. + ### Limitations -Interactive Terminals is in a beta phase and will continue to be improved upon in upcoming +Interactive Terminals is in a beta phase and will continue to be improved upon in upcoming releases. In the meantime, please note that the user is limited to having only one active terminal at a time. diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 7d0abb93262..3e24557591c 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -27,6 +27,8 @@ People have a hard time figuring out which branch has the latest code, or which Frequently, the reaction to this problem is to adopt a standardized pattern such as [Git flow](https://nvie.com/posts/a-successful-git-branching-model/) and [GitHub flow](http://scottchacon.com/2011/08/31/github-flow.html). We think there is still room for improvement. In this document, we describe a set of practices we call GitLab flow. +For a video introduction of how this works in GitLab, see [GitLab Flow](https://youtu.be/InKNIvky2KE). + ## Git flow and its problems  diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md index 9772bd385ba..5a24c254ed1 100644 --- a/doc/workflow/repository_mirroring.md +++ b/doc/workflow/repository_mirroring.md @@ -101,6 +101,9 @@ The repository will push soon. To force a push, click the appropriate button. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. > [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab-ee/issues/10871) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.11. +NOTE: **Note:** This feature [is available for free](https://gitlab.com/gitlab-org/gitlab-ee/issues/10361) to +GitLab.com users until September 22nd, 2019. + You can set up a repository to automatically have its branches, tags, and commits updated from an upstream repository. diff --git a/doc/workflow/time_tracking.md b/doc/workflow/time_tracking.md index c03dffa967d..4286a3625a2 100644 --- a/doc/workflow/time_tracking.md +++ b/doc/workflow/time_tracking.md @@ -73,7 +73,15 @@ The following time units are available: Default conversion rates are 1mo = 4w, 1w = 5d and 1d = 8h. -Other interesting links: +### Limit displayed units to hours + +> Introduced in GitLab 12.0. + +The display of time units can be limited to hours through the option in **Admin Area > Settings > Preferences** under 'Localization'. + +With this option enabled, `75h` is displayed instead of `1w 4d 3h`. + +## Other interesting links - [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/solutions/time-tracking/) diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 32907db4f46..3eac79427cf 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -34,6 +34,8 @@ A Todo appears in your Todos dashboard when: - the author, or - have set it to automatically merge once pipeline succeeds. +Todo triggers are not affected by [GitLab Notification Email settings](notifications.md). + NOTE: **Note:** When an user no longer has access to a resource related to a Todo like an issue, merge request, project or group the related Todos, for security reasons, gets deleted within the next hour. The delete is delayed to prevent data loss in case user got their access revoked by mistake. |
