diff options
| author | bikebilly <fabio@gitlab.com> | 2017-08-03 10:05:56 +0200 |
|---|---|---|
| committer | bikebilly <fabio@gitlab.com> | 2017-08-03 10:05:56 +0200 |
| commit | 40dfddd4077da4d594bd9e8956e1fcb1c99434e6 (patch) | |
| tree | d0761ed84471c22b3b0949720e3e0734e8645aac /doc | |
| parent | ed5445388de13f1d126fec14cc0a9ea9ae03b397 (diff) | |
| parent | dc412b48693668f7fba3adea57b8be76685afa76 (diff) | |
| download | gitlab-ce-40dfddd4077da4d594bd9e8956e1fcb1c99434e6.tar.gz | |
Merge branch 'master' into 33329-tech-article-deploying-maven-artifacts
Diffstat (limited to 'doc')
215 files changed, 5734 insertions, 1193 deletions
diff --git a/doc/README.md b/doc/README.md index 9f12eed1471..8bb8e147cd1 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,12 +1,20 @@ -# GitLab Community Edition +# GitLab Documentation -[GitLab](https://about.gitlab.com/) is a Git-based fully featured platform -for software development. +Welcome to [GitLab](https://about.gitlab.com/), a Git-based fully featured +platform for software development! -**GitLab Community Edition (CE)** is an opensource product, self-hosted, free to use. -All [GitLab products](https://about.gitlab.com/products/) contain the features -available in GitLab CE. Premium features are available in -[GitLab Enterprise Edition (EE)](https://about.gitlab.com/gitlab-ee/). +We offer four different products for you and your company: + +- **GitLab Community Edition (CE)** is an [opensource product](https://gitlab.com/gitlab-org/gitlab-ce/), +self-hosted, free to use. Every feature available in GitLab CE is also available on GitLab Enterprise Edition (Starter and Premium) and GitLab.com. +- **GitLab Enterprise Edition (EE)** is an [opencore product](https://gitlab.com/gitlab-org/gitlab-ee/), +self-hosted, fully featured solution of GitLab, available under distinct [subscriptions](https://about.gitlab.com/products/): **GitLab Enterprise Edition Starter (EES)** and **GitLab Enterprise Edition Premium (EEP)**. +- **GitLab.com**: SaaS GitLab solution, with [free and paid subscriptions](https://about.gitlab.com/gitlab-com/). GitLab.com is hosted by GitLab, Inc., and administrated by GitLab (users don't have access to admin settings). + +> **GitLab EE** contains all features available in **GitLab CE**, +plus premium features available in each version: **Enterprise Edition Starter** +(**EES**) and **Enterprise Edition Premium** (**EEP**). Everything available in +**EES** is also available in **EEP**. ---- @@ -24,31 +32,33 @@ Shortcuts to GitLab's most visited docs: - [GitLab Workflow](workflow/README.md): Enhance your workflow with the best of GitLab Workflow. - See also [GitLab Workflow - an overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/). - [GitLab Markdown](user/markdown.md): GitLab's advanced formatting system (GitLab Flavored Markdown). -- [GitLab Slash Commands](user/project/slash_commands.md): Textual shortcuts for common actions on issues or merge requests that are usually done by clicking buttons or dropdowns in GitLab's UI. +- [GitLab Quick Actions](user/project/quick_actions.md): Textual shortcuts for common actions on issues or merge requests that are usually done by clicking buttons or dropdowns in GitLab's UI. ### User account -- [Authentication](topics/authentication/index.md): Account security with two-factor authentication, setup your ssh keys and deploy keys for secure access to your projects. -- [Profile settings](profile/README.md): Manage your profile settings, two factor authentication and more. +- [User documentation](user/index.md): Learn how to use GitLab and explore its features +- [User account](user/profile/index.md): Manage your account + - [Authentication](topics/authentication/index.md): Account security with two-factor authentication, setup your ssh keys and deploy keys for secure access to your projects. + - [Profile settings](user/profile/index.md#profile-settings): Manage your profile settings, two factor authentication and more. - [User permissions](user/permissions.md): Learn what each role in a project (external/guest/reporter/developer/master/owner) can do. ### Projects and groups -- [Create a project](gitlab-basics/create-project.md) -- [Fork a project](gitlab-basics/fork-project.md) -- [Importing and exporting projects between instances](user/project/settings/import_export.md). -- [Project access](public_access/public_access.md): Setting up your project's visibility to public, internal, or private. -- [Groups](workflow/groups.md): Organize your projects in groups. - - [Create a group](gitlab-basics/create-group.md) - - [GitLab Subgroups](user/group/subgroups/index.md) +- [Projects](user/project/index.md): + - [Create a project](gitlab-basics/create-project.md) + - [Fork a project](gitlab-basics/fork-project.md) + - [Importing and exporting projects between instances](user/project/settings/import_export.md). + - [Project access](public_access/public_access.md): Setting up your project's visibility to public, internal, or private. + - [GitLab Pages](user/project/pages/index.md): Build, test, and deploy your static website with GitLab Pages. +- [Groups](user/group/index.md): Organize your projects in groups. + - [Subgroups](user/group/subgroups/index.md) - [Search through GitLab](user/search/index.md): Search for issues, merge requests, projects, groups, todos, and issues in Issue Boards. - [Snippets](user/snippets.md): Snippets allow you to create little bits of code. - [Wikis](user/project/wiki/index.md): Enhance your repository documentation with built-in wikis. -- [GitLab Pages](user/project/pages/index.md): Build, test, and deploy your static website with GitLab Pages. ### Repository -Manage files and branches from the UI (user interface): +Manage your [repositories](user/project/repository/index.md) from the UI (user interface): - Files - [Create a file](user/project/repository/web_editor.md#create-a-file) @@ -59,6 +69,7 @@ Manage files and branches from the UI (user interface): - Branches - [Create a branch](user/project/repository/web_editor.md#create-a-new-branch) - [Protected branches](user/project/protected_branches.md#protected-branches) + - [Delete merged branches](user/project/repository/branches/index.md#delete-merged-branches) ### Issues and Merge Requests (MRs) @@ -80,6 +91,7 @@ Manage files and branches from the UI (user interface): - [Git](topics/git/index.md): Getting started with Git, branching strategies, Git LFS, advanced use. - [Git cheatsheet](https://gitlab.com/gitlab-com/marketing/raw/master/design/print/git-cheatsheet/print-pdf/git-cheatsheet.pdf): Download a PDF describing the most used Git operations. - [GitLab Flow](workflow/gitlab_flow.md): explore the best of Git with the GitLab Flow strategy. +- [Signing commits](workflow/gpg_signed_commits/index.md): use GPG to sign your commits. ### Migrate and import your projects from other platforms @@ -103,6 +115,7 @@ Manage files and branches from the UI (user interface): - [Project Services](user/project/integrations/project_services.md): Integrate a project with external services, such as CI and chat. - [GitLab Integration](integration/README.md): Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. +- [Trello Power-Up](integration/trello_power_up.md): Integrate with GitLab's Trello Power-Up ---- @@ -124,7 +137,7 @@ have access to GitLab administration tools and settings. - [Access restrictions](user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab - [Authentication/Authorization](topics/authentication/index.md#gitlab-administrators): Enforce 2FA, configure external authentication with LDAP, SAML, CAS and additional Omniauth providers. -### GitLab admins' superpowers +### Features - [Container Registry](administration/container_registry.md): Configure Docker Registry with GitLab. - [Custom Git hooks](administration/custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough. @@ -154,6 +167,7 @@ have access to GitLab administration tools and settings. - [Operations](administration/operations.md): Keeping GitLab up and running. - [Polling](administration/polling.md): Configure how often the GitLab UI polls for updates. - [Request Profiling](administration/monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. +- [Performance Bar](administration/monitoring/performance/performance_bar.md): Get performance information for the current page. ### Customization @@ -166,6 +180,7 @@ have access to GitLab administration tools and settings. ### Admin tools +- [Gitaly](administration/gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service - [Raketasks](raketasks/README.md): Backups, maintenance, automatic webhook setup and the importing of projects. - [Backup and restore](raketasks/backup_restore.md): Backup and restore your GitLab instance. - [Reply by email](administration/reply_by_email.md): Allow users to comment on issues and merge requests by replying to notification emails. diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index fb1a16b0f96..1528f1d2b17 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -32,7 +32,7 @@ Authentiq will generate a Client ID and the accompanying Client Secret for you t "app_id" => "YOUR_CLIENT_ID", "app_secret" => "YOUR_CLIENT_SECRET", "args" => { - scope: 'aq:name email~rs aq:push' + "scope": 'aq:name email~rs address aq:push' } } ] @@ -45,21 +45,20 @@ Authentiq will generate a Client ID and the accompanying Client Secret for you t app_id: 'YOUR_CLIENT_ID', app_secret: 'YOUR_CLIENT_SECRET', args: { - scope: 'aq:name email~rs aq:push' + scope: 'aq:name email~rs address aq:push' } } ``` 5. The `scope` is set to request the user's name, email (required and signed), and permission to send push notifications to sign in on subsequent visits. -See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq#scopes-and-redirect-uri-configuration) for more information on scopes and modifiers. +See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq/wiki/Scopes,-callback-url-configuration-and-responses) for more information on scopes and modifiers. 6. Change `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` to the Client credentials you received in step 1. 7. Save the configuration file. -8. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) - for the changes to take effect if you installed GitLab via Omnibus or from source respectively. +8. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page there should now be an Authentiq icon below the regular sign in form. Click the icon to begin the authentication process. diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 725fc1f6076..425c924cdf2 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -69,14 +69,42 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # Example: 'ldap.mydomain.com' host: '_your_ldap_server' # This port is an example, it is sometimes different but it is always an integer and not a string - port: 389 + port: 389 # usually 636 for SSL uid: 'sAMAccountName' # This should be the attribute, not the value that maps to uid. - method: 'plain' # "tls" or "ssl" or "plain" # Examples: 'america\\momo' or 'CN=Gitlab Git,CN=Users,DC=mydomain,DC=com' bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' password: '_the_password_of_the_bind_user' + # Encryption method. The "method" key is deprecated in favor of + # "encryption". + # + # Examples: "start_tls" or "simple_tls" or "plain" + # + # Deprecated values: "tls" was replaced with "start_tls" and "ssl" was + # replaced with "simple_tls". + # + encryption: 'plain' + + # Enables SSL certificate verification if encryption method is + # "start_tls" or "simple_tls". (Defaults to false for backward- + # compatibility) + verify_certificates: false + + # Specifies the path to a file containing a PEM-format CA certificate, + # e.g. if you need to use an internal CA. + # + # Example: '/etc/ca.pem' + # + ca_file: '' + + # Specifies the SSL version for OpenSSL to use, if the OpenSSL default + # is not appropriate. + # + # Example: 'TLSv1_1' + # + ssl_version: '' + # Set a timeout, in seconds, for LDAP queries. This helps avoid blocking # a request if the LDAP server becomes unresponsive. # A value of 0 means there is no timeout. @@ -116,8 +144,8 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # # Note: GitLab does not support omniauth-ldap's custom filter syntax. # - # Below an example for get only specific users - # Example: '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' + # Example for getting only specific users: + # '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' # user_filter: '' @@ -228,9 +256,14 @@ Tip: If you want to limit access to the nested members of an Active Directory group you can use the following syntax: ``` -(memberOf=CN=My Group,DC=Example,DC=com) +(memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` +Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at +https://msdn.microsoft.com/en-us/library/aa746475(v=vs.85).aspx. Support for +nested members in the user filter should not be confused with +[group sync nested groups support (EE only)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#supported-ldap-group-types-attributes). + Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. @@ -245,6 +278,19 @@ In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their LDAP email address, and then sign into GitLab via their LDAP credentials. +## Encryption + +### TLS Server Authentication + +There are two encryption methods, `simple_tls` and `start_tls`. + +For either encryption method, if setting `validate_certificates: false`, TLS +encryption is established with the LDAP server before any LDAP-protocol data is +exchanged but no validation of the LDAP server's SSL certificate is performed. + +>**Note**: Before GitLab 9.5, `validate_certificates: false` is the default if +unspecified. + ## Limitations ### TLS Client Authentication @@ -254,14 +300,6 @@ You should disable anonymous LDAP authentication and enable simple or SASL authentication. The TLS client authentication setting in your LDAP server cannot be mandatory and clients cannot be authenticated with the TLS protocol. -### TLS Server Authentication - -Not supported by GitLab's configuration options. -When setting `method: ssl`, the underlying authentication method used by -`omniauth-ldap` is `simple_tls`. This method establishes TLS encryption with -the LDAP server before any LDAP-protocol data is exchanged but no validation of -the LDAP server's SSL certificate is performed. - ## Troubleshooting ### Debug LDAP user filter with ldapsearch @@ -301,9 +339,9 @@ tree and traverse it. ### Connection Refused If you are getting 'Connection Refused' errors when trying to connect to the -LDAP server please double-check the LDAP `port` and `method` settings used by -GitLab. Common combinations are `method: 'plain'` and `port: 389`, OR -`method: 'ssl'` and `port: 636`. +LDAP server please double-check the LDAP `port` and `encryption` settings used by +GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR +`encryption: 'simple_tls'` and `port: 636`. ### Troubleshooting diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index afafb6bf1f5..57e54815b68 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -112,7 +112,7 @@ GitLab from source respectively. >**Note:** Be careful to choose a port different than the one that Registry listens to (`5000` by default), -otherwise you will run into conflicts . +otherwise you will run into conflicts. --- @@ -465,23 +465,42 @@ on how to achieve that. ## Disable Container Registry but use GitLab as an auth endpoint -You can disable the embedded Container Registry to use an external one, but -still use GitLab as an auth endpoint. - **Omnibus GitLab** + +You can use GitLab as an auth endpoint and use a non-bundled Container Registry. + 1. Open `/etc/gitlab/gitlab.rb` and set necessary configurations: ```ruby - registry['enable'] = false gitlab_rails['registry_enabled'] = true gitlab_rails['registry_host'] = "registry.gitlab.example.com" gitlab_rails['registry_port'] = "5005" gitlab_rails['registry_api_url'] = "http://localhost:5000" - gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key" gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry" gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" ``` +1. A certificate keypair is required for GitLab and the Container Registry to + communicate securely. By default omnibus-gitlab will generate one keypair, + which is saved to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key`. + When using an non-bundled Container Registry, you will need to supply a + custom certificate key. To do that, add the following to + `/etc/gitlab/gitlab.rb` + + ```ruby + gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" + # registry['internal_key'] should contain the contents of the custom key + # file. Line breaks in the key file should be marked using `\n` character + # Example: + registry['internal_key'] = "---BEGIN RSA PRIVATE KEY---\nMIIEpQIBAA\n" + ``` + + **Note:** The file specified at `registry_key_path` gets populated with the + content specified by `internal_key`, each time reconfigure is executed. If + no file is specified, omnibus-gitlab will default it to + `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate + it. + 1. Save the file and [reconfigure GitLab][] for the changes to take effect. **Installations from source** diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index b6676026d06..9bcd13a52f7 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,6 +13,7 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- +`GITLAB_CDN_HOST` | string | Sets the hostname for a CDN to serve static assets (e.g. `mycdnsubdomain.fictional-cdn.com`) `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` @@ -58,6 +59,9 @@ to the naming scheme `GITLAB_#{name in 1_settings.rb in upper case}`. ## Omnibus configuration +To set environment variables, follow [these +instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.html). + It's possible to preconfigure the GitLab docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](http://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 48929910a9c..5732b6a1ca4 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,8 +2,7 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) (introduced in GitLab 9.0) is a service that provides high-level RPC access to Git -repositories. As of GitLab 9.3 it is still an optional component with -limited scope. +repositories. Gitaly is a mandatory component in GitLab 9.4 and newer. GitLab components that access Git repositories (gitlab-rails, gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do @@ -33,36 +32,155 @@ prometheus_listen_addr = "localhost:9236" Changes to `/home/git/gitaly/config.toml` are applied when you run `service gitlab restart`. -## Configuring GitLab to not use Gitaly +## Running Gitaly on its own server -Gitaly is still an optional component in GitLab 9.3. This means you -can choose to not use it. +> This is an optional way to deploy Gitaly which can benefit GitLab +installations that are larger than a single machine. Most +installations will be better served with the default configuration +used by Omnibus and the GitLab source installation guide. -In Omnibus you can make the following change in -`/etc/gitlab/gitlab.rb` and reconfigure. This will both disable the -Gitaly service and configure the rest of GitLab not to use it. +Starting with GitLab 9.4 it is possible to run Gitaly on a different +server from the rest of the application. This can improve performance +when running GitLab with its repositories stored on an NFS server. + +At the moment (GitLab 9.4) Gitaly is not yet a replacement for NFS +because some parts of GitLab still bypass Gitaly when accessing Git +repositories. If you choose to deploy Gitaly on your NFS server you +must still also mount your Git shares on your GitLab application +servers. + +Gitaly network traffic is unencrypted so you should use a firewall to +restrict access to your Gitaly server. + +Below we describe how to configure a Gitaly server at address +`gitaly.internal:9999` with secret token `abc123secret`. We assume +your GitLab installation has two repository storages, `default` and +`storage1`. + +### Client side token configuration + +Start by configuring a token on the client side. + +Omnibus installations: ```ruby -gitaly['enable'] = false +# /etc/gitlab/gitlab.rb +gitlab_rails['gitaly_token'] = 'abc123secret' +``` + +Source installations: + +```yaml +# /home/git/gitlab/config/gitlab.yml +gitlab: + gitaly: + token: 'abc123secret' +``` + +You need to reconfigure (Omnibus) or restart (source) for these +changes to be picked up. + +### Gitaly server configuration + +Next, on the Gitaly server, we need to configure storage paths, enable +the network listener and configure the token. + +Note: if you want to reduce the risk of downtime when you enable +authentication you can temporarily disable enforcement, see [the +documentation on configuring Gitaly +authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) +. + +In most or all cases the storage paths below end in `/repositories`. Check the +directory layout on your Gitaly server to be sure. + +Omnibus installations: + +```ruby +# /etc/gitlab/gitlab.rb +gitaly['listen_addr'] = '0.0.0.0:9999' +gitaly['auth_token'] = 'abc123secret' +gitaly['storage'] = [ + { 'name' => 'default', 'path' => '/path/to/default/repositories' }, + { 'name' => 'storage1', 'path' => '/path/to/storage1/repositories' }, +] +``` + +Source installations: + +```toml +# /home/git/gitaly/config.toml +listen_addr = '0.0.0.0:9999' + +[auth] +token = 'abc123secret' + +[[storage] +name = 'default' +path = '/path/to/default/repositories' + +[[storage]] +name = 'storage1' +path = '/path/to/storage1/repositories' ``` -In source installations, edit `/home/git/gitlab/config/gitlab.yml` and -make sure `enabled` in the `gitaly` section is set to 'false'. This -does not disable the Gitaly service in your init script; it only -prevents it from being used. +Again, reconfigure (Omnibus) or restart (source). + +### Converting clients to use the Gitaly server -Apply the change with `service gitlab restart`. +Now as the final step update the client machines to switch from using +their local Gitaly service to the new Gitaly server you just +configured. This is a risky step because if there is any sort of +network, firewall, or name resolution problem preventing your GitLab +server from reaching the Gitaly server then all Gitaly requests will +fail. + +We assume that your Gitaly server can be reached at +`gitaly.internal:9999` from your GitLab server, and that your GitLab +NFS shares are mounted at `/mnt/gitlab/default` and +`/mnt/gitlab/storage1` respectively. + +Omnibus installations: + +```ruby +# /etc/gitlab/gitlab.rb +git_data_dirs({ + { 'default' => { 'path' => '/mnt/gitlab/default', 'gitaly_address' => 'tcp://gitlab.internal:9999' } }, + { 'storage1' => { 'path' => '/mnt/gitlab/storage1', 'gitaly_address' => 'tcp://gitlab.internal:9999' } }, +}) + +gitlab_rails['gitaly_token'] = 'abc123secret' +``` + +Source installations: ```yaml +# /home/git/gitlab/config/gitlab.yml +gitlab: + repositories: + storages: + default: + path: /mnt/gitlab/default/repositories + gitaly_address: tcp://gitlab.internal:9999 + storage1: + path: /mnt/gitlab/storage1/repositories + gitaly_address: tcp://gitlab.internal:9999 + gitaly: - enabled: false + token: 'abc123secret' ``` +Now reconfigure (Omnibus) or restart (source). When you tail the +Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or +`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests +coming in. One sure way to trigger a Gitaly request is to clone a +repository from your GitLab server over HTTP. + ## Disabling or enabling the Gitaly service -Be careful: if you disable Gitaly without instructing the rest of your -GitLab installation not to use Gitaly, you may end up with errors -because GitLab tries to access a service that is not running. +If you are running Gitaly [as a remote +service](#running-gitaly-on-its-own-server) you may want to disable +the local Gitaly service that runs on your Gitlab server by default. To disable the Gitaly service in your Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`: @@ -81,4 +199,4 @@ following to `/etc/default/gitlab`: gitaly_enabled=false ``` -When you run `service gitlab restart` Gitaly will be disabled.
\ No newline at end of file +When you run `service gitlab restart` Gitaly will be disabled. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index da9687aa849..ca6d8d2de67 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -97,9 +97,12 @@ If you use a cloud-managed service, or provide your own PostgreSQL: Enter new password: Enter it again: ``` - -1. Enable the `pg_trgm` extension: +1. Exit from editing `template1` prompt by typing `\q` and Enter. +1. Enable the `pg_trgm` extension within the `gitlabhq_production` database: + ``` + gitlab-psql -d gitlabhq_production + CREATE EXTENSION pg_trgm; # Output: diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 137fed35a73..42666357faf 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -5,7 +5,7 @@ configure the GitLab application server(s) now. Complete the steps below for each GitLab application server in your environment. > **Note:** There is some additional configuration near the bottom for - secondary GitLab application servers. It's important to read and understand + additional GitLab application servers. It's important to read and understand these additional steps before proceeding with GitLab installation. 1. If necessary, install the NFS client utility packages using the following @@ -70,10 +70,16 @@ for each GitLab application server in your environment. gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server gitlab_rails['redis_password'] = 'Redis Password' ``` + + > **Note:** To maintain uniformity of links across HA clusters, the `external_url` + on the first application server as well as the additional application + servers should point to the external url that users will use to access GitLab. + In a typical HA setup, this will be the url of the load balancer which will + route traffic to all GitLab application servers in the HA cluster. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. -## Primary GitLab application server +## First GitLab application server As a final step, run the setup rake task on the first GitLab application server. It is not necessary to run this on additional application servers. @@ -89,10 +95,10 @@ It is not necessary to run this on additional application servers. [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for more information. -## Additional configuration for secondary GitLab application servers +## Extra configuration for additional GitLab application servers -Secondary GitLab servers (servers configured **after** the first GitLab server) -need some additional configuration. +Additional GitLab servers (servers configured **after** the first GitLab server) +need some extra configuration. 1. Configure shared secrets. These values can be obtained from the primary GitLab server in `/etc/gitlab/gitlab-secrets.json`. Add these to diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index d8e76d6ab94..90a2e9298bf 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -1,12 +1,35 @@ # NFS -## Required NFS Server features +You can view information and options set for each of the mounted NFS file +systems by running `sudo nfsstat -m`. + +## NFS Server features + +### Required features **File locking**: GitLab **requires** advisory file locking, which is only supported natively in NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not specifically test NFSv3. +### Recommended options + +When you define your NFS exports, we recommend you also add the following +options: + +- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is + a good security measure when NFS shares will be accessed by many different + users. However, in this case only GitLab will use the NFS share so it + is safe. GitLab recommends the `no_root_squash` setting because we need to + manage file permissions automatically. Without the setting you may receive + errors when the Omnibus package tries to alter permissions. Note that GitLab + and other bundled components do **not** run as `root` but as non-privileged + users. The recommendation for `no_root_squash` is to allow the Omnibus package + to set ownership and permissions on files, as needed. +- `sync` - Force synchronous behavior. Default is asynchronous and under certain + circumstances it could lead to data loss if a failure occurs before data has + synced. + ## AWS Elastic File System GitLab does not recommend using AWS Elastic File System (EFS). @@ -23,30 +46,17 @@ GitLab does not recommend using EFS with GitLab. many small files are written in a serialized manner are not well-suited for EFS. EBS with an NFS server on top will perform much better. +In addition, avoid storing GitLab log files (e.g. those in `/var/log/gitlab`) +because this will also affect performance. We recommend that the log files be +stored on a local volume. + For more details on another person's experience with EFS, see [Amazon's Elastic File System: Burst Credits](https://www.rawkode.io/2017/04/amazons-elastic-file-system-burst-credits/) -### Recommended options - -When you define your NFS exports, we recommend you also add the following -options: - -- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is - a good security measure when NFS shares will be accessed by many different - users. However, in this case only GitLab will use the NFS share so it - is safe. GitLab recommends the `no_root_squash` setting because we need to - manage file permissions automatically. Without the setting you may receive - errors when the Omnibus package tries to alter permissions. Note that GitLab - and other bundled components do **not** run as `root` but as non-privileged - users. The recommendation for `no_root_squash` is to allow the Omnibus package - to set ownership and permissions on files, as needed. -- `sync` - Force synchronous behavior. Default is asynchronous and under certain - circumstances it could lead to data loss if a failure occurs before data has - synced. - ## NFS Client mount options -Below is an example of an NFS mount point we use on GitLab.com: +Below is an example of an NFS mount point defined in `/etc/fstab` we use on +GitLab.com: ``` 10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 3629772b8af..8b7a515a076 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -4,6 +4,11 @@ This is the documentation for configuring a Highly Available Redis setup when you have installed Redis all by yourself and not using the bundled one that comes with the Omnibus packages. +Note also that you may elect to override all references to +`/home/git/gitlab/config/resque.yml` in accordance with the advanced Redis +settings outlined in +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/README.md). + We cannot stress enough the importance of reading the [Overview section](redis.md#overview) of the Omnibus Redis HA as it provides some invaluable information to the configuration of Redis. Please proceed to @@ -364,3 +369,4 @@ When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics [downloads]: https://about.gitlab.com/downloads [restart]: ../restart_gitlab.md#installations-from-source [it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png +[resque]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/resque.yml.example diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 7b0610ae414..3587696225c 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -46,7 +46,10 @@ To disable artifacts site-wide, follow the steps below. After a successful job, GitLab Runner uploads an archive containing the job artifacts to GitLab. -To change the location where the artifacts are stored, follow the steps below. +### Using local storage + +To change the location where the artifacts are stored locally, follow the steps +below. --- @@ -82,6 +85,49 @@ _The artifacts are stored by default in 1. Save the file and [restart GitLab][] for the changes to take effect. +### Using object storage + +In [GitLab Enterprise Edition Premium][eep] you can use an object storage like +AWS S3 to store the artifacts. + +[Learn how to use the object storage option.][ee-os] + +## Expiring artifacts + +If an expiry date is used for the artifacts, they are marked for deletion +right after that date passes. Artifacts are cleaned up by the +`expire_build_artifacts_worker` cron job which is run by Sidekiq every hour at +50 minutes (`50 * * * *`). + +To change the default schedule on which the artifacts are expired, follow the +steps below. + +--- + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and comment out or add the following line + + ```ruby + gitlab_rails['expire_build_artifacts_worker_cron'] = "50 * * * *" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + expire_build_artifacts_worker: + cron: "50 * * * *" + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + ## Set the maximum file size of the artifacts Provided the artifacts are enabled, you can change the maximum file size of the @@ -112,3 +158,5 @@ memory and disk I/O. [reconfigure gitlab]: restart_gitlab.md "How to restart GitLab" [restart gitlab]: restart_gitlab.md "How to restart GitLab" [gitlab workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse "GitLab Workhorse repository" +[ee-os]: https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage +[eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition Premium" diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md new file mode 100644 index 00000000000..ad2773de132 --- /dev/null +++ b/doc/administration/monitoring/ip_whitelist.md @@ -0,0 +1,39 @@ +# IP whitelist + +> Introduced in GitLab 9.4. + +GitLab provides some [monitoring endpoints] that provide health check information +when probed. + +To control access to those endpoints via IP whitelisting, you can add single +hosts or use IP ranges: + +**For Omnibus installations** + +1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: + + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` + +1. Save the file and [reconfigure] GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + monitoring: + # by default only local IPs are allowed to access monitoring resources + ip_whitelist: + - 127.0.0.0/8 + - 192.168.0.1 + ``` + +1. Save the file and [restart] GitLab for the changes to take effect. + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source +[monitoring endpoints]: ../../user/admin_area/monitoring/health_check.md diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differnew file mode 100644 index 00000000000..b3c6bc474e3 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png b/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png Binary files differnew file mode 100644 index 00000000000..2d64ef8c5fc --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png Binary files differnew file mode 100644 index 00000000000..7868e2c46d1 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png Binary files differnew file mode 100644 index 00000000000..372ae021f6b --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md new file mode 100644 index 00000000000..ee680c7b258 --- /dev/null +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -0,0 +1,35 @@ +# Performance Bar + +A Performance Bar can be displayed, to dig into the performance of a page. When +activated, it looks as follows: + + + +It allows you to: + +- see the current host serving the page +- see the timing of the page (backend, frontend) +- the number of DB queries, the time it took, and the detail of these queries + +- the number of calls to Redis, and the time it took +- the number of background jobs created by Sidekiq, and the time it took +- the number of Ruby GC calls, and the time it took +- profile the code used to generate the page, line by line + + +## Enable the Performance Bar via the Admin panel + +GitLab Performance Bar is disabled by default. To enable it for a given group, +navigate to the Admin area in **Settings > Profiling - Performance Bar** +(`/admin/application_settings`). + +The only required setting you need to set is the full path of the group that +will be allowed to display the Performance Bar. +Make sure _Enable the Performance Bar_ is checked and hit +**Save** to save the changes. + +--- + + + +--- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md new file mode 100644 index 00000000000..6baae20d16a --- /dev/null +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -0,0 +1,69 @@ +# GitLab Prometheus metrics + +>**Note:** +Available since [Omnibus GitLab 9.3][29118]. Currently experimental. For +installations from source you'll have to configure it yourself. + +To enable the GitLab Prometheus metrics: + +1. Log into GitLab as an administrator, and go to the Admin area. +1. Click on the gear, then click on Settings. +1. Find the `Metrics - Prometheus` section, and click `Enable Prometheus Metrics` +1. [Restart GitLab][restart] for the changes to take effect + +## Collecting the metrics + +GitLab monitors its own internal service metrics, and makes them available at the +`/-/metrics` endpoint. Unlike other [Prometheus] exporters, in order to access +it, the client IP needs to be [included in a whitelist][whitelist]. + +Currently the embedded Prometheus server is not automatically configured to +collect metrics from this endpoint. We recommend setting up another Prometheus +server, because the embedded server configuration is overwritten once every +[reconfigure of GitLab][reconfigure]. In the future this will not be required. + +## Metrics available + +In this experimental phase, only a few metrics are available: + +| Metric | Type | Since | Description | +|:--------------------------------- |:--------- |:----- |:----------- | +| db_ping_timeout | Gauge | 9.4 | Whether or not the last database ping timed out | +| db_ping_success | Gauge | 9.4 | Whether or not the last database ping succeeded | +| db_ping_latency_seconds | Gauge | 9.4 | Round trip time of the database ping | +| filesystem_access_latency_seconds | Gauge | 9.4 | Latency in accessing a specific filesystem | +| filesystem_accessible | Gauge | 9.4 | Whether or not a specific filesystem is accessible | +| filesystem_write_latency_seconds | Gauge | 9.4 | Write latency of a specific filesystem | +| filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | +| filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | +| filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | +| http_requests_total | Counter | 9.4 | Rack request count | +| http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | +| pipelines_created_total | Counter | 9.4 | Counter of pipelines created | +| rack_uncaught_errors_total | Counter | 9.4 | Rack connections handling uncaught errors count | +| redis_ping_timeout | Gauge | 9.4 | Whether or not the last redis ping timed out | +| redis_ping_success | Gauge | 9.4 | Whether or not the last redis ping succeeded | +| redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | +| user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | + +## Metrics shared directory + +GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. +Those files are shared among all instances running under Unicorn server. +The directory needs to be accessible to all running Unicorn's processes otherwise +metrics will not function correctly. + +For best performance its advisable that this directory will be located in `tmpfs`. + +Its location is configured using environment variable `prometheus_multiproc_dir`. + +If GitLab is installed using Omnibus and `tmpfs` is available then metrics +directory will be automatically configured. + +[← Back to the main Prometheus page](index.md) + +[29118]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29118 +[Prometheus]: https://prometheus.io +[restart]: ../../restart_gitlab.md#omnibus-gitlab-restart +[whitelist]: ../ip_whitelist.md +[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md index edb9c911aac..f68b03d1ade 100644 --- a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md @@ -4,7 +4,7 @@ Available since [Omnibus GitLab 8.17][1132]. For installations from source you'll have to install and configure it yourself. -The [GitLab monitor exporter] allows you to measure various GitLab metrics. +The [GitLab monitor exporter] allows you to measure various GitLab metrics, pulled from Redis and the database. To enable the GitLab monitor exporter: diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index b2445d1c0e5..f43c89dad87 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -95,8 +95,9 @@ Sample Prometheus queries: ## Configuring Prometheus to monitor Kubernetes > Introduced in GitLab 9.0. +> Pod monitoring introduced in GitLab 9.4. -If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes in the cluster including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them. +If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them. To disable the monitoring of Kubernetes: @@ -110,6 +111,14 @@ To disable the monitoring of Kubernetes: 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect +## GitLab Prometheus metrics + +> Introduced as an experimental feature in GitLab 9.3. + +GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it is available on the same URL and port as user traffic. + +[➔ Read more about the GitLab Metrics.](gitlab_metrics.md) + ## Prometheus exporters There are a number of libraries and servers which help in exporting existing @@ -143,7 +152,7 @@ The Postgres exporter allows you to measure various PostgreSQL metrics. ### GitLab monitor exporter -The GitLab monitor exporter allows you to measure various GitLab metrics. +The GitLab monitor exporter allows you to measure various GitLab metrics, pulled from Redis and the database. [➔ Read more about the GitLab monitor exporter.](gitlab_monitor_exporter.md) diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 93521e976d5..3a35aff8366 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -15,6 +15,12 @@ prefixed with 'session:gitlab:', so they would look like 'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to remove the keys in the old format. +**Note:** the instructions below must be modified in accordance with your +configuration settings if you have used the advanced Redis +settings outlined in +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/README.md). + + First we define a shell function with the proper Redis connection details. ``` diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index affb4d17861..04c70c3644e 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -3,7 +3,7 @@ >**Note:** > > - [Introduced][ce-10308] in GitLab 9.1. -> - You need a personal access token in order to retrieve and import GitHub +> - You need a personal access token in order to retrieve and import GitHub > projects. You can get it from: https://github.com/settings/tokens > - You also need to pass an username as the second argument to the rake task > which will become the owner of the project. @@ -19,7 +19,7 @@ bundle exec rake import:github[access_token,root,foo/bar] RAILS_ENV=production ``` In this case, `access_token` is your GitHub personal access token, `root` -is your GitLab username, and `foo/bar` is the new GitLab namespace/project that +is your GitLab username, and `foo/bar` is the new GitLab namespace/project that will get created from your GitHub project. Subgroups are also possible: `foo/foo/bar`. diff --git a/doc/api/README.md b/doc/api/README.md index e1d4009dedc..f63d395b10e 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -11,12 +11,14 @@ following locations: - [Award Emoji](award_emoji.md) - [Branches](branches.md) - [Broadcast Messages](broadcast_messages.md) -- [Build Variables](build_variables.md) +- [Project-level Variables](project_level_variables.md) +- [Group-level Variables](group_level_variables.md) - [Commits](commits.md) - [Deployments](deployments.md) - [Deploy Keys](deploy_keys.md) - [Environments](environments.md) - [Events](events.md) +- [Feature flags](features.md) - [Gitignores templates](templates/gitignores.md) - [GitLab CI Config templates](templates/gitlab_ci_ymls.md) - [Groups](groups.md) @@ -28,11 +30,12 @@ following locations: - [Keys](keys.md) - [Labels](labels.md) - [Merge Requests](merge_requests.md) -- [Milestones](milestones.md) -- [Open source license templates](templates/licenses.md) +- [Project milestones](milestones.md) +- [Group milestones](group_milestones.md) - [Namespaces](namespaces.md) - [Notes](notes.md) (comments) - [Notification settings](notification_settings.md) +- [Open source license templates](templates/licenses.md) - [Pipelines](pipelines.md) - [Pipeline Triggers](pipeline_triggers.md) - [Pipeline Schedules](pipeline_schedules.md) @@ -40,6 +43,7 @@ following locations: - [Project Access Requests](access_requests.md) - [Project Members](members.md) - [Project Snippets](project_snippets.md) +- [Protected Branches](protected_branches.md) - [Repositories](repositories.md) - [Repository Files](repository_files.md) - [Runners](runners.md) @@ -55,19 +59,35 @@ following locations: - [V3 to V4](v3_to_v4.md) - [Version](version.md) -### Internal CI API - The following documentation is for the [internal CI API](ci/README.md): - [Builds](ci/builds.md) - [Runners](ci/runners.md) +## Road to GraphQL + +Going forward, we will start on moving to +[GraphQL](http://graphql.org/learn/best-practices/) and deprecate the use of +controller-specific endpoints. GraphQL has a number of benefits: + +1. We avoid having to maintain two different APIs. +2. Callers of the API can request only what they need. +3. It is versioned by default. + +It will co-exist with the current v4 REST API. If we have a v5 API, this should +be a compatibility layer on top of GraphQL. + ## Authentication -Most API requests require authentication via a session cookie or token. For those cases where it is not required, this will be mentioned in the documentation -for each individual endpoint. For example, the [`/projects/:id` endpoint](projects.md). -There are three types of tokens available: private tokens, OAuth 2 tokens, and personal -access tokens. +Most API requests require authentication via a session cookie or token. For +those cases where it is not required, this will be mentioned in the documentation +for each individual endpoint. For example, the [`/projects/:id` endpoint](projects.md). + +There are three types of access tokens available: + +1. [OAuth2 tokens](#oauth2-tokens) +1. [Private tokens](#private-tokens) +1. [Personal access tokens](#personal-access-tokens) If authentication information is invalid or omitted, an error message will be returned with status code `401`: @@ -78,20 +98,13 @@ returned with status code `401`: } ``` -### Session Cookie +### Session cookie When signing in to GitLab as an ordinary user, a `_gitlab_session` cookie is set. The API will use this cookie for authentication if it is present, but using the API to generate a new session cookie is currently not supported. -### Private Tokens - -You need to pass a `private_token` parameter via query string or header. If passed as a -header, the header name must be `PRIVATE-TOKEN` (uppercase and with a dash instead of -an underscore). You can find or reset your private token in your account page -(`/profile/account`). - -### OAuth 2 Tokens +### OAuth2 tokens You can use an OAuth 2 token to authenticate with the API by passing it either in the `access_token` parameter or in the `Authorization` header. @@ -104,30 +117,31 @@ curl --header "Authorization: Bearer OAUTH-TOKEN" https://gitlab.example.com/api Read more about [GitLab as an OAuth2 client](oauth2.md). -### Personal Access Tokens +### Private tokens -> [Introduced][ce-3749] in GitLab 8.8. +Private tokens provide full access to the GitLab API. Anyone with access to +them can interact with GitLab as if they were you. You can find or reset your +private token in your account page (`/profile/account`). -You can create as many personal access tokens as you like from your GitLab -profile (`/profile/personal_access_tokens`); perhaps one for each application -that needs access to the GitLab API. +For examples of usage, [read the basic usage section](#basic-usage). -Once you have your token, pass it to the API using either the `private_token` -parameter or the `PRIVATE-TOKEN` header. +### Personal access tokens -> [Introduced][ce-5951] in GitLab 8.15. +Instead of using your private token which grants full access to your account, +personal access tokens could be a better fit because of their granular +permissions. -Personal Access Tokens can be created with one or more scopes that allow various actions -that a given token can perform. Although there are only two scopes available at the -moment – `read_user` and `api` – the groundwork has been laid to add more scopes easily. +Once you have your token, pass it to the API using either the `private_token` +parameter or the `PRIVATE-TOKEN` header. For examples of usage, +[read the basic usage section](#basic-usage). -At any time you can revoke any personal access token by just clicking **Revoke**. +[Read more about personal access tokens.][pat] ### Impersonation tokens > [Introduced][ce-9099] in GitLab 9.0. Needs admin permissions. -Impersonation tokens are a type of [Personal Access Token](#personal-access-tokens) +Impersonation tokens are a type of [personal access token][pat] that can only be created by an admin for a specific user. They are a better alternative to using the user's password/private token @@ -136,9 +150,11 @@ or private token, since the password/token can change over time. Impersonation tokens are a great fit if you want to build applications or tools which authenticate with the API as a specific user. -For more information about the usage please refer to the +For more information, refer to the [users API](users.md#retrieve-user-impersonation-tokens) docs. +For examples of usage, [read the basic usage section](#basic-usage). + ### Sudo > Needs admin permissions. @@ -191,11 +207,16 @@ GET /projects?private_token=9koXpg98eAheJpvBs5tK&sudo=23 curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "SUDO: 23" "https://gitlab.example.com/api/v4/projects" ``` -## Basic Usage +## Basic usage API requests should be prefixed with `api` and the API version. The API version is defined in [`lib/api.rb`][lib-api-url]. +For endpoints that require [authentication](#authentication), you need to pass +a `private_token` parameter via query string or header. If passed as a header, +the header name must be `PRIVATE-TOKEN` (uppercase and with a dash instead of +an underscore). + Example of a valid API request: ``` @@ -208,6 +229,12 @@ Example of a valid API request using cURL and authentication via header: curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects" ``` +Example of a valid API request using cURL and authentication via a query string: + +```shell +curl "https://gitlab.example.com/api/v4/projects?private_token=9koXpg98eAheJpvBs5tK" +``` + The API uses JSON to serialize data. You don't need to specify `.json` at the end of an API URL. @@ -315,7 +342,18 @@ URL-encoded. For example, `/` is represented by `%2F`: ``` -/api/v4/projects/diaspora%2Fdiaspora +GET /api/v4/projects/diaspora%2Fdiaspora +``` + +## Branches & tags name encoding + +If your branch or tag contains a `/`, make sure the branch/tag name is +URL-encoded. + +For example, `/` is represented by `%2F`: + +``` +GET /api/v4/projects/1/branches/my%2Fbranch/commits ``` ## `id` vs `iid` @@ -423,3 +461,4 @@ programming languages. Visit the [GitLab website] for a complete list. [ce-3749]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3749 [ce-5951]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951 [ce-9099]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9099 +[pat]: ../user/profile/personal_access_tokens.md diff --git a/doc/api/branches.md b/doc/api/branches.md index 325d0ea4ce3..80744258acb 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -95,6 +95,8 @@ Example response: ## Protect repository branch +>**Note:** This API endpoint is deprecated in favor of `POST /projects/:id/protected_branches`. + Protects a single project repository branch. This is an idempotent function, protecting an already protected repository branch still returns a `200 OK` status code. @@ -143,6 +145,8 @@ Example response: ## Unprotect repository branch +>**Note:** This API endpoint is deprecated in favor of `DELETE /projects/:id/protected_branches/:name` + Unprotects a single project repository branch. This is an idempotent function, unprotecting an already unprotected repository branch still returns a `200 OK` status code. @@ -251,6 +255,8 @@ curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gi Will delete all branches that are merged into the project's default branch. +Protected branches will not be deleted as part of this operation. + ``` DELETE /projects/:id/repository/merged_branches ``` diff --git a/doc/api/ci/lint.md b/doc/api/ci/lint.md index 6a4dca92cfe..e4a6dc809b1 100644 --- a/doc/api/ci/lint.md +++ b/doc/api/ci/lint.md @@ -47,3 +47,5 @@ Example responses: "error": "content is missing" } ``` + +[ce-5953]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5953 diff --git a/doc/api/commits.md b/doc/api/commits.md index 9cb58dd3ae9..c91f9ecbdaf 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -97,7 +97,7 @@ PAYLOAD=$(cat << 'JSON' }, { "action": "delete", - "file_path": "foo/bar2", + "file_path": "foo/bar2" }, { "action": "move", diff --git a/doc/api/events.md b/doc/api/events.md index e7829c9f479..6e530317f6c 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -302,6 +302,7 @@ Example response: "project_id":1, "action_name":"opened", "target_id":160, + "target_iid":160, "target_type":"Issue", "author_id":25, "data":null, @@ -322,6 +323,7 @@ Example response: "project_id":1, "action_name":"opened", "target_id":159, + "target_iid":159, "target_type":"Issue", "author_id":21, "data":null, diff --git a/doc/api/features.md b/doc/api/features.md index 89b8d3ac948..6861dbf00a2 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -1,4 +1,4 @@ -# Features API +# Features flags API All methods require administrator authorization. @@ -58,6 +58,11 @@ POST /features/:name | --------- | ---- | -------- | ----------- | | `name` | string | yes | Name of the feature to create or update | | `value` | integer/string | yes | `true` or `false` to enable/disable, or an integer for percentage of time | +| `feature_group` | string | no | A Feature group name | +| `user` | string | no | A GitLab username | + +Note that you can enable or disable a feature for both a `feature_group` and a +`user` with a single API call. ```bash curl --data "value=30" --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/features/new_library diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md new file mode 100644 index 00000000000..e19be7b35c4 --- /dev/null +++ b/doc/api/group_level_variables.md @@ -0,0 +1,125 @@ +# Group-level Variables API + +## List group variables + +Get list of a group's variables. + +``` +GET /groups/:id/variables +``` + +| Attribute | Type | required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | + +``` +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/groups/1/variables" +``` + +```json +[ + { + "key": "TEST_VARIABLE_1", + "value": "TEST_1" + }, + { + "key": "TEST_VARIABLE_2", + "value": "TEST_2" + } +] +``` + +## Show variable details + +Get the details of a group's specific variable. + +``` +GET /groups/:id/variables/:key +``` + +| Attribute | Type | required | Description | +|-----------|---------|----------|-----------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | + +``` +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/groups/1/variables/TEST_VARIABLE_1" +``` + +```json +{ + "key": "TEST_VARIABLE_1", + "value": "TEST_1" +} +``` + +## Create variable + +Create a new variable. + +``` +POST /groups/:id/variables +``` + +| Attribute | Type | required | Description | +|-------------|---------|----------|-----------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | +| `value` | string | yes | The `value` of a variable | +| `protected` | boolean | no | Whether the variable is protected | + +``` +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/groups/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" +``` + +```json +{ + "key": "NEW_VARIABLE", + "value": "new value", + "protected": false +} +``` + +## Update variable + +Update a group's variable. + +``` +PUT /groups/:id/variables/:key +``` + +| Attribute | Type | required | Description | +|-------------|---------|----------|-------------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | +| `value` | string | yes | The `value` of a variable | +| `protected` | boolean | no | Whether the variable is protected | + +``` +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/groups/1/variables/NEW_VARIABLE" --form "value=updated value" +``` + +```json +{ + "key": "NEW_VARIABLE", + "value": "updated value", + "protected": true +} +``` + +## Remove variable + +Remove a group's variable. + +``` +DELETE /groups/:id/variables/:key +``` + +| Attribute | Type | required | Description | +|-----------|---------|----------|-------------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | + +``` +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/groups/1/variables/VARIABLE_1" +``` diff --git a/doc/api/group_milestones.md b/doc/api/group_milestones.md new file mode 100644 index 00000000000..086fba7e91d --- /dev/null +++ b/doc/api/group_milestones.md @@ -0,0 +1,120 @@ +# Group milestones API + +## List group milestones + +Returns a list of group milestones. + +``` +GET /groups/:id/milestones +GET /groups/:id/milestones?iids=42 +GET /groups/:id/milestones?iids[]=42&iids[]=43 +GET /groups/:id/milestones?state=active +GET /groups/:id/milestones?state=closed +GET /groups/:id/milestones?search=version +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `iids` | Array[integer] | optional | Return only the milestones having the given `iids` | +| `state` | string | optional | Return only `active` or `closed` milestones` | +| `search` | string | optional | Return only milestones with a title or description matching the provided string | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/milestones +``` + +Example Response: + +```json +[ + { + "id": 12, + "iid": 3, + "group_id": 16, + "title": "10.0", + "description": "Version", + "due_date": "2013-11-29", + "start_date": "2013-11-10", + "state": "active", + "updated_at": "2013-10-02T09:24:18Z", + "created_at": "2013-10-02T09:24:18Z" + } +] +``` + + +## Get single milestone + +Gets a single group milestone. + +``` +GET /groups/:id/milestones/:milestone_id +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of the group milestone + +## Create new milestone + +Creates a new group milestone. + +``` +POST /groups/:id/milestones +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user +- `title` (required) - The title of an milestone +- `description` (optional) - The description of the milestone +- `due_date` (optional) - The due date of the milestone +- `start_date` (optional) - The start date of the milestone + +## Edit milestone + +Updates an existing group milestone. + +``` +PUT /groups/:id/milestones/:milestone_id +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of a group milestone +- `title` (optional) - The title of a milestone +- `description` (optional) - The description of a milestone +- `due_date` (optional) - The due date of the milestone +- `start_date` (optional) - The start date of the milestone +- `state_event` (optional) - The state event of the milestone (close|activate) + +## Get all issues assigned to a single milestone + +Gets all issues assigned to a single group milestone. + +``` +GET /groups/:id/milestones/:milestone_id/issues +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of a group milestone + +## Get all merge requests assigned to a single milestone + +Gets all merge requests assigned to a single group milestone. + +``` +GET /groups/:id/milestones/:milestone_id/merge_requests +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of a group milestone diff --git a/doc/api/issues.md b/doc/api/issues.md index 3f949ca5667..6bac2927339 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -14,7 +14,9 @@ Read more on [pagination](README.md#pagination). ## List issues -Get all issues created by the authenticated user. +Get all issues the authenticated user has access to. By default it +returns only issues created by the current user. To get all issues, +use parameter `scope=all`. ``` GET /issues @@ -26,7 +28,8 @@ GET /issues?labels=foo,bar&state=opened GET /issues?milestone=1.0.0 GET /issues?milestone=1.0.0&state=opened GET /issues?iids[]=42&iids[]=43 -GET /issues?search=issue+title+or+description +GET /issues?author_id=5 +GET /issues?assignee_id=5 ``` | Attribute | Type | Required | Description | @@ -34,9 +37,12 @@ GET /issues?search=issue+title+or+description | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `No+Label` lists all issues with no labels | | `milestone` | string | no | The milestone title | +| `scope` | string | no | Return issues for the given scope: `created-by-me`, `assigned-to-me` or `all`. Defaults to `created-by-me` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_id` | integer | no | Return issues created by the given user `id`. Combine with `scope=all` or `scope=assigned-to-me`. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `assignee_id` | integer | no | Return issues assigned to the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | | `iids` | Array[integer] | no | Return only the issues having the given `iid` | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | -| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | +| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | ```bash @@ -117,6 +123,8 @@ GET /groups/:id/issues?milestone=1.0.0 GET /groups/:id/issues?milestone=1.0.0&state=opened GET /groups/:id/issues?iids[]=42&iids[]=43 GET /groups/:id/issues?search=issue+title+or+description +GET /groups/:id/issues?author_id=5 +GET /groups/:id/issues?assignee_id=5 ``` | Attribute | Type | Required | Description | @@ -126,9 +134,12 @@ GET /groups/:id/issues?search=issue+title+or+description | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `No+Label` lists all issues with no labels | | `iids` | Array[integer] | no | Return only the issues having the given `iid` | | `milestone` | string | no | The milestone title | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | -| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | -| `search` | string | no | Search group issues against their `title` and `description` | +| `scope` | string | no | Return issues for the given scope: `created-by-me`, `assigned-to-me` or `all` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_id` | integer | no | Return issues created by the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `assignee_id` | integer | no | Return issues assigned to the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Search group issues against their `title` and `description` | ```bash @@ -209,6 +220,8 @@ GET /projects/:id/issues?milestone=1.0.0 GET /projects/:id/issues?milestone=1.0.0&state=opened GET /projects/:id/issues?iids[]=42&iids[]=43 GET /projects/:id/issues?search=issue+title+or+description +GET /projects/:id/issues?author_id=5 +GET /projects/:id/issues?assignee_id=5 ``` | Attribute | Type | Required | Description | @@ -218,10 +231,14 @@ GET /projects/:id/issues?search=issue+title+or+description | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `No+Label` lists all issues with no labels | | `milestone` | string | no | The milestone title | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | -| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | -| `search` | string | no | Search project issues against their `title` and `description` | - +| `scope` | string | no | Return issues for the given scope: `created-by-me`, `assigned-to-me` or `all` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_id` | integer | no | Return issues created by the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `assignee_id` | integer | no | Return issues assigned to the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | +| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Search project issues against their `title` and `description` | +| `created_after` | datetime | no | Return issues created after the given time (inclusive) | +| `created_before` | datetime | no | Return issues created before the given time (inclusive) | ```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/4/issues @@ -355,7 +372,13 @@ Example response: "user_notes_count": 1, "due_date": null, "web_url": "http://example.com/example/example/issues/1", - "confidential": false + "confidential": false, + "_links": { + "self": "http://example.com/api/v4/projects/1/issues/2", + "notes": "http://example.com/api/v4/projects/1/issues/2/notes", + "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", + "project": "http://example.com/api/v4/projects/1" + } } ``` @@ -417,7 +440,13 @@ Example response: "user_notes_count": 0, "due_date": null, "web_url": "http://example.com/example/example/issues/14", - "confidential": false + "confidential": false, + "_links": { + "self": "http://example.com/api/v4/projects/1/issues/2", + "notes": "http://example.com/api/v4/projects/1/issues/2/notes", + "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", + "project": "http://example.com/api/v4/projects/1" + } } ``` @@ -480,7 +509,13 @@ Example response: "user_notes_count": 0, "due_date": "2016-07-22", "web_url": "http://example.com/example/example/issues/15", - "confidential": false + "confidential": false, + "_links": { + "self": "http://example.com/api/v4/projects/1/issues/2", + "notes": "http://example.com/api/v4/projects/1/issues/2/notes", + "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", + "project": "http://example.com/api/v4/projects/1" + } } ``` @@ -566,7 +601,13 @@ Example response: }, "due_date": null, "web_url": "http://example.com/example/example/issues/11", - "confidential": false + "confidential": false, + "_links": { + "self": "http://example.com/api/v4/projects/1/issues/2", + "notes": "http://example.com/api/v4/projects/1/issues/2/notes", + "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", + "project": "http://example.com/api/v4/projects/1" + } } ``` @@ -631,7 +672,13 @@ Example response: }, "due_date": null, "web_url": "http://example.com/example/example/issues/11", - "confidential": false + "confidential": false, + "_links": { + "self": "http://example.com/api/v4/projects/1/issues/2", + "notes": "http://example.com/api/v4/projects/1/issues/2/notes", + "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", + "project": "http://example.com/api/v4/projects/1" + } } ``` @@ -963,3 +1010,32 @@ Example response: ## Comments on issues Comments are done via the [notes](notes.md) resource. + +## Get user agent details + +Available only for admins. + +``` +GET /projects/:id/issues/:issue_iid/user_agent_detail +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/93/user_agent_detail +``` + +Example response: + +```json +{ + "user_agent": "AppleWebKit/537.36", + "ip_address": "127.0.0.1", + "akismet_submitted": false +} +``` + +[ce-13004]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13004 diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index cb22b67f556..d0725b5e06e 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -1,7 +1,104 @@ # Merge requests API +Every API call to merge requests must be authenticated. + ## List merge requests +> [Introduced][ce-13060] in GitLab 9.5. + +Get all merge requests the authenticated user has access to. By +default it returns only merge requests created by the current user. To +get all merge requests, use parameter `scope=all`. + +The `state` parameter can be used to get only merge requests with a +given state (`opened`, `closed`, or `merged`) or all of them (`all`). +The pagination parameters `page` and `per_page` can be used to +restrict the list of merge requests. + +``` +GET /merge_requests +GET /merge_requests?state=opened +GET /merge_requests?state=all +GET /merge_requests?milestone=release +GET /merge_requests?labels=bug,reproduced +GET /merge_requests?author_id=5 +GET /merge_requests?scope=assigned-to-me +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, or `merged`| +| `order_by`| string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | +| `milestone` | string | no | Return merge requests for a specific milestone | +| `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | +| `labels` | string | no | Return merge requests matching a comma separated list of labels | +| `created_after` | datetime | no | Return merge requests created after the given time (inclusive) | +| `created_before` | datetime | no | Return merge requests created before the given time (inclusive) | +| `scope` | string | no | Return merge requests for the given scope: `created-by-me`, `assigned-to-me` or `all`. Defaults to `created-by-me` | +| `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned-to-me` | +| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id` | + +```json +[ + { + "id": 1, + "iid": 1, + "target_branch": "master", + "source_branch": "test1", + "project_id": 3, + "title": "test1", + "state": "opened", + "upvotes": 0, + "downvotes": 0, + "author": { + "id": 1, + "username": "admin", + "email": "admin@example.com", + "name": "Administrator", + "state": "active", + "created_at": "2012-04-29T08:46:00Z" + }, + "assignee": { + "id": 1, + "username": "admin", + "email": "admin@example.com", + "name": "Administrator", + "state": "active", + "created_at": "2012-04-29T08:46:00Z" + }, + "source_project_id": 2, + "target_project_id": 3, + "labels": [ ], + "description": "fixed login page css paddings", + "work_in_progress": false, + "milestone": { + "id": 5, + "iid": 1, + "project_id": 3, + "title": "v2.0", + "description": "Assumenda aut placeat expedita exercitationem labore sunt enim earum.", + "state": "closed", + "created_at": "2015-02-02T19:49:26.013Z", + "updated_at": "2015-02-02T19:49:26.013Z", + "due_date": null + }, + "merge_when_pipeline_succeeds": true, + "merge_status": "can_be_merged", + "sha": "8888888888888888888888888888888888888888", + "merge_commit_sha": null, + "user_notes_count": 1, + "should_remove_source_branch": true, + "force_remove_source_branch": false, + "web_url": "http://example.com/example/example/merge_requests/1" + } +] +``` + +## List project merge requests + Get all merge requests for this project. The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. @@ -25,7 +122,13 @@ Parameters: | `order_by`| string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | | `milestone` | string | no | Return merge requests for a specific milestone | +| `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | | `labels` | string | no | Return merge requests matching a comma separated list of labels | +| `created_after` | datetime | no | Return merge requests created after the given time (inclusive) | +| `created_before` | datetime | no | Return merge requests created before the given time (inclusive) | +| `scope` | string | no | Return merge requests for the given scope: `created-by-me`, `assigned-to-me` or `all` _([Introduced][ce-13060] in GitLab 9.5)_ | +| `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | +| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | ```json [ @@ -1162,3 +1265,5 @@ Example response: "total_time_spent": 3600 } ``` + +[ce-13060]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13060 diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index 4ad6071a0ed..8133251dffe 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -29,22 +29,30 @@ Example response: { "id": 1, "path": "user1", - "kind": "user" + "kind": "user", + "full_path": "user1" }, { "id": 2, "path": "group1", - "kind": "group" + "kind": "group", + "full_path": "group1", + "parent_id": "null", + "members_count_with_descendants": 2 }, { "id": 3, "path": "bar", "kind": "group", "full_path": "foo/bar", + "parent_id": "9", + "members_count_with_descendants": 5 } ] ``` +**Note**: `members_count_with_descendants` are presented only for group masters/owners. + ## Search for namespace Get all namespaces that match a string in their name or path. @@ -72,6 +80,8 @@ Example response: "path": "twitter", "kind": "group", "full_path": "twitter", + "parent_id": "null", + "members_count_with_descendants": 2 } ] ``` diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 46fe64d382e..77bb7a55d8c 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -1,59 +1,55 @@ # GitLab as an OAuth2 provider -This document covers using the OAuth2 protocol to access GitLab. +This document covers using the [OAuth2](https://oauth.net/2/) protocol to allow other services access Gitlab resources on user's behalf. -If you want GitLab to be an OAuth authentication service provider to sign into other services please see the [Oauth2 provider documentation](../integration/oauth_provider.md). +If you want GitLab to be an OAuth authentication service provider to sign into other services please see the [OAuth2 provider](../integration/oauth_provider.md) +documentation. -OAuth2 is a protocol that enables us to authenticate a user without requiring them to give their password to a third-party. +This functionality is based on [doorkeeper gem](https://github.com/doorkeeper-gem/doorkeeper). -This functionality is based on [doorkeeper gem](https://github.com/doorkeeper-gem/doorkeeper) +## Supported OAuth2 Flows -## Web Application Flow +Gitlab currently supports following authorization flows: -This is the most common type of flow and is used by server-side clients that wish to access GitLab on a user's behalf. +* *Web Application Flow* - Most secure and common type of flow, designed for the applications with secure server-side. +* *Implicit Flow* - This flow is designed for user-agent only apps (e.g. single page web application running on GitLab Pages). +* *Resource Owner Password Credentials Flow* - To be used **only** for securely hosted, first-party services. ->**Note:** -This flow **should not** be used for client-side clients as you would leak your `client_secret`. Client-side clients should use the Implicit Grant (which is currently unsupported). +Please refer to [OAuth RFC](https://tools.ietf.org/html/rfc6749) to find out in details how all those flows work and pick the right one for your use case. -For more detailed information, check out the [RFC spec](http://tools.ietf.org/html/rfc6749#section-4.1) +Both *web application* and *implicit* flows require `application` to be registered first via `/profile/applications` page +in your user's account. During registration, by enabling proper scopes you can limit the range of resources which the `application` can access. Upon creation +you'll obtain `application` credentials: _Application ID_ and _Client Secret_ - **keep them secure**. -In the following sections you will be introduced to the three steps needed for this flow. +>**Important:** OAuth specification advises sending `state` parameter with each request to `/oauth/authorize`. We highly recommended to send a unique +value with each request and validate it against the one in redirect request. This is important to prevent [CSRF attacks]. The `state` param really should +have been a requirement in the standard! -### 1. Registering the client +In the following sections you will find detailed instructions on how to obtain authorization with each flow. -First, you should create an application (`/profile/applications`) in your user's account. -Each application gets a unique App ID and App Secret parameters. +### Web Application Flow ->**Note:** -**You should not share/leak your App ID or App Secret.** +Check [RFC spec](http://tools.ietf.org/html/rfc6749#section-4.1) for a detailed flow description -### 2. Requesting authorization +#### 1. Requesting authorization code -To request the authorization code, you should redirect the user to the `/oauth/authorize` endpoint: +To request the authorization code, you should redirect the user to the `/oauth/authorize` endpoint with following GET parameters: ``` -https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=code&state=your_unique_state_hash +https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=code&state=YOUR_UNIQUE_STATE_HASH ``` -This will ask the user to approve the applications access to their account and then redirect back to the `REDIRECT_URI` you provided. +This will ask the user to approve the applications access to their account and then redirect back to the `REDIRECT_URI` you provided. The redirect will +include the GET `code` parameter, for example: -The redirect will include the GET `code` parameter, for example: - -``` -http://myapp.com/oauth/redirect?code=1234567890&state=your_unique_state_hash -``` +`http://myapp.com/oauth/redirect?code=1234567890&state=YOUR_UNIQUE_STATE_HASH` You should then use the `code` to request an access token. ->**Important:** -It is highly recommended that you send a `state` value with the request to `/oauth/authorize` and -validate that value is returned and matches in the redirect request. -This is important to prevent [CSRF attacks](http://www.oauthsecurity.com/#user-content-authorization-code-flow), -`state` really should have been a requirement in the standard! - -### 3. Requesting the access token +#### 2. Requesting access token -Once you have the authorization code you can request an `access_token` using the code, to do that you can use any HTTP client. In the following example, we are using Ruby's `rest-client`: +Once you have the authorization code you can request an `access_token` using the code, to do that you can use any HTTP client. In the following example, +we are using Ruby's `rest-client`: ``` parameters = 'client_id=APP_ID&client_secret=APP_SECRET&code=RETURNED_CODE&grant_type=authorization_code&redirect_uri=REDIRECT_URI' @@ -72,28 +68,40 @@ The `redirect_uri` must match the `redirect_uri` used in the original authorizat You can now make requests to the API with the access token returned. -### Use the access token to access the API -The access token allows you to make requests to the API on a behalf of a user. +### Implicit Grant + +Check [RFC spec](http://tools.ietf.org/html/rfc6749#section-4.2) for a detailed flow description. + +Unlike the web flow, the client receives an `access token` immediately as a result of the authorization request. The flow does not use client secret +or authorization code because all of the application code and storage is easily accessible, therefore __secrets__ can leak easily. + +>**Important:** Avoid using this flow for applications that store data outside of the Gitlab instance. If you do, make sure to verify `application id` +associated with access token before granting access to the data +(see [/oauth/token/info](https://github.com/doorkeeper-gem/doorkeeper/wiki/API-endpoint-descriptions-and-examples#get----oauthtokeninfo)). + + +#### 1. Requesting access token + +To request the access token, you should redirect the user to the `/oauth/authorize` endpoint using `token` response type: ``` -GET https://gitlab.example.com/api/v4/user?access_token=OAUTH-TOKEN +https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=token&state=YOUR_UNIQUE_STATE_HASH ``` -Or you can put the token to the Authorization header: +This will ask the user to approve the applications access to their account and then redirect back to the `REDIRECT_URI` you provided. The redirect +will include a fragment with `access_token` as well as token details in GET parameters, for example: ``` -curl --header "Authorization: Bearer OAUTH-TOKEN" https://gitlab.example.com/api/v4/user +http://myapp.com/oauth/redirect#access_token=ABCDExyz123&state=YOUR_UNIQUE_STATE_HASH&token_type=bearer&expires_in=3600 ``` -## Resource Owner Password Credentials +### Resource Owner Password Credentials -## Deprecation Notice +Check [RFC spec](http://tools.ietf.org/html/rfc6749#section-4.3) for a detailed flow description. -1. Starting in GitLab 8.11, the Resource Owner Password Credentials has been *disabled* for users with two-factor authentication turned on. -2. These users can access the API using [personal access tokens] instead. - ---- +> **Deprecation notice:** Starting in GitLab 8.11, the Resource Owner Password Credentials has been *disabled* for users with two-factor authentication +turned on. These users can access the API using [personal access tokens] instead. In this flow, a token is requested in exchange for the resource owner credentials (username and password). The credentials should only be used when there is a high degree of trust between the resource owner and the client (e.g. the @@ -101,12 +109,16 @@ client is part of the device operating system or a highly privileged application available (such as an authorization code). >**Important:** -Never store the users credentials and only use this grant type when your client is deployed to a trusted environment, in 99% of cases [personal access tokens] are a better choice. +Never store the users credentials and only use this grant type when your client is deployed to a trusted environment, in 99% of cases [personal access tokens] +are a better choice. Even though this grant type requires direct client access to the resource owner credentials, the resource owner credentials are used for a single request and are exchanged for an access token. This grant type can eliminate the need for the client to store the resource owner credentials for future use, by exchanging the credentials with a long-lived access token or refresh token. -You can do POST request to `/oauth/token` with parameters: + +#### 1. Requesting access token + +POST request to `/oauth/token` with parameters: ``` { @@ -134,4 +146,18 @@ access_token = client.password.get_token('user@example.com', 'secret') puts access_token.token ``` -[personal access tokens]: ./README.md#personal-access-tokens
\ No newline at end of file +## Access Gitlab API with `access token` + +The `access token` allows you to make requests to the API on a behalf of a user. You can pass the token either as GET parameter +``` +GET https://gitlab.example.com/api/v4/user?access_token=OAUTH-TOKEN +``` + +or you can put the token to the Authorization header: + +``` +curl --header "Authorization: Bearer OAUTH-TOKEN" https://gitlab.example.com/api/v4/user +``` + +[personal access tokens]: ../user/profile/personal_access_tokens.md +[CSRF attacks]: http://www.oauthsecurity.com/#user-content-authorization-code-flow
\ No newline at end of file diff --git a/doc/api/build_variables.md b/doc/api/project_level_variables.md index d4f00256ed3..82ac0b09027 100644 --- a/doc/api/build_variables.md +++ b/doc/api/project_level_variables.md @@ -1,8 +1,8 @@ -# Build Variables API +# Project-level Variables API ## List project variables -Get list of a project's build variables. +Get list of a project's variables. ``` GET /projects/:id/variables @@ -31,7 +31,7 @@ curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/ ## Show variable details -Get the details of a project's specific build variable. +Get the details of a project's specific variable. ``` GET /projects/:id/variables/:key @@ -55,7 +55,7 @@ curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/ ## Create variable -Create a new build variable. +Create a new variable. ``` POST /projects/:id/variables @@ -82,7 +82,7 @@ curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitl ## Update variable -Update a project's build variable. +Update a project's variable. ``` PUT /projects/:id/variables/:key @@ -109,7 +109,7 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitla ## Remove variable -Remove a project's build variable. +Remove a project's variable. ``` DELETE /projects/:id/variables/:key diff --git a/doc/api/project_snippets.md b/doc/api/project_snippets.md index ff379473961..d74398c6e65 100644 --- a/doc/api/project_snippets.md +++ b/doc/api/project_snippets.md @@ -43,6 +43,7 @@ Parameters: "id": 1, "title": "test", "file_name": "add.rb", + "description": "Ruby test snippet", "author": { "id": 1, "username": "john_smith", @@ -70,6 +71,7 @@ Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `title` (required) - The title of a snippet - `file_name` (required) - The name of a snippet file +- `description` (optional) - The description of a snippet - `code` (required) - The content of a snippet - `visibility` (required) - The snippet's visibility @@ -87,6 +89,7 @@ Parameters: - `snippet_id` (required) - The ID of a project's snippet - `title` (optional) - The title of a snippet - `file_name` (optional) - The name of a snippet file +- `description` (optional) - The description of a snippet - `code` (optional) - The content of a snippet - `visibility` (optional) - The snippet's visibility @@ -116,3 +119,35 @@ Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `snippet_id` (required) - The ID of a project's snippet + +## Get user agent details + +> **Notes:** +> [Introduced][ce-29508] in GitLab 9.4. + + +Available only for admins. + +``` +GET /projects/:id/snippets/:snippet_id/user_agent_detail +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | Integer | yes | The ID of a snippet | + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/snippets/1/user_agent_detail +``` + +Example response: + +```json +{ + "user_agent": "AppleWebKit/537.36", + "ip_address": "127.0.0.1", + "akismet_submitted": false +} +``` + +[ce-[ce-29508]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29508]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29508 diff --git a/doc/api/projects.md b/doc/api/projects.md index 0debdcfae89..d3f8e509612 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -2,10 +2,10 @@ ### Project visibility level -Project in GitLab has be either private, internal or public. -You can determine it by `visibility` field in project. +Project in GitLab can be either private, internal or public. +This is determined by the `visibility` field in the project. -Constants for project visibility levels are next: +Values for the project visibility level are: * `private`: Project access must be granted explicitly for each user. @@ -18,7 +18,7 @@ Constants for project visibility levels are next: ## List projects -Get a list of visible projects for authenticated user. When being accessed without authentication, all public projects are returned. +Get a list of visible projects for authenticated user. When accessed without authentication, only public projects are returned. ``` GET /projects @@ -99,6 +99,191 @@ Parameters: "repository_size": 1038090, "lfs_objects_size": 0, "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + }, + }, + { + "id": 6, + "description": null, + "default_branch": "master", + "visibility": "private", + "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", + "http_url_to_repo": "http://example.com/brightbox/puppet.git", + "web_url": "http://example.com/brightbox/puppet", + "tag_list": [ + "example", + "puppet" + ], + "owner": { + "id": 4, + "name": "Brightbox", + "created_at": "2013-09-30T13:46:02Z" + }, + "name": "Puppet", + "name_with_namespace": "Brightbox / Puppet", + "path": "puppet", + "path_with_namespace": "brightbox/puppet", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "jobs_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "container_registry_enabled": false, + "created_at": "2013-09-30T13:46:02Z", + "last_activity_at": "2013-09-30T13:46:02Z", + "creator_id": 3, + "namespace": { + "id": 4, + "name": "Brightbox", + "path": "brightbox", + "kind": "group", + "full_path": "brightbox" + }, + "import_status": "none", + "import_error": null, + "permissions": { + "project_access": { + "access_level": 10, + "notification_level": 3 + }, + "group_access": { + "access_level": 50, + "notification_level": 3 + } + }, + "archived": false, + "avatar_url": null, + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0, + "runners_token": "b8547b1dc37721d05889db52fa2f02", + "public_jobs": true, + "shared_with_groups": [], + "only_allow_merge_if_pipeline_succeeds": false, + "only_allow_merge_if_all_discussions_are_resolved": false, + "request_access_enabled": false, + "statistics": { + "commit_count": 12, + "storage_size": 2066080, + "repository_size": 2066080, + "lfs_objects_size": 0, + "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } + } +] +``` + +### List a user's projects + +Get a list of visible projects for the given user. When accessed without authentication, only public projects are returned. + +``` +GET /users/:user_id/projects +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `user_id` | string | yes | The ID or username of the user | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of projects matching the search criteria | +| `simple` | boolean | no | Return only the ID, URL, name, and path of each project | +| `owned` | boolean | no | Limit by projects owned by the current user | +| `membership` | boolean | no | Limit by projects that the current user is a member of | +| `starred` | boolean | no | Limit by projects starred by the current user | +| `statistics` | boolean | no | Include project statistics | +| `with_issues_enabled` | boolean | no | Limit by enabled issues feature | +| `with_merge_requests_enabled` | boolean | no | Limit by enabled merge requests feature | + +```json +[ + { + "id": 4, + "description": null, + "default_branch": "master", + "visibility": "private", + "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", + "web_url": "http://example.com/diaspora/diaspora-client", + "tag_list": [ + "example", + "disapora client" + ], + "owner": { + "id": 3, + "name": "Diaspora", + "created_at": "2013-09-30T13:46:02Z" + }, + "name": "Diaspora Client", + "name_with_namespace": "Diaspora / Diaspora Client", + "path": "diaspora-client", + "path_with_namespace": "diaspora/diaspora-client", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "jobs_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "container_registry_enabled": false, + "created_at": "2013-09-30T13:46:02Z", + "last_activity_at": "2013-09-30T13:46:02Z", + "creator_id": 3, + "namespace": { + "id": 3, + "name": "Diaspora", + "path": "diaspora", + "kind": "group", + "full_path": "diaspora" + }, + "import_status": "none", + "archived": false, + "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0, + "runners_token": "b8547b1dc37721d05889db52fa2f02", + "public_jobs": true, + "shared_with_groups": [], + "only_allow_merge_if_pipeline_succeeds": false, + "only_allow_merge_if_all_discussions_are_resolved": false, + "request_access_enabled": false, + "statistics": { + "commit_count": 37, + "storage_size": 1038090, + "repository_size": 1038090, + "lfs_objects_size": 0, + "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" } }, { @@ -168,6 +353,15 @@ Parameters: "repository_size": 2066080, "lfs_objects_size": 0, "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" } } ] @@ -261,6 +455,7 @@ Parameters: ], "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, + "printing_merge_requests_link_enabled": true, "request_access_enabled": false, "statistics": { "commit_count": 37, @@ -268,6 +463,15 @@ Parameters: "repository_size": 1038090, "lfs_objects_size": 0, "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" } } ``` @@ -335,7 +539,7 @@ Parameters: | `snippets_enabled` | boolean | no | Enable snippets for this project | | `container_registry_enabled` | boolean | no | Enable container registry for this project | | `shared_runners_enabled` | boolean | no | Enable shared runners for this project | -| `visibility` | String | no | See [project visibility level](#project-visibility-level) | +| `visibility` | string | no | See [project visibility level](#project-visibility-level) | | `import_url` | string | no | URL to import repository from | | `public_jobs` | boolean | no | If `true`, jobs can be viewed by non-project-members | | `only_allow_merge_if_pipeline_succeeds` | boolean | no | Set whether merge requests can only be merged with successful jobs | @@ -343,6 +547,9 @@ Parameters: | `lfs_enabled` | boolean | no | Enable LFS | | `request_access_enabled` | boolean | no | Allow users to request member access | | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | +| `avatar` | mixed | no | Image file for avatar of the project | +| `printing_merge_request_link_enabled` | boolean | no | Show link to create/view merge request when pushing from the command line | +| `ci_config_path` | string | no | The path to CI config file | ### Create project for user @@ -377,6 +584,9 @@ Parameters: | `lfs_enabled` | boolean | no | Enable LFS | | `request_access_enabled` | boolean | no | Allow users to request member access | | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | +| `avatar` | mixed | no | Image file for avatar of the project | +| `printing_merge_request_link_enabled` | boolean | no | Show link to create/view merge request when pushing from the command line | +| `ci_config_path` | string | no | The path to CI config file | ### Edit project @@ -410,11 +620,15 @@ Parameters: | `lfs_enabled` | boolean | no | Enable LFS | | `request_access_enabled` | boolean | no | Allow users to request member access | | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | +| `avatar` | mixed | no | Image file for avatar of the project | +| `ci_config_path` | string | no | The path to CI config file | ### Fork project Forks a project into the user namespace of the authenticated user or the one provided. +The forking operation for a project is asynchronous and is completed in a background job. The request will return immediately. To determine whether the fork of the project has completed, query the `import_status` for the new project. + ``` POST /projects/:id/fork ``` @@ -490,7 +704,16 @@ Example response: "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, - "request_access_enabled": false + "request_access_enabled": false, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } } ``` @@ -556,7 +779,16 @@ Example response: "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, - "request_access_enabled": false + "request_access_enabled": false, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } } ``` @@ -640,7 +872,16 @@ Example response: "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, - "request_access_enabled": false + "request_access_enabled": false, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } } ``` @@ -724,7 +965,16 @@ Example response: "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, - "request_access_enabled": false + "request_access_enabled": false, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } } ``` @@ -1088,17 +1338,21 @@ endpoint can be accessed without authentication if the project is publicly accessible. ``` -GET /projects/search/:query +GET /projects ``` Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `query` | string | yes | A string contained in the project name | +| `search` | string | yes | A string contained in the project name | | `order_by` | string | no | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields | | `sort` | string | no | Return requests sorted in `asc` or `desc` order | +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects?search=test +``` + ## Start the Housekeeping task for a Project >**Note:** This feature was introduced in GitLab 9.0 diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md new file mode 100644 index 00000000000..10faa95d7e8 --- /dev/null +++ b/doc/api/protected_branches.md @@ -0,0 +1,145 @@ +# Protected branches API + +>**Note:** This feature was introduced in GitLab 9.5 + +**Valid access levels** + +The access levels are defined in the `ProtectedBranchAccess::ALLOWED_ACCESS_LEVELS` constant. Currently, these levels are recognized: +``` +0 => No access +30 => Developer access +40 => Master access +``` + +## List protected branches + +Gets a list of protected branches from a project. + +``` +GET /projects/:id/protected_branches +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v4/projects/5/protected_branches' +``` + +Example response: + +```json +[ + { + "name": "master", + "push_access_levels": [ + { + "access_level": 40, + "access_level_description": "Masters" + } + ], + "merge_access_levels": [ + { + "access_level": 40, + "access_level_description": "Masters" + } + ] + }, + ... +] +``` + +## Get a single protected branch or wildcard protected branch + +Gets a single protected branch or wildcard protected branch. + +``` +GET /projects/:id/protected_branches/:name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the branch or wildcard | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v4/projects/5/protected_branches/master' +``` + +Example response: + +```json +{ + "name": "master", + "push_access_levels": [ + { + "access_level": 40, + "access_level_description": "Masters" + } + ], + "merge_access_levels": [ + { + "access_level": 40, + "access_level_description": "Masters" + } + ] +} +``` + +## Protect repository branches + +Protects a single repository branch or several project repository +branches using a wildcard protected branch. + +``` +POST /projects/:id/protected_branches +``` + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30' +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the branch or wildcard | +| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, master access level) | +| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, master access level) | + +Example response: + +```json +{ + "name": "*-stable", + "push_access_levels": [ + { + "access_level": 30, + "access_level_description": "Developers + Masters" + } + ], + "merge_access_levels": [ + { + "access_level": 30, + "access_level_description": "Developers + Masters" + } + ] +} +``` + +## Unprotect repository branches + +Unprotects the given protected branch or wildcard protected branch. + +``` +DELETE /projects/:id/protected_branches/:name +``` + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v4/projects/5/protected_branches/*-stable' +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the branch | diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 18ceb8f779e..1fc577561a0 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -61,7 +61,7 @@ POST /projects/:id/repository/files/:file_path ``` ```bash -curl --request POST --header 'PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK' 'https://gitlab.example.com/api/v4/projects/13083/repository/app%2Fprojectrb%2E?branch=master&author_email=author%40example.com&author_name=Firstname%20Lastname&content=some%20content&commit_message=create%20a%20new%20file' +curl --request POST --header 'PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fprojectrb%2E?branch=master&author_email=author%40example.com&author_name=Firstname%20Lastname&content=some%20content&commit_message=create%20a%20new%20file' ``` Example response: @@ -90,7 +90,7 @@ PUT /projects/:id/repository/files/:file_path ``` ```bash -curl --request PUT --header 'PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK' 'https://gitlab.example.com/api/v4/projects/13083/repository/app%2Fproject%2Erb?branch=master&author_email=author%40example.com&author_name=Firstname%20Lastname&content=some%20other%20content&commit_message=update%20file' +curl --request PUT --header 'PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb?branch=master&author_email=author%40example.com&author_name=Firstname%20Lastname&content=some%20other%20content&commit_message=update%20file' ``` Example response: @@ -129,7 +129,7 @@ DELETE /projects/:id/repository/files/:file_path ``` ```bash -curl --request DELETE --header 'PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK' 'https://gitlab.example.com/api/v4/projects/13083/repository/app%2Fproject%2Erb?branch=master&author_email=author%40example.com&author_name=Firstname%20Lastname&commit_message=delete%20file' +curl --request DELETE --header 'PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb?branch=master&author_email=author%40example.com&author_name=Firstname%20Lastname&commit_message=delete%20file' ``` Example response: diff --git a/doc/api/session.md b/doc/api/session.md index 7dd504b67c5..f79eac11689 100644 --- a/doc/api/session.md +++ b/doc/api/session.md @@ -1,11 +1,9 @@ # Session API -## Deprecation Notice - -1. Starting in GitLab 8.11, this feature has been *disabled* for users with two-factor authentication turned on. -2. These users can access the API using [personal access tokens] instead. - ---- +>**Deprecation notice:** +Starting in GitLab 8.11, this feature has been **disabled** for users with +[two-factor authentication][2fa] turned on. These users can access the API +using [personal access tokens] instead. You can login with both GitLab and LDAP credentials in order to obtain the private token. @@ -52,4 +50,5 @@ Example response: } ``` -[personal access tokens]: ./README.md#personal-access-tokens +[2fa]: ../user/profile/account/two_factor_authentication.md +[personal access tokens]: ../user/profile/personal_access_tokens.md diff --git a/doc/api/settings.md b/doc/api/settings.md index eefbdda42ce..94a9f8265fb 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -25,7 +25,7 @@ Example response: "id" : 1, "default_branch_protection" : 2, "restricted_visibility_levels" : [], - "signin_enabled" : true, + "password_authentication_enabled" : true, "after_sign_out_path" : null, "max_attachment_size" : 10, "user_oauth_applications" : true, @@ -42,7 +42,6 @@ Example response: "gravatar_enabled" : true, "sign_in_text" : null, "container_registry_token_expire_delay": 5, - "repository_storage": "default", "repository_storages": ["default"], "koding_enabled": false, "koding_url": null, @@ -63,7 +62,7 @@ PUT /application/settings | --------- | ---- | :------: | ----------- | | `default_projects_limit` | integer | no | Project limit per user. Default is `100000` | | `signup_enabled` | boolean | no | Enable registration. Default is `true`. | -| `signin_enabled` | boolean | no | Enable login via a GitLab account. Default is `true`. | +| `password_authentication_enabled` | boolean | no | Enable authentication via a GitLab account password. Default is `true`. | | `gravatar_enabled` | boolean | no | Enable Gravatar | | `sign_in_text` | string | no | Text on login page | | `home_page_url` | string | no | Redirect to this URL when not logged in | @@ -81,7 +80,6 @@ PUT /application/settings | `after_sign_out_path` | string | no | Where to redirect users after logout | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes | | `repository_storages` | array of strings | no | A list of names of enabled storage paths, taken from `gitlab.yml`. New projects will be created in one of these stores, chosen at random. | -| `repository_storage` | string | no | The first entry in `repository_storages`. Deprecated, but retained for compatibility reasons | | `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | | `koding_enabled` | boolean | no | Enable Koding integration. Default is `false`. | | `koding_url` | string | yes (if `koding_enabled` is `true`) | The Koding instance URL for integration. | @@ -102,7 +100,7 @@ Example response: "id": 1, "default_projects_limit": 100000, "signup_enabled": true, - "signin_enabled": true, + "password_authentication_enabled": true, "gravatar_enabled": true, "sign_in_text": "", "created_at": "2015-06-12T15:51:55.432Z", @@ -121,7 +119,7 @@ Example response: "user_oauth_applications": true, "after_sign_out_path": "", "container_registry_token_expire_delay": 5, - "repository_storage": "default", + "repository_storages": ["default"], "koding_enabled": false, "koding_url": null, "plantuml_enabled": false, diff --git a/doc/api/snippets.md b/doc/api/snippets.md index fb8cf97896c..fdafbfb5b9e 100644 --- a/doc/api/snippets.md +++ b/doc/api/snippets.md @@ -48,6 +48,7 @@ Example response: "id": 1, "title": "test", "file_name": "add.rb", + "description": "Ruby test snippet", "author": { "id": 1, "username": "john_smith", @@ -73,16 +74,17 @@ POST /snippets Parameters: -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `title` | String | yes | The title of a snippet | -| `file_name` | String | yes | The name of a snippet file | -| `content` | String | yes | The content of a snippet | -| `visibility` | String | yes | The snippet's visibility | +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `title` | String | yes | The title of a snippet | +| `file_name` | String | yes | The name of a snippet file | +| `content` | String | yes | The content of a snippet | +| `description` | String | no | The description of a snippet | +| `visibility` | String | no | The snippet's visibility | ``` bash -curl --request POST --data '{"title": "This is a snippet", "content": "Hello world", "file_name": "test.txt", "visibility": "internal" }' --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/snippets +curl --request POST --data '{"title": "This is a snippet", "content": "Hello world", "description": "Hello World snippet", "file_name": "test.txt", "visibility": "internal" }' --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/snippets ``` Example response: @@ -92,6 +94,7 @@ Example response: "id": 1, "title": "This is a snippet", "file_name": "test.txt", + "description": "Hello World snippet", "author": { "id": 1, "username": "john_smith", @@ -117,13 +120,14 @@ PUT /snippets/:id Parameters: -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | Integer | yes | The ID of a snippet | -| `title` | String | no | The title of a snippet | -| `file_name` | String | no | The name of a snippet file | -| `content` | String | no | The content of a snippet | -| `visibility` | String | no | The snippet's visibility | +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | Integer | yes | The ID of a snippet | +| `title` | String | no | The title of a snippet | +| `file_name` | String | no | The name of a snippet file | +| `description` | String | no | The description of a snippet | +| `content` | String | no | The content of a snippet | +| `visibility` | String | no | The snippet's visibility | ``` bash @@ -137,6 +141,7 @@ Example response: "id": 1, "title": "test", "file_name": "add.rb", + "description": "description of snippet", "author": { "id": 1, "username": "john_smith", @@ -229,3 +234,35 @@ Example response: } ] ``` + +## Get user agent details + +> **Notes:** +> [Introduced][ce-29508] in GitLab 9.4. + + +Available only for admins. + +``` +GET /snippets/:id/user_agent_detail +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | Integer | yes | The ID of a snippet | + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/snippets/1/user_agent_detail +``` + +Example response: + +```json +{ + "user_agent": "AppleWebKit/537.36", + "ip_address": "127.0.0.1", + "akismet_submitted": false +} +``` + +[ce-[ce-29508]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29508]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29508 diff --git a/doc/api/users.md b/doc/api/users.md index f4167ba2605..57a13eb477d 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -62,6 +62,7 @@ GET /users "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", "web_url": "http://localhost:3000/john_smith", "created_at": "2012-05-23T08:00:58Z", + "is_admin": false, "bio": null, "location": null, "skype": "", @@ -94,6 +95,7 @@ GET /users "avatar_url": "http://localhost:3000/uploads/user/avatar/2/index.jpg", "web_url": "http://localhost:3000/jack_smith", "created_at": "2012-05-23T08:01:01Z", + "is_admin": false, "bio": null, "location": null, "skype": "", @@ -144,6 +146,12 @@ GET /users?extern_uid=1234567&provider=github You can search for users who are external with: `/users?external=true` +You can search users by creation date time range with: + +``` +GET /users?created_before=2001-01-02T00:00:00.060Z&created_after=1999-01-02T00:00:00.060 +``` + ## Single user Get a single user. @@ -197,6 +205,7 @@ Parameters: "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", "web_url": "http://localhost:3000/john_smith", "created_at": "2012-05-23T08:00:58Z", + "is_admin": false, "bio": null, "location": null, "skype": "", @@ -232,25 +241,26 @@ POST /users Parameters: -- `email` (required) - Email -- `password` (optional) - Password -- `reset_password` (optional) - Send user password reset link - true or false(default) -- `username` (required) - Username -- `name` (required) - Name -- `skype` (optional) - Skype ID -- `linkedin` (optional) - LinkedIn -- `twitter` (optional) - Twitter account -- `website_url` (optional) - Website URL -- `organization` (optional) - Organization name -- `projects_limit` (optional) - Number of projects user can create -- `extern_uid` (optional) - External UID -- `provider` (optional) - External provider name -- `bio` (optional) - User's biography -- `location` (optional) - User's location -- `admin` (optional) - User is admin - true or false (default) -- `can_create_group` (optional) - User can create groups - true or false -- `confirm` (optional) - Require confirmation - true (default) or false -- `external` (optional) - Flags the user as external - true or false(default) +- `email` (required) - Email +- `password` (optional) - Password +- `reset_password` (optional) - Send user password reset link - true or false(default) +- `username` (required) - Username +- `name` (required) - Name +- `skype` (optional) - Skype ID +- `linkedin` (optional) - LinkedIn +- `twitter` (optional) - Twitter account +- `website_url` (optional) - Website URL +- `organization` (optional) - Organization name +- `projects_limit` (optional) - Number of projects user can create +- `extern_uid` (optional) - External UID +- `provider` (optional) - External provider name +- `bio` (optional) - User's biography +- `location` (optional) - User's location +- `admin` (optional) - User is admin - true or false (default) +- `can_create_group` (optional) - User can create groups - true or false +- `skip_confirmation` (optional) - Skip confirmation - true or false (default) +- `external` (optional) - Flags the user as external - true or false(default) +- `avatar` (optional) - Image file for user's avatar ## User modification @@ -279,6 +289,7 @@ Parameters: - `admin` (optional) - User is admin - true or false (default) - `can_create_group` (optional) - User can create groups - true or false - `external` (optional) - Flags the user as external - true or false(default) +- `avatar` (optional) - Image file for user's avatar On password update, user will be forced to change it upon next login. Note, at the moment this method does only return a `404` error, @@ -353,7 +364,7 @@ GET /user Parameters: -- `sudo` (required) - the ID of a user +- `sudo` (optional) - the ID of a user to make the call in their place ``` GET /user @@ -804,7 +815,7 @@ Example response: It creates a new impersonation token. Note that only administrators can do this. You are only able to create impersonation tokens to impersonate the user and perform -both API calls and Git reads and writes. The user will not see these tokens in his profile +both API calls and Git reads and writes. The user will not see these tokens in their profile settings page. ``` diff --git a/doc/api/v3_to_v4.md b/doc/api/v3_to_v4.md index 9db8e0351cf..9835fab7c98 100644 --- a/doc/api/v3_to_v4.md +++ b/doc/api/v3_to_v4.md @@ -2,9 +2,11 @@ Since GitLab 9.0, API V4 is the preferred version to be used. -API V3 will be removed in GitLab 9.5, to be released on August 22, 2017. In the -meantime, we advise you to make any necessary changes to applications that use -V3. The V3 API documentation is still [available](https://gitlab.com/gitlab-org/gitlab-ce/blob/8-16-stable/doc/api/README.md). +API V3 will be unsupported from GitLab 9.5, to be released on August +22, 2017. It will be removed in GitLab 9.5 or later. In the meantime, we advise +you to make any necessary changes to applications that use V3. The V3 API +documentation is still +[available](https://gitlab.com/gitlab-org/gitlab-ce/blob/8-16-stable/doc/api/README.md). Below are the changes made between V3 and V4. diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md index 6892905dd94..130e8f542b4 100644 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md @@ -120,7 +120,8 @@ gitlab_rails['ldap_servers'] = { 'host' => 'ad.example.org', 'port' => 636, 'uid' => 'sAMAccountName', - 'method' => 'ssl', + 'encryption' => 'simple_tls', + 'verify_certificates' => true, 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', 'password' => 'Password1', 'active_directory' => true, @@ -255,7 +256,7 @@ If `allow_username_or_email_login` is enabled in the LDAP configuration, GitLab ## LDAP extended features on GitLab EE -With [GitLab Enterprise Edition (EE)](https://about.gitlab.com/giltab-ee/), besides everything we just described, you'll +With [GitLab Enterprise Edition (EE)](https://about.gitlab.com/gitlab-ee/), besides everything we just described, you'll have extended functionalities with LDAP, such as: - Group sync diff --git a/doc/articles/index.md b/doc/articles/index.md index 342fa88e80f..9d2e5956029 100644 --- a/doc/articles/index.md +++ b/doc/articles/index.md @@ -7,19 +7,107 @@ to provide the community with guidance on specific processes to achieve certain They are written by members of the GitLab Team and by [Community Writers](https://about.gitlab.com/handbook/product/technical-writing/community-writers/). +Part of the articles listed below link to the [GitLab Blog](https://about.gitlab.com/blog/), +where they were originally published. + ## Authentication -- **LDAP** - - [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md) +Explore GitLab's supported [authentications methods](../topics/authentication/index.md): + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| **LDAP** | +| [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md)| Admin guide | 2017/05/03 | +| [How to configure LDAP with GitLab EE](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/) | Admin guide | 2017/05/03 | + +## Build, test, and deploy with GitLab CI/CD + +Build, test, and deploy the software you develop with [GitLab CI/CD](../ci/README.md): + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| [Making CI Easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/) | Concepts | 2017/07/13 | +| [Dockerizing GitLab Review Apps](https://about.gitlab.com/2017/07/11/dockerizing-review-apps/) | Concepts | 2017/07/11 | +| [Continuous Integration: From Jenkins to GitLab Using Docker](https://about.gitlab.com/2017/07/27/docker-my-precious/) | Concepts | 2017/07/27 | +| [Continuous Delivery of a Spring Boot application with GitLab CI and Kubernetes](https://about.gitlab.com/2016/12/14/continuous-delivery-of-a-spring-boot-application-with-gitlab-ci-and-kubernetes/) | Tutorial | 2016/12/14 | +| [Setting up GitLab CI for Android projects](https://about.gitlab.com/2016/11/30/setting-up-gitlab-ci-for-android-projects/) | Tutorial | 2016/11/30 | +| [Automated Debian Package Build with GitLab CI](https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/) | Tutorial | 2016/10/12 | +| [Building an Elixir Release into a Docker image using GitLab CI](https://about.gitlab.com/2016/08/11/building-an-elixir-release-into-docker-image-using-gitlab-ci-part-1/) | Tutorial | 2016/08/11 | +| [Continuous Delivery with GitLab and Convox](https://about.gitlab.com/2016/06/09/continuous-delivery-with-gitlab-and-convox/) | Technical overview | 2016/06/09 | +| [GitLab Container Registry](https://about.gitlab.com/2016/05/23/gitlab-container-registry/) | Technical overview | 2016/05/23 | +| [How to use GitLab CI and MacStadium to build your macOS or iOS projects](https://about.gitlab.com/2017/05/15/how-to-use-macstadium-and-gitlab-ci-to-build-your-macos-or-ios-projects/) | Technical overview | 2017/05/15 | +| [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) | Tutorial | 2016/03/10 | ## Git -- [How to install Git](how_to_install_git/index.md) +Learn how to use [Git with GitLab](../topics/git/index.md): + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| [Why Git is Worth the Learning Curve](https://about.gitlab.com/2017/05/17/learning-curve-is-the-biggest-challenge-developers-face-with-git/) | Concepts | 2017/05/17 | +| [How to install Git](how_to_install_git/index.md) | Tutorial | 2017/05/15 | +| [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) | Tutorial | 2017/01/30 | +| [Git Tips & Tricks](https://about.gitlab.com/2016/12/08/git-tips-and-tricks/) | Technical overview | 2016/12/08 | ## GitLab Pages -- **GitLab Pages from A to Z** - - [Part 1: Static sites and GitLab Pages domains](../user/project/pages/getting_started_part_one.md) - - [Part 2: Quick start guide - Setting up GitLab Pages](../user/project/pages/getting_started_part_two.md) - - [Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](../user/project/pages/getting_started_part_three.md) - - [Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](../user/project/pages/getting_started_part_four.md) +Learn how to deploy a static website with [GitLab Pages](../user/project/pages/index.md#getting-started): + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| **Series: GitLab Pages from A to Z:** | +| [- Part 1: Static sites and GitLab Pages domains](../user/project/pages/getting_started_part_one.md)| User guide | 2017/02/22 | +| [- Part 2: Quick start guide - Setting up GitLab Pages](../user/project/pages/getting_started_part_two.md)| User guide | 2017/02/22 | +| [- Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](../user/project/pages/getting_started_part_three.md)| User guide | 2017/02/22 | +| [- Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](../user/project/pages/getting_started_part_four.md)| User guide | 2017/02/22 | +| [Setting up GitLab Pages with CloudFlare Certificates](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) | Tutorial | 2017/02/07 | +| [Building a new GitLab Docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) | Tutorial | 2016/12/07 | +| [Publish Code Coverage Report with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/) | Tutorial | 2016/11/03 | +| [GitLab CI: Deployment & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) | Tutorial | 2016/08/26 | +| [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/) | Tutorial | 2016/08/19 | +| **Series: Static Site Generator:** | +| [- Part 1: Dynamic vs Static Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) | Tutorial | 2016/06/03 | +| [- Part 2: Modern Static Site Generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) | Tutorial | 2016/06/10 | +| [- Part 3: Build any SSG site with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) | Tutorial | 2016/06/17 | +| [Securing your GitLab Pages with TLS and Let's Encrypt](https://about.gitlab.com/2016/04/11/tutorial-securing-your-gitlab-pages-with-tls-and-letsencrypt/) | Tutorial | 2016/04/11 | +| [Hosting on GitLab.com with GitLab Pages](https://about.gitlab.com/2016/04/07/gitlab-pages-setup/) | Tutorial | 2016/04/07 | + +## Install and maintain GitLab + +Install, upgrade, integrate, migrate to GitLab: + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| [Video Tutorial: Idea to Production on Google Container Engine (GKE)](https://about.gitlab.com/2017/01/23/video-tutorial-idea-to-production-on-google-container-engine-gke/) | Tutorial | 2017/01/23 | +| [How to Setup a GitLab Instance on Microsoft Azure](https://about.gitlab.com/2016/07/13/how-to-setup-a-gitlab-instance-on-microsoft-azure/) | Tutorial | 2016/07/13 | +| [Get started with OpenShift Origin 3 and GitLab](https://about.gitlab.com/2016/06/28/get-started-with-openshift-origin-3-and-gitlab/) | Tutorial | 2016/06/28 | +| [Getting started with GitLab and DigitalOcean](https://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/) | Tutorial | 2016/04/27 | + +## Software development + +Explore the best of GitLab's software development's capabilities: + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| [Making CI Easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/) | Concepts | 2017/07/13 | +| [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/)| Concepts | 2017/06/29 | +| [Fast and Natural Continuous Integration with GitLab CI](https://about.gitlab.com/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/) | Concepts | 2017/05/22 | +| [Demo: Auto-Deploy from GitLab to an OpenShift Container Cluster](https://about.gitlab.com/2017/05/16/devops-containers-gitlab-openshift/) | Technical overview | 2017/05/16 | +| [Demo: GitLab Service Desk](https://about.gitlab.com/2017/05/09/demo-service-desk/) | Feature highlight | 2017/05/09 | +| [Demo: Mapping Work Versus Time, With Burndown Charts](https://about.gitlab.com/2017/04/25/mapping-work-to-do-versus-time-with-burndown-charts/) | Feature highlight | 2017/04/25 | +| [Demo: Cloud Native Development with GitLab](https://about.gitlab.com/2017/04/18/cloud-native-demo/) | Feature highlight | 2017/04/18 | +| [Demo: Mastering Code Review With GitLab](https://about.gitlab.com/2017/03/17/demo-mastering-code-review-with-gitlab/) | Feature highlight | 2017/03/17 | +| [In 13 minutes from Kubernetes to a complete application development tool](https://about.gitlab.com/2016/11/14/idea-to-production/) | Technical overview | 2016/11/14 | +| [GitLab Workflow, an Overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/) | Technical overview | 2016/10/25 | +| [Trends in Version Control Land: Microservices](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/) | Concepts | 2016/08/16 | +| [Continuous Integration, Delivery, and Deployment with GitLab](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) | Concepts | 2016/08/05 | +| [Trends in Version Control Land: Innersourcing](https://about.gitlab.com/2016/07/07/trends-version-control-innersourcing/) | Concepts | 2016/07/07 | +| [Tutorial: It's all connected in GitLab](https://about.gitlab.com/2016/03/08/gitlab-tutorial-its-all-connected/) | Technical overview | 2016/03/08 | + +## Technologies + +| Article title | Category | Publishing date | +| :------------ | :------: | --------------: | +| [Why we are not leaving the cloud](https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/) | Concepts | 2017/03/02 | +| [Why We Chose Vue.js](https://about.gitlab.com/2016/10/20/why-we-chose-vue/) | Concepts | 2016/10/20 | +| [Markdown Kramdown Tips & Tricks](https://about.gitlab.com/2016/07/19/markdown-kramdown-tips-and-tricks/) | Technical overview | 2016/07/19 | diff --git a/doc/ci/README.md b/doc/ci/README.md index ca7266ac68f..10ea9467942 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -23,6 +23,7 @@ The first steps towards your GitLab CI journey. - [Setting up GitLab Runner For Continuous Integration](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) - [GitLab CI: Deployment & environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - **Videos:** + - [Demo (Streamed live on Jul 17, 2017): GitLab CI/CD Deep Dive](https://youtu.be/pBe4t1CD8Fc?t=195) - [Demo (March, 2017): how to get started using CI/CD with GitLab](https://about.gitlab.com/2017/03/13/ci-cd-demo/) - [Webcast (April, 2016): getting started with CI in GitLab](https://about.gitlab.com/2016/04/20/webcast-recording-and-slides-introduction-to-ci-in-gitlab/) - **Third-party videos:** diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 408d46a756c..f7c2a0ef0ca 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -282,9 +282,9 @@ which can be avoided if a different driver is used, for example `overlay`. > **Notes:** - This feature requires GitLab 8.8 and GitLab Runner 1.2. -- Starting from GitLab 8.12, if you have 2FA enabled in your account, you need - to pass a personal access token instead of your password in order to login to - GitLab's Container Registry. +- Starting from GitLab 8.12, if you have [2FA] enabled in your account, you need + to pass a [personal access token][pat] instead of your password in order to + login to GitLab's Container Registry. Once you've built a Docker image, you can push it up to the built-in [GitLab Container Registry](../../user/project/container_registry.md). For example, @@ -409,3 +409,5 @@ Some things you should be aware of when using the Container Registry: [docker-in-docker]: https://blog.docker.com/2013/09/docker-can-now-run-within-docker/ [docker-cap]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities +[2fa]: ../../user/profile/account/two_factor_authentication.md +[pat]: ../../user/profile/personal_access_tokens.md diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 96834e15bb9..d3433594eb7 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,4 +1,4 @@ -# Using Docker Images +# Using Docker images GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use [Docker Engine](https://www.docker.com/) to test and build any application. @@ -17,14 +17,16 @@ can also run on your workstation. The added benefit is that you can test all the commands that we will explore later from your shell, rather than having to test them on a dedicated CI server. -## Register docker runner +## Register Docker Runner -To use GitLab Runner with docker you need to register a new runner to use the -`docker` executor: +To use GitLab Runner with Docker you need to [register a new Runner][register] +to use the `docker` executor. + +A one-line example can be seen below: ```bash -gitlab-ci-multi-runner register \ - --url "https://gitlab.com/" \ +sudo gitlab-runner register \ + --url "https://gitlab.example.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby-2.1" \ --executor "docker" \ @@ -33,26 +35,26 @@ gitlab-ci-multi-runner register \ --docker-mysql latest ``` -The registered runner will use the `ruby:2.1` docker image and will run two +The registered runner will use the `ruby:2.1` Docker image and will run two services, `postgres:latest` and `mysql:latest`, both of which will be accessible during the build process. ## What is an image -The `image` keyword is the name of the docker image the docker executor -will run to perform the CI tasks. +The `image` keyword is the name of the Docker image the Docker executor +will run to perform the CI tasks. -By default the executor will only pull images from [Docker Hub][hub], +By default, the executor will only pull images from [Docker Hub][hub], but this can be configured in the `gitlab-runner/config.toml` by setting -the [docker pull policy][] to allow using local images. +the [Docker pull policy][] to allow using local images. For more information about images and Docker Hub please read the [Docker Fundamentals][] documentation. ## What is a service -The `services` keyword defines just another docker image that is run during -your job and is linked to the docker image that the `image` keyword defines. +The `services` keyword defines just another Docker image that is run during +your job and is linked to the Docker image that the `image` keyword defines. This allows you to access the service image during build time. The service image can run any application, but the most common use case is to @@ -60,6 +62,11 @@ run a database container, eg. `mysql`. It's easier and faster to use an existing image and run it as an additional container than install `mysql` every time the project is built. +You are not limited to have only database services. You can add as many +services you need to `.gitlab-ci.yml` or manually modify `config.toml`. +Any image found at [Docker Hub][hub] or your private Container Registry can be +used as a service. + You can see some widely used services examples in the relevant documentation of [CI services examples](../services/README.md). @@ -73,22 +80,49 @@ then be used to create a container that is linked to the job container. The service container for MySQL will be accessible under the hostname `mysql`. So, in order to access your database service you have to connect to the host -named `mysql` instead of a socket or `localhost`. +named `mysql` instead of a socket or `localhost`. Read more in [accessing the +services](#accessing-the-services). -## Overwrite image and services +### Accessing the services -See [How to use other images as services](#how-to-use-other-images-as-services). +Let's say that you need a Wordpress instance to test some API integration with +your application. -## How to use other images as services +You can then use for example the [tutum/wordpress][] image in your +`.gitlab-ci.yml`: -You are not limited to have only database services. You can add as many -services you need to `.gitlab-ci.yml` or manually modify `config.toml`. -Any image found at [Docker Hub][hub] can be used as a service. +```yaml +services: +- tutum/wordpress:latest +``` -## Define image and services from `.gitlab-ci.yml` +If you don't [specify a service alias](#available-settings-for-services-entry), +when the job is run, `tutum/wordpress` will be started and you will have +access to it from your build container under two hostnames to choose from: + +- `tutum-wordpress` +- `tutum__wordpress` + +>**Note:** +Hostnames with underscores are not RFC valid and may cause problems in 3rd party +applications. + +The default aliases for the service's hostname are created from its image name +following these rules: + +- Everything after the colon (`:`) is stripped +- Slash (`/`) is replaced with double underscores (`__`) and the primary alias + is created +- Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is + created (requires GitLab Runner v1.1.0 or higher) + +To override the default behavior, you can +[specify a service alias](#available-settings-for-services-entry). + +## Define `image` and `services` from `.gitlab-ci.yml` You can simply define an image that will be used for all jobs and a list of -services that you want to use during build time. +services that you want to use during build time: ```yaml image: ruby:2.2 @@ -125,6 +159,203 @@ test:2.2: - bundle exec rake spec ``` +Or you can pass some [extended configuration options](#extended-docker-configuration-options) +for `image` and `services`: + +```yaml +image: + name: ruby:2.2 + entrypoint: ["/bin/bash"] + +services: +- name: my-postgres:9.4 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + +before_script: +- bundle install + +test: + script: + - bundle exec rake spec +``` + +## Extended Docker configuration options + +> **Note:** +This feature requires GitLab 9.4 and GitLab Runner 9.4 or higher. + +When configuring the `image` or `services` entries, you can use a string or a map as +options: + +- when using a string as an option, it must be the full name of the image to use + (including the Registry part if you want to download the image from a Registry + other than Docker Hub) +- when using a map as an option, then it must contain at least the `name` + option, which is the same name of the image as used for the string setting + +For example, the following two definitions are equal: + +1. Using a string as an option to `image` and `services`: + + ```yaml + image: "registry.example.com/my/image:latest" + + services: + - postgresql:9.4 + - redis:latest + ``` + +1. Using a map as an option to `image` and `services`. The use of `image:name` is + required: + + ```yaml + image: + name: "registry.example.com/my/image:latest" + + services: + - name: postgresql:9.4 + - name: redis:latest + ``` + +### Available settings for `image` + +> **Note:** +This feature requires GitLab 9.4 and GitLab Runner 9.4 or higher. + +| Setting | Required | Description | +|------------|----------|-------------| +| `name` | yes, when used with any other option | Full name of the image that should be used. It should contain the Registry part if needed. | +| `entrypoint` | no | Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`][entrypoint] directive, where each shell token is a separate string in the array. | + +### Available settings for `services` + +> **Note:** +This feature requires GitLab 9.4 and GitLab Runner 9.4 or higher. + +| Setting | Required | Description | +|------------|----------|-------------| +| `name` | yes, when used with any other option | Full name of the image that should be used. It should contain the Registry part if needed. | +| `entrypoint` | no | Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`][entrypoint] directive, where each shell token is a separate string in the array. | +| `command` | no | Command or script that should be used as the container's command. It will be translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`][cmd] directive, where each shell token is a separate string in the array. | +| `alias` | no | Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | + +### Starting multiple services from the same image + +Before the new extended Docker configuration options, the following configuration +would not work properly: + +```yaml +services: +- mysql:latest +- mysql:latest +``` + +The Runner would start two containers using the `mysql:latest` image, but both +of them would be added to the job's container with the `mysql` alias based on +the [default hostname naming](#accessing-the-services). This would end with one +of the services not being accessible. + +After the new extended Docker configuration options, the above example would +look like: + +```yaml +services: +- name: mysql:latest + alias: mysql-1 +- name: mysql:latest + alias: mysql-2 +``` + +The Runner will still start two containers using the `mysql:latest` image, +but now each of them will also be accessible with the alias configured +in `.gitlab-ci.yml` file. + +### Setting a command for the service + +Let's assume you have a `super/sql:latest` image with some SQL database +inside it and you would like to use it as a service for your job. Let's also +assume that this image doesn't start the database process while starting +the container and the user needs to manually use `/usr/bin/super-sql run` as +a command to start the database. + +Before the new extended Docker configuration options, you would need to create +your own image based on the `super/sql:latest` image, add the default command, +and then use it in job's configuration, like: + +```Dockerfile +# my-super-sql:latest image's Dockerfile + +FROM super/sql:latest +CMD ["/usr/bin/super-sql", "run"] +``` + +```yaml +# .gitlab-ci.yml + +services: +- my-super-sql:latest +``` + +After the new extended Docker configuration options, you can now simply +set a `command` in `.gitlab-ci.yml`, like: + +```yaml +# .gitlab-ci.yml + +services: +- name: super/sql:latest + command: ["/usr/bin/super-sql", "run"] +``` + +As you can see, the syntax of `command` is similar to [Dockerfile's `CMD`][cmd]. + +### Overriding the entrypoint of an image + +Let's assume you have a `super/sql:experimental` image with some SQL database +inside it and you would like to use it as a base image for your job because you +want to execute some tests with this database binary. Let's also assume that +this image is configured with `/usr/bin/super-sql run` as an entrypoint. That +means, that when starting the container without additional options, it will run +the database's process, while Runner expects that the image will have no +entrypoint or at least will start with a shell as its entrypoint. + +Previously we would need to create our own image based on the +`super/sql:experimental` image, set the entrypoint to a shell, and then use +it in job's configuration, e.g.: + +Before the new extended Docker configuration options, you would need to create +your own image based on the `super/sql:experimental` image, set the entrypoint +to a shell and then use it in job's configuration, like: + +```Dockerfile +# my-super-sql:experimental image's Dockerfile + +FROM super/sql:experimental +ENTRYPOINT ["/bin/sh"] +``` + +```yaml +# .gitlab-ci.yml + +image: my-super-sql:experimental +``` + +After the new extended Docker configuration options, you can now simply +set an `entrypoint` in `.gitlab-ci.yml`, like: + +```yaml +# .gitlab-ci.yml + +image: + name: super/sql:experimental + entrypoint: ["/bin/sh"] +``` + +As you can see the syntax of `entrypoint` is similar to +[Dockerfile's `ENTRYPOINT`][entrypoint]. + ## Define image and services in `config.toml` Look for the `[runners.docker]` section: @@ -138,49 +369,60 @@ Look for the `[runners.docker]` section: The image and services defined this way will be added to all job run by that runner. -## Define an image from a private Docker registry +## Define an image from a private Container Registry -Starting with GitLab Runner 0.6.0, you are able to define images located to -private registries that could also require authentication. +> **Notes:** +- This feature requires GitLab Runner **1.8** or higher +- For GitLab Runner versions **>= 0.6, <1.8** there was a partial + support for using private registries, which required manual configuration + of credentials on runner's host. We recommend to upgrade your Runner to + at least version **1.8** if you want to use private registries. +- If the repository is private you need to authenticate your GitLab Runner in the + registry. Learn more about how [GitLab Runner works in this case][runner-priv-reg]. -All you have to do is be explicit on the image definition in `.gitlab-ci.yml`. +As an example, let's assume that you want to use the `registry.example.com/private/image:latest` +image which is private and requires you to login into a private container registry. +To configure access for `registry.example.com`, follow these steps: -```yaml -image: my.registry.tld:5000/namespace/image:tag -``` +1. Do a `docker login` on your computer: -In the example above, GitLab Runner will look at `my.registry.tld:5000` for the -image `namespace/image:tag`. + ```bash + docker login registry.example.com --username my_username --password my_password + ``` -If the repository is private you need to authenticate your GitLab Runner in the -registry. Learn how to do that on -[GitLab Runner's documentation][runner-priv-reg]. +1. Copy the content of `~/.docker/config.json` +1. Create a [secret variable] `DOCKER_AUTH_CONFIG` with the content of the + Docker configuration file as the value: -## Accessing the services + ```json + { + "auths": { + "registry.example.com": { + "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=" + } + } + } + ``` -Let's say that you need a Wordpress instance to test some API integration with -your application. +1. Do a `docker logout` on your computer if you don't need access to the + registry from it: -You can then use for example the [tutum/wordpress][] image in your -`.gitlab-ci.yml`: - -```yaml -services: -- tutum/wordpress:latest -``` + ```bash + docker logout registry.example.com + ``` -When the job is run, `tutum/wordpress` will be started and you will have -access to it from your build container under the hostnames `tutum-wordpress` -(requires GitLab Runner v1.1.0 or newer) and `tutum__wordpress`. +1. You can now use any private image from `registry.example.com` defined in + `image` and/or `services` in your [`.gitlab-ci.yml` file][yaml-priv-reg]: -*Note: hostname with underscores is not RFC valid and may cause problems in 3rd party applications.* + ```yaml + image: my.registry.tld:5000/namespace/image:tag + ``` -The alias hostnames for the service are made from the image name following these -rules: + In the example above, GitLab Runner will look at `my.registry.tld:5000` for the + image `namespace/image:tag`. -1. Everything after `:` is stripped -2. Slash (`/`) is replaced with double underscores (`__`) - primary alias -3. Slash (`/`) is replaced with dash (`-`) - secondary alias, requires GitLab Runner v1.1.0 or newer +You can add configuration for as many registries as you want, adding more +registries to the `"auths"` hash as described above. ## Configuring services @@ -208,7 +450,7 @@ See the specific documentation for ## How Docker integration works -Below is a high level overview of the steps performed by docker during job +Below is a high level overview of the steps performed by Docker during job time. 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. @@ -225,7 +467,7 @@ time. ## How to debug a job locally *Note: The following commands are run without root privileges. You should be -able to run docker with your regular user account.* +able to run Docker with your regular user account.* First start with creating a file named `build_script`: @@ -283,4 +525,8 @@ creation. [tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/ [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ -[runner-priv-reg]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry +[runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry +[secret variable]: ../variables/README.md#secret-variables +[entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint +[cmd]: https://docs.docker.com/engine/reference/builder/#cmd +[register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 3393030210e..6a7f694d705 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -602,14 +602,12 @@ exist, you should see something like: >**Notes:** > - For the monitor dashboard to appear, you need to: - - Have enabled the [Kubernetes integration][kube] - - Have your app deployed on Kubernetes - Have enabled the [Prometheus integration][prom] + - Configured Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/metrics.md) - With GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard -If your application is deployed on Kubernetes and you have enabled Prometheus -collecting metrics, you can monitor the performance behavior of your app +If you have enabled Prometheus for collecting metrics, you can monitor the performance behavior of your app through the environments. Once configured, GitLab will attempt to retrieve performance metrics for any diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md index a047e809788..5659a8c2a2a 100644 --- a/doc/ci/examples/code_climate.md +++ b/doc/ci/examples/code_climate.md @@ -27,7 +27,7 @@ download and analyze the report artifact in JSON format. For GitLab [Enterprise Edition Starter][ee] users, this information can be automatically extracted and shown right in the merge request widget. [Learn more on code quality -diffs in merge requests](http://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.md). +diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). [cli]: https://github.com/codeclimate/codeclimate [dind]: ../docker/using_docker_build.md#use-docker-in-docker-executor diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md index 8b0d8a003fd..b9f0485290e 100644 --- a/doc/ci/examples/deployment/composer-npm-deploy.md +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -20,12 +20,12 @@ before_script: - php -r "unlink('composer-setup.php');" ``` -This will make sure we have all requirements ready. Next, we want to run `composer update` to fetch all PHP dependencies and `npm install` to load node packages, then run the `npm` script. We need to append them into `before_script` section: +This will make sure we have all requirements ready. Next, we want to run `composer install` to fetch all PHP dependencies and `npm install` to load node packages, then run the `npm` script. We need to append them into `before_script` section: ```yaml before_script: # ... - - php composer.phar update + - php composer.phar install - npm install - npm run deploy ``` @@ -133,7 +133,7 @@ before_script: - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" - php composer-setup.php - php -r "unlink('composer-setup.php');" - - php composer.phar update + - php composer.phar install - npm install - npm run deploy - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' diff --git a/doc/ci/img/environments_monitoring.png b/doc/ci/img/environments_monitoring.png Binary files differindex 387b6c54b61..d9c46ea4c95 100644 --- a/doc/ci/img/environments_monitoring.png +++ b/doc/ci/img/environments_monitoring.png diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 41cae58782d..88e53ff40e8 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -155,7 +155,7 @@ Find more information about different Runners in the [Runners](../runners/README.md) documentation. You can find whether any Runners are assigned to your project by going to -**Settings ➔ CI/CD Pipelines**. Setting up a Runner is easy and straightforward. The +**Settings ➔ Pipelines**. Setting up a Runner is easy and straightforward. The official Runner supported by GitLab is written in Go and its documentation can be found at <https://docs.gitlab.com/runner/>. @@ -168,7 +168,7 @@ Follow the links above to set up your own Runner or use a Shared Runner as described in the next section. Once the Runner has been set up, you should see it on the Runners page of your -project, following **Settings ➔ CI/CD Pipelines**. +project, following **Settings ➔ Pipelines**.  @@ -181,7 +181,7 @@ These are special virtual machines that run on GitLab's infrastructure and can build any project. To enable the **Shared Runners** you have to go to your project's -**Settings ➔ CI/CD Pipelines** and click **Enable shared runners**. +**Settings ➔ Pipelines** and click **Enable shared runners**. [Read more on Shared Runners](../runners/README.md). diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 1bd1ee93ac5..76d746155eb 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -1,124 +1,168 @@ # Runners -In GitLab CI, Runners run your [yaml](../yaml/README.md). -A Runner is an isolated (virtual) machine that picks up jobs -through the coordinator API of GitLab CI. +In GitLab CI, Runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). +They are isolated (virtual) machines that pick up jobs through the coordinator +API of GitLab CI. A Runner can be specific to a certain project or serve any project in GitLab CI. A Runner that serves all projects is called a shared Runner. -Ideally, GitLab Runner should not be installed on the same machine as GitLab. +Ideally, the GitLab Runner should not be installed on the same machine as GitLab. Read the [requirements documentation](../../install/requirements.md#gitlab-runner) for more information. -## Shared vs. Specific Runners - -A Runner that is specific only runs for the specified project. A shared Runner -can run jobs for every project that has enabled the option **Allow shared Runners**. - -**Shared Runners** are useful for jobs that have similar requirements, -between multiple projects. Rather than having multiple Runners idling for -many projects, you can have a single or a small number of Runners that handle -multiple projects. This makes it easier to maintain and update Runners. - -**Specific Runners** are useful for jobs that have special requirements or for -projects with a specific demand. If a job has certain requirements, you can set -up the specific Runner with this in mind, while not having to do this for all -Runners. For example, if you want to deploy a certain project, you can setup -a specific Runner to have the right credentials for this. - -Projects with high demand of CI activity can also benefit from using specific Runners. -By having dedicated Runners you are guaranteed that the Runner is not being held -up by another project's jobs. +## Shared vs specific Runners + +After [installing the Runner][install], you can either register it as shared or +specific. You can only register a shared Runner if you have admin access to +the GitLab instance. The main differences between a shared and a specific Runner +are: + +- **Shared Runners** are useful for jobs that have similar requirements, + between multiple projects. Rather than having multiple Runners idling for + many projects, you can have a single or a small number of Runners that handle + multiple projects. This makes it easier to maintain and update them. + Shared Runners process jobs using a [fair usage queue](#how-shared-runners-pick-jobs). + In contrast to specific Runners that use a FIFO queue, this prevents + cases where projects create hundreds of jobs which can lead to eating all + available shared Runners resources. +- **Specific Runners** are useful for jobs that have special requirements or for + projects with a specific demand. If a job has certain requirements, you can set + up the specific Runner with this in mind, while not having to do this for all + Runners. For example, if you want to deploy a certain project, you can setup + a specific Runner to have the right credentials for this. The [usage of tags](#using-tags) + may be useful in this case. Specific Runners process jobs using a [FIFO] queue. + +A Runner that is specific only runs for the specified project(s). A shared Runner +can run jobs for every project that has enabled the option **Allow shared Runners** +under **Settings ➔ Pipelines**. + +Projects with high demand of CI activity can also benefit from using specific +Runners. By having dedicated Runners you are guaranteed that the Runner is not +being held up by another project's jobs. You can set up a specific Runner to be used by multiple projects. The difference with a shared Runner is that you have to enable each project explicitly for the Runner to be able to run its jobs. Specific Runners do not get shared with forked projects automatically. -A fork does copy the CI settings (jobs, allow shared, etc) of the cloned repository. - -# Creating and Registering a Runner - -There are several ways to create a Runner. Only after creation, upon -registration its status as Shared or Specific is determined. - -[See the documentation for](https://docs.gitlab.com/runner/install) -the different methods of installing a Runner instance. +A fork does copy the CI settings (jobs, allow shared, etc) of the cloned +repository. -After installing the Runner, you can either register it as `Shared` or as `Specific`. -You can only register a Shared Runner if you have admin access to the GitLab instance. +## Registering a shared Runner -## Registering a Shared Runner +You can only register a shared Runner if you are an admin of the GitLab instance. -You can only register a shared Runner if you are an admin on the linked -GitLab instance. +1. Grab the shared-Runner token on the `admin/runners` page -Grab the shared-Runner token on the `admin/runners` page of your GitLab CI -instance. +  - +1. [Register the Runner][register] -Now simply register the Runner as any Runner: +Shared Runners are enabled by default as of GitLab 8.2, but can be disabled +with the **Disable shared Runners** button which is present under each project's +**Settings ➔ Pipelines** page. Previous versions of GitLab defaulted shared +Runners to disabled. -``` -sudo gitlab-ci-multi-runner register -``` - -Shared Runners are enabled by default as of GitLab 8.2, but can be disabled with the -`DISABLE SHARED RUNNERS` button. Previous versions of GitLab defaulted shared Runners to -disabled. - -## Registering a Specific Runner +## Registering a specific Runner Registering a specific can be done in two ways: 1. Creating a Runner with the project registration token 1. Converting a shared Runner into a specific Runner (one-way, admin only) -There are several ways to create a Runner instance. The steps below only -concern registering the Runner on GitLab CI. - -### Registering a Specific Runner with a Project Registration token +### Registering a specific Runner with a project registration token To create a specific Runner without having admin rights to the GitLab instance, -visit the project you want to make the Runner work for in GitLab CI. +visit the project you want to make the Runner work for in GitLab: -Click on the Runner tab and use the registration token you find there to -setup a specific Runner for this project. +1. Go to **Settings ➔ Pipelines** to obtain the token +1. [Register the Runner][register] - +### Making an existing shared Runner specific -To register the Runner, run the command below and follow instructions: +If you are an admin on your GitLab instance, you can turn any shared Runner into +a specific one, but not the other way around. Keep in mind that this is a one +way transition. -``` -sudo gitlab-ci-multi-runner register -``` +1. Go to the Runners in the admin area **Overview ➔ Runners** (`/admin/runners`) + and find your Runner +1. Enable any projects under **Restrict projects for this Runner** to be used + with the Runner -### Lock a specific Runner from being enabled for other projects +From now on, the shared Runner will be specific to those projects. + +## Locking a specific Runner from being enabled for other projects You can configure a Runner to assign it exclusively to a project. When a Runner is locked this way, it can no longer be enabled for other projects. -This setting is available on each Runner in *Project Settings* > *Runners*. +This setting can be enabled the first time you [register a Runner][register] and +can be changed afterwards under each Runner's settings. + +To lock/unlock a Runner: + +1. Visit your project's **Settings ➔ Pipelines** +1. Find the Runner you wish to lock/unlock and make sure it's enabled +1. Click the pencil button +1. Check the **Lock to current projects** option +1. Click **Save changes** for the changes to take effect -### Making an existing Shared Runner Specific +## How shared Runners pick jobs -If you are an admin on your GitLab instance, -you can make any shared Runner a specific Runner, _but you can not -make a specific Runner a shared Runner_. +Shared Runners abide to a process queue we call fair usage. The fair usage +algorithm tries to assign jobs to shared Runners from projects that have the +lowest number of jobs currently running on shared Runners. -To make a shared Runner specific, go to the Runner page (`/admin/runners`) -and find your Runner. Add any projects on the left to make this Runner -run exclusively for these projects, therefore making it a specific Runner. +**Example 1** - +We have following jobs in queue: -## Using Shared Runners Effectively +- Job 1 for Project 1 +- Job 2 for Project 1 +- Job 3 for Project 1 +- Job 4 for Project 2 +- Job 5 for Project 2 +- Job 6 for Project 3 + +With the fair usage algorithm jobs are assigned in following order: + +1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (i.e. all projects) +1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running) +1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running) +1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number +1. Job 5 is next, because Project 1 now has 2 jobs running, and between Projects 2 and 3, Job 5 is the lowest remaining job number +1. Lastly we choose Job 3... because it's the only job left + +--- + +**Example 2** + +We have following jobs in queue: + +- Job 1 for project 1 +- Job 2 for project 1 +- Job 3 for project 1 +- Job 4 for project 2 +- Job 5 for project 2 +- Job 6 for project 3 + +With the fair usage algorithm jobs are assigned in following order: + +1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (i.e. all projects) +1. We finish job 1 +1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number +1. Job 4 is next, because with Project 1 running a job, 4 is the lowest number from projects running no jobs (Projects 2 and 3) +1. We finish job 4 +1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again +1. Job 6 is next, because Project 3 is the only project left with no running jobs +1. Lastly we choose Job 3... because, again, it's the only job left (who says 1 is the loneliest number?) + +## Using shared Runners effectively If you are planning to use shared Runners, there are several things you should keep in mind. -### Use Tags +### Using tags You must setup a Runner to be able to run all the different types of jobs that it may encounter on the projects it's shared over. This would be @@ -130,17 +174,27 @@ shared Runners will only run the jobs they are equipped to run. For instance, at GitLab we have Runners tagged with "rails" if they contain the appropriate dependencies to run Rails test suites. -### Prevent Runner with tags from picking jobs without tags +### Preventing Runners with tags from picking jobs without tags You can configure a Runner to prevent it from picking jobs with tags when -the Runner does not have tags assigned. This setting is available on each -Runner in *Project Settings* > *Runners*. +the Runner does not have tags assigned. This setting can be enabled the first +time you [register a Runner][register] and can be changed afterwards under +each Runner's settings. + +To make a Runner pick tagged/untagged jobs: + +1. Visit your project's **Settings ➔ Pipelines** +1. Find the Runner you wish and make sure it's enabled +1. Click the pencil button +1. Check the **Run untagged jobs** option +1. Click **Save changes** for the changes to take effect ### Be careful with sensitive information If you can run a job on a Runner, you can get access to any code it runs and get the token of the Runner. With shared Runners, this means that anyone -that runs jobs on the Runner, can access anyone else's code that runs on the Runner. +that runs jobs on the Runner, can access anyone else's code that runs on the +Runner. In addition, because you can get access to the Runner token, it is possible to create a clone of a Runner and submit false jobs, for example. @@ -160,3 +214,7 @@ project. Mentioned briefly earlier, but the following things of Runners can be exploited. We're always looking for contributions that can mitigate these [Security Considerations](https://docs.gitlab.com/runner/security/). + +[install]: http://docs.gitlab.com/runner/install/ +[fifo]: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics) +[register]: http://docs.gitlab.com/runner/register/ diff --git a/doc/ci/runners/img/shared_runners_admin.png b/doc/ci/runners/img/shared_runners_admin.png Binary files differnew file mode 100644 index 00000000000..e049b339b36 --- /dev/null +++ b/doc/ci/runners/img/shared_runners_admin.png diff --git a/doc/ci/runners/project_specific.png b/doc/ci/runners/project_specific.png Binary files differdeleted file mode 100644 index c812defa67b..00000000000 --- a/doc/ci/runners/project_specific.png +++ /dev/null diff --git a/doc/ci/runners/shared_runner.png b/doc/ci/runners/shared_runner.png Binary files differdeleted file mode 100644 index 31574a17764..00000000000 --- a/doc/ci/runners/shared_runner.png +++ /dev/null diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index befaa06e918..cf25a8b618f 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -34,9 +34,9 @@ instructions to [generate an SSH key](../../ssh/README.md). Do not add a passphrase to the SSH key, or the `before_script` will prompt for it. Then, create a new **Secret Variable** in your project settings on GitLab -following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY` -and in the **Value** field paste the content of your _private_ key that you -created earlier. +following **Settings > Pipelines** and look for the "Secret Variables" section. +As **Key** add the name `SSH_PRIVATE_KEY` and in the **Value** field paste the +content of your _private_ key that you created earlier. It is also good practice to check the server's own public key to make sure you are not being targeted by a man-in-the-middle attack. To do this, add another diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index cb646827fb4..7ec7136d8c6 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -4,15 +4,22 @@ - [Introduced][ci-229] in GitLab CE 7.14. - GitLab 8.12 has a completely redesigned job permissions system. Read all about the [new model and its implications](../../user/project/new_ci_build_permissions_model.md#job-triggers). -- GitLab 9.0 introduced a trigger ownership to solve permission problems. -Triggers can be used to force a rebuild of a specific `ref` (branch or tag) -with an API call. +Triggers can be used to force a pipeline rerun of a specific `ref` (branch or +tag) with an API call. -## Add a trigger +## Authentication tokens + +The following methods of authentication are supported. + +### Trigger token + +A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). + +## Adding a new trigger You can add a new trigger by going to your project's -**Settings ➔ Pipelines ➔ Triggers**. The **Add trigger** button will +**Settings ➔ Pipelines** under **Triggers**. The **Add trigger** button will create a new token which you can then use to trigger a rerun of this particular project's pipeline. @@ -22,7 +29,10 @@ overview of the time the triggers were last used.  -## Take ownership +## Taking ownership of a trigger + +> **Note**: +GitLab 9.0 introduced a trigger ownership to solve permission problems. Each created trigger when run will impersonate their associated user including their access to projects and their project permissions. @@ -30,26 +40,20 @@ their access to projects and their project permissions. You can take ownership of existing triggers by clicking *Take ownership*. From now on the trigger will be run as you. -## Legacy triggers - -Old triggers, created before 9.0 will be marked as Legacy. Triggers with -the legacy label do not have an associated user and only have access -to the current project. - -Legacy trigger are considered deprecated and will be removed -with one of the future versions of GitLab. - -## Revoke a trigger +## Revoking a trigger You can revoke a trigger any time by going at your project's -**Settings > Triggers** and hitting the **Revoke** button. The action is -irreversible. +**Settings ➔ Pipelines** under **Triggers** and hitting the **Revoke** button. +The action is irreversible. -## Trigger a pipeline +## Triggering a pipeline -> **Note**: -Valid refs are only the branches and tags. If you pass a commit SHA as a ref, -it will not trigger a job. +> **Notes**: +- Valid refs are only the branches and tags. If you pass a commit SHA as a ref, + it will not trigger a job. +- If your project is public, passing the token in plain text is probably not the + wisest idea, so you might want to use a + [secret variable](../variables/README.md#secret-variables) for that purpose. To trigger a job you need to send a `POST` request to GitLab's API endpoint: @@ -57,11 +61,11 @@ To trigger a job you need to send a `POST` request to GitLab's API endpoint: POST /projects/:id/trigger/pipeline ``` -The required parameters are the trigger's `token` and the Git `ref` on which -the trigger will be performed. Valid refs are the branch and the tag. The `:id` -of a project can be found by [querying the API](../../api/projects.md) -or by visiting the **Pipelines** settings page which provides -self-explanatory examples. +The required parameters are the [trigger's `token`](#authentication-tokens) +and the Git `ref` on which the trigger will be performed. Valid refs are the +branch and the tag. The `:id` of a project can be found by +[querying the API](../../api/projects.md) or by visiting the **Pipelines** +settings page which provides self-explanatory examples. When a rerun of a pipeline is triggered, the information is exposed in GitLab's UI under the **Jobs** page and the jobs are marked as triggered 'by API'. @@ -78,46 +82,7 @@ below. --- -See the [Examples](#examples) section for more details on how to actually -trigger a rebuild. - -## Trigger a pipeline from webhook - -> Introduced in GitLab 8.14. - -To trigger a job from webhook of another project you need to add the following -webhook url for Push and Tag push events: - -``` -https://gitlab.example.com/api/v4/projects/:id/ref/:ref/trigger/pipeline?token=TOKEN -``` - -> **Note**: -- `ref` should be passed as part of url in order to take precedence over `ref` - from webhook body that designates the branchref that fired the trigger in the source repository. -- `ref` should be url encoded if contains slashes. - -## Pass job variables to a trigger - -You can pass any number of arbitrary variables in the trigger API call and they -will be available in GitLab CI so that they can be used in your `.gitlab-ci.yml` -file. The parameter is of the form: - -``` -variables[key]=value -``` - -This information is also exposed in the UI. - - - ---- - -See the [Examples](#examples) section below for more details. - -## Examples - -Using cURL you can trigger a rebuild with minimal effort, for example: +By using cURL you can trigger a pipeline rerun with minimal effort, for example: ```bash curl --request POST \ @@ -135,8 +100,6 @@ curl --request POST \ "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=master" ``` -### Triggering a pipeline within `.gitlab-ci.yml` - You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that you have two projects, A and B, and you want to trigger a rebuild on the `master` branch of project B whenever a tag on project A is created. This is the job you @@ -156,14 +119,37 @@ Now, whenever a new tag is pushed on project A, the job will run and the `stage: deploy` ensures that this job will run only after all jobs with `stage: test` complete successfully. -_**Note:** If your project is public, passing the token in plain text is -probably not the wisest idea, so you might want to use a -[secure variable](../variables/README.md#user-defined-variables-secure-variables) -for that purpose._ +## Triggering a pipeline from a webhook -### Making use of trigger variables +> **Notes**: +- Introduced in GitLab 8.14. +- `ref` should be passed as part of the URL in order to take precedence over + `ref` from the webhook body that designates the branch ref that fired the + trigger in the source repository. +- `ref` should be URL-encoded if it contains slashes. -Using trigger variables can be proven useful for a variety of reasons. +To trigger a job from a webhook of another project you need to add the following +webhook URL for Push and Tag events (change the project ID, ref and token): + +``` +https://gitlab.example.com/api/v4/projects/9/ref/master/trigger/pipeline?token=TOKEN +``` + +## Making use of trigger variables + +You can pass any number of arbitrary variables in the trigger API call and they +will be available in GitLab CI so that they can be used in your `.gitlab-ci.yml` +file. The parameter is of the form: + +``` +variables[key]=value +``` + +This information is also exposed in the UI. + + + +Using trigger variables can be proven useful for a variety of reasons: * Identifiable jobs. Since the variable is exposed in the UI you can know why the rebuild was triggered if you pass a variable that explains the @@ -208,15 +194,7 @@ curl --request POST \ https://gitlab.example.com/api/v4/projects/9/trigger/pipeline ``` -### Using a webhook to trigger a pipeline - -You can add the following webhook to another project in order to trigger a job: - -``` -https://gitlab.example.com/api/v4/projects/9/ref/master/trigger/pipeline?token=TOKEN&variables[UPLOAD_TO_S3]=true -``` - -### Using cron to trigger nightly pipelines +## Using cron to trigger nightly pipelines >**Note:** The following behavior can also be achieved through GitLab's UI with @@ -230,4 +208,18 @@ branch of project with ID `9` every night at `00:30`: 30 0 * * * curl --request POST --form token=TOKEN --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline ``` +## Legacy triggers + +Old triggers, created before GitLab 9.0 will be marked as legacy. + +Triggers with the legacy label do not have an associated user and only have +access to the current project. They are considered deprecated and will be +removed with one of the future versions of GitLab. You are advised to +[take ownership](#taking-ownership) of any legacy triggers. + +[ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 [ci-229]: https://gitlab.com/gitlab-org/gitlab-ci/merge_requests/229 +[ee]: https://about.gitlab.com/gitlab-ee/ +[variables]: ../variables/README.md +[predef]: ../variables/README.md#predefined-variables-environment-variables +[registry]: ../../user/project/container_registry.md diff --git a/doc/ci/triggers/img/triggers_page.png b/doc/ci/triggers/img/triggers_page.png Binary files differindex eafd8519a23..7dc8f91cf7e 100644 --- a/doc/ci/triggers/img/triggers_page.png +++ b/doc/ci/triggers/img/triggers_page.png diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 76ba78ea7be..22e7f6879ed 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -9,8 +9,9 @@ and a list of **user-defined variables**. The variables can be overwritten and they take precedence over each other in this order: -1. [Trigger variables][triggers] (take precedence over all) -1. [Secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) +1. [Trigger variables][triggers] or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables) (take precedence over all) +1. Project-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) +1. Group-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) 1. YAML-defined [job-level variables](../yaml/README.md#job-variables) 1. YAML-defined [global variables](../yaml/README.md#variables) 1. [Deployment variables](#deployment-variables) @@ -37,9 +38,10 @@ future GitLab releases.** |-------------------------------- |--------|--------|-------------| | **CI** | all | 0.4 | Mark that job is executed in CI environment | | **CI_COMMIT_REF_NAME** | 9.0 | all | The branch or tag name for which project is built | -| **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | +| **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | | **CI_COMMIT_SHA** | 9.0 | all | The commit revision for which project is built | | **CI_COMMIT_TAG** | 9.0 | 0.5 | The commit tag name. Present only when building tags. | +| **CI_CONFIG_PATH** | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | | **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | | **CI_ENVIRONMENT_NAME** | 8.15 | all | The name of the environment for this job | | **CI_ENVIRONMENT_SLUG** | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. | @@ -141,25 +143,30 @@ script: >**Notes:** - This feature requires GitLab Runner 0.4.0 or higher. +- Group-level secret variables added in GitLab 9.4. - Be aware that secret variables are not masked, and their values can be shown in the job logs if explicitly asked to do so. If your project is public or internal, you can set the pipelines private from your project's Pipelines settings. Follow the discussion in issue [#13784][ce-13784] for masking the secret variables. -GitLab CI allows you to define per-project **secret variables** that are set in -the build environment. The secret variables are stored out of the repository -(`.gitlab-ci.yml`) and are securely passed to GitLab Runner making them -available in the build environment. It's the recommended method to use for -storing things like passwords, secret keys and credentials. +GitLab CI allows you to define per-project or per-group **secret variables** +that are set in the build environment. The secret variables are stored out of +the repository (`.gitlab-ci.yml`) and are securely passed to GitLab Runner +making them available in the build environment. It's the recommended method to +use for storing things like passwords, secret keys and credentials. -Secret variables can be added by going to your project's -**Settings ➔ Pipelines**, then finding the section called -**Secret variables**. +Project-level secret variables can be added by going to your project's +**Settings ➔ Pipelines**, then finding the section called **Secret variables**. -Once you set them, they will be available for all subsequent pipelines. +Likewise, group-level secret variables can be added by going to your group's +**Settings ➔ Pipelines**, then finding the section called **Secret variables**. +Any variables of [subgroups] will be inherited recursively. + +Once you set them, they will be available for all subsequent pipelines. You can also +[protect your variables](#protected-secret-variables). -## Protected secret variables +### Protected secret variables >**Notes:** This feature requires GitLab 9.3 or higher. @@ -425,9 +432,12 @@ export CI_REGISTRY_PASSWORD="longalfanumstring" ``` [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 -[runner]: https://docs.gitlab.com/runner/ -[triggered]: ../triggers/README.md -[triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger +[eep]: https://about.gitlab.com/gitlab-ee/ "Available only in GitLab Enterprise Edition Premium" +[envs]: ../environments.md [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md +[runner]: https://docs.gitlab.com/runner/ [shellexecutors]: https://docs.gitlab.com/runner/executors/ +[triggered]: ../triggers/README.md +[triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger +[subgroups]: ../../user/group/subgroups/index.md diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index fc813694ff2..1869782fe6e 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -306,6 +306,53 @@ cache: untracked: true ``` +### cache:policy + +> Introduced in GitLab 9.4. + +The default behaviour of a caching job is to download the files at the start of +execution, and to re-upload them at the end. This allows any changes made by the +job to be persisted for future runs, and is known as the `pull-push` cache +policy. + +If you know the job doesn't alter the cached files, you can skip the upload step +by setting `policy: pull` in the job specification. Typically, this would be +twinned with an ordinary cache job at an earlier stage to ensure the cache +is updated from time to time: + +```yaml +stages: + - setup + - test + +prepare: + stage: setup + cache: + key: gems + paths: + - vendor/bundle + script: + - bundle install --deployment + +rspec: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - bundle exec rspec ... +``` + +This helps to speed up job execution and reduce load on the cache server, +especially when you have a large number of cache-using jobs executing in +parallel. + +Additionally, if you have a job that unconditionally recreates the cache without +reference to its previous contents, you can use `policy: push` in that job to +skip the download step. + ## Jobs `.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job @@ -348,6 +395,7 @@ job_name: | after_script | no | Override a set of commands that are executed after job | | environment | no | Defines a name of environment to which deployment is done by this job | | coverage | no | Define code coverage settings for a given job | +| retry | no | Define how many times a job can be auto-retried in case of a failure | ### script @@ -393,12 +441,25 @@ There are a few rules that apply to the usage of refs policy: * `only` and `except` are inclusive. If both `only` and `except` are defined in a job specification, the ref is filtered by `only` and `except`. * `only` and `except` allow the use of regular expressions. -* `only` and `except` allow the use of special keywords: `branches`, `tags`, and `triggers`. * `only` and `except` allow to specify a repository path to filter jobs for forks. +In addition, `only` and `except` allow the use of special keywords: + +| **Value** | **Description** | +| --------- | ---------------- | +| `branches` | When a branch is pushed. | +| `tags` | When a tag is pushed. | +| `api` | When pipeline has been triggered by a second pipelines API (not triggers API). | +| `external` | When using CI services other than GitLab. | +| `pipelines` | For multi-project triggers, created using the API with `CI_JOB_TOKEN`. | +| `pushes` | Pipeline is triggered by a `git push` by the user. | +| `schedules` | For [scheduled pipelines][schedules]. | +| `triggers` | For pipelines created using a trigger token. | +| `web` | For pipelines created using **Run pipeline** button in GitLab UI (under your project's **Pipelines**). | + In the example below, `job` will run only for refs that start with `issue-`, -whereas all branches will be skipped. +whereas all branches will be skipped: ```yaml job: @@ -411,7 +472,7 @@ job: ``` In this example, `job` will run only for refs that are tagged, or if a build is -explicitly requested via an API trigger. +explicitly requested via an API trigger or a [Pipeline Schedule][schedules]: ```yaml job: @@ -419,6 +480,7 @@ job: only: - tags - triggers + - schedules ``` The repository path can be used to have jobs executed only for the parent @@ -1080,9 +1142,33 @@ A simple example: ```yaml job1: + script: rspec coverage: '/Code coverage: \d+\.\d+/' ``` +### retry + +**Notes:** +- [Introduced][ce-3442] in GitLab 9.5. + +`retry` allows you to configure how many times a job is going to be retried in +case of a failure. + +When a job fails, and has `retry` configured it is going to be processed again +up to the amount of times specified by the `retry` keyword. + +If `retry` is set to 2, and a job succeeds in a second run (first retry), it won't be retried +again. `retry` value has to be a positive integer, equal or larger than 0, but +lower or equal to 2 (two retries maximum, three runs in total). + +A simple example: + +```yaml +test: + script: rspec + retry: 2 +``` + ## Git Strategy > Introduced in GitLab 8.9 as an experimental feature. May change or be removed @@ -1457,3 +1543,5 @@ CI with various languages. [variables]: ../variables/README.md [ce-7983]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7983 [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 +[ce-3442]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3442 +[schedules]: ../../user/project/pipelines/schedules.md diff --git a/doc/development/README.md b/doc/development/README.md index af4131c4a8f..58993c52dcd 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -51,6 +51,11 @@ - [Post Deployment Migrations](post_deployment_migrations.md) - [Foreign Keys & Associations](foreign_keys.md) - [Serializing Data](serializing_data.md) +- [Polymorphic Associations](polymorphic_associations.md) +- [Single Table Inheritance](single_table_inheritance.md) +- [Background Migrations](background_migrations.md) +- [Storing SHA1 Hashes As Binary](sha1_as_binary.md) +- [Iterating Tables In Batches](iterating_tables_in_batches.md) ## i18n diff --git a/doc/development/architecture.md b/doc/development/architecture.md index acd5e3c2093..54029e00507 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -194,3 +194,7 @@ bundle exec rake gitlab:check RAILS_ENV=production ``` Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While the sudo commands provided by gitlabhq work in Ubuntu they do not always work in RHEL. + +## GitLab.com + +We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/handbook/infrastructure/production-architecture/) but this is probably over the top unless you have millions of users. diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md new file mode 100644 index 00000000000..e67db9ff142 --- /dev/null +++ b/doc/development/background_migrations.md @@ -0,0 +1,250 @@ +# Background Migrations + +Background migrations can be used to perform data migrations that would +otherwise take a very long time (hours, days, years, etc) to complete. For +example, you can use background migrations to migrate data so that instead of +storing data in a single JSON column the data is stored in a separate table. + +## When To Use Background Migrations + +In the vast majority of cases you will want to use a regular Rails migration +instead. Background migrations should _only_ be used when migrating _data_ in +tables that have so many rows this process would take hours when performed in a +regular Rails migration. + +Background migrations _may not_ be used to perform schema migrations, they +should only be used for data migrations. + +Some examples where background migrations can be useful: + +* Migrating events from one table to multiple separate tables. +* Populating one column based on JSON stored in another column. +* Migrating data that depends on the output of exernal services (e.g. an API). + +## Isolation + +Background migrations must be isolated and can not use application code (e.g. +models defined in `app/models`). Since these migrations can take a long time to +run it's possible for new versions to be deployed while they are still running. + +It's also possible for different migrations to be executed at the same time. +This means that different background migrations should not migrate data in a +way that would cause conflicts. + +## Idempotence + +Background migrations are executed in a context of a Sidekiq process. +Usual Sidekiq rules apply, especially the rule that jobs should be small +and idempotent. + +See [Sidekiq best practices guidelines](https://github.com/mperham/sidekiq/wiki/Best-Practices) +for more details. + +Make sure that in case that your migration job is going to be retried data +integrity is guarateed. + +## How It Works + +Background migrations are simple classes that define a `perform` method. A +Sidekiq worker will then execute such a class, passing any arguments to it. All +migration classes must be defined in the namespace +`Gitlab::BackgroundMigration`, the files should be placed in the directory +`lib/gitlab/background_migration/`. + +## Scheduling + +Scheduling a migration can be done in either a regular migration or a +post-deployment migration. To do so, simply use the following code while +replacing the class name and arguments with whatever values are necessary for +your migration: + +```ruby +BackgroundMigrationWorker.perform_async('BackgroundMigrationClassName', [arg1, arg2, ...]) +``` + +Usually it's better to enqueue jobs in bulk, for this you can use +`BackgroundMigrationWorker.perform_bulk`: + +```ruby +BackgroundMigrationWorker.perform_bulk( + [['BackgroundMigrationClassName', [1]], + ['BackgroundMigrationClassName', [2]]] +) +``` + +You'll also need to make sure that newly created data is either migrated, or +saved in both the old and new version upon creation. For complex and time +consuming migrations it's best to schedule a background job using an +`after_create` hook so this doesn't affect response timings. The same applies to +updates. Removals in turn can be handled by simply defining foreign keys with +cascading deletes. + +If you would like to schedule jobs in bulk with a delay, you can use +`BackgroundMigrationWorker.perform_bulk_in`: + +```ruby +jobs = [['BackgroundMigrationClassName', [1]], + ['BackgroundMigrationClassName', [2]]] + +BackgroundMigrationWorker.perform_bulk_in(5.minutes, jobs) +``` + +## Cleaning Up + +Because background migrations can take a long time you can't immediately clean +things up after scheduling them. For example, you can't drop a column that's +used in the migration process as this would cause jobs to fail. This means that +you'll need to add a separate _post deployment_ migration in a future release +that finishes any remaining jobs before cleaning things up (e.g. removing a +column). + +As an example, say you want to migrate the data from column `foo` (containing a +big JSON blob) to column `bar` (containing a string). The process for this would +roughly be as follows: + +1. Release A: + 1. Create a migration class that perform the migration for a row with a given ID. + 1. Deploy the code for this release, this should include some code that will + schedule jobs for newly created data (e.g. using an `after_create` hook). + 1. Schedule jobs for all existing rows in a post-deployment migration. It's + possible some newly created rows may be scheduled twice so your migration + should take care of this. +1. Release B: + 1. Deploy code so that the application starts using the new column and stops + scheduling jobs for newly created data. + 1. In a post-deployment migration you'll need to ensure no jobs remain. To do + so you can use `Gitlab::BackgroundMigration.steal` to process any remaining + jobs before continueing. + 1. Remove the old column. + +## Example + +To explain all this, let's use the following example: the table `services` has a +field called `properties` which is stored in JSON. For all rows you want to +extract the `url` key from this JSON object and store it in the `services.url` +column. There are millions of services and parsing JSON is slow, thus you can't +do this in a regular migration. + +To do this using a background migration we'll start with defining our migration +class: + +```ruby +class Gitlab::BackgroundMigration::ExtractServicesUrl + class Service < ActiveRecord::Base + self.table_name = 'services' + end + + def perform(service_id) + # A row may be removed between scheduling and starting of a job, thus we + # need to make sure the data is still present before doing any work. + service = Service.select(:properties).find_by(id: service_id) + + return unless service + + begin + json = JSON.load(service.properties) + rescue JSON::ParserError + # If the JSON is invalid we don't want to keep the job around forever, + # instead we'll just leave the "url" field to whatever the default value + # is. + return + end + + service.update(url: json['url']) if json['url'] + end +end +``` + +Next we'll need to adjust our code so we schedule the above migration for newly +created and updated services. We can do this using something along the lines of +the following: + +```ruby +class Service < ActiveRecord::Base + after_commit :schedule_service_migration, on: :update + after_commit :schedule_service_migration, on: :create + + def schedule_service_migration + BackgroundMigrationWorker.perform_async('ExtractServicesUrl', [id]) + end +end +``` + +We're using `after_commit` here to ensure the Sidekiq job is not scheduled +before the transaction completes as doing so can lead to race conditions where +the changes are not yet visible to the worker. + +Next we'll need a post-deployment migration that schedules the migration for +existing data. Since we're dealing with a lot of rows we'll schedule jobs in +batches instead of doing this one by one: + +```ruby +class ScheduleExtractServicesUrl < ActiveRecord::Migration + disable_ddl_transaction! + + class Service < ActiveRecord::Base + self.table_name = 'services' + end + + def up + Service.select(:id).in_batches do |relation| + jobs = relation.pluck(:id).map do |id| + ['ExtractServicesUrl', [id]] + end + + BackgroundMigrationWorker.perform_bulk(jobs) + end + end + + def down + end +end +``` + +Once deployed our application will continue using the data as before but at the +same time will ensure that both existing and new data is migrated. + +In the next release we can remove the `after_commit` hooks and related code. We +will also need to add a post-deployment migration that consumes any remaining +jobs. Such a migration would look like this: + +```ruby +class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration + disable_ddl_transaction! + + def up + Gitlab::BackgroundMigration.steal('ExtractServicesUrl') + end + + def down + end +end +``` + +This migration will then process any jobs for the ExtractServicesUrl migration +and continue once all jobs have been processed. Once done you can safely remove +the `services.properties` column. + +## Testing + +It is required to write tests for background migrations' scheduling migration +(either a regular migration or a post deployment migration), background +migration itself and a cleanup migration. You can use the `:migration` RSpec +tag when testing a regular / post deployment migration. +See [README][migrations-readme]. + +When you do that, keep in mind that `before` and `after` RSpec hooks are going +to migrate you database down and up, which can result in other background +migrations being called. That means that using `spy` test doubles with +`have_received` is encouraged, instead of using regular test doubles, because +your expectations defined in a `it` block can conflict with what is being +called in RSpec hooks. See [gitlab-org/gitlab-ce#35351][issue-rspec-hooks] +for more details. + +## Best practices + +1. Make sure that background migration jobs are idempotent. +1. Make sure that tests you write are not false positives. + +[migrations-readme]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md +[issue-rspec-hooks]: https://gitlab.com/gitlab-org/gitlab-ce/issues/35351 diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 4ed89146072..64a89976300 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -11,6 +11,8 @@ There are a few rules to get your merge request accepted: **approved by a [frontend maintainer][projects]**. 1. If your merge request includes frontend and backend changes [^1], it must be **approved by a [frontend and a backend maintainer][projects]**. + 1. If your merge request includes a new dependency or a filesystem change, it must + be **approved by a [Build team member][team]**. See [how to work with the Build team][build handbook] for more details. 1. To lower the amount of merge requests maintainers need to review, you can ask or assign any [reviewers][projects] for a first review. 1. If you need some guidance (e.g. it's your first merge request), feel free @@ -133,6 +135,55 @@ reviewee. tomorrow. When you are not able to find the right balance, ask other people about their opinion. +### GitLab-specific concerns + +GitLab is used in a lot of places. Many users use +our [Omnibus packages](https://about.gitlab.com/installation/), but some use +the [Docker images](https://docs.gitlab.com/omnibus/docker/), some are +[installed from source](https://docs.gitlab.com/ce/install/installation.html), +and there are other installation methods available. GitLab.com itself is a large +Enterprise Edition instance. This has some implications: + +1. **Query changes** should be tested to ensure that they don't result in worse + performance at the scale of GitLab.com: + 1. Generating large quantities of data locally can help. + 2. Asking for query plans from GitLab.com is the most reliable way to validate + these. +2. **Database migrations** must be: + 1. Reversible. + 2. Performant at the scale of GitLab.com - ask a maintainer to test the + migration on the staging environment if you aren't sure. + 3. Categorised correctly: + - Regular migrations run before the new code is running on the instance. + - [Post-deployment migrations](post_deployment_migrations.md) run _after_ + the new code is deployed, when the instance is configured to do that. + - [Background migrations](background_migrations.md) run in Sidekiq, and + should only be done for migrations that would take an extreme amount of + time at GitLab.com scale. +3. **Sidekiq workers** + [cannot change in a backwards-incompatible way](sidekiq_style_guide.md#removing-or-renaming-queues): + 1. Sidekiq queues are not drained before a deploy happens, so there will be + workers in the queue from the previous version of GitLab. + 2. If you need to change a method signature, try to do so across two releases, + and accept both the old and new arguments in the first of those. + 3. Similarly, if you need to remove a worker, stop it from being scheduled in + one release, then remove it in the next. This will allow existing jobs to + execute. + 4. Don't forget, not every instance will upgrade to every intermediate version + (some people may go from X.1.0 to X.10.0, or even try bigger upgrades!), so + try to be liberal in accepting the old format if it is cheap to do so. +4. **Cached values** may persist across releases. If you are changing the type a + cached value returns (say, from a string or nil to an array), change the + cache key at the same time. +5. **Settings** should be added as a + [last resort](https://about.gitlab.com/handbook/product/#convention-over-configuration). + If you're adding a new setting in `gitlab.yml`: + 1. Try to avoid that, and add to `ApplicationSetting` instead. + 2. Ensure that it is also + [added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml.html#adding-a-new-setting-to-gitlab-yml). +6. **Filesystem access** can be slow, so try to avoid + [shared files](shared_files.md) when an alternative solution is available. + ### Credits Largely based on the [thoughtbot code review guide]. @@ -145,3 +196,4 @@ Largely based on the [thoughtbot code review guide]. [projects]: https://about.gitlab.com/handbook/engineering/projects/ [team]: https://about.gitlab.com/team/ +[build handbook]: https://about.gitlab.com/handbook/build/handbook/build#how-to-work-with-build diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md index 5b09f79f143..90d1d9657b9 100644 --- a/doc/development/doc_styleguide.md +++ b/doc/development/doc_styleguide.md @@ -105,8 +105,8 @@ merge request. considered beta or experimental, put this info in a note, not in the heading. - When introducing a new document, be careful for the headings to be grammatically and syntactically correct. It is advised to mention one or all - of the following GitLab members for a review: `@axil`, `@rspeicher`, `@marcia`, - `@SeanPackham`. This is to ensure that no document with wrong heading is going + of the following GitLab members for a review: `@axil`, `@rspeicher`, `@marcia`. + This is to ensure that no document with wrong heading is going live without an audit, thus preventing dead links and redirection issues when corrected - Leave exactly one newline after a heading @@ -388,8 +388,8 @@ the style below as a guide: 1. Save the file and [restart] GitLab for the changes to take effect. -[reconfigure]: path/to/administration/gitlab_restart.md#omnibus-gitlab-reconfigure -[restart]: path/to/administration/gitlab_restart.md#installations-from-source +[reconfigure]: path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: path/to/administration/restart_gitlab.md#installations-from-source ```` In this case: diff --git a/doc/development/fe_guide/droplab/plugins/input_setter.md b/doc/development/fe_guide/droplab/plugins/input_setter.md index a549603c20d..94f3c8254a8 100644 --- a/doc/development/fe_guide/droplab/plugins/input_setter.md +++ b/doc/development/fe_guide/droplab/plugins/input_setter.md @@ -1,6 +1,6 @@ # InputSetter -`InputSetter` is a plugin that allows for udating DOM out of the scope of droplab when a list item is clicked. +`InputSetter` is a plugin that allows for updating DOM out of the scope of droplab when a list item is clicked. ## Usage diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 2ddcbe13afa..14ac1133cc0 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -23,6 +23,19 @@ controlled by the server. 1. The backend code will most likely be using etags. You do not and should not check for status `304 Not Modified`. The browser will transform it for you. +### Lazy Loading + +To improve the time to first render we are using lazy loading for images. This works by setting +the actual image source on the `data-src` attribute. After the HTML is rendered and JavaScript is loaded, +the value of `data-src` will be moved to `src` automatically if the image is in the current viewport. + +* Prepare images in HTML for lazy loading by renaming the `src` attribute to `data-src` AND adding the class `lazy` +* If you are using the Rails `image_tag` helper, all images will be lazy-loaded by default unless `lazy: false` is provided. + +If you are asynchronously adding content which contains lazy images then you need to call the function +`gl.lazyLoader.searchLazyImages()` which will search for lazy images and load them if needed. +But in general it should be handled automatically through a `MutationObserver` in the lazy loading function. + ## Reducing Asset Footprint ### Page-specific JavaScript diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index d2d89517241..149a0159680 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -447,6 +447,7 @@ A forEach will cause side effects, it will be mutating the array being iterated. 1. `name` 1. `props` 1. `mixins` + 1. `directives` 1. `data` 1. `components` 1. `computedProps` @@ -463,20 +464,24 @@ A forEach will cause side effects, it will be mutating the array being iterated. 1. `destroyed` #### Vue and Boostrap -1. Tooltips: Do not rely on `has-tooltip` class name for vue components +1. Tooltips: Do not rely on `has-tooltip` class name for Vue components ``` javascript // bad - <span class="has-tooltip"> + <span + class="has-tooltip" + title="Some tooltip text"> Text </span> // good - <span data-toggle="tooltip"> + <span + v-tooltip + title="Some tooltip text"> Text </span> ``` -1. Tooltips: When using a tooltip, include the tooltip mixin +1. Tooltips: When using a tooltip, include the tooltip directive, `./app/assets/javascripts/vue_shared/directives/tooltip.js` 1. Don't change `data-original-title`. ```javascript diff --git a/doc/development/fe_guide/testing.md b/doc/development/fe_guide/testing.md index 0ef9fc61a61..867c83f1e72 100644 --- a/doc/development/fe_guide/testing.md +++ b/doc/development/fe_guide/testing.md @@ -7,7 +7,7 @@ feature tests with Capybara for e2e (end-to-end) integration testing. Unit and feature tests need to be written for all new features. Most of the time, you should use rspec for your feature tests. There are cases where the behaviour you are testing is not worth the time spent running the full application, -for example, if you are testing styling, animation or small actions that don't involve the backend, +for example, if you are testing styling, animation, edge cases or small actions that don't involve the backend, you should write an integration test using Jasmine.  diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index a984bb6c94c..0742b202807 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -112,7 +112,50 @@ Vue Resource should only be imported in the service file. Vue.use(VueResource); ``` -### CSRF token +#### Vue-resource gotchas +#### Headers +Headers are being parsed into a plain object in an interceptor. +In Vue-resource 1.x `headers` object was changed into an `Headers` object. In order to not change all old code, an interceptor was added. + +If you need to write a unit test that takes the headers in consideration, you need to include an interceptor to parse the headers after your test interceptor. +You can see an example in `spec/javascripts/environments/environment_spec.js`: + ```javascript + import { headersInterceptor } from './helpers/vue_resource_helper'; + + beforeEach(() => { + Vue.http.interceptors.push(myInterceptor); + Vue.http.interceptors.push(headersInterceptor); + }); + + afterEach(() => { + Vue.http.interceptors = _.without(Vue.http.interceptors, myInterceptor); + Vue.http.interceptors = _.without(Vue.http.interceptors, headersInterceptor); + }); + ``` + +#### `.json()` +When making a request to the server, you will most likely need to access the body of the response. +Use `.json()` to convert. Because `.json()` returns a Promise the follwoing structure should be used: + + ```javascript + service.get('url') + .then(resp => resp.json()) + .then((data) => { + this.store.storeData(data); + }) + .catch(() => new Flash('Something went wrong')); + ``` + +When using `Poll` (`app/assets/javascripts/lib/utils/poll.js`), the `successCallback` needs to handle `.json()` as a Promise: + ```javascript + successCallback: (response) => { + return response.json().then((data) => { + // handle the response + }); + } + ``` + +#### CSRF token We use a Vue Resource interceptor to manage the CSRF token. `app/assets/javascripts/vue_shared/vue_resource_interceptor.js` holds all our common interceptors. Note: You don't need to load `app/assets/javascripts/vue_shared/vue_resource_interceptor.js` @@ -126,13 +169,13 @@ The following example shows an application: // store.js export default class Store { - /** + /** * This is where we will iniatialize the state of our data. * Usually in a small SPA you don't need any options when starting the store. In the case you do * need guarantee it's an Object and it's documented. - * - * @param {Object} options - */ + * + * @param {Object} options + */ constructor(options) { this.options = options; @@ -205,14 +248,14 @@ import Store from 'store'; import Service from 'service'; import TodoComponent from 'todoComponent'; export default { - /** + /** * Although most data belongs in the store, each component it's own state. * We want to show a loading spinner while we are fetching the todos, this state belong * in the component. * * We need to access the store methods through all methods of our component. * We need to access the state of our store. - */ + */ data() { const store = new Store(); @@ -396,42 +439,46 @@ need to test the rendered output. [Vue][vue-test] guide's to unit test show us e [Vue Resource Interceptors][vue-resource-interceptor] allow us to add a interceptor with the response we need: -```javascript - // Mock the service to return data - const interceptor = (request, next) => { - next(request.respondWith(JSON.stringify([{ - title: 'This is a todo', - body: 'This is the text' - }]), { - status: 200, - })); - }; + ```javascript + // Mock the service to return data + const interceptor = (request, next) => { + next(request.respondWith(JSON.stringify([{ + title: 'This is a todo', + body: 'This is the text' + }]), { + status: 200, + })); + }; - beforeEach(() => { - Vue.http.interceptors.push(interceptor); - }); + beforeEach(() => { + Vue.http.interceptors.push(interceptor); + }); - afterEach(() => { - Vue.http.interceptors = _.without(Vue.http.interceptors, interceptor); - }); + afterEach(() => { + Vue.http.interceptors = _.without(Vue.http.interceptors, interceptor); + }); - it('should do something', (done) => { - setTimeout(() => { - // Test received data - done(); - }, 0); - }); -``` + it('should do something', (done) => { + setTimeout(() => { + // Test received data + done(); + }, 0); + }); + ``` + +1. Headers interceptor +Refer to [this section](vue.md#headers) 1. Use `$.mount()` to mount the component + ```javascript - // bad - new Component({ - el: document.createElement('div') - }); +// bad +new Component({ + el: document.createElement('div') +}); - // good - new Component().$mount(); +// good +new Component().$mount(); ``` [vue-docs]: http://vuejs.org/guide/index.html diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 5c6316b9ac6..59e8a087e02 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -3,5 +3,19 @@ Starting from GitLab 9.3 we support feature flags via [Flipper](https://github.com/jnunemaker/flipper/). You should use the `Feature` class (defined in `lib/feature.rb`) in your code to get, set and list feature -flags. During runtime you can set the values for the gates via the -[admin API](../api/features.md). +flags. + +During runtime you can set the values for the gates via the +[features API](../api/features.md) (accessible to admins only). + +## Feature groups + +Starting from GitLab 9.4 we support feature groups via +[Flipper groups](https://github.com/jnunemaker/flipper/blob/v0.10.2/docs/Gates.md#2-group). + +Feature groups must be defined statically in `lib/feature.rb` (in the +`.register_feature_groups` method), but their implementation can obviously be +dynamic (querying the DB etc.). + +Once defined in `lib/feature.rb`, you will be able to activate a +feature for a given feature group via the [`feature_group` param of the features API](../api/features.md#set-or-create-a-feature) diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index 565d4b33457..c2ca8966a3f 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -3,35 +3,6 @@ The purpose of this guide is to document potential "gotchas" that contributors might encounter or should avoid during development of GitLab CE and EE. -## Do not `describe` symbols - -Consider the following model spec: - -```ruby -require 'rails_helper' - -describe User do - describe :to_param do - it 'converts the username to a param' do - user = described_class.new(username: 'John Smith') - - expect(user.to_param).to eq 'john-smith' - end - end -end -``` - -When run, this spec doesn't do what we might expect: - -```sh -spec/models/user_spec.rb|6 error| Failure/Error: u = described_class.new NoMethodError: undefined method `new' for :to_param:Symbol -``` - -### Solution - -Except for the top-level `describe` block, always provide a String argument to -`describe`. - ## Do not assert against the absolute value of a sequence-generated attribute Consider the following factory: diff --git a/doc/development/i18n_guide.md b/doc/development/i18n_guide.md index bfb0779fbfa..756535e28bc 100644 --- a/doc/development/i18n_guide.md +++ b/doc/development/i18n_guide.md @@ -127,6 +127,14 @@ New translations will be added with their default content and will be marked fuzzy. To use the translation, look for the `#, fuzzy` mention in `gitlab.edit.po` and remove it. +We need to make sure we remove the `fuzzy` translations before generating the +`locale/**/gitlab.po` file. When they aren't removed, the resulting `.po` will +be treated as a binary file which could overwrite translations that were merged +before the new translations. + +When we are just preparing a page to be translated, but not actually adding any +translations. There's no need to generate `.po` files. + Translations that aren't used in the source code anymore will be marked with `~#`; these can be removed to keep our translation files clutter-free. diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md new file mode 100644 index 00000000000..590c8cbba2d --- /dev/null +++ b/doc/development/iterating_tables_in_batches.md @@ -0,0 +1,37 @@ +# Iterating Tables In Batches + +Rails provides a method called `in_batches` that can be used to iterate over +rows in batches. For example: + +```ruby +User.in_batches(of: 10) do |relation| + relation.update_all(updated_at: Time.now) +end +``` + +Unfortunately this method is implemented in a way that is not very efficient, +both query and memory usage wise. + +To work around this you can include the `EachBatch` module into your models, +then use the `each_batch` class method. For example: + +```ruby +class User < ActiveRecord::Base + include EachBatch +end + +User.each_batch(of: 10) do |relation| + relation.update_all(updated_at: Time.now) +end +``` + +This will end up producing queries such as: + +``` +User Load (0.7ms) SELECT "users"."id" FROM "users" WHERE ("users"."id" >= 41654) ORDER BY "users"."id" ASC LIMIT 1 OFFSET 1000 + (0.7ms) SELECT COUNT(*) FROM "users" WHERE ("users"."id" >= 41654) AND ("users"."id" < 42687) +``` + +The API of this method is similar to `in_batches`, though it doesn't support +all of the arguments that `in_batches` supports. You should always use +`each_batch` _unless_ you have a specific need for `in_batches`. diff --git a/doc/development/limit_ee_conflicts.md b/doc/development/limit_ee_conflicts.md index 51b4b398f2c..899be9eae4b 100644 --- a/doc/development/limit_ee_conflicts.md +++ b/doc/development/limit_ee_conflicts.md @@ -166,8 +166,8 @@ For instance this kind of thing: = render 'projects/zen', f: form, attr: :description, classes: 'note-textarea', placeholder: "Write a comment or drag your files here...", - supports_slash_commands: !issuable.persisted? - = render 'projects/notes/hints', supports_slash_commands: !issuable.persisted? + supports_quick_actions: !issuable.persisted? + = render 'projects/notes/hints', supports_quick_actions: !issuable.persisted? .clearfix .error-alert - if issuable.is_a?(Issue) diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 77ba2a5fd87..161d2544169 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -122,7 +122,7 @@ limit can vary from installation to installation. As a result it's recommended you do not use more than 32 threads in a single migration. Usually 4-8 threads should be more than enough. -## Removing indices +## Removing indexes When removing an index make sure to use the method `remove_concurrent_index` instead of the regular `remove_index` method. The `remove_concurrent_index` method @@ -142,7 +142,7 @@ class MyMigration < ActiveRecord::Migration end ``` -## Adding indices +## Adding indexes If you need to add a unique index please keep in mind there is the possibility of existing duplicates being present in the database. This means that should @@ -222,6 +222,41 @@ add_column_with_default(:projects, :foo, :integer, default: 10, limit: 8) add_column(:projects, :foo, :integer, default: 10, limit: 8) ``` +## Timestamp column type + +By default, Rails uses the `timestamp` data type that stores timestamp data without timezone information. +The `timestamp` data type is used by calling either the `add_timestamps` or the `timestamps` method. +Also Rails converts the `:datetime` data type to the `timestamp` one. + +Example: + +```ruby +# timestamps +create_table :users do |t| + t.timestamps +end + +# add_timestamps +def up + add_timestamps :users +end + +# :datetime +def up + add_column :users, :last_sign_in, :datetime +end +``` + +Instead of using these methods one should use the following methods to store timestamps with timezones: + +* `add_timestamps_with_timezone` +* `timestamps_with_timezone` + +This ensures all timestamps have a time zone specified. This in turn means existing timestamps won't +suddenly use a different timezone when the system's timezone changes. It also makes it very clear which +timezone was used in the first place. + + ## Testing Make sure that your migration works with MySQL and PostgreSQL with data. An diff --git a/doc/development/policies.md b/doc/development/policies.md new file mode 100644 index 00000000000..62141356f59 --- /dev/null +++ b/doc/development/policies.md @@ -0,0 +1,116 @@ +# `DeclarativePolicy` framework + +The DeclarativePolicy framework is designed to assist in performance of policy checks, and to enable ease of extension for EE. The DSL code in `app/policies` is what `Ability.allowed?` uses to check whether a particular action is allowed on a subject. + +The policy used is based on the subject's class name - so `Ability.allowed?(user, :some_ability, project)` will create a `ProjectPolicy` and check permissions on that. + +## Managing Permission Rules + +Permissions are broken into two parts: `conditions` and `rules`. Conditions are boolean expressions that can access the database and the environment, while rules are statically configured combinations of expressions and other rules that enable or prevent certain abilities. For an ability to be allowed, it must be enabled by at least one rule, and not prevented by any. + + +### Conditions + +Conditions are defined by the `condition` method, and are given a name and a block. The block will be executed in the context of the policy object - so it can access `@user` and `@subject`, as well as call any methods defined on the policy. Note that `@user` may be nil (in the anonymous case), but `@subject` is guaranteed to be a real instance of the subject class. + +``` ruby +class FooPolicy < BasePolicy + condition(:is_public) do + # @subject guaranteed to be an instance of Foo + @subject.public? + end + + # instance methods can be called from the condition as well + condition(:thing) { check_thing } + + def check_thing + # ... + end +end +``` + +When you define a condition, a predicate method is defined on the policy to check whether that condition passes - so in the above example, an instance of `FooPolicy` will also respond to `#is_public?` and `#thing?`. + +Conditions are cached according to their scope. Scope and ordering will be covered later. + +### Rules + +A `rule` is a logical combination of conditions and other rules, that are configured to enable or prevent certain abilities. It is important to note that the rule configuration is static - a rule's logic cannot touch the database or know about `@user` or `@subject`. This allows us to cache only at the condition level. Rules are specified through the `rule` method, which takes a block of DSL configuration, and returns an object that responds to `#enable` or `#prevent`: + +``` ruby +class FooPolicy < BasePolicy + # ... + + rule { is_public }.enable :read + rule { thing }.prevent :read + + # equivalently, + rule { is_public }.policy do + enable :read + end + + rule { ~thing }.policy do + prevent :read + end +end +``` + +Within the rule DSL, you can use: + +* A regular word mentions a condition by name - a rule that is in effect when that condition is truthy. +* `~` indicates negation +* `&` and `|` are logical combinations, also available as `all?(...)` and `any?(...)` +* `can?(:other_ability)` delegates to the rules that apply to `:other_ability`. Note that this is distinct from the instance method `can?`, which can check dynamically - this only configures a delegation to another ability. + +## Scores, Order, Performance + +To see how the rules get evaluated into a judgment, it is useful in a console to use `policy.debug(:some_ability)`. This will print the rules in the order they are evaluated. + +When a policy is asked whether a particular ability is allowed (`policy.allowed?(:some_ability)`), it does not necessarily have to compute all the conditions on the policy. First, only the rules relevant to that particular ability are selected. Then, the execution model takes advantage of short-circuiting, and attempts to sort rules based on a heuristic of how expensive they will be to calculate. The sorting is dynamic and cache-aware, so that previously calculated conditions will be considered first, before computing other conditions. + +## Scope + +Sometimes, a condition will only use data from `@user` or only from `@subject`. In this case, we want to change the scope of the caching, so that we don't recalculate conditions unnecessarily. For example, given: + +``` ruby +class FooPolicy < BasePolicy + condition(:expensive_condition) { @subject.expensive_query? } + + rule { expensive_condition }.enable :some_ability +end +``` + +Naively, if we call `Ability.can?(user1, :some_ability, foo)` and `Ability.can?(user2, :some_ability, foo)`, we would have to calculate the condition twice - since they are for different users. But if we use the `scope: :subject` option: + +``` ruby + condition(:expensive_condition, scope: :subject) { @subject.expensive_query? } +``` + +then the result of the condition will be cached globally only based on the subject - so it will not be calculated repeatedly for different users. Similarly, `scope: :user` will cache only based on the user. + +**DANGER**: If you use a `:scope` option when the condition actually uses data from +both user and subject (including a simple anonymous check!) your result will be cached at too global of a scope and will result in cache bugs. + +Sometimes we are checking permissions for a lot of users for one subject, or a lot of subjects for one user. In this case, we want to set a *preferred scope* - i.e. tell the system that we prefer rules that can be cached on the repeated parameter. For example, in `Ability.users_that_can_read_project`: + +``` ruby +def users_that_can_read_project(users, project) + DeclarativePolicy.subject_scope do + users.select { |u| allowed?(u, :read_project, project) } + end +end +``` + +This will, for example, prefer checking `project.public?` to checking `user.admin?`. + +## Delegation + +Delegation is the inclusion of rules from another policy, on a different subject. For example, + +``` ruby +class FooPolicy < BasePolicy + delegate { @subject.project } +end +``` + +will include all rules from `ProjectPolicy`. The delegated conditions will be evaluated with the correct delegated subject, and will be sorted along with the regular rules in the policy. Note that only the relevant rules for a particular ability will actually be considered. diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md new file mode 100644 index 00000000000..d63b9fb115f --- /dev/null +++ b/doc/development/polymorphic_associations.md @@ -0,0 +1,146 @@ +# Polymorphic Associations + +**Summary:** always use separate tables instead of polymorphic associations. + +Rails makes it possible to define so called "polymorphic associations". This +usually works by adding two columns to a table: a target type column, and a +target id. For example, at the time of writing we have such a setup for +`members` with the following columns: + +* `source_type`: a string defining the model to use, can be either `Project` or + `Namespace`. +* `source_id`: the ID of the row to retrieve based on `source_type`. For + example, when `source_type` is `Project` then `source_id` will contain a + project ID. + +While such a setup may appear to be useful, it comes with many drawbacks; enough +that you should avoid this at all costs. + +## Space Wasted + +Because this setup relies on string values to determine the model to use it will +end up wasting a lot of space. For example, for `Project` and `Namespace` the +maximum size is 9 bytes, plus 1 extra byte for every string when using +PostgreSQL. While this may only be 10 bytes per row, given enough tables and +rows using such a setup we can end up wasting quite a bit of disk space and +memory (for any indexes). + +## Indexes + +Because our associations are broken up into two columns this may result in +requiring composite indexes for queries to be performed efficiently. While +composite indexes are not wrong at all, they can be tricky to set up as the +ordering of columns in these indexes is important to ensure optimal performance. + +## Consistency + +One really big problem with polymorphic associations is being unable to enforce +data consistency on the database level using foreign keys. For consistency to be +enforced on the database level one would have to write their own foreign key +logic to support polymorphic associations. + +Enforcing consistency on the database level is absolutely crucial for +maintaining a healthy environment, and thus is another reason to avoid +polymorphic associations. + +## Query Overhead + +When using polymorphic associations you always need to filter using both +columns. For example, you may end up writing a query like this: + +```sql +SELECT * +FROM members +WHERE source_type = 'Project' +AND source_id = 13083; +``` + +Here PostgreSQL can perform the query quite efficiently if both columns are +indexed, but as the query gets more complex it may not be able to use these +indexes efficiently. + +## Mixed Responsibilities + +Similar to functions and classes a table should have a single responsibility: +storing data with a certain set of pre-defined columns. When using polymorphic +associations you are instead storing different types of data (possibly with +different columns set) in the same table. + +## The Solution + +Fortunately there is a very simple solution to these problems: simply use a +separate table for every type you would otherwise store in the same table. Using +a separate table allows you to use everything a database may provide to ensure +consistency and query data efficiently, without any additional application logic +being necessary. + +Let's say you have a `members` table storing both approved and pending members, +for both projects and groups, and the pending state is determined by the column +`requested_at` being set or not. Schema wise such a setup can lead to various +columns only being set for certain rows, wasting space. It's also possible that +certain indexes will only be set for certain rows, again wasting space. Finally, +querying such a table requires less than ideal queries. For example: + +```sql +SELECT * +FROM members +WHERE requested_at IS NULL +AND source_type = 'GroupMember' +AND source_id = 4 +``` + +Instead such a table should be broken up into separate tables. For example, you +may end up with 4 tables in this case: + +* project_members +* group_members +* pending_project_members +* pending_group_members + +This makes querying data trivial. For example, to get the members of a group +you'd run: + +```sql +SELECT * +FROM group_members +WHERE group_id = 4 +``` + +To get all the pending members of a group in turn you'd run: + +```sql +SELECT * +FROM pending_group_members +WHERE group_id = 4 +``` + +If you want to get both you can use a UNION, though you need to be explicit +about what columns you want to SELECT as otherwise the result set will use the +columns of the first query. For example: + +```sql +SELECT id, 'Group' AS target_type, group_id AS target_id +FROM group_members + +UNION ALL + +SELECT id, 'Project' AS target_type, project_id AS target_id +FROM project_members +``` + +The above example is perhaps a bit silly, but it shows that there's nothing +stopping you from merging the data together and presenting it on the same page. +Selecting columns explicitly can also speed up queries as the database has to do +less work to get the data (compared to selecting all columns, even ones you're +not using). + +Our schema also becomes easier. No longer do we need to both store and index the +`source_type` column, we can define foreign keys easily, and we don't need to +filter rows using the `IS NULL` condition. + +To summarize: using separate tables allows us to use foreign keys effectively, +create indexes only where necessary, conserve space, query data more +efficiently, and scale these tables more easily (e.g. by storing them on +separate disks). A nice side effect of this is that code can also become easier +as you won't end up with a single model having to handle different kinds of +data. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index fdaaa65fa28..42bb5e8619c 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -12,6 +12,56 @@ The `setup` task is a alias for `gitlab:setup`. This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database. Note: `db:setup` calls `db:seed` but this does nothing. +### Automation + +If you're very sure that you want to **wipe the current database** and refill +seeds, you could: + +``` shell +echo 'yes' | bundle exec rake setup +``` + +To save you from answering `yes` manually. + +### Discard stdout + +Since the script would print a lot of information, it could be slowing down +your terminal, and it would generate more than 20G logs if you just redirect +it to a file. If we don't care about the output, we could just redirect it to +`/dev/null`: + +``` shell +echo 'yes' | bundle exec rake setup > /dev/null +``` + +Note that since you can't see the questions from stdout, you might just want +to `echo 'yes'` to keep it running. It would still print the errors on stderr +so no worries about missing errors. + +### Notes for MySQL + +Since the seeds would contain various UTF-8 characters, such as emojis or so, +we'll need to make sure that we're using `utf8mb4` for all the encoding +settings and `utf8mb4_unicode_ci` for collation. Please check +[MySQL utf8mb4 support](../install/database_mysql.md#mysql-utf8mb4-support) + +Make sure that `config/database.yml` has `encoding: utf8mb4`, too. + +Next, we'll need to update the schema to make the indices fit: + +``` shell +sed -i 's/limit: 255/limit: 191/g' db/schema.rb +``` + +Then run the setup script: + +``` shell +bundle exec rake setup +``` + +To make sure that indices still fit. You could find great details in: +[How to support full Unicode in MySQL databases](https://mathiasbynens.be/notes/mysql-utf8mb4) + ## Run tests In order to run the test you can use the following commands: diff --git a/doc/development/sha1_as_binary.md b/doc/development/sha1_as_binary.md new file mode 100644 index 00000000000..3151cc29bbc --- /dev/null +++ b/doc/development/sha1_as_binary.md @@ -0,0 +1,36 @@ +# Storing SHA1 Hashes As Binary + +Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string +requires at least 40 bytes, an additional byte to store the encoding, and +perhaps more space depending on the internals of PostgreSQL and MySQL. + +On the other hand, if one were to store a SHA1 as binary one would only need 20 +bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending +on database internals). This means that in the best case scenario we can reduce +the space usage by 50%. + +To make this easier to work with you can include the concern `ShaAttribute` into +a model and define a SHA attribute using the `sha_attribute` class method. For +example: + +```ruby +class Commit < ActiveRecord::Base + include ShaAttribute + + sha_attribute :sha +end +``` + +This allows you to use the value of the `sha` attribute as if it were a string, +while storing it as binary. This means that you can do something like this, +without having to worry about converting data to the right binary format: + +```ruby +commit = Commit.find_by(sha: '88c60307bd1f215095834f09a1a5cb18701ac8ad') +commit.sha = '971604de4cfa324d91c41650fabc129420c8d1cc' +commit.save +``` + +There is however one requirement: the column used to store the SHA has _must_ be +a binary type. For Rails this means you need to use the `:binary` type instead +of `:text` or `:string`. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e3a20f29a09..1e9fdbc65e2 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -36,3 +36,10 @@ slow jobs blocking work (even for different jobs) on the shared queue. Each Sidekiq worker must be tested using RSpec, just like any other class. These tests should be placed in `spec/workers`. + +## Removing or renaming queues + +Try to avoid renaming or removing queues in minor and patch releases. +During online update instance can have pending jobs and removing the queue can +lead to those jobs being stuck forever. If you can't write migration for those +Sidekiq jobs, please consider doing rename or remove queue in major release only.
\ No newline at end of file diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md new file mode 100644 index 00000000000..27c3c4f3199 --- /dev/null +++ b/doc/development/single_table_inheritance.md @@ -0,0 +1,18 @@ +# Single Table Inheritance + +**Summary:** don't use Single Table Inheritance (STI), use separate tables +instead. + +Rails makes it possible to have multiple models stored in the same table and map +these rows to the correct models using a `type` column. This can be used to for +example store two different types of SSH keys in the same table. + +While tempting to use one should avoid this at all costs for the same reasons as +outlined in the document ["Polymorphic Associations"](polymorphic_associations.md). + +## Solution + +The solution is very simple: just use a separate table for every type you'd +otherwise store in the same table. For example, instead of having a `keys` table +with `type` set to either `Key` or `DeployKey` you'd have two separate tables: +`keys` and `deploy_keys`. diff --git a/doc/development/testing.md b/doc/development/testing.md index 6d8b846d27f..3d5aa3d45e9 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -25,7 +25,7 @@ records should use stubs/doubles as much as possible. | --------- | ---------- | -------------- | ----- | | `app/finders/` | `spec/finders/` | RSpec | | | `app/helpers/` | `spec/helpers/` | RSpec | | -| `app/db/{post_,}migrate/` | `spec/migrations/` | RSpec | | +| `app/db/{post_,}migrate/` | `spec/migrations/` | RSpec | More details at [`spec/migrations/README.md`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md). | | `app/policies/` | `spec/policies/` | RSpec | | | `app/presenters/` | `spec/presenters/` | RSpec | | | `app/routing/` | `spec/routing/` | RSpec | | @@ -195,7 +195,6 @@ Please consult the [dedicated "Frontend testing" guide](./fe_guide/testing.md). - Use `context` to test branching logic. - Use multi-line `do...end` blocks for `before` and `after`, even when it would fit on a single line. -- Don't `describe` symbols (see [Gotchas](gotchas.md#dont-describe-symbols)). - Don't assert against the absolute value of a sequence-generated attribute (see [Gotchas](gotchas.md#dont-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Don't supply the `:each` argument to hooks since it's the default. - Prefer `not_to` to `to_not` (_this is enforced by RuboCop_). @@ -427,8 +426,6 @@ Here are some things to keep in mind regarding test performance: - `FactoryGirl.build(...)` and `.build_stubbed` are faster than `.create`. - Don't `create` an object when `build`, `build_stubbed`, `attributes_for`, `spy`, or `double` will do. Database persistence is slow! -- Use `create(:empty_project)` instead of `create(:project)` when you don't need - the underlying Git repository. Filesystem operations are slow! - Don't mark a feature as requiring JavaScript (through `@javascript` in Spinach or `:js` in RSpec) unless it's _actually_ required for the test to be valid. Headless browser testing is slow! @@ -479,6 +476,11 @@ slowest test files and try to improve them. run the suite against MySQL for tags, `master`, and any branch that includes `mysql` in the name. - On EE, the test suite always runs both PostgreSQL and MySQL. +- Rails logging to `log/test.log` is disabled by default in CI [for + performance reasons][logging]. To override this setting, provide the + `RAILS_ENABLE_TEST_LOG` environment variable. + +[logging]: https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4 ## Spinach (feature) tests diff --git a/doc/development/ux_guide/copy.md b/doc/development/ux_guide/copy.md index 794c8eb6bfe..12e8d0a31bb 100644 --- a/doc/development/ux_guide/copy.md +++ b/doc/development/ux_guide/copy.md @@ -106,6 +106,14 @@ When using verbs or adjectives: * If the context clearly refers to the object, use them alone. Example: `Edit` or `Closed`
* If the context isn’t clear enough, use them with the object. Example: `Edit issue` or `Closed issues`
+### Search
+
+| Term | Use |
+| ---- | --- |
+| Search | When using all metadata to add criteria that match/don't match. Search can also affect ordering, by ranking best results. |
+| Filter | When taking a single criteria that removes items within a list that match/don't match. Filters do not affect ordering. |
+| Sort | Orders a list based on a single or grouped criteria |
+
### Projects and Groups
| Term | Use | :no_entry_sign: Don't |
diff --git a/doc/downgrade_ee_to_ce/README.md b/doc/downgrade_ee_to_ce/README.md index fe4b6d73771..75bae324585 100644 --- a/doc/downgrade_ee_to_ce/README.md +++ b/doc/downgrade_ee_to_ce/README.md @@ -46,6 +46,19 @@ $ sudo gitlab-rails runner "Service.where(type: ['JenkinsService', 'JenkinsDepre $ bundle exec rails runner "Service.where(type: ['JenkinsService', 'JenkinsDeprecatedService']).delete_all" production ``` +### Secret variables environment scopes + +If you're using this feature and there are variables sharing the same +key, but they have different scopes in a project, then you might want to +revisit the environment scope setting for those variables. + +In CE, environment scopes are completely ignored, therefore you could +accidentally get a variable which you're not expecting for a particular +environment. Make sure that you have the right variables in this case. + +Data is completely preserved, so you could always upgrade back to EE and +restore the behavior if you leave it alone. + ## Downgrade to CE After performing the above mentioned steps, you are now ready to downgrade your diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index 12466437edc..3d893ba53dd 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -6,7 +6,7 @@ Step-by-step guides on the basics of working with Git and GitLab. - [Start using Git on the command line](start-using-git.md) - [Create and add your SSH Keys](create-your-ssh-keys.md) - [Create a project](create-project.md) -- [Create a group](create-group.md) +- [Create a group](../user/group/index.md#create-a-new-group) - [Create a branch](create-branch.md) - [Fork a project](fork-project.md) - [Add a file](add-file.md) diff --git a/doc/gitlab-basics/create-group.md b/doc/gitlab-basics/create-group.md index b4889bb8818..985a52d88f5 100644 --- a/doc/gitlab-basics/create-group.md +++ b/doc/gitlab-basics/create-group.md @@ -1,50 +1,2 @@ -# How to create a group in GitLab -Your projects in GitLab can be organized in 2 different ways: -under your own namespace for single projects, such as `your-name/project-1` or -under groups. - -If you organize your projects under a group, it works like a folder. You can -manage your group members' permissions and access to the projects. - ---- - -To create a group: - -1. Expand the left sidebar by clicking the three bars at the upper left corner - and then navigate to **Groups**. - -  - -1. Once in your groups dashboard, click on **New group**. - -  - -1. Fill out the needed information: - - 1. Set the "Group path" which will be the namespace under which your projects - will be hosted (path can contain only letters, digits, underscores, dashes - and dots; it cannot start with dashes or end in dot). - 1. The "Group name" will populate with the path. Optionally, you can change - it. This is the name that will display in the group views. - 1. Optionally, you can add a description so that others can briefly understand - what this group is about. - 1. Optionally, choose and avatar for your project. - 1. Choose the [visibility level](../public_access/public_access.md). - -1. Finally, click the **Create group** button. - -## Add a new project to a group - -There are 2 different ways to add a new project to a group: - -- Select a group and then click on the **New project** button. - -  - - You can then continue on [creating a project](create-project.md). - -- While you are [creating a project](create-project.md), select a group namespace - you've already created from the dropdown menu. - -  +This document was moved to [another location](../user/group/index.md#create-a-new-group). diff --git a/doc/gitlab-basics/img/create_new_group_sidebar.png b/doc/gitlab-basics/img/create_new_group_sidebar.png Binary files differdeleted file mode 100644 index fa88d1d51c0..00000000000 --- a/doc/gitlab-basics/img/create_new_group_sidebar.png +++ /dev/null diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md index 9a171d34671..bc75dc1447e 100644 --- a/doc/install/database_mysql.md +++ b/doc/install/database_mysql.md @@ -39,11 +39,14 @@ mysql> SET storage_engine=INNODB; # If you have MySQL < 5.7.7 and want to enable utf8mb4 character set support with your GitLab install, you must set the following NOW: mysql> SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1; +# If you use MySQL with replication, or just have MySQL configured with binary logging, you need to run the following to allow the use of `TRIGGER`: +mysql> SET GLOBAL log_bin_trust_function_creators = 1; + # Create the GitLab production database mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`; # Grant the GitLab user necessary permissions on the database -mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES ON `gitlabhq_production`.* TO 'git'@'localhost'; +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES, TRIGGER ON `gitlabhq_production`.* TO 'git'@'localhost'; # Quit the database session mysql> \q @@ -60,7 +63,15 @@ mysql> \q ``` You are done installing the database for now and can go back to the rest of the installation. -Please proceed to the rest of the installation before running through the utf8mb4 support section. +Please proceed to the rest of the installation **before** running through the steps below. + +### `log_bin_trust_function_creators` + +If you use MySQL with replication, or just have MySQL configured with binary logging, all of your MySQL servers will need to have `log_bin_trust_function_creators` enabled to allow the use of `TRIGGER` in migrations. You have already set this global variable in the steps above, but to make it persistent, add the following to your `my.cnf` file: + +``` +log_bin_trust_function_creators=1 +``` ### MySQL utf8mb4 support diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index 35220119e9b..c6b767fff02 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -2,13 +2,13 @@  ->**Important note:** -GitLab has no official images in Google Cloud Platform yet. This guide serves -as a template for when the GitLab VM will be available. - The fastest way to get started on [Google Cloud Platform (GCP)][gcp] is through the [Google Cloud Launcher][launcher] program. +GitLab's official Google Launcher apps: +1. [GitLab Community Edition](https://console.cloud.google.com/launcher/details/gitlab-public/gitlab-community-edition?project=gitlab-public) +2. [GitLab Enterprise Edition](https://console.cloud.google.com/launcher/details/gitlab-public/gitlab-enterprise-edition?project=gitlab-public) + ## Prerequisites There are only two prerequisites in order to install GitLab on GCP: diff --git a/doc/install/installation.md b/doc/install/installation.md index 84af6432889..8ded607bcab 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -64,7 +64,7 @@ up-to-date and install it. Install the required packages (needed to compile Ruby and native extensions to Ruby gems): - sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake + sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libre2-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake If you want to use Kerberos for user authentication, then install libkrb5-dev: @@ -294,9 +294,9 @@ sudo usermod -aG redis git ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 9-2-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 9-4-stable gitlab -**Note:** You can change `9-2-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `9-4-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -420,6 +420,12 @@ GitLab Shell is an SSH access and repository management software developed speci **Note:** Make sure your hostname can be resolved on the machine itself by either a proper DNS record or an additional line in /etc/hosts ("127.0.0.1 hostname"). This might be necessary for example if you set up GitLab behind a reverse proxy. If the hostname cannot be resolved, the final installation check will fail with "Check GitLab API access: FAILED. code: 401" and pushing commits will be rejected with "[remote rejected] master -> master (hook declined)". +**Note:** GitLab Shell application startup time can be greatly reduced by disabling RubyGems. This can be done in several manners: + +* Export `RUBYOPT=--disable-gems` environment variable for the processes +* Compile Ruby with `configure --disable-rubygems` to disable RubyGems by default. Not recommened for system-wide Ruby. +* Omnibus GitLab [replaces the *shebang* line of the `gitlab-shell/bin/*` scripts](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1707) + ### Install gitlab-workhorse GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/). The diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index b8bc0795f2e..515b2841d08 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -54,6 +54,13 @@ gitlabURL: http://gitlab.your-domain.com/ ## runnerRegistrationToken: "" +## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use +## Provide resource name for a Kubernetes Secret Object in the same namespace, +## this is used to populate the /etc/gitlab-runner/certs directory +## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates +## +#certsSecretName: + ## Configure the maximum number of concurrent jobs ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section ## @@ -135,6 +142,52 @@ runners: privileged: true ``` +### Providing a custom certificate for accessing GitLab + +You can provide a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) +to the GitLab Runner Helm Chart, which will be used to populate the container's +`/etc/gitlab-runner/certs` directory. + +Each key name in the Secret will be used as a filename in the directory, with the +file content being the value associated with the key. + +More information on how GitLab Runner uses these certificates can be found in the +[Runner Documentation](https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates). + + - The key/file name used should be in the format `<gitlab-hostname>.crt`. For example: `gitlab.your-domain.com.crt`. + - Any intermediate certificates need to be concatenated to your server certificate in the same file. + - The hostname used should be the one the certificate is registered for. + +The GitLab Runner Helm Chart does not create a secret for you. In order to create +the secret, you can prepare your certificate on you local machine, and then run +the `kubectl create secret` command from the directory with the certificate + +```bash +kubectl + --namespace <NAMESPACE> + create secret generic <SECRET_NAME> + --from-file=<CERTFICATE_FILENAME> +``` + +- `<NAMESPACE>` is the Kubernetes namespace where you want to install the GitLab Runner. +- `<SECRET_NAME>` is the Kubernetes Secret resource name. For example: `gitlab-domain-cert` +- `<CERTFICATE_FILENAME>` is the filename for the certificate in your current directory that will be imported into the secret + +You then need to provide the secret's name to the GitLab Runner chart. + +Add the following to your `values.yaml` + +```yaml +## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use +## Provide resource name for a Kubernetes Secret Object in the same namespace, +## this is used to populate the /etc/gitlab-runner/certs directory +## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates +## +certsSecretName: <SECRET NAME> +``` + +- `<SECRET_NAME>` is the Kubernetes Secret resource name. For example: `gitlab-domain-cert` + ## Installing GitLab Runner using the Helm Chart Once you [have configured](#configuration) GitLab Runner in your `values.yml` file, diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index 88c56a1d17c..5ea08869a9b 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -1,7 +1,7 @@ # Installing GitLab on Kubernetes > Officially supported cloud providers are Google Container Service and Azure Container Service. -> Officially supported schedulers are Kubernetes and Terraform. +> Officially supported schedulers are Kubernetes, Terraform and Tectonic. The easiest method to deploy GitLab in [Kubernetes](https://kubernetes.io/) is to take advantage of the official GitLab Helm charts. [Helm] is a package diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 5338ccb9d3a..141df55f6bc 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -69,7 +69,7 @@ so keep in mind that you need at least 4GB available before running GitLab. With less memory GitLab will give strange errors during the reconfigure run and 500 errors during usage. -- 1GB RAM + 3GB of swap is the absolute minimum but we strongly **advise against** this amount of memory. See the unicorn worker section below for more advice. +- 1GB RAM + 3GB of swap is the absolute minimum but we strongly **advise against** this amount of memory. See the [unicorn worker section below](#unicorn-workers) for more advice. - 2GB RAM + 2GB swap supports up to 100 users but it will be very slow - **4GB RAM** is the **recommended** memory size for all installations and supports up to 100 users - 8GB RAM supports up to 1,000 users @@ -86,55 +86,32 @@ if your available memory changes. Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as top or htop) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about many you need of those. -## GitLab Runner - -We strongly advise against installing GitLab Runner on the same machine you plan -to install GitLab on. Depending on how you decide to configure GitLab Runner and -what tools you use to exercise your application in the CI environment, GitLab -Runner can consume significant amount of available memory. - -Memory consumption calculations, that are available above, will not be valid if -you decide to run GitLab Runner and the GitLab Rails application on the same -machine. - -It is also not safe to install everything on a single machine, because of the -[security reasons] - especially when you plan to use shell executor with GitLab -Runner. - -We recommend using a separate machine for each GitLab Runner, if you plan to -use the CI features. - -[security reasons]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md - -## Unicorn Workers - -It's possible to increase the amount of unicorn workers and this will usually help to reduce the response time of the applications and increase the ability to handle parallel requests. +## Database -For most instances we recommend using: CPU cores + 1 = unicorn workers. -So for a machine with 2 cores, 3 unicorn workers is ideal. +The server running the database should have _at least_ 5-10 GB of storage +available, though the exact requirements depend on the size of the GitLab +installation (e.g. the number of users, projects, etc). -For all machines that have 2GB and up we recommend a minimum of three unicorn workers. -If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive swapping. +We currently support the following databases: -To change the Unicorn workers when you have the Omnibus package please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). +- PostgreSQL (highly recommended) +- MySQL/MariaDB (strongly discouraged, not all GitLab features are supported, no support for [MySQL/MariaDB GTID](https://mariadb.com/kb/en/mariadb/gtid/)) -## Database +We highly recommend the use of PostgreSQL instead of MySQL/MariaDB as not all +features of GitLab work with MySQL/MariaDB: -We currently support the following databases: +1. MySQL support for subgroups was [dropped with GitLab 9.3][post]. + See [issue #30472][30472] for more information. +1. GitLab Geo does [not support MySQL](https://docs.gitlab.com/ee/gitlab-geo/database.html#mysql-replication). +1. [Zero downtime migrations][zero] do not work with MySQL +1. We expect this list to grow over time. -- PostgreSQL -- MySQL/MariaDB +Existing users using GitLab with MySQL/MariaDB are advised to +[migrate to PostgreSQL](../update/mysql_to_postgresql.md) instead. -We _highly_ recommend the use of PostgreSQL instead of MySQL/MariaDB as not all -features of GitLab may work with MySQL/MariaDB. For example, MySQL does not have -the right features to support nested groups in an efficient manner; see -<https://gitlab.com/gitlab-org/gitlab-ce/issues/30472> for more information -about this. Existing users using GitLab with MySQL/MariaDB are advised to -migrate to PostgreSQL instead. - -The server running the database should have _at least_ 5-10 GB of storage -available, though the exact requirements depend on the size of the GitLab -installation (e.g. the number of users, projects, etc). +[30472]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30472 +[zero]: ../update/README.md#upgrading-without-downtime +[post]: https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#dropping-support-for-subgroups-in-mysql ### PostgreSQL Requirements @@ -153,6 +130,18 @@ CREATE EXTENSION pg_trgm; On some systems you may need to install an additional package (e.g. `postgresql-contrib`) for this extension to become available. +## Unicorn Workers + +It's possible to increase the amount of unicorn workers and this will usually help to reduce the response time of the applications and increase the ability to handle parallel requests. + +For most instances we recommend using: CPU cores + 1 = unicorn workers. +So for a machine with 2 cores, 3 unicorn workers is ideal. + +For all machines that have 2GB and up we recommend a minimum of three unicorn workers. +If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive swapping. + +To change the Unicorn workers when you have the Omnibus package please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). + ## Redis and Sidekiq Redis stores all user sessions and the background task queue. @@ -171,6 +160,26 @@ default settings. If you would like to disable Prometheus and it's exporters or read more information about it, check the [Prometheus documentation](../administration/monitoring/prometheus/index.md). +## GitLab Runner + +We strongly advise against installing GitLab Runner on the same machine you plan +to install GitLab on. Depending on how you decide to configure GitLab Runner and +what tools you use to exercise your application in the CI environment, GitLab +Runner can consume significant amount of available memory. + +Memory consumption calculations, that are available above, will not be valid if +you decide to run GitLab Runner and the GitLab Rails application on the same +machine. + +It is also not safe to install everything on a single machine, because of the +[security reasons] - especially when you plan to use shell executor with GitLab +Runner. + +We recommend using a separate machine for each GitLab Runner, if you plan to +use the CI features. + +[security reasons]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md + ## Supported web browsers We support the current and the previous major release of Firefox, Chrome/Chromium, Safari and Microsoft browsers (Microsoft Edge and Internet Explorer 11). diff --git a/doc/integration/README.md b/doc/integration/README.md index e56e58498a6..d70b9a7f54b 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -5,19 +5,23 @@ trackers and external authentication. See the documentation below for details on how to configure these services. -- [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Akismet](akismet.md) Configure Akismet to stop spam +- [Auth0 OmniAuth](auth0.md) Enable the Auth0 OmniAuth provider +- [Bitbucket](bitbucket.md) Import projects from Bitbucket.org and login to your GitLab instance with your +Bitbucket.org account +- [CAS](cas.md) Configure GitLab to sign in using CAS - [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. +- [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages +- [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Koding](../administration/integration/koding.md) Configure Koding to use IDE integration - [LDAP](ldap.md) Set up sign in via LDAP -- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure and Authentiq ID -- [SAML](saml.md) Configure GitLab as a SAML 2.0 Service Provider -- [CAS](cas.md) Configure GitLab to sign in using CAS - [OAuth2 provider](oauth_provider.md) OAuth2 application creation +- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure and Authentiq ID - [OpenID Connect](openid_connect_provider.md) Use GitLab as an identity provider -- [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages -- [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users -- [Akismet](akismet.md) Configure Akismet to stop spam -- [Koding](../administration/integration/koding.md) Configure Koding to use IDE integration - [PlantUML](../administration/integration/plantuml.md) Configure PlantUML to use diagrams in AsciiDoc documents. +- [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users +- [SAML](saml.md) Configure GitLab as a SAML 2.0 Service Provider +- [Trello](trello_power_up.md) Integrate Trello with GitLab > GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. diff --git a/doc/integration/chat_commands.md b/doc/integration/chat_commands.md index c878dc7e650..2856992ee25 100644 --- a/doc/integration/chat_commands.md +++ b/doc/integration/chat_commands.md @@ -1,14 +1 @@ -# Chat Commands - -Chat commands in Mattermost and Slack (also called Slack slash commands) allow you to control GitLab and view GitLab content right inside your chat client, without having to leave it. For Slack, this requires a [project service configuration](../user/project/integrations/slack_slash_commands.md). Simply type the command as a message in your chat client to activate it. - -Commands are scoped to a project, with a trigger term that is specified during configuration. (We suggest you use the project name as the trigger term for simplicty and clarity.) Taking the trigger term as `project-name`, the commands are: - - -| Command | Effect | -| ------- | ------ | -| `/project-name help` | Shows all available chat commands | -| `/project-name issue new <title> <shift+return> <description>` | Creates a new issue with title `<title>` and description `<description>` | -| `/project-name issue show <id>` | Shows the issue with id `<id>` | -| `/project-name issue search <query>` | Shows up to 5 issues matching `<query>` | -| `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment |
\ No newline at end of file +This document was moved to [integration/slash_commands.md](slash_commands.md). diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index 265c891cf83..372e1909330 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -4,10 +4,11 @@ GitLab has a great issue tracker but you can also use an external one such as Jira, Redmine, or Bugzilla. Issue trackers are configurable per GitLab project and allow you to do the following: -- the **Issues** link on the GitLab project pages takes you to the appropriate - issue index of the external tracker -- clicking **New issue** on the project dashboard creates a new issue on the - external tracker +- you can reference these external issues inside GitLab interface + (merge requests, commits, comments) and they will be automatically converted + into links + +You can have enabled both external and internal GitLab issue trackers in parallel. The **Issues** link always opens the internal issue tracker and in case the internal issue tracker is disabled the link is not visible in the menu. ## Configuration diff --git a/doc/integration/google.md b/doc/integration/google.md index 1e7ad90c5a8..d5b523e6dc0 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -72,6 +72,21 @@ To enable the Google OAuth2 OmniAuth provider you must register your application 1. Change 'YOUR_APP_SECRET' to the client secret from the Google Developer page from step 10. +1. Make sure that you configure GitLab to use an FQDN as Google will not accept raw IP addresses. + + For Omnibus packages: + + ```ruby + external_url 'https://gitlab.example.com' + ``` + + For installations from source: + + ```yaml + gitlab: + host: https://gitlab.example.com + ``` + 1. Save the configuration file. 1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you diff --git a/doc/integration/img/enable_trello_powerup.png b/doc/integration/img/enable_trello_powerup.png Binary files differnew file mode 100644 index 00000000000..65d01f1c38c --- /dev/null +++ b/doc/integration/img/enable_trello_powerup.png diff --git a/doc/integration/img/trello_card_with_gitlab_powerup.png b/doc/integration/img/trello_card_with_gitlab_powerup.png Binary files differnew file mode 100644 index 00000000000..2965dc35855 --- /dev/null +++ b/doc/integration/img/trello_card_with_gitlab_powerup.png diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index af8a1c4e5ed..8ba2e8731c8 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -63,6 +63,9 @@ it from the admin area.  +You're also able to mark an application as _trusted_ when creating it through the admin area. By doing that, +the user authorization step is automatically skipped for this application. + --- ## Authorized applications diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md new file mode 100644 index 00000000000..aa52b5415cf --- /dev/null +++ b/doc/integration/slash_commands.md @@ -0,0 +1,33 @@ +# Slash Commands + +Slash commands in Mattermost and Slack allow you to control GitLab and view GitLab content right inside your chat client, without having to leave it. For Slack, this requires a [project service configuration](../user/project/integrations/slack_slash_commands.md). Simply type the command as a message in your chat client to activate it. + +Commands are scoped to a project, with a trigger term that is specified during configuration. + +We suggest you use the project name as the trigger term for simplicity and clarity. + +Taking the trigger term as `project-name`, the commands are: + + +| Command | Effect | +| ------- | ------ | +| `/project-name help` | Shows all available slash commands | +| `/project-name issue new <title> <shift+return> <description>` | Creates a new issue with title `<title>` and description `<description>` | +| `/project-name issue show <id>` | Shows the issue with id `<id>` | +| `/project-name issue search <query>` | Shows up to 5 issues matching `<query>` | +| `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment | + +## Issue commands + +It is possible to create new issue, display issue details and search up to 5 issues. + +## Deploy command + +In order to deploy to an environment, GitLab will try to find a deployment +manual action in the pipeline. + +If there is only one action for a given environment, it is going to be triggered. +If there is more than one action defined, GitLab will try to find an action +which name equals the environment name we want to deploy to. + +Command will return an error when no matching action has been found. diff --git a/doc/integration/trello_power_up.md b/doc/integration/trello_power_up.md new file mode 100644 index 00000000000..d264486a872 --- /dev/null +++ b/doc/integration/trello_power_up.md @@ -0,0 +1,42 @@ +# Trello Power-Up + +GitLab's Trello Power-Up enables you to seamlessly attach +GitLab **merge requests** to Trello cards. + + + +## Configuring the Power-Up + +In order to get started, you will need to configure your Power-Up. + +In Trello: + +1. Go to your Trello board +1. Select `Power-Ups` to see a listing of all the available Power-Ups +1. Look for a row that says `GitLab` and select the `Enable` button +1. Select the `Settings` (gear) icon +1. In the popup menu, select `Authorize Account` + +In this popup, fill in your `API URL` and `Personal Access Token`. After that, you will be able to attach any merge request to any Trello card on your selected Trello board. + +## What is my API URL? + +Your API URL should be your GitLab instance URL with `/api/v4` appended in the end of the URL. +For example, if your GitLab instance URL is `https://gitlab.com`, your API URL would be `https://gitlab.com/api/v4`. +If your instance's URL is `https://example.com`, your API URL will be `https://example.com/api/v4`. + + + +## What is my Personal Access Token? + +Your GitLab's personal access token will enable your GitLab account to be accessed +from Trello. + +> Find it in GitLab by clicking on your avatar (upright corner), from which you access +your user **Settings** > **Access Tokens**. + +Learn more about generating a personal access token in the +[Personal Access Token Documentation][personal-access-token-documentation]. +Don't forget to check the API scope checkbox! + +[personal-access-token-documentation]: ../user/profile/personal_access_tokens.html diff --git a/doc/profile/README.md b/doc/profile/README.md index aed64ac1228..fda6d85a84c 100644 --- a/doc/profile/README.md +++ b/doc/profile/README.md @@ -1,5 +1 @@ -# Profile settings - -- [Preferences](../user/profile/preferences.md) -- [Two-factor Authentication (2FA)](../user/profile/account/two_factor_authentication.md) -- [Deleting your account](../user/profile/account/delete_account.md) +This document was moved to [user/profile/account](../user/profile/index.md). diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 855f437cd73..10f5ab3370d 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -5,9 +5,9 @@ An application data backup creates an archive file that contains the database, all repositories and all attachments. -You can only restore a backup to **exactly the same version** of GitLab on which -it was created. The best way to migrate your repositories from one server to -another is through backup restore. +You can only restore a backup to **exactly the same version and type (CE/EE)** +of GitLab on which it was created. The best way to migrate your repositories +from one server to another is through backup restore. ## Backup @@ -270,6 +270,15 @@ For installations from source: remote_directory: 'gitlab_backups' ``` +### Specifying a custom directory for backups + +If you want to group your backups you can pass a `DIRECTORY` environment variable: + +``` +sudo gitlab-rake gitlab:backup:create DIRECTORY=daily +sudo gitlab-rake gitlab:backup:create DIRECTORY=weekly +``` + ### Backup archive permissions The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) @@ -369,8 +378,8 @@ The [restore prerequisites section](#restore-prerequisites) includes crucial information. Make sure to read and test the whole restore process at least once before attempting to perform it in a production environment. -You can only restore a backup to **exactly the same version** of GitLab that -you created it on, for example 9.1.0. +You can only restore a backup to **exactly the same version and type (CE/EE)** of +GitLab that you created it on, for example CE 9.1.0. ### Restore prerequisites @@ -441,8 +450,8 @@ Deleting tmp directories...[DONE] This procedure assumes that: -- You have installed the **exact same version** of GitLab Omnibus with which the - backup was created. +- You have installed the **exact same version and type (CE/EE)** of GitLab + Omnibus with which the backup was created. - You have run `sudo gitlab-ctl reconfigure` at least once. - GitLab is running. If not, start it using `sudo gitlab-ctl start`. diff --git a/doc/university/README.md b/doc/university/README.md index 399d54bcf23..170582bcd0c 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -122,6 +122,7 @@ The curriculum is composed of GitLab videos, screencasts, presentations, project 1. [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) 1. [IBM: Continuous Delivery vs Continuous Deployment - Video](https://www.youtube.com/watch?v=igwFj8PPSnw) 1. [Amazon: Transition to Continuous Delivery - Video](https://www.youtube.com/watch?v=esEFaY0FDKc) +2. [TechBeacon: Doing continuous delivery? Focus first on reducing release cycle times](https://techbeacon.com/doing-continuous-delivery-focus-first-reducing-release-cycle-times) 1. See **[Integrations](#integrations)** for integrations with other CI services. #### 2.4. Workflow diff --git a/doc/university/glossary/README.md b/doc/university/glossary/README.md index 591d1524061..9544de41b9a 100644 --- a/doc/university/glossary/README.md +++ b/doc/university/glossary/README.md @@ -1,4 +1,3 @@ - ## What is the Glossary This contains a simplified list and definitions of some of the terms that you will encounter in your day to day activities when working with GitLab. @@ -10,7 +9,7 @@ User authentication by combination of 2 different steps during login. This allow ### Access Levels -Process of selective restriction to create, view, modify or delete a resource based on a set of assigned permissions. See [GitLab's Permission Guidelines](../../permissions/permissions.md +Process of selective restriction to create, view, modify or delete a resource based on a set of assigned permissions. See [GitLab's Permission Guidelines](../../user/permissions.md) ### Active Directory (AD) diff --git a/doc/university/process/README.md b/doc/university/process/README.md index 7ff53c2cc3f..04f2d52514f 100644 --- a/doc/university/process/README.md +++ b/doc/university/process/README.md @@ -27,4 +27,4 @@ please submit a merge request to add an upcoming class, assign to 1. Please upload any video recordings to our Youtube channel. We prefer them to be public, if needed they can be unlisted but if so they should be linked from this page. -1. Please create a merge request and assign to [SeanPackham](https://gitlab.com/u/SeanPackham). +1. Please create a merge request and assign to [Erica](https://gitlab.com/u/Erica). diff --git a/doc/update/8.9-to-8.10.md b/doc/update/8.9-to-8.10.md index d6b2f11d49a..42132f690d8 100644 --- a/doc/update/8.9-to-8.10.md +++ b/doc/update/8.9-to-8.10.md @@ -156,7 +156,7 @@ See [smtp_settings.rb.sample] as an example. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab - + For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload diff --git a/doc/update/9.1-to-9.2.md b/doc/update/9.1-to-9.2.md index 19db6e5763e..225a4dcc924 100644 --- a/doc/update/9.1-to-9.2.md +++ b/doc/update/9.1-to-9.2.md @@ -70,7 +70,27 @@ curl --location https://yarnpkg.com/install.sh | bash - More information can be found on the [yarn website](https://yarnpkg.com/en/docs/install). -### 5. Get latest code +### 5. Update Go + +NOTE: GitLab 9.2 and higher only supports Go 1.8.3 and dropped support for Go +1.5.x through 1.7.x. Be sure to upgrade your installation if necessary. + +You can check which version you are running with `go version`. + +Download and install Go: + +```bash +# Remove former Go installation folder +sudo rm -rf /usr/local/go + +curl --remote-name --progress https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz +echo '1862f4c3d3907e59b04a757cfda0ea7aa9ef39274af99a784f5be843c80c6772 go1.8.3.linux-amd64.tar.gz' | shasum -a256 -c - && \ + sudo tar -C /usr/local -xzf go1.8.3.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.8.3.linux-amd64.tar.gz +``` + +### 6. Get latest code ```bash cd /home/git/gitlab @@ -97,7 +117,7 @@ cd /home/git/gitlab sudo -u git -H git checkout 9-2-stable-ee ``` -### 6. Update gitlab-shell +### 7. Update gitlab-shell ```bash cd /home/git/gitlab-shell @@ -107,11 +127,10 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) sudo -u git -H bin/compile ``` -### 7. Update gitlab-workhorse +### 8. Update gitlab-workhorse -Install and compile gitlab-workhorse. This requires -[Go 1.5](https://golang.org/dl) which should already be on your system from -GitLab 8.1. GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/). +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). If you are not using Linux you may have to run `gmake` instead of `make` below. @@ -123,7 +142,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) sudo -u git -H make ``` -### 8. Update configuration files +### 9. Update configuration files #### New configuration options for `gitlab.yml` @@ -197,7 +216,7 @@ For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload ``` -### 9. Install libs, migrations, etc. +### 10. Install libs, migrations, etc. ```bash cd /home/git/gitlab @@ -223,7 +242,7 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production **MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). -### 10. Optional: install Gitaly +### 11. Optional: install Gitaly Gitaly is still an optional component of GitLab. If you want to save time during your 9.2 upgrade **you can skip this step**. @@ -240,14 +259,14 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) sudo -u git -H make ``` -### 11. Start application +### 12. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 12. Check application status +### 13. Check application status Check if GitLab and its environment are configured correctly: diff --git a/doc/update/9.2-to-9.3.md b/doc/update/9.2-to-9.3.md index 0c32e4db53f..910539acc70 100644 --- a/doc/update/9.2-to-9.3.md +++ b/doc/update/9.2-to-9.3.md @@ -72,8 +72,8 @@ More information can be found on the [yarn website](https://yarnpkg.com/en/docs/ ### 5. Update Go -NOTE: GitLab 9.3 and higher only supports Go 1.8.3 and dropped support for Go 1.5.x through 1.7.x. Be -sure to upgrade your installation if necessary +NOTE: GitLab 9.2 and higher only supports Go 1.8.3 and dropped support for Go +1.5.x through 1.7.x. Be sure to upgrade your installation if necessary. You can check which version you are running with `go version`. @@ -117,7 +117,7 @@ cd /home/git/gitlab sudo -u git -H git checkout 9-3-stable-ee ``` -### 5. Update gitlab-shell +### 7. Update gitlab-shell ```bash cd /home/git/gitlab-shell @@ -127,11 +127,10 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) sudo -u git -H bin/compile ``` -### 6. Update gitlab-workhorse +### 8. Update gitlab-workhorse -Install and compile gitlab-workhorse. This requires -[Go 1.5](https://golang.org/dl) which should already be on your system from -GitLab 8.1. GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/). +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). If you are not using Linux you may have to run `gmake` instead of `make` below. @@ -143,7 +142,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) sudo -u git -H make ``` -### 7. Update Gitaly +### 9. Update Gitaly If you have not yet set up Gitaly then follow [Gitaly section of the installation guide](../install/installation.md#install-gitaly). @@ -157,7 +156,29 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) sudo -u git -H make ``` -### 10. Update configuration files +### 10. Update MySQL permissions + +If you are using MySQL you need to grant the GitLab user the necessary +permissions on the database: + +```bash +mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" +``` + +If you use MySQL with replication, or just have MySQL configured with binary logging, +you will need to also run the following on all of your MySQL servers: + +```bash +mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" +``` + +You can make this setting permanent by adding it to your `my.cnf`: + +``` +log_bin_trust_function_creators=1 +``` + +### 11. Update configuration files #### New configuration options for `gitlab.yml` @@ -231,7 +252,7 @@ For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload ``` -### 11. Install libs, migrations, etc. +### 12. Install libs, migrations, etc. ```bash cd /home/git/gitlab @@ -257,14 +278,14 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production **MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). -### 12. Start application +### 13. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 13. Check application status +### 14. Check application status Check if GitLab and its environment are configured correctly: diff --git a/doc/update/9.3-to-9.4.md b/doc/update/9.3-to-9.4.md new file mode 100644 index 00000000000..9540c36e7d0 --- /dev/null +++ b/doc/update/9.3-to-9.4.md @@ -0,0 +1,339 @@ +# From 9.3 to 9.4 + +Make sure you view this update guide from the tag (version) of GitLab you would +like to install. In most cases this should be the highest numbered production +tag (without rc in it). You can select the tag in the version dropdown at the +top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + +```bash +sudo service gitlab stop +``` + +### 2. Backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Update Ruby + +NOTE: GitLab 9.0 and higher only support Ruby 2.3.x and dropped support for Ruby 2.1.x. Be +sure to upgrade your interpreter if necessary. + +You can check which version you are running with `ruby -v`. + +Download and compile Ruby: + +```bash +mkdir /tmp/ruby && cd /tmp/ruby +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.3.tar.gz +echo '1014ee699071aa2ddd501907d18cbe15399c997d ruby-2.3.3.tar.gz' | shasum -c - && tar xzf ruby-2.3.3.tar.gz +cd ruby-2.3.3 +./configure --disable-install-rdoc +make +sudo make install +``` + +Install Bundler: + +```bash +sudo gem install bundler --no-ri --no-rdoc +``` + +### 4. Update Node + +GitLab now runs [webpack](http://webpack.js.org) to compile frontend assets and +it has a minimum requirement of node v4.3.0. + +You can check which version you are running with `node -v`. If you are running +a version older than `v4.3.0` you will need to update to a newer version. You +can find instructions to install from community maintained packages or compile +from source at the nodejs.org website. + +<https://nodejs.org/en/download/> + + +Since 8.17, GitLab requires the use of yarn `>= v0.17.0` to manage +JavaScript dependencies. + +```bash +curl --location https://yarnpkg.com/install.sh | bash - +``` + +More information can be found on the [yarn website](https://yarnpkg.com/en/docs/install). + +### 5. Update Go + +NOTE: GitLab 9.2 and higher only supports Go 1.8.3 and dropped support for Go +1.5.x through 1.7.x. Be sure to upgrade your installation if necessary. + +You can check which version you are running with `go version`. + +Download and install Go: + +```bash +# Remove former Go installation folder +sudo rm -rf /usr/local/go + +curl --remote-name --progress https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz +echo '1862f4c3d3907e59b04a757cfda0ea7aa9ef39274af99a784f5be843c80c6772 go1.8.3.linux-amd64.tar.gz' | shasum -a256 -c - && \ + sudo tar -C /usr/local -xzf go1.8.3.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.8.3.linux-amd64.tar.gz +``` + +### 6. Get latest code + +```bash +cd /home/git/gitlab + +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 9-4-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 9-4-stable-ee +``` + +### 7. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell + +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) +sudo -u git -H bin/compile +``` + +### 8. Update gitlab-workhorse + +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). +If you are not using Linux you may have to run `gmake` instead of +`make` below. + +```bash +cd /home/git/gitlab-workhorse + +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) +sudo -u git -H make +``` + +### 9. Update Gitaly + +If you have not yet set up Gitaly then follow [Gitaly section of the installation +guide](../install/installation.md#install-gitaly). + +As of GitLab 9.4, Gitaly is a mandatory component of GitLab. + +#### Check Gitaly configuration + +Due to a bug in the `rake gitlab:gitaly:install` script your Gitaly +configuration file may contain syntax errors. The block name +`[[storages]]`, which may occur more than once in your `config.toml` +file, should be `[[storage]]` instead. + +```shell +sudo -u git -H sed -i.pre-9.4 's/\[\[storages\]\]/[[storage]]/' /home/git/gitaly/config.toml +``` + +#### Compile Gitaly + +```shell +cd /home/git/gitaly +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) +sudo -u git -H make +``` + +### 10. Update MySQL permissions + +If you are using MySQL you need to grant the GitLab user the necessary +permissions on the database: + +```bash +mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" +``` + +If you use MySQL with replication, or just have MySQL configured with binary logging, +you will need to also run the following on all of your MySQL servers: + +```bash +mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" +``` + +You can make this setting permanent by adding it to your `my.cnf`: + +``` +log_bin_trust_function_creators=1 +``` + +### 11. Update configuration files + +#### New configuration options for `gitlab.yml` + +There might be configuration options available for [`gitlab.yml`][yaml]. View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +cd /home/git/gitlab + +git diff origin/9-3-stable:config/gitlab.yml.example origin/9-4-stable:config/gitlab.yml.example +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +cd /home/git/gitlab + +# For HTTPS configurations +git diff origin/9-3-stable:lib/support/nginx/gitlab-ssl origin/9-4-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/9-3-stable:lib/support/nginx/gitlab origin/9-4-stable:lib/support/nginx/gitlab +``` + +If you are using Strict-Transport-Security in your installation to continue using it you must enable it in your Nginx +configuration as GitLab application no longer handles setting it. + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/9-4-stable/lib/support/init.d/gitlab.default.example#L38 + +#### SMTP configuration + +If you're installing from source and use SMTP to deliver mail, you will need to add the following line +to config/initializers/smtp_settings.rb: + +```ruby +ActionMailer::Base.delivery_method = :smtp +``` + +See [smtp_settings.rb.sample] as an example. + +[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/9-3-stable/config/initializers/smtp_settings.rb.sample#L13 + +#### Init script + +There might be new configuration options available for [`gitlab.default.example`][gl-example]. View them with the command below and apply them manually to your current `/etc/default/gitlab`: + +```sh +cd /home/git/gitlab + +git diff origin/9-3-stable:lib/support/init.d/gitlab.default.example origin/9-4-stable:lib/support/init.d/gitlab.default.example +``` + +Ensure you're still up-to-date with the latest init script changes: + +```bash +cd /home/git/gitlab + +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +For Ubuntu 16.04.1 LTS: + +```bash +sudo systemctl daemon-reload +``` + +### 12. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Update node dependencies and recompile assets +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production + +# Clean up cache +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + +### 13. Start application + +```bash +sudo service gitlab start +sudo service nginx restart +``` + +### 14. Check application status + +Check if GitLab and its environment are configured correctly: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +To make sure you didn't miss anything run a more thorough check: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (9.3) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 9.2 to 9.3](9.2-to-9.3.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +[yaml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/9-4-stable/config/gitlab.yml.example +[gl-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/9-4-stable/lib/support/init.d/gitlab.default.example diff --git a/doc/update/README.md b/doc/update/README.md index d024a809f24..22dbc7c750f 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -11,22 +11,6 @@ There are currently 3 official ways to install GitLab: Based on your installation, choose a section below that fits your needs. ---- - -<!-- START doctoc generated TOC please keep comment here to allow auto update --> -<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> -**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - -- [Omnibus Packages](#omnibus-packages) -- [Installation from source](#installation-from-source) -- [Installation using Docker](#installation-using-docker) -- [Upgrading between editions](#upgrading-between-editions) - - [Community to Enterprise Edition](#community-to-enterprise-edition) - - [Enterprise to Community Edition](#enterprise-to-community-edition) -- [Miscellaneous](#miscellaneous) - -<!-- END doctoc generated TOC please keep comment here to allow auto update --> - ## Omnibus Packages - The [Omnibus update guide](http://docs.gitlab.com/omnibus/update/README.html) diff --git a/doc/user/admin_area/monitoring/convdev.md b/doc/user/admin_area/monitoring/convdev.md new file mode 100644 index 00000000000..3d93c7557a4 --- /dev/null +++ b/doc/user/admin_area/monitoring/convdev.md @@ -0,0 +1,29 @@ +# Conversational Development Index + +> [Introduced][ce-30469] in GitLab 9.3. + +Conversational Development Index (ConvDev) gives you an overview of your entire +instance's feature usage, from idea to production. It looks at your usage in the +past 30 days, averaged over the number of active users in that time period. It also +provides a lead score per feature, which is calculated based on GitLab's analysis +of top performing instances, based on [usage ping data][ping] that GitLab has +collected. Your score is compared to the lead score, expressed as a percentage. +The overall index score is an average over all your feature scores. + + + +The page also provides helpful links to articles and GitLab docs, to help you +improve your scores. + +Your GitLab instance's usage ping must be activated in order to use this feature. +Usage ping data is aggregated on GitLab's servers for analysis. Your usage +information is **not sent** to any other GitLab instances. + +If you have just started using GitLab, it may take a few weeks for data to be +collected before this feature is available. + +This feature is accessible only to a system admin, at +**Admin area > Monitoring > ConvDev Index**. + +[ce-30469]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30469 +[ping]: ../settings/usage_statistics.md#usage-ping diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index a954840b8a6..70934f9960a 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -5,6 +5,8 @@ - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and will be deprecated in GitLab 9.1. Read more in the [old behavior](#old-behavior) section. + - [Access token](#access-token) has been deprecated in GitLab 9.4 + in favor of [IP whitelist](#ip-whitelist) GitLab provides liveness and readiness probes to indicate service health and reachability to required services. These probes report on the status of the @@ -12,97 +14,101 @@ database connection, Redis connection, and access to the filesystem. These endpoints [can be provided to schedulers like Kubernetes][kubernetes] to hold traffic until the system is ready or restart the container as needed. -## Access Token +## IP whitelist -An access token needs to be provided while accessing the probe endpoints. The current -accepted token can be found under the **Admin area ➔ Monitoring ➔ Health check** -(`admin/health_check`) page of your GitLab instance. +To access monitoring resources, the client IP needs to be included in a whitelist. - +[Read how to add IPs to a whitelist for the monitoring endpoints.][admin]. -The access token can be passed as a URL parameter: +## Using the endpoint -``` -https://gitlab.example.com/-/readiness?token=ACCESS_TOKEN -``` +With default whitelist settings, the probes can be accessed from localhost: + +- `http://localhost/-/readiness` +- `http://localhost/-/liveness` -which will then provide a report of system health in JSON format: +which will then provide a report of system health in JSON format. + +Readiness example output: ``` { - "db_check": { - "status": "ok" - }, - "redis_check": { - "status": "ok" - }, - "fs_shards_check": { - "status": "ok", - "labels": { - "shard": "default" - } - } + "queues_check" : { + "status" : "ok" + }, + "redis_check" : { + "status" : "ok" + }, + "shared_state_check" : { + "status" : "ok" + }, + "fs_shards_check" : { + "labels" : { + "shard" : "default" + }, + "status" : "ok" + }, + "db_check" : { + "status" : "ok" + }, + "cache_check" : { + "status" : "ok" + } } ``` -## Using the Endpoint - -Once you have the access token, the probes can be accessed: +Liveness example output: -- `https://gitlab.example.com/-/readiness?token=ACCESS_TOKEN` -- `https://gitlab.example.com/-/liveness?token=ACCESS_TOKEN` +``` +{ + "fs_shards_check" : { + "status" : "ok" + }, + "cache_check" : { + "status" : "ok" + }, + "db_check" : { + "status" : "ok" + }, + "redis_check" : { + "status" : "ok" + }, + "queues_check" : { + "status" : "ok" + }, + "shared_state_check" : { + "status" : "ok" + } +} +``` ## Status On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint will return a valid successful HTTP status code, and a `success` message. -## Old behavior - ->**Notes:** - - Liveness and readiness probes were [introduced][ce-10416] in GitLab 9.1. - - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and will - be deprecated in GitLab 9.1. Read more in the [old behavior](#old-behavior) - section. - -GitLab provides a health check endpoint for uptime monitoring on the `health_check` web -endpoint. The health check reports on the overall system status based on the status of -the database connection, the state of the database migrations, and the ability to write -and access the cache. This endpoint can be provided to uptime monitoring services like -[Pingdom][pingdom], [Nagios][nagios-health], and [NewRelic][newrelic-health]. +## Access token (Deprecated) -Once you have the [access token](#access-token), health information can be -retrieved as plain text, JSON, or XML using the `health_check` endpoint: +>**Note:** +Access token has been deprecated in GitLab 9.4 +in favor of [IP whitelist](#ip-whitelist) -- `https://gitlab.example.com/health_check?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check.xml?token=ACCESS_TOKEN` - -You can also ask for the status of specific services: - -- `https://gitlab.example.com/health_check/cache.json?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check/database.json?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check/migrations.json?token=ACCESS_TOKEN` - -For example, the JSON output of the following health check: +An access token needs to be provided while accessing the probe endpoints. The current +accepted token can be found under the **Admin area ➔ Monitoring ➔ Health check** +(`admin/health_check`) page of your GitLab instance. -```bash -curl --header "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json -``` + -would be like: +The access token can be passed as a URL parameter: ``` -{"healthy":true,"message":"success"} +https://gitlab.example.com/-/readiness?token=ACCESS_TOKEN ``` -On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint -will return a valid successful HTTP status code, and a `success` message. Ideally your -uptime monitoring should look for the success message. - [ce-10416]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10416 [ce-3888]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3888 [pingdom]: https://www.pingdom.com [nagios-health]: https://nagios-plugins.org/doc/man/check_http.html [newrelic-health]: https://docs.newrelic.com/docs/alerts/alert-policies/downtime-alerts/availability-monitoring [kubernetes]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ +[admin]: ../../../administration/monitoring/ip_whitelist.md diff --git a/doc/user/admin_area/monitoring/img/convdev_index.png b/doc/user/admin_area/monitoring/img/convdev_index.png Binary files differnew file mode 100644 index 00000000000..4e47ff2228d --- /dev/null +++ b/doc/user/admin_area/monitoring/img/convdev_index.png diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index f3745d0efa7..d874688cc29 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -3,7 +3,8 @@ GitLab Inc. will periodically collect information about your instance in order to perform various actions. -All statistics are opt-out, you can disable them from the admin panel. +All statistics are opt-out, you can enable/disable them from the admin panel +under **Admin area > Settings > Usage statistics**. ## Version check diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 59e343ebe51..8b1d299484c 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -10,7 +10,7 @@ You can leave a comment in the following places: - commits - commit diffs -The comment area supports [Markdown] and [slash commands]. One can edit their +The comment area supports [Markdown] and [quick actions]. One can edit their own comment at any time, and anyone with [Master access level][permissions] or higher can also edit a comment made by someone else. @@ -146,5 +146,5 @@ comments in greater detail. [discussion-view]: img/discussion_view.png [discussions-resolved]: img/discussions_resolved.png [markdown]: ../markdown.md -[slash commands]: ../project/slash_commands.md +[quick actions]: ../project/quick_actions.md [permissions]: ../permissions.md diff --git a/doc/workflow/groups/access_requests_management.png b/doc/user/group/img/access_requests_management.png Binary files differindex 36deaa89a70..36deaa89a70 100644 --- a/doc/workflow/groups/access_requests_management.png +++ b/doc/user/group/img/access_requests_management.png diff --git a/doc/user/group/img/add_new_members.png b/doc/user/group/img/add_new_members.png Binary files differnew file mode 100644 index 00000000000..53f5596de23 --- /dev/null +++ b/doc/user/group/img/add_new_members.png diff --git a/doc/gitlab-basics/img/create_new_group_info.png b/doc/user/group/img/create_new_group_info.png Binary files differindex 8d2501d9f7a..8d2501d9f7a 100644 --- a/doc/gitlab-basics/img/create_new_group_info.png +++ b/doc/user/group/img/create_new_group_info.png diff --git a/doc/gitlab-basics/img/create_new_project_from_group.png b/doc/user/group/img/create_new_project_from_group.png Binary files differindex c35234660db..c35234660db 100644 --- a/doc/gitlab-basics/img/create_new_project_from_group.png +++ b/doc/user/group/img/create_new_project_from_group.png diff --git a/doc/user/group/img/group_settings.png b/doc/user/group/img/group_settings.png Binary files differnew file mode 100644 index 00000000000..629cd0729aa --- /dev/null +++ b/doc/user/group/img/group_settings.png diff --git a/doc/user/group/img/groups.png b/doc/user/group/img/groups.png Binary files differnew file mode 100644 index 00000000000..6211f999d5e --- /dev/null +++ b/doc/user/group/img/groups.png diff --git a/doc/user/group/img/membership_lock.png b/doc/user/group/img/membership_lock.png Binary files differnew file mode 100644 index 00000000000..d31fbb43375 --- /dev/null +++ b/doc/user/group/img/membership_lock.png diff --git a/doc/workflow/groups/new_group_form.png b/doc/user/group/img/new_group_form.png Binary files differindex 91727ab5336..91727ab5336 100644 --- a/doc/workflow/groups/new_group_form.png +++ b/doc/user/group/img/new_group_form.png diff --git a/doc/user/group/img/new_group_from_groups.png b/doc/user/group/img/new_group_from_groups.png Binary files differnew file mode 100644 index 00000000000..baf34244cb2 --- /dev/null +++ b/doc/user/group/img/new_group_from_groups.png diff --git a/doc/user/group/img/new_group_from_other_pages.png b/doc/user/group/img/new_group_from_other_pages.png Binary files differnew file mode 100644 index 00000000000..014a7088af2 --- /dev/null +++ b/doc/user/group/img/new_group_from_other_pages.png diff --git a/doc/workflow/groups/request_access_button.png b/doc/user/group/img/request_access_button.png Binary files differindex f1aae6afed7..f1aae6afed7 100644 --- a/doc/workflow/groups/request_access_button.png +++ b/doc/user/group/img/request_access_button.png diff --git a/doc/gitlab-basics/img/select_group_dropdown.png b/doc/user/group/img/select_group_dropdown.png Binary files differindex 68fc950304c..68fc950304c 100644 --- a/doc/gitlab-basics/img/select_group_dropdown.png +++ b/doc/user/group/img/select_group_dropdown.png diff --git a/doc/user/group/img/share_with_group_lock.png b/doc/user/group/img/share_with_group_lock.png Binary files differnew file mode 100644 index 00000000000..8df41bf9465 --- /dev/null +++ b/doc/user/group/img/share_with_group_lock.png diff --git a/doc/user/group/img/transfer_project_to_other_group.png b/doc/user/group/img/transfer_project_to_other_group.png Binary files differnew file mode 100644 index 00000000000..042c002f83f --- /dev/null +++ b/doc/user/group/img/transfer_project_to_other_group.png diff --git a/doc/workflow/groups/withdraw_access_request_button.png b/doc/user/group/img/withdraw_access_request_button.png Binary files differindex c5d8ef6c04f..c5d8ef6c04f 100644 --- a/doc/workflow/groups/withdraw_access_request_button.png +++ b/doc/user/group/img/withdraw_access_request_button.png diff --git a/doc/user/group/index.md b/doc/user/group/index.md new file mode 100644 index 00000000000..08da721c71d --- /dev/null +++ b/doc/user/group/index.md @@ -0,0 +1,208 @@ +# Groups + +With GitLab Groups you can assemble related projects together +and grant members access to several projects at once. + +Groups can also be nested in [subgroups](subgroups/index.md). + +Find your groups by expanding the left menu and clicking **Groups**: + + + +The Groups page displays all groups you are a member of, how many projects it holds, +how many members it has, the group visibility, and, if you have enough permissions, +a link to the group settings. By clicking the last button you can leave that group. + +## Use cases + +You can create groups for numerous reasons. To name a few: + +- Organize related projects under the same [namespace](#namespaces), add members to that +group and grant access to all their projects at once +- Create a group, include members of your team, and make it easier to +`@mention` all the team at once in issues and merge requests + - Create a group for your company members, and create [subgroups](subgroups/index.md) + for each individual team. Let's say you create a group called `company-team`, and among others, + you created subgroups in this group for each individual team `backend-team`, + `frontend-team`, and `production-team`: + 1. When you start a new implementation from an issue, you add a comment: + _"`@company-team`, let's do it! `@company-team/backend-team` you're good to go!"_ + 1. When your backend team needs help from frontend, they add a comment: + _"`@company-team/frontend-team` could you help us here please?"_ + 1. When the frontend team completes their implementation, they comment: + _"`@company-team/backend-team`, it's done! Let's ship it `@company-team/production-team`!"_ + +## Namespaces + +In GitLab, a namespace is a unique name to be used as a user name, a group name, or a subgroup name. + +- `http://gitlab.example.com/username` +- `http://gitlab.example.com/groupname` +- `http://gitlab.example.com/groupname/subgroup_name` + +For example, consider a user called John: + +1. John creates his account on GitLab.com with the username `john`; +his profile will be accessed under `https://gitlab.example.com/john` +1. John creates a group for his team with the groupname `john-team`; +his group and its projects will be accessed under `https://gitlab.example.com/john-team` +1. John creates a subgroup of `john-team` with the subgroup name `marketing`; +his subgroup and its projects will be accessed under `https://gitlab.example.com/john-team/marketing` + +By doing so: + +- Any team member mentions John with `@john` +- John mentions everyone from his team with `@john-team` +- John mentions only his marketing team with `@john-team/marketing` + +## Create a new group + +You can create a group in GitLab from: + +1. The Groups page: expand the left menu, click **Groups**, and click the green button **New group**: + +  + +1. Elsewhere: expand the `plus` sign button on the top navbar and choose **New group**: + +  + +Add the following information: + + + +1. Set the **Group path** which will be the **namespace** under which your projects + will be hosted (path can contain only letters, digits, underscores, dashes + and dots; it cannot start with dashes or end in dot). +1. The **Group name** will populate with the path. Optionally, you can change + it. This is the name that will display in the group views. +1. Optionally, you can add a description so that others can briefly understand + what this group is about. +1. Optionally, choose an avatar for your project. +1. Choose the [visibility level](../../public_access/public_access.md). + +## Add users to a group + +Add members to a group by navigating to the group's dashboard, and clicking **Members**: + + + +Select the [permission level][permissions] and add the new member. You can also set the expiring +date for that user, from which they will no longer have access to your group. + +One of the benefits of putting multiple projects in one group is that you can +give a user to access to all projects in the group with one action. + +Consider we have a group with two projects: + +- On the **Group Members** page we can now add a new user to the group. +- Now because this user is a **Developer** member of the group, he automatically +gets **Developer** access to **all projects** within that group. + +If necessary, you can increase the access level of an individual user for a specific project, +by adding them again as a new member to the project with the new permission levels. + +## Request access to a group + +As a group owner you can enable or disable non members to request access to +your group. Go to the group settings and click on **Allow users to request access**. + +As a user, you can request to be a member of a group. Go to the group you'd +like to be a member of, and click the **Request Access** button on the right +side of your screen. + + + +--- + +Group owners and masters will be notified of your request and will be able to approve or +decline it on the members page. + + + +--- + +If you change your mind before your request is approved, just click the +**Withdraw Access Request** button. + + + +## Add projects to a group + +There are two different ways to add a new project to a group: + +- Select a group and then click on the **New project** button. + +  + + You can then continue on [creating a project](../../gitlab-basics/create-project.md). + +- While you are creating a project, select a group namespace + you've already created from the dropdown menu. + +  + +## Transfer an existing project into a group + +You can transfer an existing project into a group as long as you have at least **Master** [permissions][permissions] to that group +and if you are an **Owner** of the project. + + + +Find this option under your project's settings. + +GitLab administrators can use the admin interface to move any project to any namespace if needed. + +## Manage group memberships via LDAP + +In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. +See [the GitLab Enterprise Edition documentation](../../integration/ldap.md) for more information. + +## Group settings + +Once you have created a group, you can manage its settings by navigating to +the group's dashboard, and clicking **Settings**. + + + +### General settings + +Besides giving you the option to edit any settings you've previously +set when [creating the group](#create-a-new-group), you can also +access further configurations for your group. + +#### Enforce 2FA to group members + +Add a secury layer to your group by +[enforcing two-factor authentication (2FA)](../../security/two_factor_authentication.md#enforcing-2fa-for-all-users-in-a-group) +to all group members. + +#### Member Lock (EES/EEP) + +Available in [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/), +with **Member Lock** it is possible to lock membership in project to the +level of members in group. + +Learn more about [Member Lock](https://docs.gitlab.com/ee/user/group/index.html#member-lock-ees-eep). + +#### Share with group lock (EES/EEP) + +In [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/) +it is possible to prevent projects in a group from [sharing +a project with another group](../../workflow/share_projects_with_other_groups.md). +This allows for tighter control over project access. + +Learn more about [Share with group lock](https://docs.gitlab.com/ee/user/group/index.html#share-with-group-lock-ees-eep). + +### Advanced settings + +- **Projects**: view all projects within that group, add members to each project, +access each project's settings, and remove any project from the same screen. +- **Webhooks**: configure [webhooks](../project/integrations/webhooks.md) +and [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html#push-rules) to your group (Push Rules is available in [GitLab Enteprise Edition Starter][ee].) +- **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html#audit-events) +for the group (GitLab admins only, available in [GitLab Enterprise Edition Starter][ee]). +- **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group + +[permissions]: ../permissions.md#permissions +[ee]: https://about.gitlab.com/products/
\ No newline at end of file diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index c4921c74a17..5724dcfab48 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -1,6 +1,9 @@ # Subgroups -> [Introduced][ce-2772] in GitLab 9.0. +>**Notes:** +- [Introduced][ce-2772] in GitLab 9.0. +- Not available when using MySQL as external database (support removed in + GitLab 9.3 [due to performance reasons][issue]). With subgroups (aka nested groups or hierarchical groups) you can have up to 20 levels of nested groups, which among other things can help you to: @@ -173,3 +176,4 @@ Here's a list of what you can't do with subgroups: [ce-2772]: https://gitlab.com/gitlab-org/gitlab-ce/issues/2772 [permissions]: ../../permissions.md#group [reserved]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/path_regex.rb +[issue]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30472#note_27747600 diff --git a/doc/user/index.md b/doc/user/index.md new file mode 100644 index 00000000000..1281cc6e4f0 --- /dev/null +++ b/doc/user/index.md @@ -0,0 +1,180 @@ +# User documentation + +Welcome to GitLab! We're glad to have you here! + +As a GitLab user you'll have access to all the features +your [subscription](https://about.gitlab.com/products/) +includes, except [GitLab administrator](../README.md#administrator-documentation) +settings, unless you have admin privileges to install, configure, +and upgrade your GitLab instance. + +For GitLab.com, admin privileges are restricted to the GitLab team. + +If you run your own GitLab instance and are looking for the administration settings, +please refer to the [administration](../README.md#administrator-documentation) +documentation. + +## Overview + +GitLab is a fully integrated software development platform that enables you +and your team to work cohesively, faster, transparently, and effectively, +since the discussion of a new idea until taking that idea to production all +all the way through, from within the same platform. + +Please check this page for an overview on [GitLab's features](https://about.gitlab.com/features/). + +## Use cases + +GitLab is a git-based platforms that integrates a great number of essential tools for software development and deployment, and project management: + +- Code hosting in repositories with version control +- Track proposals for new implementations, bug reports, and feedback with a +fully featured [Issue Tracker](project/issues/index.md#issue-tracker) +- Organize and prioritize with [Issue Boards](project/issues/index.md#issue-boards) +- Code review in [Merge Requests](project/merge_requests/index.md) with live-preview changes per +branch with [Review Apps](../ci/review_apps/index.md) +- Build, test and deploy with built-in [Continuous Integration](../ci/README.md) +- Deploy your personal and professional static websites with [GitLab Pages](project/pages/index.md) +- Integrate with Docker with [GitLab Container Registry](project/container_registry.md) +- Track the development lifecycle with [GitLab Cycle Analytics](project/cycle_analytics.md) + +With GitLab Enterprise Edition, you can also: + +- Provide support with [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) +- Improve collaboration with +[Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/index.html#merge-request-approvals), +[Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), +and [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards) +- Create formal relashionships between issues with [Related Issues](https://docs.gitlab.com/ee/user/project/issues/related_issues.html) +- Use [Burndown Charts](https://docs.gitlab.com/ee/user/project/milestones/burndown_charts.html) to track progress during a sprint or while working on a new version of their software. +- Leverage [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) with [Advanced Global Search](https://docs.gitlab.com/ee/user/search/advanced_global_search.html) and [Advanced Syntax Search](https://docs.gitlab.com/ee/user/search/advanced_search_syntax.html) for faster, more advanced code search across your entire GitLab instance +- [Authenticate users with Kerberos](https://docs.gitlab.com/ee/integration/kerberos.html) +- [Mirror a repository](https://docs.gitlab.com/ee/workflow/repository_mirroring.html) from elsewhere on your local server. +- [Export issues as CSV](https://docs.gitlab.com/ee/user/project/issues/csv_export.html) +- View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html) +- [Lock files](https://docs.gitlab.com/ee/user/project/file_lock.html) to prevent conflicts +- View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) +- Leverage your continuous delivery method with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) + +You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, JIRA, and a lot more. + +### Articles + +For a complete workflow use case please check [GitLab Workflow, an Overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/#gitlab-workflow-use-case-scenario). + +For more use cases please check our [Technical Articles](../articles/index.md). + +## Projects + +In GitLab, you can create [projects](project/index.md) for numerous reasons, such as, host +your code, use it as an issue tracker, collaborate on code, and continuously +build, test, and deploy your app with built-in GitLab CI/CD. Or, you can do +it all at once, from one single project. + +### Repository + +Host your codebase in [GitLab repositories](project/repository/index.md) with version control +and as part of a fully integrated platform. + +### Issues + +Explore the best of GitLab [Issues](project/issues/index.md). + +### Merge Requests + +Collanorate on code, gather reviews, live preview changes per branch, and +request approvals with [Merge Requests](project/merge_requests/index.md). + +### Milestones + +Work on multiple issues and merge requests towards the same target date +with [Milestones](project/milestones/index.md). + +### GitLab Pages + +Publish your static site directly from GitLab with [GitLab Pages](project/pages/index.md). You +can [build, test, and deploy any Static Site Generator](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) with Pages. + +### Container Registry + +Build and deploy Docker images with [GitLab Container Registry](project/container_registry.md). + +## GitLab CI/CD + +Use built-in [GitLab CI/CD](../ci/README.md) to test, build, and deploy your applications +directly from GitLab. No third-party integrations needed. + +### Auto Deploy + +Deploy your application out-of-the-box with [GitLab Auto Deploy](../ci/autodeploy/index.md). + +### Review Apps + +Live-preview the changes introduced by a merge request with [Review Apps](../ci/review_apps/index.md). + +## Groups + +With GitLab [Groups](group/index.md) you can assemble related projects together +and grant members access to several projects at once. + +### Subgroups + +Groups can also be nested in [subgroups](group/subgroups/index.md). + +## Account + +There is a lot you can customize and configure +to enjoy the best of GitLab. + +[Manage your user settings](profile/index.md) to change your personal info, +personal access tokens, authorized applications, etc. + +### Authentication + +Read through the [authentication](../topics/authentication/index.md) methods available in GitLab. + +### Permissions + +Learn the different set of [permissions](permissions.md) for user type (guest, reporter, developer, master, owner). + +## Integrations + +[Integrate GitLab](../integration/README.md) with your preferred tool, +such as Trello, JIRA, etc. + +## Git and GitLab + +Learn what is [Git](../topics/git/index.md) and its best practices. + +## Discussions + +In GitLab, you can comment and mention collaborators in issues, +merge requests, code snippets, and commits. + +When performing inline reviews to implementations +to your codebase through merge requests you can +gather feedback through [resolvable discussions](discussions/index.md#resolvable-discussions). + +## Todos + +Never forget to reply to your collaborators. [GitLab Todos](../workflow/todos.md) +are a tool for working faster and more effectively with your team, +by listing all user or group mentions, as well as issues and merge +requests you're assigned to. + +## Snippets + +[Snippets](snippets.md) are code blocks that you want to store in GitLab, from which +you have quick access to. You can also gather feedback on them through +[discussions](#discussions). + +## Webhooks + +Configure [webhooks](project/integrations/webhooks.html) to listen for +specific events like pushes, issues or merge requests. GitLab will send a +POST request with data to the webhook URL. + +## API + +Automate GitLab via [API](../api/README.html). + diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 3fda47b9e34..3d47e644ad2 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -89,6 +89,7 @@ group. | Create project in group | | | | ✓ | ✓ | | Manage group members | | | | | ✓ | | Remove group | | | | | ✓ | +| Manage group labels | | ✓ | ✓ | ✓ | ✓ | ## External Users diff --git a/doc/user/profile/account/index.md b/doc/user/profile/account/index.md index 764354e1e96..06667bfc5f1 100644 --- a/doc/user/profile/account/index.md +++ b/doc/user/profile/account/index.md @@ -1,5 +1,2 @@ -# Profile settings -## Account - -Set up [two-factor authentication](two_factor_authentication.md). +This document was moved to [../index.md#profile-settings](../index.md#profile-settings). diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index fb69d934ae1..590c3f862fb 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -125,23 +125,14 @@ applications and U2F devices. ## Personal access tokens When 2FA is enabled, you can no longer use your normal account password to -authenticate with Git over HTTPS on the command line, you must use a personal -access token instead. - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Access Tokens**. -1. Choose a name and expiry date for the token. -1. Click on **Create Personal Access Token**. -1. Save the personal access token somewhere safe. - -When using Git over HTTPS on the command line, enter the personal access token -into the password field. +authenticate with Git over HTTPS on the command line or when using +[GitLab's API][api], you must use a [personal access token][pat] instead. ## Recovery options To disable two-factor authentication on your account (for example, if you have lost your code generation device) you can: + * [Use a saved recovery code](#use-a-saved-recovery-code) * [Generate new recovery codes using SSH](#generate-new-recovery-codes-using-ssh) * [Ask a GitLab administrator to disable two-factor authentication on your account](#ask-a-gitlab-administrator-to-disable-two-factor-authentication-on-your-account) @@ -154,8 +145,9 @@ codes. If you saved these codes, you can use one of them to sign in. To use a recovery code, enter your username/email and password on the GitLab sign-in page. When prompted for a two-factor code, enter the recovery code. -> **Note:** Once you use a recovery code, you cannot re-use it. You can still - use the other recovery codes you saved. +>**Note:** +Once you use a recovery code, you cannot re-use it. You can still use the other +recovery codes you saved. ### Generate new recovery codes using SSH @@ -190,11 +182,14 @@ a new set of recovery codes with SSH. two-factor code. Then, visit your Profile Settings and add a new device so you do not lose access to your account again. ``` -3. Go to the GitLab sign-in page and enter your username/email and password. When prompted for a two-factor code, enter one of the recovery codes obtained -from the command-line output. -> **Note:** After signing in, visit your **Profile Settings -> Account** immediately to set up two-factor authentication with a new - device. +3. Go to the GitLab sign-in page and enter your username/email and password. + When prompted for a two-factor code, enter one of the recovery codes obtained + from the command-line output. + +>**Note:** +After signing in, visit your **Profile settings > Account** immediately to set +up two-factor authentication with a new device. ### Ask a GitLab administrator to disable two-factor authentication on your account @@ -206,23 +201,23 @@ Sign in and re-enable two-factor authentication as soon as possible. ## Note to GitLab administrators - You need to take special care to that 2FA keeps working after -[restoring a GitLab backup](../../../raketasks/backup_restore.md). - + [restoring a GitLab backup](../../../raketasks/backup_restore.md). - To ensure 2FA authorizes correctly with TOTP server, you may want to ensure -your GitLab server's time is synchronized via a service like NTP. Otherwise, -you may have cases where authorization always fails because of time differences. - -[Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en -[FreeOTP]: https://freeotp.github.io/ -[YubiKey]: https://www.yubico.com/products/yubikey-hardware/ - + your GitLab server's time is synchronized via a service like NTP. Otherwise, + you may have cases where authorization always fails because of time differences. - The GitLab U2F implementation does _not_ work when the GitLab instance is accessed from -multiple hostnames, or FQDNs. Each U2F registration is linked to the _current hostname_ at -the time of registration, and cannot be used for other hostnames/FQDNs. + multiple hostnames, or FQDNs. Each U2F registration is linked to the _current hostname_ at + the time of registration, and cannot be used for other hostnames/FQDNs. For example, if a user is trying to access a GitLab instance from `first.host.xyz` and `second.host.xyz`: - The user logs in via `first.host.xyz` and registers their U2F key. - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication suceeds. - - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because + - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because the U2F key has only been registered on `first.host.xyz`. + +[Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en +[FreeOTP]: https://freeotp.github.io/ +[YubiKey]: https://www.yubico.com/products/yubikey-hardware/ +[api]: ../../../api/README.md +[pat]: ../personal_access_tokens.md diff --git a/doc/user/profile/img/personal_access_tokens.png b/doc/user/profile/img/personal_access_tokens.png Binary files differnew file mode 100644 index 00000000000..6aa63dbe342 --- /dev/null +++ b/doc/user/profile/img/personal_access_tokens.png diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md new file mode 100644 index 00000000000..7d25970fcb1 --- /dev/null +++ b/doc/user/profile/index.md @@ -0,0 +1,47 @@ +# User account + +When logged into their GitLab account, users can customize their +experience according to the best approach to their cases. + +## Username + +Your `username` is a unique [`namespace`](../group/index.md#namespaces) +related to your user ID. + +You can change your `username` from your +[profile settings](#profile-settings). To avoid breaking +paths when you change your `username`, we suggest you follow +[this procedure from the GitLab Team Handbook](https://about.gitlab.com/handbook/tools-and-tips/#how-to-change-your-username-at-gitlabcom). + +## User profile + +Your profile is available from the up-right corner menu bar (user's avatar) > **Profile**, +or from `https://example.gitlab.com/username`. + +On your profile page, you will see the following information: + +- Personal information +- Activity stream: see your activity streamline and the history of your contributions +- Groups: [groups](../group/index.md) you're a member of +- Contributed projects: [projects](../project/index.md) you contributed to +- Personal projects: your personal projects (respecting the project's visibility level) +- Snippets: your personal code [snippets](../snippets.md#personal-snippets) + +## Profile settings + +You can edit your account settings by navigating from the up-right corner menu bar +(user's avatar) > **Settings**, or visiting `https://example.gitlab.com/profile`. + +From there, you can: + +- Update your personal information +- Manage [private tokens](../../api/README.md#private-tokens), email tokens, [2FA](account/two_factor_authentication.md) +- Change your username and [delete your account](account/delete_account.md) +- Manage applications that can +[use GitLab as an OAuth provider](../../integration/oauth_provider.md#introduction-to-oauth) +- Manage [personal access tokens](personal_access_tokens.md) to access your account via API and authorized applications +- Add and delete emails linked to your account +- Manage [SSH keys](../../ssh/README.md#ssh) to access your account via SSH +- Manage your [preferences](preferences.md#syntax-highlighting-theme) +to customize your own GitLab experience +- Acess your audit log, a security log of important events involving your account diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md new file mode 100644 index 00000000000..f28c034e74c --- /dev/null +++ b/doc/user/profile/personal_access_tokens.md @@ -0,0 +1,60 @@ +# Personal access tokens + +> [Introduced][ce-3749] in GitLab 8.8. + +Personal access tokens are useful if you need access to the [GitLab API][api]. +Instead of using your private token which grants full access to your account, +personal access tokens could be a better fit because of their +[granular permissions](#limiting-scopes-of-a-personal-access-token). + +You can also use them to authenticate against Git over HTTP. They are the only +accepted method of authentication when you have +[Two-Factor Authentication (2FA)][2fa] enabled. + +Once you have your token, [pass it to the API][usage] using either the +`private_token` parameter or the `PRIVATE-TOKEN` header. + +The expiration of personal access tokens happens on the date you define, +at midnight UTC. + +## Creating a personal access token + +You can create as many personal access tokens as you like from your GitLab +profile. + +1. Log in to your GitLab account. +1. Go to your **Profile settings**. +1. Go to **Access tokens**. +1. Choose a name and optionally an expiry date for the token. +1. Choose the [desired scopes](#limiting-scopes-of-a-personal-access-token). +1. Click on **Create personal access token**. +1. Save the personal access token somewhere safe. Once you leave or refresh + the page, you won't be able to access it again. + + + +## Revoking a personal access token + +At any time, you can revoke any personal access token by just clicking the +respective **Revoke** button under the 'Active personal access tokens' area. + +## Limiting scopes of a personal access token + +Personal access tokens can be created with one or more scopes that allow various +actions that a given token can perform. The available scopes are depicted in +the following table. + +| Scope | Description | +| ----- | ----------- | +|`read_user` | Allows access to the read-only endpoints under `/users`. Essentially, any of the `GET` requests in the [Users API][users] are allowed ([introduced][ce-5951] in GitLab 8.15). | +| `api` | Grants complete access to the API (read/write) ([introduced][ce-5951] in GitLab 8.15). Required for accessing Git repositories over HTTP when 2FA is enabled. | +| `read_registry` | Allows to read [container registry] images if a project is private and authorization is required ([introduced][ce-11845] in GitLab 9.3). | + +[2fa]: ../account/two_factor_authentication.md +[api]: ../../api/README.md +[ce-3749]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3749 +[ce-5951]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951 +[ce-11845]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845 +[container registry]: ../project/container_registry.md +[users]: ../../api/users.md +[usage]: ../../api/README.md#basic-usage diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md index 3cbb0b5196d..629d69d8aea 100644 --- a/doc/user/project/container_registry.md +++ b/doc/user/project/container_registry.md @@ -8,8 +8,8 @@ Registry across your GitLab instance, visit the [administrator documentation](../../administration/container_registry.md). - Starting from GitLab 8.12, if you have 2FA enabled in your account, you need - to pass a personal access token instead of your password in order to login to - GitLab's Container Registry. + to pass a [personal access token][pat] instead of your password in order to + login to GitLab's Container Registry. - Multiple level image names support was added in GitLab 9.1 With the Docker Container Registry integrated into GitLab, every project can @@ -39,6 +39,14 @@ You can read more about Docker Registry at https://docs.docker.com/registry/intr ## Build and push images +>**Notes:** +- Moving or renaming existing container registry repositories is not supported +once you have pushed images because the images are signed, and the +signature includes the repository name. +- To move or rename a repository with a container registry you will have to +delete all existing images. + + If you visit the **Registry** link under your project's menu, you can see the explicit instructions to login to the Container Registry using your GitLab credentials. @@ -104,12 +112,13 @@ Make sure that your GitLab Runner is configured to allow building Docker images following the [Using Docker Build](../../ci/docker/using_docker_build.md) and [Using the GitLab Container Registry documentation](../../ci/docker/using_docker_build.md#using-the-gitlab-container-registry). -## Limitations +## Using with private projects + +> [Introduced][ce-11845] in GitLab 9.3. -In order to use a container image from your private project as an `image:` in -your `.gitlab-ci.yml`, you have to follow the -[Using a private Docker Registry][private-docker] -documentation. This workflow will be simplified in the future. +If a project is private, credentials will need to be provided for authorization. +The preferred way to do this, is by using [personal access tokens][pat]. +The minimal scope needed is `read_registry`. ## Troubleshooting the GitLab Container Registry @@ -254,5 +263,6 @@ The solution: check the [IAM permissions again](https://docs.docker.com/registry Once the right permissions were set, the error will go away. [ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040 +[ce-11845]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845 [docker-docs]: https://docs.docker.com/engine/userguide/intro/ -[private-docker]: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry +[pat]: ../profile/personal_access_tokens.md diff --git a/doc/user/project/img/issue_board.png b/doc/user/project/img/issue_board.png Binary files differindex b636cb294b8..cf7f519f783 100644 --- a/doc/user/project/img/issue_board.png +++ b/doc/user/project/img/issue_board.png diff --git a/doc/user/project/img/issue_board_add_list.png b/doc/user/project/img/issue_board_add_list.png Binary files differindex cdfc466d23f..973d9f7cde4 100644 --- a/doc/user/project/img/issue_board_add_list.png +++ b/doc/user/project/img/issue_board_add_list.png diff --git a/doc/user/project/img/issue_board_move_issue_card_list.png b/doc/user/project/img/issue_board_move_issue_card_list.png Binary files differnew file mode 100644 index 00000000000..c6b17ada40e --- /dev/null +++ b/doc/user/project/img/issue_board_move_issue_card_list.png diff --git a/doc/user/project/img/issue_board_welcome_message.png b/doc/user/project/img/issue_board_welcome_message.png Binary files differindex 5318e6ea4a9..127b9b08cc7 100644 --- a/doc/user/project/img/issue_board_welcome_message.png +++ b/doc/user/project/img/issue_board_welcome_message.png diff --git a/doc/user/project/img/issue_boards_add_issues_modal.png b/doc/user/project/img/issue_boards_add_issues_modal.png Binary files differindex 33049dce74f..bedaf724a15 100644 --- a/doc/user/project/img/issue_boards_add_issues_modal.png +++ b/doc/user/project/img/issue_boards_add_issues_modal.png diff --git a/doc/user/project/img/protected_branches_delete.png b/doc/user/project/img/protected_branches_delete.png Binary files differnew file mode 100644 index 00000000000..cfdfe6c6c29 --- /dev/null +++ b/doc/user/project/img/protected_branches_delete.png diff --git a/doc/user/project/index.md b/doc/user/project/index.md new file mode 100644 index 00000000000..91a19600951 --- /dev/null +++ b/doc/user/project/index.md @@ -0,0 +1,107 @@ +# Projects + +In GitLab, you can create projects for hosting +your codebase, use it as an issue tracker, collaborate on code, and continuously +build, test, and deploy your app with built-in GitLab CI/CD. + +Your projects can be [available](../../public_access/public_access.md) +publicly, internally, or privately, at your choice. GitLab does not limit +the number of private projects you create. + +## Project's features + +When you create a project in GitLab, you'll have access to a large number of +[features](https://about.gitlab.com/features/): + +**Issues and merge requests:** + +- [Issue tracker](issues/index.md): Discuss implementations with your team within issues + - [Issue Boards](issue_board.md): Organize and prioritize your workflow + - [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards) (**EES/EEP**): Allow your teams to create their own workflows (Issue Boards) for the same project +- [Repositories](repository/index.md): Host your code in a fully +integrated platform + - [Protected branches](protected_branches.md): Prevent collaborators + from messing with history or pushing code without review + - [Protected tags](protected_tags.md): Control over who has + permission to create tags, and prevent accidental update or deletion +- [Merge Requests](merge_requests/index.md): Apply your branching +strategy and get reviewed by your team + - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) (**EES/EEP**): Ask for approval before + implementing a change + - [Fix merge conflicts from the UI](merge_requests/resolve_conflicts.md): + Your Git diff tool right from GitLab's UI + - [Review Apps](../../ci/review_apps/index.md): Live preview the results + of the changes proposed in a merge request in a per-branch basis +- [Labels](labels.md): Organize issues and merge requests by labels +- [Time Tracking](../../workflow/time_tracking.md): Track estimate time +and time spent on + the conclusion of an issue or merge request +- [Milestones](milestones/index.md): Work towards a target date +- [Description templates](description_templates.md): Define context-specific +templates for issue and merge request description fields for your project +- [Slash commands (quick actions)](quick_actions.md): Textual shortcuts for +common actions on issues or merge requests + +**GitLab CI/CD:** + +- [GitLab CI/CD](../../ci/README.md): GitLab's built-in [Continuous Integration, Delivery, and Deployment](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) tool + - [Container Registry](container_registry.md): Build and push Docker + images out-of-the-box + - [Auto Deploy](../../ci/autodeploy/index.md): Configure GitLab CI/CD + to automatically set up your app's deployment + - [Enable and disable GitLab CI](../../ci/enable_or_disable_ci.md) + - [Pipelines](../../ci/pipelines.md#pipelines): Configure and visualize + your GitLab CI/CD pipelines from the UI + - [Scheduled Pipelines](pipelines/schedules.md): Schedule a pipeline + to start at a chosen time + - [Pipeline Graphs](../../ci/pipelines.md#pipeline-graphs): View your + entire pipeline from the UI + - [Job artifacts](pipelines/job_artifacts.md): Define, + browse, and download job artifacts + - [Pipeline settings](pipelines/settings.md): Set up Git strategy (choose the default way your repository is fetched from GitLab in a job), + timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more +- [GitLab Pages](pages/index.md): Build, test, and deploy your static +website with GitLab Pages + +**Other features:** + +- [Cycle Analytics](cycle_analytics.md): Review your development lifecycle +- [Koding integration](koding.md) (not available on GitLab.com): Integrate +with Koding to have access to a web terminal right from the GitLab UI +- [Syntax highlighting](highlighting.md): An alternative to customize +your code blocks, overriding GitLab's default choice of language + +### Project's integrations + +[Integrate your project](integrations/index.md) with Jira, Mattermost, +Kubernetes, Slack, and a lot more. + +## New project + +Learn how to [create a new project](../../gitlab-basics/create-project.md) in GitLab. + +### Fork a project + +You can [fork a project](../../gitlab-basics/fork-project.md) in order to: + +- Collaborate on code by forking a project and creating a merge request +from your fork to the upstream project +- Fork a sample project to work on the top of that + +## Import or export a project + +- Import a project from: + - [GitHub to GitLab](../../workflow/importing/import_projects_from_github.md) + - [BitBucket to GitLab](../../workflow/importing/import_projects_from_bitbucket.md) + - [Gitea to GitLab](../../workflow/importing/import_projects_from_gitea.md) + - [FogBugz to GitLab](../../workflow/importing/import_projects_from_fogbugz.md) +- [Export a project from GitLab](settings/import_export.md#exporting-a-project-and-its-data) +- [Importing and exporting projects between GitLab instances](settings/import_export.md) + +## Leave a project + +**Leave project** will only display on the project's dashboard +when a project is part of a group (under a +[group namespace](../group/index.md#namespaces)). +If you choose to leave a project you will no longer be a project +member, therefore, unable to contribute. diff --git a/doc/user/project/integrations/bugzilla.md b/doc/user/project/integrations/bugzilla.md index 0b219e84478..ba2adc1afda 100644 --- a/doc/user/project/integrations/bugzilla.md +++ b/doc/user/project/integrations/bugzilla.md @@ -16,3 +16,16 @@ Once you have configured and enabled Bugzilla: - the **Issues** link on the GitLab project pages takes you to the appropriate Bugzilla product page - clicking **New issue** on the project dashboard takes you to Bugzilla for entering a new issue + +## Referencing issues in Bugzilla + +Issues in Bugzilla can be referenced in two alternative ways: +1. `#<ID>` where `<ID>` is a number (example `#143`). +2. `<PROJECT>-<ID>` where `<PROJECT>` starts with a capital letter which is + then followed by capital letters, numbers or underscores, and `<ID>` is + a number (example `API_32-143`). + +We suggest using the longer format if you have both internal and external issue trackers enabled. If you use the shorter format and an issue with the same ID exists in the internal issue tracker the internal issue will be linked. + +Please note that `<PROJECT>` part is ignored and links always point to the +address specified in `issues_url`. diff --git a/doc/user/project/integrations/img/jira_service_page.png b/doc/user/project/integrations/img/jira_service_page.png Binary files differindex c74351b57b8..e69376f74c4 100644 --- a/doc/user/project/integrations/img/jira_service_page.png +++ b/doc/user/project/integrations/img/jira_service_page.png diff --git a/doc/user/project/integrations/img/merge_request_performance.png b/doc/user/project/integrations/img/merge_request_performance.png Binary files differindex 93b2626fed7..eba6515a6ae 100644 --- a/doc/user/project/integrations/img/merge_request_performance.png +++ b/doc/user/project/integrations/img/merge_request_performance.png diff --git a/doc/user/project/integrations/img/webhook_testing.png b/doc/user/project/integrations/img/webhook_testing.png Binary files differnew file mode 100644 index 00000000000..176dcec9d8a --- /dev/null +++ b/doc/user/project/integrations/img/webhook_testing.png diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index a048260b033..4f583879a4e 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -1,18 +1,22 @@ # GitLab JIRA integration -GitLab can be configured to interact with JIRA. Configuration happens via -user name and password. Connecting to a JIRA server via CAS is not possible. +GitLab can be configured to interact with [JIRA], a project management platform. -Each project can be configured to connect to a different JIRA instance, see the -[configuration](#configuration) section. If you have one JIRA instance you can -pre-fill the settings page with a default template. To configure the template -see the [Services Templates][services-templates] document. +Once your GitLab project is connected to JIRA, you can reference and close the +issues in JIRA directly from GitLab. -Once the project is connected to JIRA, you can reference and close the issues -in JIRA directly from GitLab. +For a use case, check out this article of [How and why to integrate GitLab with +JIRA](https://www.programmableweb.com/news/how-and-why-to-integrate-gitlab-jira/how-to/2017/04/25). ## Configuration +Each GitLab project can be configured to connect to a different JIRA instance. +If you have one JIRA instance you can pre-fill the settings page with a default +template, see the [Services Templates][services-templates] docs. + +Configuration happens via user name and password. Connecting to a JIRA server +via CAS is not possible. + In order to enable the JIRA service in GitLab, you need to first configure the project in JIRA and then enter the correct values in GitLab. @@ -76,7 +80,7 @@ We have split this stage in steps so it is easier to follow.  ---- + --- The JIRA configuration is over. Write down the new JIRA username and its password as they will be needed when configuring GitLab in the next section. @@ -99,13 +103,13 @@ in the table below. | ----- | ----------- | | `Web URL` | The base URL to the JIRA instance web interface which is being linked to this GitLab project. E.g., `https://jira.example.com`. | | `JIRA API URL` | The base URL to the JIRA instance API. Web URL value will be used if not set. E.g., `https://jira-api.example.com`. | -| `Project key` | The short identifier for your JIRA project, all uppercase, e.g., `PROJ`. | +| `Project key` | Put a JIRA project key (in uppercase), e.g. `MARS` in this field. This is only for testing the configuration settings. JIRA integration in GitLab works with _all_ JIRA projects in your JIRA instance. This field will be removed in a future release. | | `Username` | The user name created in [configuring JIRA step](#configuring-jira). | | `Password` |The password of the user created in [configuring JIRA step](#configuring-jira). | -| `JIRA issue transition` | This is the ID of a transition that moves issues to a closed state. You can find this number under JIRA workflow administration ([see screenshot](img/jira_workflow_screenshot.png)). **Closing JIRA issues via commits or Merge Requests won't work if you don't set the ID correctly.** | +| `Transition ID` | This is the ID of a transition that moves issues to a closed state. You can find this number under JIRA workflow administration ([see screenshot](img/jira_workflow_screenshot.png)). **Closing JIRA issues via commits or Merge Requests won't work if you don't set the ID correctly.** | After saving the configuration, your GitLab project will be able to interact -with the linked JIRA project. +with all JIRA projects in your JIRA instance.  @@ -213,3 +217,4 @@ your project needs to close a ticket. [services-templates]: services_templates.md [jira-repo-old-docs]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-13-stable/doc/project_services/jira.md +[jira]: https://www.atlassian.com/software/jira diff --git a/doc/user/project/integrations/kubernetes.md b/doc/user/project/integrations/kubernetes.md index 73fa83d72a8..f4000523938 100644 --- a/doc/user/project/integrations/kubernetes.md +++ b/doc/user/project/integrations/kubernetes.md @@ -19,10 +19,10 @@ of your project and select the **Kubernetes** service to configure it. The Kubernetes service takes the following arguments: -1. Kubernetes namespace 1. API URL -1. Service token 1. Custom CA bundle +1. Kubernetes namespace +1. Service token The API URL is the URL that GitLab uses to access the Kubernetes API. Kubernetes exposes several APIs - we want the "base" URL that is common to all of them, @@ -55,6 +55,7 @@ GitLab CI build environment: - `KUBE_CA_PEM_FILE` - only present if a custom CA bundle was specified. Path to a file containing PEM data. - `KUBE_CA_PEM` (deprecated)- only if a custom CA bundle was specified. Raw PEM data. +- `KUBECONFIG` - Path to a file containing kubeconfig for this deployment. CA bundle would be embedded if specified. ## Web terminals diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index d3fb5916dc6..6f15765751c 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -17,35 +17,30 @@ the settings page with a default template. To configure the template, see the Integration with Prometheus requires the following: 1. GitLab 9.0 or higher -1. The [Kubernetes integration must be enabled][kube] on your project -1. Your app must be deployed on [Kubernetes][] -1. Prometheus must be configured to collect Kubernetes metrics +1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md) 1. Each metric must be have a label to indicate the environment -1. GitLab must have network connectivity to the Prometheus sever +1. GitLab must have network connectivity to the Prometheus server -There are a few steps necessary to set up integration between Prometheus and -GitLab. +## Getting started with Prometheus monitoring -## Configuring Prometheus to collect Kubernetes metrics +Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring. -In order for Prometheus to collect Kubernetes metrics, you first must have a -Prometheus server up and running. You have two options here: +* If both GitLab and your applications are installed in the same Kubernetes cluster, you can leverage the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes). +* If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes). +* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes). -- If you installed Omnibus GitLab inside of Kubernetes, you can simply use the - [bundled version of Prometheus][promgldocs]. In that case, follow the info in the - [Omnibus GitLab section](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes) - below. -- If you are using GitLab.com or installed GitLab outside of Kubernetes, you - will likely need to run a Prometheus server within the Kubernetes cluster. - Once installed, the easiest way to monitor Kubernetes is to simply use - Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd]. - In that case, follow the instructions on - [configuring your own Prometheus server within Kubernetes](#configuring-your-own-prometheus-server-within-kubernetes). +### Getting started with Prometheus outside of Kubernetes -### Configuring Omnibus GitLab Prometheus to monitor Kubernetes +Installing and configuring Prometheus to monitor applications is fairly straight forward. + +1. [Install Prometheus](https://prometheus.io/docs/introduction/install/) +1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md) +1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config) + +### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled -version of Prometheus to collect the required metrics. +version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>). 1. Read how to configure the bundled Prometheus server in the [Administration guide][gitlab-prometheus-k8s-monitor]. @@ -74,7 +69,7 @@ kubectl apply -f path/to/prometheus.yml Once deployed, you should see the Prometheus service, deployment, and pod start within the `prometheus` namespace. The server will begin to collect metrics from each Kubernetes Node in the cluster, based on the configuration -provided in the template. +provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod). Since GitLab is not running within Kubernetes, the template provides external network access via a `NodePort` running on `30090`. This method allows access @@ -133,30 +128,6 @@ to integrate with.  -## Metrics and Labels - -GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total` -and `container_memory_usage_bytes`. These metrics are collected from the -Kubernetes pods via Prometheus, and report CPU and Memory utilization of each -container or Pod running in the cluster. - -In order to isolate and only display relevant metrics for a given environment -however, GitLab needs a method to detect which pods are associated. To do that, -GitLab will specifically request metrics that have an `environment` tag that -matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug]. - -If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of -configuring Prometheus above, the `environment` will be automatically added. - -### GitLab Prometheus queries - -The queries utilized by GitLab are shown in the following table. - -| Metric | Query | -| ------ | ----- | -| Average Memory (MB) | `(sum(container_memory_usage_bytes{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}) / count(container_memory_usage_bytes{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"})) /1024/1024` | -| Average CPU Utilization (%) | `sum(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",environment="$CI_ENVIRONMENT_SLUG"}) * 100` | - ## Monitoring CI/CD Environments Once configured, GitLab will attempt to retrieve performance metrics for any @@ -167,15 +138,16 @@ environment which has had a successful deployment. ## Determining the performance impact of a merge > [Introduced][ce-10408] in GitLab 9.2. +> GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages. +> Requires [Kubernetes](prometheus_library/kubernetes.md) metrics -Developers can view the performance impact of their changes within the merge -request workflow. When a source branch has been deployed to an environment, a -sparkline will appear showing the average memory consumption of the app. The dot +Developers can view theperformance impact of their changes within the merge +request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot indicates when the current changes were deployed, with up to 30 minutes of -performance data displayed before and after. The sparkline will be updated after +performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after each commit has been deployed. -Once merged and the target branch has been redeployed, the sparkline will switch +Once merged and the target branch has been redeployed, the metrics will switch to show the new environments this revision has been deployed to. Performance data will be available for the duration it is persisted on the diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md new file mode 100644 index 00000000000..cc5cee36d28 --- /dev/null +++ b/doc/user/project/integrations/prometheus_library/cloudwatch.md @@ -0,0 +1,25 @@ +# Monitoring AWS Resources +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4 + +GitLab has support for automatically detecting and monitoring AWS resources, starting with the [Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/). This is provided by leveraging the official [Cloudwatch exporter](https://github.com/prometheus/cloudwatch_exporter), which translates [Cloudwatch metrics](https://aws.amazon.com/cloudwatch/) into a Prometheus readable form. + +## Metrics supported + +| Name | Query | +| ---- | ----- | +| Throughput (req/sec) | sum(aws_elb_request_count_sum{%{environment_filter}}) / 60 | +| Latency (ms) | avg(aws_elb_latency_average{%{environment_filter}}) * 1000 | +| HTTP Error Rate (%) | sum(aws_elb_httpcode_backend_5_xx_sum{%{environment_filter}}) / sum(aws_elb_request_count_sum{%{environment_filter}}) | + +## Configuring Prometheus to monitor for Cloudwatch metrics + +To get started with Cloudwatch monitoring, you should install and configure the [Cloudwatch exporter](https://github.com/hnlq715/nginx-vts-exporter) which retrieves and parses the specified Cloudwatch metrics and translates them into a Prometheus monitoring endpoint. + +Right now, the only AWS resource supported is the Elastic Load Balancer, whose Cloudwatch metrics can be found [here](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-cloudwatch-metrics.html). + +A sample Cloudwatch Exporter configuration file, configured for basic AWS ELB monitoring, is [available for download](../samples/cloudwatch.yml). + +## Specifying the Environment label + +In order to isolate and only display relevant metrics for a given environment +however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments). diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md new file mode 100644 index 00000000000..f2939f047a3 --- /dev/null +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -0,0 +1,20 @@ +# Monitoring HAProxy +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4 + +GitLab has support for automatically detecting and monitoring HAProxy. This is provided by leveraging the [HAProxy Exporter](https://github.com/prometheus/haproxy_exporter), which translates HAProxy statistics into a Prometheus readable form. + +## Metrics supported + +| Name | Query | +| ---- | ----- | +| Throughput (req/sec) | sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) | +| HTTP Error Rate (%) | sum(rate(haproxy_frontend_http_requests_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) | + +## Configuring Prometheus to monitor for HAProxy metrics + +To get started with NGINX monitoring, you should install and configure the [HAProxy exporter](https://github.com/prometheus/haproxy_exporter) which parses these statistics and translates them into a Prometheus monitoring endpoint. + +## Specifying the Environment label + +In order to isolate and only display relevant metrics for a given environment +however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments). diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md new file mode 100644 index 00000000000..eb8cd821ddc --- /dev/null +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -0,0 +1,26 @@ +# Monitoring Kubernetes +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0 + +GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor). No additional services or exporters are needed. + +## Metrics supported + +| Name | Query | +| ---- | ----- | +| Average Memory Usage (MB) | (sum(container_memory_usage_bytes{container_name!="POD",%{environment_filter}}) / count(container_memory_usage_bytes{container_name!="POD",%{environment_filter}})) /1024/1024 | +| Average CPU Utilization (%) | sum(rate(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",%{environment_filter}}) * 100 | + +## Configuring Prometheus to monitor for Kubernetes node metrics + +In order for Prometheus to collect Kubernetes metrics, you first must have a +Prometheus server up and running. You have two options here: + +- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes). +- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes). + +## Specifying the Environment label + +In order to isolate and only display relevant metrics for a given environment +however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments). + +If you are using [GitLab Auto-Deploy][autodeploy] and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added. diff --git a/doc/user/project/integrations/prometheus_library/metrics.md b/doc/user/project/integrations/prometheus_library/metrics.md new file mode 100644 index 00000000000..6bdffce9c55 --- /dev/null +++ b/doc/user/project/integrations/prometheus_library/metrics.md @@ -0,0 +1,26 @@ +# Prometheus Metrics library +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0 + +GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are: +* [Kubernetes](kubernetes.md) +* [NGINX](nginx.md) +* [HAProxy](haproxy.md) +* [Amazon Cloud Watch](cloudwatch.md) + +We have tried to surface the most important metrics for each exporter, and will be continuing to add support for additional exporters in future releases. If you would like to add support for other official exporters, [contributions](#adding-to-the-library) are welcome. + +## Identifying Environments + +GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment. + +In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that, +GitLab will look for the required metrics which have a label that +matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug]. + +For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`. + +## Adding to the library + +We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `additional_metrics.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/prometheus/additional_metrics.yml) file. + +> Note: The library is only for monitoring public, common, system services which all customers can benefit from. Support for monitoring [customer proprietary metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/2273) will be added in a subsequent release. diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md new file mode 100644 index 00000000000..b3470773996 --- /dev/null +++ b/doc/user/project/integrations/prometheus_library/nginx.md @@ -0,0 +1,23 @@ +# Monitoring NGINX +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4 + +GitLab has support for automatically detecting and monitoring NGINX. This is provided by leveraging the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter), which translates [VTS statistics](https://github.com/vozlt/nginx-module-vts) into a Prometheus readable form. + +## Metrics supported + +| Name | Query | +| ---- | ----- | +| Throughput (req/sec) | sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) | +| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000 | +| HTTP Error Rate (%) | sum(rate(haproxy_frontend_http_responses_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_responses_total{%{environment_filter}}[2m])) | + +## Configuring Prometheus to monitor for NGINX metrics + +To get started with NGINX monitoring, you should first enable the [VTS statistics](https://github.com/vozlt/nginx-module-vts)) module for your NGINX server. This will capture and display statistics in an HTML readable form. Next, you should install and configure the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter) which parses these statistics and translates them into a Prometheus monitoring endpoint. + +If you are using NGINX as your Kubernetes ingress, there is [upcoming direct support](https://github.com/kubernetes/ingress/pull/423) for enabling Prometheus monitoring in the 0.9.0 release. + +## Specifying the Environment label + +In order to isolate and only display relevant metrics for a given environment +however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments). diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index 89c0312d3c2..cf92465da53 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -21,3 +21,16 @@ Once you have configured and enabled Redmine: As an example, below is a configuration for a project named gitlab-ci.  + +## Referencing issues in Redmine + +Issues in Redmine can be referenced in two alternative ways: +1. `#<ID>` where `<ID>` is a number (example `#143`) +2. `<PROJECT>-<ID>` where `<PROJECT>` starts with a capital letter which is + then followed by capital letters, numbers or underscores, and `<ID>` is + a number (example `API_32-143`). + +We suggest using the longer format if you have both internal and external issue trackers enabled. If you use the shorter format and an issue with the same ID exists in the internal issue tracker the internal issue will be linked. + +Please note that `<PROJECT>` part is ignored and links always point to the +address specified in `issues_url`. diff --git a/doc/user/project/integrations/samples/cloudwatch.yml b/doc/user/project/integrations/samples/cloudwatch.yml new file mode 100644 index 00000000000..d9b58f52c32 --- /dev/null +++ b/doc/user/project/integrations/samples/cloudwatch.yml @@ -0,0 +1,26 @@ +region: us-east-1 + metrics: + - aws_namespace: AWS/ELB + aws_metric_name: RequestCount + aws_dimensions: [AvailabilityZone, LoadBalancerName] + aws_dimension_select: + LoadBalancerName: [gitlab-ha-lb] + aws_statistics: [Sum] + - aws_namespace: AWS/ELB + aws_metric_name: Latency + aws_dimensions: [AvailabilityZone, LoadBalancerName] + aws_dimension_select: + LoadBalancerName: [gitlab-ha-lb] + aws_statistics: [Average] + - aws_namespace: AWS/ELB + aws_metric_name: HTTPCode_Backend_2XX + aws_dimensions: [AvailabilityZone, LoadBalancerName] + aws_dimension_select: + LoadBalancerName: [gitlab-ha-lb] + aws_statistics: [Sum] + - aws_namespace: AWS/ELB + aws_metric_name: HTTPCode_Backend_5XX + aws_dimensions: [AvailabilityZone, LoadBalancerName] + aws_dimension_select: + LoadBalancerName: [gitlab-ha-lb] + aws_statistics: [Sum] diff --git a/doc/user/project/integrations/samples/prometheus.yml b/doc/user/project/integrations/samples/prometheus.yml index 01bbcaffe1e..30b59e172a1 100644 --- a/doc/user/project/integrations/samples/prometheus.yml +++ b/doc/user/project/integrations/samples/prometheus.yml @@ -24,6 +24,44 @@ data: target_label: environment regex: (.+)-.+-.+ replacement: $1 + - job_name: kubernetes-pods + tls_config: + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + insecure_skip_verify: true + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token" + kubernetes_sd_configs: + - role: pod + api_server: https://kubernetes.default.svc:443 + tls_config: + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token" + relabel_configs: + - source_labels: + - __meta_kubernetes_pod_annotation_prometheus_io_scrape + action: keep + regex: 'true' + - source_labels: + - __meta_kubernetes_pod_annotation_prometheus_io_path + action: replace + target_label: __metrics_path__ + regex: "(.+)" + - source_labels: + - __address__ + - __meta_kubernetes_pod_annotation_prometheus_io_port + action: replace + regex: "([^:]+)(?::[0-9]+)?;([0-9]+)" + replacement: "$1:$2" + target_label: __address__ + - action: labelmap + regex: __meta_kubernetes_pod_label_(.+) + - source_labels: + - __meta_kubernetes_namespace + action: replace + target_label: kubernetes_namespace + - source_labels: + - __meta_kubernetes_pod_name + action: replace + target_label: kubernetes_pod_name --- apiVersion: v1 kind: Service diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index 54e0ee611cb..c267da69bb3 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -2,7 +2,7 @@ > Introduced in GitLab 8.15 -Slack slash commands (also known as chat commmands) allow you to control GitLab and view content right inside Slack, without having to leave it. This requires configurations in both Slack and GitLab. +Slack slash commands allow you to control GitLab and view content right inside Slack, without having to leave it. This requires configurations in both Slack and GitLab. > Note: GitLab can also send events (e.g. issue created) to Slack as notifications. This is the separately configured [Slack Notifications Service](slack.md). @@ -20,4 +20,4 @@ Slack slash commands (also known as chat commmands) allow you to control GitLab ## Usage -You can now use the [Slack slash commands](../../../integration/chat_commands.md).
\ No newline at end of file +You can now use the [Slack slash commands](../../../integration/slash_commands.md). diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 0517ed3ec18..c03a2df9a72 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -736,7 +736,7 @@ X-Gitlab-Event: Merge Request Hook ### Wiki Page events -Triggered when a wiki page is created, edited or deleted. +Triggered when a wiki page is created, updated or deleted. **Request Header**: @@ -1014,6 +1014,13 @@ X-Gitlab-Event: Build Hook } ``` +## Testing webhooks + +You can trigger the webhook manually. Sample data from the project will be used.Sample data will take from the project. +> For example: for triggering `Push Events` your project should have at least one commit. + + + ## Troubleshoot webhooks Gitlab stores each perform of the webhook. @@ -1056,7 +1063,7 @@ Pick an unused port (e.g. 8000) and start the script: `ruby print_http_body.rb 8000`. Then add your server as a webhook receiver in GitLab as `http://my.host:8000/`. -When you press 'Test Hook' in GitLab, you should see something like this in the +When you press 'Test' in GitLab, you should see something like this in the console: ``` diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 5aa8337b75d..e2cc67726e0 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -1,8 +1,7 @@ -# Issue board +# Issue Board ->**Notes:** -- [Introduced][ce-5554] in GitLab 8.11. -- The Backlog column was replaced by the **Add issues** button in GitLab 8.17. +>**Note:** +[Introduced][ce-5554] in [GitLab 8.11](https://about.gitlab.com/2016/08/22/gitlab-8-11-released/#issue-board). The GitLab Issue Board is a software project management tool used to plan, organize, and visualize a workflow for a feature or product release. @@ -15,12 +14,65 @@ Other interesting links: ## Overview -The Issue Board builds on GitLab's existing issue tracking functionality and +The Issue Board builds on GitLab's existing +[issue tracking functionality](issues/index.md#issue-tracker) and leverages the power of [labels] by utilizing them as lists of the scrum board. -With the Issue Board you can have a different view of your issues while also +With the Issue Board you can have a different view of your issues while maintaining the same filtering and sorting abilities you see across the -issue tracker. +issue tracker. An Issue Board is based on its project's label structure, therefore, it +applies the same descriptive labels to indicate placement on the board, keeping +consistency throughout the entire development lifecycle. + +An Issue Board shows you what issues your team is working on, who is assigned to each, +and where in the workflow those issues are. + +You create issues, host code, perform reviews, build, test, +and deploy from one single platform. Issue Boards help you to visualize +and manage the entire process _in_ GitLab. + +With [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards), available +only in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/), +you go even further, as you can not only keep yourself and your project +organized from a broader perspective with one Issue Board per project, +but also allow your team members to organize their own workflow by creating +multiple Issue Boards within the same project. + +## Use cases + +GitLab Workflow allows you to discuss proposals in issues, categorize them +with labels, and from there organize and prioritize them with Issue Boards. + +For example, let's consider this simplified development workflow: + +1. You have a repository hosting your app's codebase +and your team actively contributing to code +1. Your **backend** team starts working a new +implementation, gathers feedback and approval, and pass it over to **frontend** +1. When frontend is complete, the new feature is deployed to **staging** to be tested +1. When successful, it is deployed to **production** + +If we have the labels "**backend**", "**frontend**", "**staging**", and +"**production**", and an Issue Board with a list for each, we can: + +- Visualize the entire flow of implementations since the +beginning of the development lifecycle until deployed to production +- Prioritize the issues in a list by moving them vertically +- Move issues between lists to organize them according to the labels you've set +- Add multiple issues to lists in the board by selecting one or more existing issues + + + +> **Notes:** +> +>- For a broader use case, please check the blog post +[GitLab Workflow, an Overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/#gitlab-workflow-use-case-scenario). +> +>- For a real use case, please check why +[Codepen decided to adopt Issue Boards](https://about.gitlab.com/2017/01/27/codepen-welcome-to-gitlab/#project-management-everything-in-one-place) +to improve their workflow with [multiple boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards). + +## Issue Board terminology Below is a table of the definitions used for GitLab's Issue Board. @@ -31,10 +83,11 @@ Below is a table of the definitions used for GitLab's Issue Board. | **Card** | Every card represents an issue and it is shown under the list for which it has a label. The information you can see on a card consists of the issue number, the issue title, the assignee and the labels associated with it. You can drag cards around from one list to another. You can re-order cards within a list. | There are two types of lists, the ones you create based on your labels, and -one default: +two defaults: - Label list: a list based on a label. It shows all opened issues with that label. -- **Done** (default): shows all closed issues. Always appears on the very right. +- **Backlog** (default): shows all open issues that does not belong to one of lists. Always appears on the very left. +- **Closed** (default): shows all closed issues. Always appears on the very right.  @@ -56,7 +109,7 @@ In short, here's a list of actions you can take in an Issue Board: If you are not able to perform one or more of the things above, make sure you have the right [permissions](#permissions). -## First time using the issue board +## First time using the Issue Board The first time you navigate to your Issue Board, you will be presented with a default list (**Done**) and a welcoming message that gives @@ -97,7 +150,7 @@ list view that is removed. You can always add it back later if you need. ## Adding issues to a list You can add issues to a list by clicking the **Add issues** button that is -present in the upper right corner of the issue board. This will open up a modal +present in the upper right corner of the Issue Board. This will open up a modal window where you can see all the issues that do not belong to any list. Select one or more issues by clicking on the cards and then click **Add issues** diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index fe87e6f9495..1f78849a92c 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -1,17 +1,17 @@ -# Issues documentation +# Issues The GitLab Issue Tracker is an advanced and complete tool for tracking the evolution of a new idea or the process of solving a problem. It allows you, your team, and your collaborators to share -and discuss proposals, before and while implementing them. +and discuss proposals before and while implementing them. Issues and the GitLab Issue Tracker are available in all [GitLab Products](https://about.gitlab.com/products/) as part of the [GitLab Workflow](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/). -## Use-Cases +## Use cases Issues can have endless applications. Just to exemplify, these are some cases for which creating issues are most used: @@ -23,7 +23,28 @@ some cases for which creating issues are most used: - Obtaining support - Elaborating new code implementations -See also the blog post [Always start a discussion with an issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/). +See also the blog post "[Always start a discussion with an issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/)". + +### Keep private things private + +For instance, let's assume you have a public project but want to start a discussion on something +you don't want to be public. With [Confidential Issues](#confidential-issues), +you can discuss private matters among the project members, and still keep +your project public, open to collaboration. + +### Streamline collaboration + +With [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), +available in [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/) +you can streamline collaboration and allow shared responsibilities to be clearly displayed. +All assignees are shown across your workflows and receive notifications (as they +would as single assignees), simplifying communication and ownership. + +### Consistent collaboration + +Create [issue templates](#issue-templates) to make collaboration consistent and +containing all information you need. For example, you can create a template +for feature proposals and another one for bug reports. ## Issue Tracker @@ -96,8 +117,8 @@ Find GitLab Issue Boards by navigating to your **Project's Dashboard** > **Issue Read through the documentation for [Issue Boards](../issue_board.md) to find out more about this feature. -[Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards) -are available only in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/). +With [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/), you can also +create various boards per project with [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards). ### Issue's API diff --git a/doc/user/project/issues/issues_functionalities.md b/doc/user/project/issues/issues_functionalities.md index ba843201e1a..074b2c19c43 100644 --- a/doc/user/project/issues/issues_functionalities.md +++ b/doc/user/project/issues/issues_functionalities.md @@ -43,7 +43,7 @@ assigned to them if they created the issue themselves. ##### 3.1. Multiple Assignees (EES/EEP) -Issue Weights are only available in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/). +Multiple Assignees are only available in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/). Often multiple people likely work on the same issue together, which can especially be difficult to track in large teams @@ -52,25 +52,21 @@ where there is shared ownership of an issue. In GitLab Enterprise Edition, you can also select multiple assignees to an issue. -> **Note:** -Multiple Assignees was [introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1904) -in [GitLab Enterprise Edition 9.2](https://about.gitlab.com/2017/05/22/gitlab-9-2-released/#multiple-assignees-for-issues). +Learn more on the [Multiple Assignees documentation](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html). #### 4. Milestone - Select a [milestone](../milestones/index.md) to attribute that issue to. -#### 5. Time Tracking (EES/EEP) - -This feature is available only in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/). +#### 5. Time Tracking - Estimate time: add an estimate time in which the issue will be implemented - Spend: add the time spent on the implementation of that issue > **Note:** -both estimate and spend times are set via [GitLab Slash Commands](../slash_commands.md). +Both estimate and spend times are set via [GitLab Quick Actions](../quick_actions.md). -Learn more on the [Time Tracking documentation](https://docs.gitlab.com/ee/workflow/time_tracking.html). +Learn more on the [Time Tracking documentation](../../../workflow/time_tracking.md). #### 6. Due date @@ -147,7 +143,7 @@ or in the issue thread. #### 15. Award emoji -- Award an emoji to that issue. +- Award an emoji to that issue. > **Tip:** Posting "+1" as comments in threads spam all diff --git a/doc/user/project/koding.md b/doc/user/project/koding.md index c56a1efe3c2..455e2ee47b4 100644 --- a/doc/user/project/koding.md +++ b/doc/user/project/koding.md @@ -1,4 +1,4 @@ -# Koding & GitLab +# Koding integration > [Introduced][ce-5909] in GitLab 8.11. diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 954454f7e7a..9bdf2a998d3 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -3,6 +3,59 @@ Merge requests allow you to exchange changes you made to source code and collaborate with other people on the same project. +## Overview + +A Merge Request (**MR**) is the basis of GitLab as a code collaboration +and version control platform. +Is it simple as the name implies: a _request_ to _merge_ one branch into another. + +With GitLab merge requests, you can: + +- Compare the changes between two [branches](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell#_git_branching) +- [Review and discuss](../../discussions/index.md#discussions) the proposed modifications inline +- Live preview the changes when [Review Apps](../../../ci/review_apps/index.md) is configured for your project +- Build, test, and deploy your code in a per-branch basis with built-in [GitLab CI/CD](../../../ci/README.md) +- Prevent the merge request from being merged before it's ready with [WIP MRs](#work-in-progress-merge-requests) +- View the deployment process through [Pipeline Graphs](../../../ci/pipelines.md#pipeline-graphs) +- [Automatically close the issue(s)](../../project/issues/closing_issues.md#via-merge-request) that originated the implementation proposed in the merge request +- Assign it to any registered user, and change the assignee how many times you need +- Assign a [milestone](../../project/milestones/index.md) and track the development of a broader implementation +- Organize your issues and merge requests consistently throughout the project with [labels](../../project/labels.md) +- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.html#time-tracking) +- [Resolve merge conflicts from the UI](#resolve-conflicts) + +With **[GitLab Enterprise Edition][ee]**, you can also: + +- View the deployment process across projects with [Multi-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html#multi-project-pipeline-graphs) (available only in GitLab Enterprise Edition Premium) +- Request [approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers (available in GitLab Enterprise Edition Starter) +- Enable [fast-forward merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/fast_forward_merge.html) (available in GitLab Enterprise Edition Starter) +- [Squash and merge](https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html) for a cleaner commit history (available in GitLab Enterprise Edition Starter) +- Enable [semi-linear history merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/index.html#semi-linear-history-merge-requests) as another security layer to guarantee the pipeline is passing in the target branch (available in GitLab Enterprise Edition Starter) +- Analise the impact of your changes with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html) (available in GitLab Enterprise Edition Starter) + +## Use cases + +A. Consider you are a software developer working in a team: + +1. You checkout a new branch, and submit your changes through a merge request +1. You gather feedback from your team +1. You work on the implementation optimizing code with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html) (available in GitLab Enterprise Edition Starter) +1. You build and test your changes with GitLab CI/CD +1. You request the approval from your manager +1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Enterprise Edition Starter) +1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#manual-actions) for GitLab CI/CD +1. Your implementations were successfully shipped to your customer + +B. Consider you're a web developer writing a webpage for your company's: + +1. You checkout a new branch, and submit a new page through a merge request +1. You gather feedback from your reviewers +1. Your changes are previewed with [Review Apps](../../../ci/review_apps/index.md) +1. You request your web designers for their implementation +1. You request the [approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your manager (available in GitLab Enterprise Edition Starter) +1. Once approved, your merge request is [squashed and merged](https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html), and [deployed to staging with GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) (Squash and Merge is available in GitLab Enterprise Edition Starter) +1. Your production team [cherry picks](#cherry-pick-changes) the merge commit into production + ## Authorization for merge requests There are two main ways to have a merge request flow with GitLab: @@ -79,6 +132,16 @@ specific commit page. You can append `?w=1` while on the diffs page of a merge request to ignore any whitespace changes. +## Live preview with Review Apps + +If you configured [Review Apps](https://about.gitlab.com/features/review-apps/) for your project, +you can preview the changes submitted to a feature-branch through a merge request +in a per-branch basis. No need to checkout the branch, install and preview locally; +all your changes will be available to preview by anyone with the Review Apps link. + +[Read more about Review Apps.](../../../ci/review_apps/index.md) + + ## Tips Here are some tips that will help you be more efficient with merge requests in @@ -167,3 +230,4 @@ git checkout origin/merge-requests/1 ``` [protected branches]: ../protected_branches.md +[ee]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition" diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 99233ed5ae2..23ffde4e8bd 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -21,14 +21,15 @@ Once you fill in all the details, hit the **Create milestone** button. >**Note:** You need [Master permissions](../../permissions.md) in order to create a milestone. -You can create a milestone for several projects in the same group simultaneously. -On the group's **Issues ➔ Milestones** page, you will be able to see the status -of that milestone across all of the selected projects. To create a new milestone -for selected projects in the group, click the **New milestone** button. The -form is the same as when creating a milestone for a specific project with the -addition of the selection of the projects you want to inherit this milestone. +You can create a milestone for a group that will be shared across group projects. +On the group's **Issues ➔ Milestones** page, you will be able to see the state +of that milestone and the issues/merge requests count that it shares across the group projects. To create a new milestone click the **New milestone** button. The form is the same as when creating a milestone for a specific project which you can find in the previous item. - +In addition to that you will be able to filter issues or merge requests by group milestones in all projects that belongs to the milestone group. + +## Milestone promotion + +You will be able to promote a project milestone to a group milestone [in the future](https://gitlab.com/gitlab-org/gitlab-ce/issues/35833). ## Special milestone filters @@ -52,3 +53,7 @@ is calculated as; closed and merged merge requests plus all closed issues divide total merge requests and issues.  + +## Quick actions + +[Quick actions](../quick_actions.md) are available for assigning and removing project milestones only. [In the future](https://gitlab.com/gitlab-org/gitlab-ce/issues/34778), this will also apply to group milestones. diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index e9512497d6c..271adee7da1 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -212,9 +212,9 @@ Container Registries for private projects. access token created explicitly for this purpose). This issue is resolved with latest changes in GitLab Runner 1.8 which receives GitLab credentials with build data. -- Starting with GitLab 8.12, if you have 2FA enabled in your account, you need - to pass a personal access token instead of your password in order to login to - GitLab's Container Registry. +- Starting from GitLab 8.12, if you have [2FA] enabled in your account, you need + to pass a [personal access token][pat] instead of your password in order to + login to GitLab's Container Registry. Your jobs can access all container images that you would normally have access to. The only implication is that you can push to the Container Registry of the @@ -239,3 +239,5 @@ test: [update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update [workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse [jobenv]: ../../ci/variables/README.md#predefined-variables-environment-variables +[2fa]: ../profile/account/two_factor_authentication.md +[pat]: ../profile/personal_access_tokens.md diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 2f104c7becc..46fa4378fe7 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -41,7 +41,7 @@ server up and running for your GitLab instance. Before we begin, let's understand a few concepts first. -### Static sites +## Static sites GitLab Pages only supports static websites, meaning, your output files must be HTML, CSS, and JavaScript only. @@ -51,14 +51,14 @@ CSS, and JS, or use a [Static Site Generator (SSG)](https://www.staticgen.com/) to simplify your code and build the static site for you, which is highly recommendable and much faster than hardcoding. -#### Further Reading +### Further reading - Read through this technical overview on [Static versus Dynamic Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) - Understand [how modern Static Site Generators work](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) and what you can add to your static site - You can use [any SSG with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) - Fork an [example project](https://gitlab.com/pages) to build your website based upon -### GitLab Pages domain +## GitLab Pages domain If you set up a GitLab Pages project on GitLab.com, it will automatically be accessible under a @@ -73,9 +73,9 @@ Pages wildcard domain. This guide is valid for any GitLab instance, you just need to replace Pages wildcard domain on GitLab.com (`*.gitlab.io`) with your own. -#### Practical examples +### Practical examples -**Project Websites:** +#### Project Websites - You created a project called `blog` under your username `john`, therefore your project URL is `https://gitlab.com/john/blog/`. @@ -87,16 +87,21 @@ URL is `https://gitlab.com/websites/blog/`. Once you enable GitLab Pages for this project, the site will live under `https://websites.gitlab.io/blog/`. -**User and Group Websites:** +#### User and Group Websites - Under your username, `john`, you created a project called `john.gitlab.io`. Your project URL will be `https://gitlab.com/john/john.gitlab.io`. Once you enable GitLab Pages for your project, your website will be published under `https://john.gitlab.io`. - Under your group `websites`, you created a project called -`websites.gitlab.io`. your project's URL will be `https://gitlab.com/websites/websites.gitlab.io`. Once you enable GitLab Pages for your project, +`websites.gitlab.io`. your project's URL will be `https://gitlab.com/websites/websites.gitlab.io`. +Once you enable GitLab Pages for your project, your website will be published under `https://websites.gitlab.io`. +>**Note:** +GitLab Pages [does **not** support subgroups](../../group/subgroups/index.md#limitations). +You can only create the highest level group website. + **General example:** - On GitLab.com, a project site will always be available under diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index deaceabb7c5..9ecf7a3a8e7 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -398,6 +398,9 @@ don't redirect HTTP to HTTPS. [rfc]: https://tools.ietf.org/html/rfc2818#section-3.1 "HTTP Over TLS RFC" +GitLab Pages [does **not** support subgroups](../../group/subgroups/index.md#limitations). +You can only create the highest level group website. + ## Redirects in GitLab Pages Since you cannot use any custom server configuration files, like `.htaccess` or diff --git a/doc/user/project/pipelines/img/pipeline_schedule_variables.png b/doc/user/project/pipelines/img/pipeline_schedule_variables.png Binary files differnew file mode 100644 index 00000000000..47a0c6f3697 --- /dev/null +++ b/doc/user/project/pipelines/img/pipeline_schedule_variables.png diff --git a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png Binary files differindex ea5394fa8a6..5a0e5965992 100644 --- a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png +++ b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png diff --git a/doc/user/project/pipelines/job_artifacts.md b/doc/user/project/pipelines/job_artifacts.md index 151ee4728ad..e853bfff444 100644 --- a/doc/user/project/pipelines/job_artifacts.md +++ b/doc/user/project/pipelines/job_artifacts.md @@ -12,7 +12,7 @@ to GitLab using GitLab Runner version 1.0 and up. It will not be possible to browse old artifacts already uploaded to GitLab. >- This is the user documentation. For the administration guide see - [administration/job_artifacts.md](../../../administration/job_artifacts.md). + [administration/job_artifacts](../../../administration/job_artifacts.md). Artifacts is a list of files and directories which are attached to a job after it completes successfully. This feature is enabled by default in all @@ -29,25 +29,31 @@ pdf: artifacts: paths: - mycv.pdf + expire_in: 1 week ``` A job named `pdf` calls the `xelatex` command in order to build a pdf file from the latex source file `mycv.tex`. We then define the `artifacts` paths which in turn are defined with the `paths` keyword. All paths to files and directories -are relative to the repository that was cloned during the build. +are relative to the repository that was cloned during the build. These uploaded +artifacts will be kept in GitLab for 1 week as defined by the `expire_in` +definition. You have the option to keep the artifacts from expiring via the +[web interface](#browsing-job-artifacts). If you don't define an expiry date, +the artifacts will be kept forever. -For more examples on artifacts, follow the artifacts reference in -[`.gitlab-ci.yml` documentation](../../../ci/yaml/README.md#artifacts). +For more examples on artifacts, follow the [artifacts reference in +`.gitlab-ci.yml`](../../../ci/yaml/README.md#artifacts). ## Browsing job artifacts >**Note:** -With GitLab 9.2, PDFs, images, videos and other formats can be previewed directly -in the job artifacts browser without the need to download them. +With GitLab 9.2, PDFs, images, videos and other formats can be previewed +directly in the job artifacts browser without the need to download them. -After a job finishes, if you visit the job's specific page, you can see -that there are two buttons. One is for downloading the artifacts archive and -the other for browsing its contents. +After a job finishes, if you visit the job's specific page, there are three +buttons. You can download the artifacts archive or browse its contents, whereas +the **Keep** button appears only if you have set an [expiry date] to the +artifacts in case you changed your mind and want to keep them.  @@ -103,7 +109,7 @@ https://example.com/<namespace>/<project>/builds/artifacts/<ref>/download?job=<j To download a single file from the artifacts use the following URL: ``` -https://example.com/<namespace>/<project>/builds/artifacts/<ref>/file/<path_to_file>?job=<job_name> +https://example.com/<namespace>/<project>/builds/artifacts/<ref>/raw/<path_to_file>?job=<job_name> ``` For example, to download the latest artifacts of the job named `coverage` of @@ -118,7 +124,7 @@ To download the file `coverage/index.html` from the same artifacts use the following URL: ``` -https://gitlab.com/gitlab-org/gitlab-ce/builds/artifacts/master/file/coverage/index.html?job=coverage +https://gitlab.com/gitlab-org/gitlab-ce/builds/artifacts/master/raw/coverage/index.html?job=coverage ``` There is also a URL to browse the latest job artifacts: @@ -145,3 +151,5 @@ information in the UI.  + +[expiry date]: ../../../ci/yaml/README.md#artifacts-expire_in diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index d19d184f9b0..9ad15a12c3c 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -31,6 +31,35 @@ is installed on.  +### Making use of scheduled pipeline variables + +> [Introduced][ce-12328] in GitLab 9.4. + +You can pass any number of arbitrary variables and they will be available in +GitLab CI so that they can be used in your `.gitlab-ci.yml` file. + + + +## Using only and except + +To configure that a job can be executed only when the pipeline has been +scheduled (or the opposite), you can use +[only and except](../../../ci/yaml/README.md#only-and-except) configuration keywords. + +``` +job:on-schedule: + only: + - schedules + script: + - make world + +job: + except: + - schedules + script: + - make build +``` + ## Taking ownership Pipelines are executed as a user, who owns a schedule. This influences what @@ -42,9 +71,10 @@ The next time a pipeline is scheduled, your credentials will be used. >**Note:** When the owner of the schedule doesn't have the ability to create pipelines -anymore, due to e.g., being blocked or removed from the project, the schedule -is deactivated. Another user can take ownership and activate it, so the -schedule can be run again. +anymore, due to e.g., being blocked or removed from the project, or lacking +the permission to run on protected branches or tags. When this happened, the +schedule is deactivated. Another user can take ownership and activate it, so +the schedule can be run again. ## Advanced admin configuration @@ -59,4 +89,5 @@ don't have admin access to the server, ask your administrator. [ce-10533]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10533 [ce-10853]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10853 +[ce-12328]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12328 [settings]: https://about.gitlab.com/gitlab-com/settings/#cron-jobs diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 1d2eba4f74b..3ff5a08d72c 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -1,7 +1,7 @@ # Pipelines settings To reach the pipelines settings navigate to your project's -**Settings ➔ CI/CD Pipelines**. +**Settings ➔ Pipelines**. The following settings can be configured per project. @@ -27,6 +27,22 @@ The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs' running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed. +## Custom CI config path + +> - [Introduced][ce-12509] in GitLab 9.4. + +By default we look for the `.gitlab-ci.yml` file in the project's root +directory. If you require a different location **within** the repository, +you can set a custom filepath that will be used to lookup the config file, +this filepath should be **relative** to the root. + +Here are some valid examples: + +> * .gitlab-ci.yml +> * .my-custom-file.yml +> * my/path/.gitlab-ci.yml +> * my/path/.my-custom-file.yml + ## Test coverage parsing If you use test coverage in your code, GitLab can capture its output in the @@ -59,8 +75,8 @@ pipelines** checkbox and save the changes. > [Introduced][ce-9362] in GitLab 9.1. -If you want to auto-cancel all pending non-HEAD pipelines on branch, when -new pipeline will be created (after your git push or manually from UI), +If you want to auto-cancel all pending non-HEAD pipelines on branch, when +new pipeline will be created (after your git push or manually from UI), check **Auto-cancel pending pipelines** checkbox and save the changes. ## Badges @@ -115,3 +131,4 @@ into your `README.md`: [var]: ../../../ci/yaml/README.md#git-strategy [coverage report]: #test-coverage-parsing [ce-9362]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9362 +[ce-12509]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12509 diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 7650020b37e..0570d9f471f 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -94,8 +94,33 @@ all matching branches:  +## Deleting a protected branch + +> [Introduced][ce-21393] in GitLab 9.3. + +From time to time, it may be required to delete or clean up branches that are +protected. + +User with [Master permissions][perm] and up can manually delete protected +branches via GitLab's web interface: + +1. Visit **Repository > Branches** +1. Click on the delete icon next to the branch you wish to delete +1. In order to prevent accidental deletion, an additional confirmation is + required + +  + +Deleting a protected branch is only allowed via the web interface, not via Git. +This means that you can't accidentally delete a protected branch from your +command line or a Git client application. + ## Changelog +**9.2** + +- Allow deletion of protected branches via the web interface [gitlab-org/gitlab-ce#21393][ce-21393] + **8.11** - Allow creating protected branches that can't be pushed to [gitlab-org/gitlab-ce!5081][ce-5081] @@ -110,4 +135,6 @@ all matching branches: [ce-4665]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4665 "Allow specifying protected branches using wildcards" [ce-4892]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4892 "Allow developers to merge into a protected branch without having push access" [ce-5081]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5081 "Allow creating protected branches that can't be pushed to" +[ce-21393]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21393 [ee-restrict]: http://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users +[perm]: ../permissions.md diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md new file mode 100644 index 00000000000..ce4dd4e99d5 --- /dev/null +++ b/doc/user/project/quick_actions.md @@ -0,0 +1,40 @@ +# GitLab quick actions + +Quick actions are textual shortcuts for common actions on issues or merge +requests that are usually done by clicking buttons or dropdowns in GitLab's UI. +You can enter these commands while creating a new issue or merge request, and +in comments. Each command should be on a separate line in order to be properly +detected and executed. The commands are removed from the issue, merge request or +comment body before it is saved and will not be visible to anyone else. + +Below is a list of all of the available commands and descriptions about what they +do. + +| Command | Action | +|:---------------------------|:-------------| +| `/close` | Close the issue or merge request | +| `/reopen` | Reopen the issue or merge request | +| `/merge` | Merge (when pipeline succeeds) | +| `/title <New title>` | Change title | +| `/assign @username` | Assign | +| `/unassign` | Remove assignee | +| `/milestone %milestone` | Set milestone | +| `/remove_milestone` | Remove milestone | +| `/label ~foo ~"bar baz"` | Add label(s) | +| `/unlabel ~foo ~"bar baz"` | Remove all or specific label(s) | +| `/relabel ~foo ~"bar baz"` | Replace all label(s) | +| `/todo` | Add a todo | +| `/done` | Mark todo as done | +| `/subscribe` | Subscribe | +| `/unsubscribe` | Unsubscribe | +| <code>/due <in 2 days | this Friday | December 31st></code> | Set due date | +| `/remove_due_date` | Remove due date | +| `/wip` | Toggle the Work In Progress status | +| <code>/estimate <1w 3d 2h 14m></code> | Set time estimate | +| `/remove_estimate` | Remove estimated time | +| <code>/spend <1h 30m | -1h 5m></code> | Add or subtract spent time | +| `/remove_time_spent` | Remove time spent | +| `/target_branch <Branch Name>` | Set target branch for current merge request | +| `/award :emoji:` | Toggle award for :emoji: | +| `/board_move ~column` | Move issue to column on the board | +| `/duplicate #issue` | Closes this issue and marks it as a duplicate of another issue | diff --git a/doc/user/project/repository/branches/img/delete_merged_branches.png b/doc/user/project/repository/branches/img/delete_merged_branches.png Binary files differnew file mode 100644 index 00000000000..1856a624f74 --- /dev/null +++ b/doc/user/project/repository/branches/img/delete_merged_branches.png diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md new file mode 100644 index 00000000000..1948627ee79 --- /dev/null +++ b/doc/user/project/repository/branches/index.md @@ -0,0 +1,17 @@ +# Branches + +## Delete merged branches + +> [Introduced][ce-6449] in GitLab 8.14. + + + +This feature allows merged branches to be deleted in bulk. Only branches that +have been merged and [are not protected][protected] will be deleted as part of +this operation. + +It's particularly useful to clean up old branches that were not deleting +automatically when a merge request was merged. + +[ce-6449]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6449 "Add button to delete all merged branches" +[protected]: ../../protected_branches.md diff --git a/doc/user/project/repository/img/compare_branches.png b/doc/user/project/repository/img/compare_branches.png Binary files differnew file mode 100755 index 00000000000..353bd72ef4e --- /dev/null +++ b/doc/user/project/repository/img/compare_branches.png diff --git a/doc/user/project/repository/img/contributors_graph.png b/doc/user/project/repository/img/contributors_graph.png Binary files differnew file mode 100755 index 00000000000..c31da7aa1ff --- /dev/null +++ b/doc/user/project/repository/img/contributors_graph.png diff --git a/doc/user/project/repository/img/repo_graph.png b/doc/user/project/repository/img/repo_graph.png Binary files differnew file mode 100755 index 00000000000..28da8ad9589 --- /dev/null +++ b/doc/user/project/repository/img/repo_graph.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md new file mode 100644 index 00000000000..4b2c435a120 --- /dev/null +++ b/doc/user/project/repository/index.md @@ -0,0 +1,150 @@ +# Repository + +A [repository](https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository) +is what you use to store your codebase in GitLab and change it with version control. +A repository is part of a [project](../index.md), which has a lot of other features. + +## Create a repository + +To create a new repository, all you need to do is +[create a new project](../../../gitlab-basics/create-project.md). + +Once you create a new project, you can add new files via UI +(read the section below) or via command line. +To add files from the command line, follow the instructions that will +be presented on the screen when you create a new project, or read +through them in the [command line basics](../../../gitlab-basics/start-using-git.md) +documentation. + +> **Important:** +For security reasons, when using the command line, we strongly recommend +you to [connect with GitLab via SSH](../../../ssh/README.md). + +## Create and edit files + +Host your codebase in GitLab repositories by pushing your files to GitLab. +You can either use the user interface (UI), or connect your local computer +with GitLab [through the command line](../../../gitlab-basics/command-line-commands.md#start-working-on-your-project). + +To configure [GitLab CI/CD](../../../ci/README.md) to build, test, and deploy +you code, add a file called [.`gitlab-ci.yml`](../../../ci/quick_start/README.md) +to your repository's root. + +**From the user interface:** + +GitLab's UI allows you to perform lots of Git commands without having to +touch the command line. Even if you use the command line regularly, sometimes +it's easier to do so [via GitLab UI](web_editor.md): + +- [Create a file](web_editor.md#create-a-file) +- [Upload a file](web_editor.md#upload-a-file) +- [File templates](web_editor.md#template-dropdowns) +- [Create a directory](web_editor.md#create-a-directory) +- [Start a merge request](web_editor.md#tips) + +**From the command line:** + +To get started with the command line, please read through the +[command line basics documentation](../../../gitlab-basics/command-line-commands.md). + +## Branches + +When you submit changes in a new branch, you create a new version +of that project's file tree. Your branch contains all the changes +you are presenting, which are detected by Git line by line. + +To continue your workflow, once you pushed your changes to a new branch, +you can create a [merge request](../merge_requests/index.md), perform +inline code review, and [discuss](../../discussions/index.md) +your implementation with your team. +You can live preview changes submitted to a new branch with +[Review Apps](../../../ci/review_apps/index.md). + +With [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/) +subscriptions, you can also request +[approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#merge-request-approvals) from your managers. + +To create, delete, and branches via GitLab's UI: + +- [Create a branch](web_editor.md#create-a-new-branch) +- [Protected branches](../protected_branches.md#protected-branches) +- [Delete merged branches](branches/index.md#delete-merged-branches) + +Alternatively, you can use the +[command line](../../../gitlab-basics/start-using-git.md#create-a-branch). + +To learn more about branching strategies read through the +[GitLab Flow](../../../university/training/gitlab_flow.md) documentation. + +## Commits + +When you [commit your changes](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository), +you are introducing those changes to your branch. +Via command line, you can commit multiple times before pushing. + +- **Commit message:** +A commit message is important to identity what is being changed and, +more importantly, why. In GitLab, you can add keywords to the commit +message that will perform one of the actions below: + - **Trigger a GitLab CI/CD pipeline:** + If you have your project configured with [GitLab CI/CD](../../../ci/README.md), + you will trigger a pipeline per push, not per commit. + - **Skip pipelines:** + You can add to you commit message the keyword + [`[ci skip]`](../../../ci/yaml/README.html#skipping-jobs) + and GitLab CI will skip that pipeline. + - **Cross-link issues and merge requests:** + [Cross-linking](../issues/crosslinking_issues.md#from-commit-messages) + is great to keep track of what's is somehow related in your workflow. + If you mention an issue or a merge request in a commit message, they will be shown + on their respective thread. +- **Cherry-pick a commit:** +In GitLab, you can +[cherry-pick a commit](../merge_requests/cherry_pick_changes.md#cherry-picking-a-commit) +right from the UI. +- **Revert a commit:** +Easily [revert a commit](../merge_requests/revert_changes.md#reverting-a-commit) +from the UI to a selected branch. + +## Repository size + +In GitLab.com, your repository size limit it 10GB. For other instances, +the repository size is limited by your system administrators. + +You can also [reduce a repository size using Git](reducing_the_repo_size_using_git.md). + +## Contributors + +All the contributors to your codebase are displayed under your project's **Settings > Contributors**. + +They are ordered from the collaborator with the greatest number +of commits to the fewest, and displayed on a nice graph: + + + +## Repository graph + +The repository graph displays visually the Git flow strategy used in that repository: + + + +Find it under your project's **Repository > Graph**. + +## Compare + +Select branches to compare and view the changes inline: + + + +Find it under your project's **Repository > Compare**. + +## Locked files (EEP) + +Lock your files to prevent any conflicting changes. + +[File Locking](https://docs.gitlab.com/ee/user/project/file_lock.html) is available only in +[GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/). + +## Repository's API + +You can access your repos via [repository API](../../../api/repositories.md). diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md new file mode 100644 index 00000000000..08805a4dc99 --- /dev/null +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -0,0 +1,81 @@ +# Reducing the repository size using Git + +A GitLab Entrerprise Edition administrator can set a [repository size limit][admin-repo-size] +which will prevent you to exceed it. + +When a project has reached its size limit, you will not be able to push to it, +create a new merge request, or merge existing ones. You will still be able to +create new issues, and clone the project though. Uploading LFS objects will +also be denied. + +In order to lift these restrictions, the administrator of the GitLab instance +needs to increase the limit on the particular project that exceeded it or you +need to instruct Git to rewrite changes. + +If you exceed the repository size limit, your first thought might be to remove +some data, make a new commit and push back to the repository. Unfortunately, +it's not so easy and that workflow won't work. Deleting files in a commit doesn't +actually reduce the size of the repo since the earlier commits and blobs are +still around. What you need to do is rewrite history with Git's +[`filter-branch` option][gitscm]. + +Note that even with that method, until `git gc` runs on the GitLab side, the +"removed" commits and blobs will still be around. And if a commit was ever +included in an MR, or if a build was run for a commit, or if a user commented +on it, it will be kept around too. So, in these cases the size will not decrease. + +The only fool proof way to actually decrease the repository size is to prune all +the unneeded stuff locally, and then create a new project on GitLab and start +using that instead. + +With that being said, you can try reducing your repository size with the +following method. + +## Using `git filter-branch` to purge files + +> +**Warning:** +Make sure to first make a copy of your repository since rewriting history will +purge the files and information you are about to delete. Also make sure to +inform any collaborators to not use `pull` after your changes, but use `rebase`. + +1. Navigate to your repository: + + ``` + cd my_repository/ + ``` + +1. Change to the branch you want to remove the big file from: + + ``` + git checkout master + ``` + +1. Use `filter-branch` to remove the big file: + + ``` + git filter-branch --force --tree-filter 'rm -f path/to/big_file.mpg' HEAD + ``` + +1. Instruct Git to purge the unwanted data: + + ``` + git reflog expire --expire=now --all && git gc --prune=now --aggressive + ``` + +1. Lastly, force push to the repository: + + ``` + git push --force origin master + ``` + +Your repository should now be below the size limit. + +>**Note:** +As an alternative to `filter-branch`, you can use the `bfg` tool with a +command like: `bfg --delete-files path/to/big_file.mpg`. Read the +[BFG Repo-Cleaner][bfg] documentation for more information. + +[admin-repo-size]: https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html#repository-size-limit +[bfg]: https://rtyley.github.io/bfg-repo-cleaner/ +[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 58d2fd76c61..35960ade3d4 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -27,14 +27,15 @@ with all their related data and be moved into a new GitLab instance. | GitLab version | Import/Export version | | -------- | -------- | -| 9.2.0 to current | 0.1.7 | -| 8.17.0 | 0.1.6 | -| 8.13.0 | 0.1.5 | -| 8.12.0 | 0.1.4 | -| 8.10.3 | 0.1.3 | -| 8.10.0 | 0.1.2 | -| 8.9.5 | 0.1.1 | -| 8.9.0 | 0.1.0 | +| 9.4.0 to current | 0.1.8 | +| 9.2.0 | 0.1.7 | +| 8.17.0 | 0.1.6 | +| 8.13.0 | 0.1.5 | +| 8.12.0 | 0.1.4 | +| 8.10.3 | 0.1.3 | +| 8.10.0 | 0.1.2 | +| 8.9.5 | 0.1.1 | +| 8.9.0 | 0.1.0 | > The table reflects what GitLab version we updated the Import/Export version at. > For instance, 8.10.3 and 8.11 will have the same Import/Export version (0.1.3) diff --git a/doc/user/project/slash_commands.md b/doc/user/project/slash_commands.md index 08452ca75cd..e9103a3f49c 100644 --- a/doc/user/project/slash_commands.md +++ b/doc/user/project/slash_commands.md @@ -1,39 +1 @@ -# GitLab slash commands - -Slash commands are textual shortcuts for common actions on issues or merge -requests that are usually done by clicking buttons or dropdowns in GitLab's UI. -You can enter these commands while creating a new issue or merge request, and -in comments. Each command should be on a separate line in order to be properly -detected and executed. The commands are removed from the issue, merge request or -comment body before it is saved and will not be visible to anyone else. - -Below is a list of all of the available commands and descriptions about what they -do. - -| Command | Action | -|:---------------------------|:-------------| -| `/close` | Close the issue or merge request | -| `/reopen` | Reopen the issue or merge request | -| `/merge` | Merge (when pipeline succeeds) | -| `/title <New title>` | Change title | -| `/assign @username` | Assign | -| `/unassign` | Remove assignee | -| `/milestone %milestone` | Set milestone | -| `/remove_milestone` | Remove milestone | -| `/label ~foo ~"bar baz"` | Add label(s) | -| `/unlabel ~foo ~"bar baz"` | Remove all or specific label(s) | -| `/relabel ~foo ~"bar baz"` | Replace all label(s) | -| `/todo` | Add a todo | -| `/done` | Mark todo as done | -| `/subscribe` | Subscribe | -| `/unsubscribe` | Unsubscribe | -| <code>/due <in 2 days | this Friday | December 31st></code> | Set due date | -| `/remove_due_date` | Remove due date | -| `/wip` | Toggle the Work In Progress status | -| <code>/estimate <1w 3d 2h 14m></code> | Set time estimate | -| `/remove_estimate` | Remove estimated time | -| <code>/spend <1h 30m | -1h 5m></code> | Add or subtract spent time | -| `/remove_time_spent` | Remove time spent | -| `/target_branch <Branch Name>` | Set target branch for current merge request | -| `/award :emoji:` | Toggle award for :emoji: | -| `/board_move ~column` | Move issue to column on the board | +This document was moved to [user/project/quick_actions.md](quick_actions.md). diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 604c7d5cefb..925bbf76d49 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -6,7 +6,7 @@ - [Description templates](../user/project/description_templates.md) - [Feature branch workflow](workflow.md) - [GitLab Flow](gitlab_flow.md) -- [Groups](groups.md) +- [Groups](../user/group/index.md) - Issues - The GitLab Issue Tracker is an advanced and complete tool for tracking the evolution of a new idea or the process of solving a problem. - [Confidential issues](../user/project/issues/confidential_issues.md) @@ -21,7 +21,7 @@ - [Project users](add-user/add-user.md) - [Protected branches](../user/project/protected_branches.md) - [Protected tags](../user/project/protected_tags.md) -- [Slash commands](../user/project/slash_commands.md) +- [Quick Actions](../user/project/quick_actions.md) - [Sharing a project with a group](share_with_group.md) - [Share projects with other groups](share_projects_with_other_groups.md) - [Time tracking](time_tracking.md) diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index e10ccc4fc46..9d466ae1971 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -91,7 +91,6 @@ This workflow where commits only flow downstream ensures that everything has bee If you need to cherry-pick a commit with a hotfix it is common to develop it on a feature branch and merge it into master with a merge request, do not delete the feature branch. If master is good to go (it should be if you are practicing [continuous delivery](http://martinfowler.com/bliki/ContinuousDelivery.html)) you then merge it to the other branches. If this is not possible because more manual testing is required you can send merge requests from the feature branch to the downstream branches. -An 'extreme' version of environment branches are setting up an environment for each feature branch as done by [Teatro](https://teatro.io/). ## Release branches with GitLab flow @@ -300,7 +299,7 @@ If there are no merge conflicts and the feature branches are short lived the ris If there are merge conflicts you merge the master branch into the feature branch and the CI server will rerun the tests. If you have long lived feature branches that last for more than a few days you should make your issues smaller. -## Working wih feature branches +## Working with feature branches  diff --git a/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys.png b/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys.png Binary files differnew file mode 100644 index 00000000000..e525083918b --- /dev/null +++ b/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys.png diff --git a/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys_paste_pub.png b/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys_paste_pub.png Binary files differnew file mode 100644 index 00000000000..8e26d98f1b0 --- /dev/null +++ b/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys_paste_pub.png diff --git a/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png b/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png Binary files differnew file mode 100644 index 00000000000..f715c46adc3 --- /dev/null +++ b/doc/workflow/gpg_signed_commits/img/profile_settings_gpg_keys_single_key.png diff --git a/doc/workflow/gpg_signed_commits/img/project_signed_and_unsigned_commits.png b/doc/workflow/gpg_signed_commits/img/project_signed_and_unsigned_commits.png Binary files differnew file mode 100644 index 00000000000..16ec2d031ae --- /dev/null +++ b/doc/workflow/gpg_signed_commits/img/project_signed_and_unsigned_commits.png diff --git a/doc/workflow/gpg_signed_commits/img/project_signed_commit_unverified_signature.png b/doc/workflow/gpg_signed_commits/img/project_signed_commit_unverified_signature.png Binary files differnew file mode 100644 index 00000000000..22565cf7c7e --- /dev/null +++ b/doc/workflow/gpg_signed_commits/img/project_signed_commit_unverified_signature.png diff --git a/doc/workflow/gpg_signed_commits/img/project_signed_commit_verified_signature.png b/doc/workflow/gpg_signed_commits/img/project_signed_commit_verified_signature.png Binary files differnew file mode 100644 index 00000000000..1778b2ddf2b --- /dev/null +++ b/doc/workflow/gpg_signed_commits/img/project_signed_commit_verified_signature.png diff --git a/doc/workflow/gpg_signed_commits/index.md b/doc/workflow/gpg_signed_commits/index.md new file mode 100644 index 00000000000..7d5762d2b9d --- /dev/null +++ b/doc/workflow/gpg_signed_commits/index.md @@ -0,0 +1,84 @@ +# Signing commits with GPG + +## Getting started + +- [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) +- [Git Tools - Signing Your Work: GPG introduction](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work#_gpg_introduction) +- [Git Tools - Signing Your Work: Signing commits](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work#_signing_commits) + +## How GitLab handles GPG + +GitLab uses its own keyring to verify the GPG signature. It does not access any +public key server. + +In order to have a commit verified on GitLab the corresponding public key needs +to be uploaded to GitLab. + +For a signature to be verified two prerequisites need to be met: + +1. The public key needs to be added to GitLab +1. One of the emails in the GPG key matches your **primary** email + +## Add a GPG key + +1. On the upper right corner, click on your avatar and go to your **Settings**. + +  + +1. Navigate to the **GPG keys** tab. + +  + +1. Paste your **public** key in the 'Key' box. + +  + +1. Finally, click on **Add key** to add it to GitLab. You will be able to see + its fingerprint, the corresponding email address and creation date. + +  + +>**Note:** +Once you add a key, you cannot edit it, only remove it. In case the paste +didn't work, you will have to remove the offending key and re-add it. + +## Remove a GPG key + +1. On the upper right corner, click on your avatar and go to your **Settings**. + +1. Navigate to the **GPG keys** tab. + +1. Click on the trash icon besides the GPG key you want to delete. + +>**Note:** +Removing a key **does not unverify** already signed commits. Commits that were +verified by using this key will stay verified. Only unpushed commits will stay +unverified once you remove this key. + +## Revoke a GPG key + +1. On the upper right corner, click on your avatar and go to your **Settings**. + +1. Navigate to the **GPG keys** tab. + +1. Click on **Revoke** besides the GPG key you want to delete. + +>**Note:** +Revoking a key **unverifies** already signed commits. Commits that were +verified by using this key will change to an unverified state. Future commits +will also stay unverified once you revoke this key. This action should be used +in case your key has been compromised. + +## Verifying commits + +1. Within a project navigate to the **Commits** tag. Signed commits will show a + badge containing either "Verified" or "Unverified", depending on the + verification status of the GPG signature. + +  + +1. By clicking on the GPG badge details of the signature are displayed. + +  + +  diff --git a/doc/workflow/groups.md b/doc/workflow/groups.md index 1cb3c940f00..06eec1ed928 100644 --- a/doc/workflow/groups.md +++ b/doc/workflow/groups.md @@ -1,96 +1,2 @@ -# GitLab Groups -GitLab groups allow you to group projects into directories and give users access to several projects at once. - -When you create a new project in GitLab, the default namespace for the project is the personal namespace associated with your GitLab user. -In this document we will see how to create groups, put projects in groups and manage who can access the projects in a group. - -## Creating groups - -You can create a group by going to the 'Groups' tab of the GitLab dashboard and clicking the 'New group' button. - - - -Next, enter the path and name (required) and the optional description and group avatar. - - - -When your group has been created you are presented with the group dashboard feed, which will be empty. - - - -You can use the 'New project' button to add a project to the new group. - -## Transferring an existing project into a group - -You can transfer an existing project into a group you own from the project settings page. The option to transfer a project is only available if you are the Owner of the project. -First scroll down to the 'Dangerous settings' and click 'Show them to me'. -Now you can pick any of the groups you manage as the new namespace for the group. - - - -GitLab administrators can use the admin interface to move any project to any namespace if needed. - -## Adding users to a group - -One of the benefits of putting multiple projects in one group is that you can give a user to access to all projects in the group with one action. - -Suppose we have a group with two projects. - - - -On the 'Group Members' page we can now add a new user Barry to the group. - - - -Now because Barry is a 'Developer' member of the 'Open Source' group, he automatically gets 'Developer' access to all projects in the 'Open Source' group. - - - -If necessary, you can increase the access level of an individual user for a specific project, by adding them as a Member to the project. - - - -## Requesting access to a group - -As a group owner you can enable or disable non members to request access to -your group. Go to the group settings and click on **Allow users to request access**. - -As a user, you can request to be a member of a group. Go to the group you'd -like to be a member of, and click the **Request Access** button on the right -side of your screen. - - - ---- - -Group owners & masters will be notified of your request and will be able to approve or -decline it on the members page. - - - ---- - -If you change your mind before your request is approved, just click the -**Withdraw Access Request** button. - - - -## Managing group memberships via LDAP - -In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. -See [the GitLab Enterprise Edition documentation](http://docs.gitlab.com/ee/integration/ldap.html) for more information. - -## Allowing only admins to create groups - -By default, any GitLab user can create new groups. -This ability can be disabled for individual users from the admin panel. -It is also possible to configure GitLab so that new users default to not being able to create groups: - -``` -# For omnibus-gitlab, put the following in /etc/gitlab/gitlab.rb -gitlab_rails['gitlab_default_can_create_group'] = false - -# For installations from source, uncomment the 'default_can_create_group' -# line in /home/git/gitlab/config/gitlab.yml -``` +This document was moved to [another location](../user/group/index.md). diff --git a/doc/workflow/groups/add_member_to_group.png b/doc/workflow/groups/add_member_to_group.png Binary files differdeleted file mode 100644 index a10d5032bb0..00000000000 --- a/doc/workflow/groups/add_member_to_group.png +++ /dev/null diff --git a/doc/workflow/groups/group_dashboard.png b/doc/workflow/groups/group_dashboard.png Binary files differdeleted file mode 100644 index a5829f25808..00000000000 --- a/doc/workflow/groups/group_dashboard.png +++ /dev/null diff --git a/doc/workflow/groups/group_with_two_projects.png b/doc/workflow/groups/group_with_two_projects.png Binary files differdeleted file mode 100644 index 76d0a1b8ab2..00000000000 --- a/doc/workflow/groups/group_with_two_projects.png +++ /dev/null diff --git a/doc/workflow/groups/new_group_button.png b/doc/workflow/groups/new_group_button.png Binary files differdeleted file mode 100644 index 7155d6280bd..00000000000 --- a/doc/workflow/groups/new_group_button.png +++ /dev/null diff --git a/doc/workflow/groups/override_access_level.png b/doc/workflow/groups/override_access_level.png Binary files differdeleted file mode 100644 index 2b3e9a49842..00000000000 --- a/doc/workflow/groups/override_access_level.png +++ /dev/null diff --git a/doc/workflow/groups/project_members_via_group.png b/doc/workflow/groups/project_members_via_group.png Binary files differdeleted file mode 100644 index 878c9a03ac9..00000000000 --- a/doc/workflow/groups/project_members_via_group.png +++ /dev/null diff --git a/doc/workflow/groups/transfer_project.png b/doc/workflow/groups/transfer_project.png Binary files differdeleted file mode 100644 index 52161817f11..00000000000 --- a/doc/workflow/groups/transfer_project.png +++ /dev/null diff --git a/doc/workflow/share_projects_with_other_groups.md b/doc/workflow/share_projects_with_other_groups.md index 8e50cb03e63..40d756bc199 100644 --- a/doc/workflow/share_projects_with_other_groups.md +++ b/doc/workflow/share_projects_with_other_groups.md @@ -5,7 +5,7 @@ to a project with a single action. ## Groups as collections of users -Groups are used primarily to [create collections of projects](groups.md), but you can also +Groups are used primarily to [create collections of projects](../user/group/index.md), but you can also take advantage of the fact that groups define collections of _users_, namely the group members. diff --git a/doc/workflow/shortcuts.md b/doc/workflow/shortcuts.md index c5b7488be69..87416008e98 100644 --- a/doc/workflow/shortcuts.md +++ b/doc/workflow/shortcuts.md @@ -6,7 +6,10 @@ You can see GitLab's keyboard shortcuts by using 'shift + ?' | Keyboard Shortcut | Description | | ----------------- | ----------- | +| <kbd>n</kbd> | Main navigation | | <kbd>s</kbd> | Focus search | +| <kbd>f</kbd> | Focus filter | +| <kbd>p b</kbd> | Show/hide the Performance Bar | | <kbd>?</kbd> | Show/hide this dialog | | <kbd>⌘</kbd> + <kbd>shift</kbd> + <kbd>p</kbd> | Toggle markdown preview | | <kbd>↑</kbd> | Edit last comment (when focused on an empty textarea) | diff --git a/doc/workflow/time_tracking.md b/doc/workflow/time_tracking.md index de12994c516..bfe87bb2ceb 100644 --- a/doc/workflow/time_tracking.md +++ b/doc/workflow/time_tracking.md @@ -21,13 +21,13 @@ below. ## How to enter data -Time Tracking uses two [slash commands] that GitLab introduced with this new +Time Tracking uses two [quick actions] that GitLab introduced with this new feature: `/spend` and `/estimate`. -Slash commands can be used in the body of an issue or a merge request, but also +Quick actions can be used in the body of an issue or a merge request, but also in a comment in both an issue or a merge request. -Below is an example of how you can use those new slash commands inside a comment. +Below is an example of how you can use those new quick actions inside a comment.  @@ -70,4 +70,4 @@ The following time units are available: Default conversion rates are 1w = 5d and 1d = 8h. [landing]: https://about.gitlab.com/features/time-tracking -[slash-commands]: ../user/project/slash_commands.md +[quick actions]: ../user/project/quick_actions.md |
