diff options
Diffstat (limited to 'doc/administration')
17 files changed, 398 insertions, 96 deletions
diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index fb1a16b0f96..1528f1d2b17 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -32,7 +32,7 @@ Authentiq will generate a Client ID and the accompanying Client Secret for you t "app_id" => "YOUR_CLIENT_ID", "app_secret" => "YOUR_CLIENT_SECRET", "args" => { - scope: 'aq:name email~rs aq:push' + "scope": 'aq:name email~rs address aq:push' } } ] @@ -45,21 +45,20 @@ Authentiq will generate a Client ID and the accompanying Client Secret for you t app_id: 'YOUR_CLIENT_ID', app_secret: 'YOUR_CLIENT_SECRET', args: { - scope: 'aq:name email~rs aq:push' + scope: 'aq:name email~rs address aq:push' } } ``` 5. The `scope` is set to request the user's name, email (required and signed), and permission to send push notifications to sign in on subsequent visits. -See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq#scopes-and-redirect-uri-configuration) for more information on scopes and modifiers. +See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq/wiki/Scopes,-callback-url-configuration-and-responses) for more information on scopes and modifiers. 6. Change `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` to the Client credentials you received in step 1. 7. Save the configuration file. -8. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) - for the changes to take effect if you installed GitLab via Omnibus or from source respectively. +8. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page there should now be an Authentiq icon below the regular sign in form. Click the icon to begin the authentication process. diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 725fc1f6076..a7395e03d1c 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -69,14 +69,42 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # Example: 'ldap.mydomain.com' host: '_your_ldap_server' # This port is an example, it is sometimes different but it is always an integer and not a string - port: 389 + port: 389 # usually 636 for SSL uid: 'sAMAccountName' # This should be the attribute, not the value that maps to uid. - method: 'plain' # "tls" or "ssl" or "plain" # Examples: 'america\\momo' or 'CN=Gitlab Git,CN=Users,DC=mydomain,DC=com' bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' password: '_the_password_of_the_bind_user' + # Encryption method. The "method" key is deprecated in favor of + # "encryption". + # + # Examples: "start_tls" or "simple_tls" or "plain" + # + # Deprecated values: "tls" was replaced with "start_tls" and "ssl" was + # replaced with "simple_tls". + # + encryption: 'plain' + + # Enables SSL certificate verification if encryption method is + # "start_tls" or "simple_tls". (Defaults to false for backward- + # compatibility) + verify_certificates: false + + # Specifies the path to a file containing a PEM-format CA certificate, + # e.g. if you need to use an internal CA. + # + # Example: '/etc/ca.pem' + # + ca_cert: '' + + # Specifies the SSL version for OpenSSL to use, if the OpenSSL default + # is not appropriate. + # + # Example: 'TLSv1_1' + # + ssl_version: '' + # Set a timeout, in seconds, for LDAP queries. This helps avoid blocking # a request if the LDAP server becomes unresponsive. # A value of 0 means there is no timeout. @@ -116,8 +144,8 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # # Note: GitLab does not support omniauth-ldap's custom filter syntax. # - # Below an example for get only specific users - # Example: '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' + # Example for getting only specific users: + # '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' # user_filter: '' @@ -228,9 +256,14 @@ Tip: If you want to limit access to the nested members of an Active Directory group you can use the following syntax: ``` -(memberOf=CN=My Group,DC=Example,DC=com) +(memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` +Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at +https://msdn.microsoft.com/en-us/library/aa746475(v=vs.85).aspx. Support for +nested members in the user filter should not be confused with +[group sync nested groups support (EE only)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#supported-ldap-group-types-attributes). + Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. @@ -245,6 +278,19 @@ In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their LDAP email address, and then sign into GitLab via their LDAP credentials. +## Encryption + +### TLS Server Authentication + +There are two encryption methods, `simple_tls` and `start_tls`. + +For either encryption method, if setting `validate_certificates: false`, TLS +encryption is established with the LDAP server before any LDAP-protocol data is +exchanged but no validation of the LDAP server's SSL certificate is performed. + +>**Note**: Before GitLab 9.5, `validate_certificates: false` is the default if +unspecified. + ## Limitations ### TLS Client Authentication @@ -254,14 +300,6 @@ You should disable anonymous LDAP authentication and enable simple or SASL authentication. The TLS client authentication setting in your LDAP server cannot be mandatory and clients cannot be authenticated with the TLS protocol. -### TLS Server Authentication - -Not supported by GitLab's configuration options. -When setting `method: ssl`, the underlying authentication method used by -`omniauth-ldap` is `simple_tls`. This method establishes TLS encryption with -the LDAP server before any LDAP-protocol data is exchanged but no validation of -the LDAP server's SSL certificate is performed. - ## Troubleshooting ### Debug LDAP user filter with ldapsearch @@ -301,9 +339,9 @@ tree and traverse it. ### Connection Refused If you are getting 'Connection Refused' errors when trying to connect to the -LDAP server please double-check the LDAP `port` and `method` settings used by -GitLab. Common combinations are `method: 'plain'` and `port: 389`, OR -`method: 'ssl'` and `port: 636`. +LDAP server please double-check the LDAP `port` and `encryption` settings used by +GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR +`encryption: 'simple_tls'` and `port: 636`. ### Troubleshooting diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index afafb6bf1f5..57e54815b68 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -112,7 +112,7 @@ GitLab from source respectively. >**Note:** Be careful to choose a port different than the one that Registry listens to (`5000` by default), -otherwise you will run into conflicts . +otherwise you will run into conflicts. --- @@ -465,23 +465,42 @@ on how to achieve that. ## Disable Container Registry but use GitLab as an auth endpoint -You can disable the embedded Container Registry to use an external one, but -still use GitLab as an auth endpoint. - **Omnibus GitLab** + +You can use GitLab as an auth endpoint and use a non-bundled Container Registry. + 1. Open `/etc/gitlab/gitlab.rb` and set necessary configurations: ```ruby - registry['enable'] = false gitlab_rails['registry_enabled'] = true gitlab_rails['registry_host'] = "registry.gitlab.example.com" gitlab_rails['registry_port'] = "5005" gitlab_rails['registry_api_url'] = "http://localhost:5000" - gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key" gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry" gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" ``` +1. A certificate keypair is required for GitLab and the Container Registry to + communicate securely. By default omnibus-gitlab will generate one keypair, + which is saved to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key`. + When using an non-bundled Container Registry, you will need to supply a + custom certificate key. To do that, add the following to + `/etc/gitlab/gitlab.rb` + + ```ruby + gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" + # registry['internal_key'] should contain the contents of the custom key + # file. Line breaks in the key file should be marked using `\n` character + # Example: + registry['internal_key'] = "---BEGIN RSA PRIVATE KEY---\nMIIEpQIBAA\n" + ``` + + **Note:** The file specified at `registry_key_path` gets populated with the + content specified by `internal_key`, each time reconfigure is executed. If + no file is specified, omnibus-gitlab will default it to + `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate + it. + 1. Save the file and [reconfigure GitLab][] for the changes to take effect. **Installations from source** diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 48929910a9c..5732b6a1ca4 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,8 +2,7 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) (introduced in GitLab 9.0) is a service that provides high-level RPC access to Git -repositories. As of GitLab 9.3 it is still an optional component with -limited scope. +repositories. Gitaly is a mandatory component in GitLab 9.4 and newer. GitLab components that access Git repositories (gitlab-rails, gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do @@ -33,36 +32,155 @@ prometheus_listen_addr = "localhost:9236" Changes to `/home/git/gitaly/config.toml` are applied when you run `service gitlab restart`. -## Configuring GitLab to not use Gitaly +## Running Gitaly on its own server -Gitaly is still an optional component in GitLab 9.3. This means you -can choose to not use it. +> This is an optional way to deploy Gitaly which can benefit GitLab +installations that are larger than a single machine. Most +installations will be better served with the default configuration +used by Omnibus and the GitLab source installation guide. -In Omnibus you can make the following change in -`/etc/gitlab/gitlab.rb` and reconfigure. This will both disable the -Gitaly service and configure the rest of GitLab not to use it. +Starting with GitLab 9.4 it is possible to run Gitaly on a different +server from the rest of the application. This can improve performance +when running GitLab with its repositories stored on an NFS server. + +At the moment (GitLab 9.4) Gitaly is not yet a replacement for NFS +because some parts of GitLab still bypass Gitaly when accessing Git +repositories. If you choose to deploy Gitaly on your NFS server you +must still also mount your Git shares on your GitLab application +servers. + +Gitaly network traffic is unencrypted so you should use a firewall to +restrict access to your Gitaly server. + +Below we describe how to configure a Gitaly server at address +`gitaly.internal:9999` with secret token `abc123secret`. We assume +your GitLab installation has two repository storages, `default` and +`storage1`. + +### Client side token configuration + +Start by configuring a token on the client side. + +Omnibus installations: ```ruby -gitaly['enable'] = false +# /etc/gitlab/gitlab.rb +gitlab_rails['gitaly_token'] = 'abc123secret' +``` + +Source installations: + +```yaml +# /home/git/gitlab/config/gitlab.yml +gitlab: + gitaly: + token: 'abc123secret' +``` + +You need to reconfigure (Omnibus) or restart (source) for these +changes to be picked up. + +### Gitaly server configuration + +Next, on the Gitaly server, we need to configure storage paths, enable +the network listener and configure the token. + +Note: if you want to reduce the risk of downtime when you enable +authentication you can temporarily disable enforcement, see [the +documentation on configuring Gitaly +authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) +. + +In most or all cases the storage paths below end in `/repositories`. Check the +directory layout on your Gitaly server to be sure. + +Omnibus installations: + +```ruby +# /etc/gitlab/gitlab.rb +gitaly['listen_addr'] = '0.0.0.0:9999' +gitaly['auth_token'] = 'abc123secret' +gitaly['storage'] = [ + { 'name' => 'default', 'path' => '/path/to/default/repositories' }, + { 'name' => 'storage1', 'path' => '/path/to/storage1/repositories' }, +] +``` + +Source installations: + +```toml +# /home/git/gitaly/config.toml +listen_addr = '0.0.0.0:9999' + +[auth] +token = 'abc123secret' + +[[storage] +name = 'default' +path = '/path/to/default/repositories' + +[[storage]] +name = 'storage1' +path = '/path/to/storage1/repositories' ``` -In source installations, edit `/home/git/gitlab/config/gitlab.yml` and -make sure `enabled` in the `gitaly` section is set to 'false'. This -does not disable the Gitaly service in your init script; it only -prevents it from being used. +Again, reconfigure (Omnibus) or restart (source). + +### Converting clients to use the Gitaly server -Apply the change with `service gitlab restart`. +Now as the final step update the client machines to switch from using +their local Gitaly service to the new Gitaly server you just +configured. This is a risky step because if there is any sort of +network, firewall, or name resolution problem preventing your GitLab +server from reaching the Gitaly server then all Gitaly requests will +fail. + +We assume that your Gitaly server can be reached at +`gitaly.internal:9999` from your GitLab server, and that your GitLab +NFS shares are mounted at `/mnt/gitlab/default` and +`/mnt/gitlab/storage1` respectively. + +Omnibus installations: + +```ruby +# /etc/gitlab/gitlab.rb +git_data_dirs({ + { 'default' => { 'path' => '/mnt/gitlab/default', 'gitaly_address' => 'tcp://gitlab.internal:9999' } }, + { 'storage1' => { 'path' => '/mnt/gitlab/storage1', 'gitaly_address' => 'tcp://gitlab.internal:9999' } }, +}) + +gitlab_rails['gitaly_token'] = 'abc123secret' +``` + +Source installations: ```yaml +# /home/git/gitlab/config/gitlab.yml +gitlab: + repositories: + storages: + default: + path: /mnt/gitlab/default/repositories + gitaly_address: tcp://gitlab.internal:9999 + storage1: + path: /mnt/gitlab/storage1/repositories + gitaly_address: tcp://gitlab.internal:9999 + gitaly: - enabled: false + token: 'abc123secret' ``` +Now reconfigure (Omnibus) or restart (source). When you tail the +Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or +`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests +coming in. One sure way to trigger a Gitaly request is to clone a +repository from your GitLab server over HTTP. + ## Disabling or enabling the Gitaly service -Be careful: if you disable Gitaly without instructing the rest of your -GitLab installation not to use Gitaly, you may end up with errors -because GitLab tries to access a service that is not running. +If you are running Gitaly [as a remote +service](#running-gitaly-on-its-own-server) you may want to disable +the local Gitaly service that runs on your Gitlab server by default. To disable the Gitaly service in your Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`: @@ -81,4 +199,4 @@ following to `/etc/default/gitlab`: gitaly_enabled=false ``` -When you run `service gitlab restart` Gitaly will be disabled.
\ No newline at end of file +When you run `service gitlab restart` Gitaly will be disabled. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index da9687aa849..ca6d8d2de67 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -97,9 +97,12 @@ If you use a cloud-managed service, or provide your own PostgreSQL: Enter new password: Enter it again: ``` - -1. Enable the `pg_trgm` extension: +1. Exit from editing `template1` prompt by typing `\q` and Enter. +1. Enable the `pg_trgm` extension within the `gitlabhq_production` database: + ``` + gitlab-psql -d gitlabhq_production + CREATE EXTENSION pg_trgm; # Output: diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 137fed35a73..42666357faf 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -5,7 +5,7 @@ configure the GitLab application server(s) now. Complete the steps below for each GitLab application server in your environment. > **Note:** There is some additional configuration near the bottom for - secondary GitLab application servers. It's important to read and understand + additional GitLab application servers. It's important to read and understand these additional steps before proceeding with GitLab installation. 1. If necessary, install the NFS client utility packages using the following @@ -70,10 +70,16 @@ for each GitLab application server in your environment. gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server gitlab_rails['redis_password'] = 'Redis Password' ``` + + > **Note:** To maintain uniformity of links across HA clusters, the `external_url` + on the first application server as well as the additional application + servers should point to the external url that users will use to access GitLab. + In a typical HA setup, this will be the url of the load balancer which will + route traffic to all GitLab application servers in the HA cluster. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. -## Primary GitLab application server +## First GitLab application server As a final step, run the setup rake task on the first GitLab application server. It is not necessary to run this on additional application servers. @@ -89,10 +95,10 @@ It is not necessary to run this on additional application servers. [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for more information. -## Additional configuration for secondary GitLab application servers +## Extra configuration for additional GitLab application servers -Secondary GitLab servers (servers configured **after** the first GitLab server) -need some additional configuration. +Additional GitLab servers (servers configured **after** the first GitLab server) +need some extra configuration. 1. Configure shared secrets. These values can be obtained from the primary GitLab server in `/etc/gitlab/gitlab-secrets.json`. Add these to diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index d8e76d6ab94..90a2e9298bf 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -1,12 +1,35 @@ # NFS -## Required NFS Server features +You can view information and options set for each of the mounted NFS file +systems by running `sudo nfsstat -m`. + +## NFS Server features + +### Required features **File locking**: GitLab **requires** advisory file locking, which is only supported natively in NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not specifically test NFSv3. +### Recommended options + +When you define your NFS exports, we recommend you also add the following +options: + +- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is + a good security measure when NFS shares will be accessed by many different + users. However, in this case only GitLab will use the NFS share so it + is safe. GitLab recommends the `no_root_squash` setting because we need to + manage file permissions automatically. Without the setting you may receive + errors when the Omnibus package tries to alter permissions. Note that GitLab + and other bundled components do **not** run as `root` but as non-privileged + users. The recommendation for `no_root_squash` is to allow the Omnibus package + to set ownership and permissions on files, as needed. +- `sync` - Force synchronous behavior. Default is asynchronous and under certain + circumstances it could lead to data loss if a failure occurs before data has + synced. + ## AWS Elastic File System GitLab does not recommend using AWS Elastic File System (EFS). @@ -23,30 +46,17 @@ GitLab does not recommend using EFS with GitLab. many small files are written in a serialized manner are not well-suited for EFS. EBS with an NFS server on top will perform much better. +In addition, avoid storing GitLab log files (e.g. those in `/var/log/gitlab`) +because this will also affect performance. We recommend that the log files be +stored on a local volume. + For more details on another person's experience with EFS, see [Amazon's Elastic File System: Burst Credits](https://www.rawkode.io/2017/04/amazons-elastic-file-system-burst-credits/) -### Recommended options - -When you define your NFS exports, we recommend you also add the following -options: - -- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is - a good security measure when NFS shares will be accessed by many different - users. However, in this case only GitLab will use the NFS share so it - is safe. GitLab recommends the `no_root_squash` setting because we need to - manage file permissions automatically. Without the setting you may receive - errors when the Omnibus package tries to alter permissions. Note that GitLab - and other bundled components do **not** run as `root` but as non-privileged - users. The recommendation for `no_root_squash` is to allow the Omnibus package - to set ownership and permissions on files, as needed. -- `sync` - Force synchronous behavior. Default is asynchronous and under certain - circumstances it could lead to data loss if a failure occurs before data has - synced. - ## NFS Client mount options -Below is an example of an NFS mount point we use on GitLab.com: +Below is an example of an NFS mount point defined in `/etc/fstab` we use on +GitLab.com: ``` 10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 3629772b8af..8b7a515a076 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -4,6 +4,11 @@ This is the documentation for configuring a Highly Available Redis setup when you have installed Redis all by yourself and not using the bundled one that comes with the Omnibus packages. +Note also that you may elect to override all references to +`/home/git/gitlab/config/resque.yml` in accordance with the advanced Redis +settings outlined in +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/README.md). + We cannot stress enough the importance of reading the [Overview section](redis.md#overview) of the Omnibus Redis HA as it provides some invaluable information to the configuration of Redis. Please proceed to @@ -364,3 +369,4 @@ When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics [downloads]: https://about.gitlab.com/downloads [restart]: ../restart_gitlab.md#installations-from-source [it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png +[resque]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/resque.yml.example diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md new file mode 100644 index 00000000000..ad2773de132 --- /dev/null +++ b/doc/administration/monitoring/ip_whitelist.md @@ -0,0 +1,39 @@ +# IP whitelist + +> Introduced in GitLab 9.4. + +GitLab provides some [monitoring endpoints] that provide health check information +when probed. + +To control access to those endpoints via IP whitelisting, you can add single +hosts or use IP ranges: + +**For Omnibus installations** + +1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: + + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` + +1. Save the file and [reconfigure] GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + monitoring: + # by default only local IPs are allowed to access monitoring resources + ip_whitelist: + - 127.0.0.0/8 + - 192.168.0.1 + ``` + +1. Save the file and [restart] GitLab for the changes to take effect. + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source +[monitoring endpoints]: ../../user/admin_area/monitoring/health_check.md diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differnew file mode 100644 index 00000000000..b3c6bc474e3 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png b/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png Binary files differnew file mode 100644 index 00000000000..2d64ef8c5fc --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png Binary files differnew file mode 100644 index 00000000000..7868e2c46d1 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png Binary files differnew file mode 100644 index 00000000000..372ae021f6b --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md new file mode 100644 index 00000000000..ee680c7b258 --- /dev/null +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -0,0 +1,35 @@ +# Performance Bar + +A Performance Bar can be displayed, to dig into the performance of a page. When +activated, it looks as follows: + + + +It allows you to: + +- see the current host serving the page +- see the timing of the page (backend, frontend) +- the number of DB queries, the time it took, and the detail of these queries + +- the number of calls to Redis, and the time it took +- the number of background jobs created by Sidekiq, and the time it took +- the number of Ruby GC calls, and the time it took +- profile the code used to generate the page, line by line + + +## Enable the Performance Bar via the Admin panel + +GitLab Performance Bar is disabled by default. To enable it for a given group, +navigate to the Admin area in **Settings > Profiling - Performance Bar** +(`/admin/application_settings`). + +The only required setting you need to set is the full path of the group that +will be allowed to display the Performance Bar. +Make sure _Enable the Performance Bar_ is checked and hit +**Save** to save the changes. + +--- + + + +--- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 07c05b5a6fb..6baae20d16a 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,10 +1,8 @@ # GitLab Prometheus metrics >**Note:** -Available since [Omnibus GitLab 9.3][29118]. Currently experimental. For installations from source -you'll have to configure it yourself. - -GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other [Prometheus] exporters, this endpoint requires authentication as it is available on the same URL and port as user traffic. +Available since [Omnibus GitLab 9.3][29118]. Currently experimental. For +installations from source you'll have to configure it yourself. To enable the GitLab Prometheus metrics: @@ -15,33 +13,57 @@ To enable the GitLab Prometheus metrics: ## Collecting the metrics -Since the metrics endpoint is available on the same host and port as other traffic, it requires authentication. The token and URL to access is displayed on the [Health Check][health-check] page. +GitLab monitors its own internal service metrics, and makes them available at the +`/-/metrics` endpoint. Unlike other [Prometheus] exporters, in order to access +it, the client IP needs to be [included in a whitelist][whitelist]. -Currently the embedded Prometheus server is not automatically configured to collect metrics from this endpoint. We recommend setting up another Prometheus server, because the embedded server configuration is overwritten one every reconfigure of GitLab. In the future this will not be required. +Currently the embedded Prometheus server is not automatically configured to +collect metrics from this endpoint. We recommend setting up another Prometheus +server, because the embedded server configuration is overwritten once every +[reconfigure of GitLab][reconfigure]. In the future this will not be required. ## Metrics available In this experimental phase, only a few metrics are available: -| Metric | Type | Description | -| ------ | ---- | ----------- | -| db_ping_timeout | Gauge | Whether or not the last database ping timed out | -| db_ping_success | Gauge | Whether or not the last database ping succeeded | -| db_ping_latency | Gauge | Round trip time of the database ping | -| redis_ping_timeout | Gauge | Whether or not the last redis ping timed out | -| redis_ping_success | Gauge | Whether or not the last redis ping succeeded | -| redis_ping_latency | Gauge | Round trip time of the redis ping | -| filesystem_access_latency | gauge | Latency in accessing a specific filesystem | -| filesystem_accessible | gauge | Whether or not a specific filesystem is accessible | -| filesystem_write_latency | gauge | Write latency of a specific filesystem | -| filesystem_writable | gauge | Whether or not the filesystem is writable | -| filesystem_read_latency | gauge | Read latency of a specific filesystem | -| filesystem_readable | gauge | Whether or not the filesystem is readable | -| user_sessions_logins | Counter | Counter of how many users have logged in | +| Metric | Type | Since | Description | +|:--------------------------------- |:--------- |:----- |:----------- | +| db_ping_timeout | Gauge | 9.4 | Whether or not the last database ping timed out | +| db_ping_success | Gauge | 9.4 | Whether or not the last database ping succeeded | +| db_ping_latency_seconds | Gauge | 9.4 | Round trip time of the database ping | +| filesystem_access_latency_seconds | Gauge | 9.4 | Latency in accessing a specific filesystem | +| filesystem_accessible | Gauge | 9.4 | Whether or not a specific filesystem is accessible | +| filesystem_write_latency_seconds | Gauge | 9.4 | Write latency of a specific filesystem | +| filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | +| filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | +| filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | +| http_requests_total | Counter | 9.4 | Rack request count | +| http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | +| pipelines_created_total | Counter | 9.4 | Counter of pipelines created | +| rack_uncaught_errors_total | Counter | 9.4 | Rack connections handling uncaught errors count | +| redis_ping_timeout | Gauge | 9.4 | Whether or not the last redis ping timed out | +| redis_ping_success | Gauge | 9.4 | Whether or not the last redis ping succeeded | +| redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | +| user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | + +## Metrics shared directory + +GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. +Those files are shared among all instances running under Unicorn server. +The directory needs to be accessible to all running Unicorn's processes otherwise +metrics will not function correctly. + +For best performance its advisable that this directory will be located in `tmpfs`. + +Its location is configured using environment variable `prometheus_multiproc_dir`. + +If GitLab is installed using Omnibus and `tmpfs` is available then metrics +directory will be automatically configured. [← Back to the main Prometheus page](index.md) [29118]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29118 [Prometheus]: https://prometheus.io [restart]: ../../restart_gitlab.md#omnibus-gitlab-restart -[health-check]: ../../../user/admin_area/monitoring/health_check.md +[whitelist]: ../ip_whitelist.md +[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 695fdf09a87..f43c89dad87 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -95,8 +95,9 @@ Sample Prometheus queries: ## Configuring Prometheus to monitor Kubernetes > Introduced in GitLab 9.0. +> Pod monitoring introduced in GitLab 9.4. -If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes in the cluster including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them. +If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them. To disable the monitoring of Kubernetes: diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 93521e976d5..3a35aff8366 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -15,6 +15,12 @@ prefixed with 'session:gitlab:', so they would look like 'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to remove the keys in the old format. +**Note:** the instructions below must be modified in accordance with your +configuration settings if you have used the advanced Redis +settings outlined in +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/README.md). + + First we define a shell function with the proper Redis connection details. ``` |
