From c76ff8cf32801198e93122fdcf689f3ac1987e41 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 22 Sep 2016 06:05:28 -0300 Subject: Improved redis sentinel documentation for CE --- doc/administration/high_availability/redis.md | 62 +++++++++++---------------- 1 file changed, 26 insertions(+), 36 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index bc424330656..e347959ebbc 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -143,6 +143,7 @@ the master, and `masterauth` in slaves. redis['port'] = 6379 ## Slave redis instance + redis['master'] = false redis['master_ip'] = '10.10.10.10' # IP of master Redis server redis['master_port'] = 6379 # Port of master Redis server redis['master_password'] = "" @@ -157,31 +158,19 @@ servers. ### Sentinel setup -We don't provide yet an automated way to setup and run the Sentinel daemon -from Omnibus installation method. You must follow the instructions below and -run it by yourself. +We provide an automated way to setup and run the Sentinel daemon +with GitLab EE. -The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531). -While you can give any name for the `master-group-name` part of the -configuration, as in this example: - -```conf -sentinel monitor -``` - -,for it to work in Ruby, you have to use the "hostname" of the master Redis -server, otherwise you will get an error message like: -`Redis::CannotConnectError: No sentinels available.`. Read -[Sentinel troubleshooting](#sentinel-troubleshooting) for more information. +See the instructions below how to setup it by yourself. Here is an example configuration file (`sentinel.conf`) for a Sentinel node: ```conf port 26379 -sentinel monitor master-redis.example.com 10.10.10.10 6379 1 -sentinel down-after-milliseconds master-redis.example.com 10000 -sentinel config-epoch master-redis.example.com 0 -sentinel leader-epoch master-redis.example.com 0 +sentinel monitor gitlab-redis 10.0.0.1 6379 1 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 ``` --- @@ -213,10 +202,11 @@ The following steps should be performed in the [GitLab application server](gitla 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: ```ruby - gitlab-rails['redis_host'] = "master-redis.example.com" - gitlab-rails['redis_port'] = 6379 - gitlab-rails['redis_password'] = '' - gitlab-rails['redis_sentinels'] = [ + redis['master_name'] = "gitlab-redis" + redis['master_ip'] = "10.0.0.1" + redis['master_port'] = 6379 + redis['master_password'] = '' + gitlab_rails['redis_sentinels'] = [ {'host' => '10.10.10.1', 'port' => 26379}, {'host' => '10.10.10.2', 'port' => 26379}, {'host' => '10.10.10.3', 'port' => 26379} @@ -229,33 +219,33 @@ The following steps should be performed in the [GitLab application server](gitla If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related -to [this issue][gh-531] ([pull request][gh-534] that should make things better). +to [this issue][gh-531]. -It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`, -otherwise `redis-rb` will not work properly. +It's a bit non-intuitive the way you have to config `resque.yml` and +`sentinel.conf`, otherwise `redis-rb` will not work properly. -The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`) -**must** match the one configured in GitLab (`resque.yml` for source installations -or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex: +The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) +**must** be used as the hostname in GitLab (`resque.yml` for source installations +or `gitlab-rails['redis_*']` in Omnibus): ```conf # sentinel.conf: -sentinel monitor my-primary-redis 10.10.10.10 6379 1 -sentinel down-after-milliseconds my-primary-redis 10000 -sentinel config-epoch my-primary-redis 0 -sentinel leader-epoch my-primary-redis 0 +sentinel monitor gitlab-redis 10.10.10.10 6379 1 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 ``` ```yaml # resque.yaml production: - url: redis://my-primary-redis:6378 + url: redis://:myredispassword@gitlab-redis/ sentinels: - - host: slave1 + host: slave1.example.com # or use ip port: 26380 # point to sentinel, not to redis port - - host: slave2 + host: slave2.exampl.com # or use ip port: 26381 # point to sentinel, not to redis port ``` -- cgit v1.2.1 From f6d29583b02ffbb35b3c344f78b5a3575bf9c656 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Fri, 7 Oct 2016 01:16:32 +0200 Subject: Small fixes on Sentinel documentation for CE --- doc/administration/high_availability/redis.md | 2 -- 1 file changed, 2 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index e347959ebbc..69184ae9723 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -203,8 +203,6 @@ The following steps should be performed in the [GitLab application server](gitla ```ruby redis['master_name'] = "gitlab-redis" - redis['master_ip'] = "10.0.0.1" - redis['master_port'] = 6379 redis['master_password'] = '' gitlab_rails['redis_sentinels'] = [ {'host' => '10.10.10.1', 'port' => 26379}, -- cgit v1.2.1 From f54d60b41df3d30181a371d3799fa8b9451d4c5b Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Fri, 7 Oct 2016 01:47:22 +0200 Subject: Updated password examples and improved omnibus troubleshooting --- doc/administration/high_availability/redis.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 69184ae9723..9fca7bfb8b4 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -45,7 +45,7 @@ Redis. redis['bind'] = '0.0.0.0' # If you wish to use Redis authentication (recommended) - redis['password'] = 'Redis Password' + redis['password'] = 'redis-password-goes-here' ``` 1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. @@ -132,7 +132,7 @@ the master, and `masterauth` in slaves. redis['port'] = 6379 ## Master redis instance - redis['password'] = '' + redis['password'] = 'redis-password-goes-here' ``` 1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines): @@ -146,7 +146,7 @@ the master, and `masterauth` in slaves. redis['master'] = false redis['master_ip'] = '10.10.10.10' # IP of master Redis server redis['master_port'] = 6379 # Port of master Redis server - redis['master_password'] = "" + redis['master_password'] = "redis-password-goes-here" ``` 1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` @@ -203,7 +203,7 @@ The following steps should be performed in the [GitLab application server](gitla ```ruby redis['master_name'] = "gitlab-redis" - redis['master_password'] = '' + redis['master_password'] = 'redis-password-goes-here' gitlab_rails['redis_sentinels'] = [ {'host' => '10.10.10.1', 'port' => 26379}, {'host' => '10.10.10.2', 'port' => 26379}, @@ -215,6 +215,21 @@ The following steps should be performed in the [GitLab application server](gitla ### Sentinel troubleshooting +#### Omnibus install + +If you get an error like: `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this issue][gh-531]. + +You must make sure you are defining the same value in `redis['master_name']` +and `redis['master_pasword']` as you defined for your sentinel node. + +The way the redis connector `redis-rb` works with sentinel is a bit +non-intuitive. We try to hide the complexity in omnibus, but it still requires +a few extra configs. + +#### Source install + If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related to [this issue][gh-531]. -- cgit v1.2.1 From c4d3c0de1f489639e1e2f1a12b7b4d88384d3e06 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Sat, 15 Oct 2016 05:40:15 +0200 Subject: Improved documentation on HA sentinel part and Redis replication troubleshooting. --- doc/administration/high_availability/redis.md | 316 +++++++++++++++++++++----- 1 file changed, 256 insertions(+), 60 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 9fca7bfb8b4..840f5896bd7 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -8,6 +8,27 @@ that comes bundled with GitLab Omnibus packages. information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. + + +**Table of Contents** + +- [Configure your own Redis server](#configure-your-own-redis-server) +- [Configure Redis using Omnibus](#configure-redis-using-omnibus) +- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) + - [Redis setup](#redis-setup) + - [Source install](#source-install) + - [Omnibus Install](#omnibus-install) + - [Troubleshooting Replication](#troubleshooting-replication) + - [Sentinel](#sentinel) + - [Sentinel setup (Community Edition)](#sentinel-setup-community-edition) + - [Sentinel setup (EE Only)](#sentinel-setup-ee-only) + - [GitLab setup](#gitlab-setup) + - [Sentinel troubleshooting](#sentinel-troubleshooting) + - [Omnibus install](#omnibus-install) + - [Source install](#source-install-1) + + + ## Configure your own Redis server If you're hosting GitLab on a cloud provider, you can optionally use a @@ -37,6 +58,7 @@ Redis. unicorn['enable'] = false sidekiq['enable'] = false postgresql['enable'] = false + gitlab_rails['enable'] = false gitlab_workhorse['enable'] = false mailroom['enable'] = false @@ -59,120 +81,294 @@ Redis. ## Experimental Redis Sentinel support -> [Introduced][ce-1877] in GitLab 8.11. +> [Introduced][ce-1877] in GitLab 8.11, improved in 8.13. Since GitLab 8.11, you can configure a list of Redis Sentinel servers that will monitor a group of Redis servers to provide you with a standard failover support. -There is currently one exception to the Sentinel support: `mail_room`, the -component that processes incoming emails. It doesn't support Sentinel yet, but -we hope to integrate a future release that does support it. - To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as failing to configure it correctly can lead to data loss. +Redis Sentinel can handle the most important tasks in a HA environment to help +keep servers online with minimal to no downtime: + +- Monitors master and slave instances to see if they are available +- Promote a slave to master when the master fails. +- Demote a master to slave when failed master comes back online (to prevent + data-partitioning). +- Can be queried by clients to always connect to the correct master server. + +There is currently one exception to the Sentinel support: `mail_room`, the +component that processes incoming emails. It doesn't support Sentinel yet, but +we hope to integrate a future release that does support it soon. + The configuration consists of three parts: -- Redis setup -- Sentinel setup -- GitLab setup +- Setup Redis Master and Slave nodes +- Setup Sentinel nodes +- Setup GitLab + +> **IMPORTANT**: You need at least 3 independent machines: physical, or VMs +running into distinct physical machines. If you fail to provision the +machines in that specific way, any issue with the shared environment can +bring your entire setup down. Read carefully how to configure those components below. ### Redis setup -You must have at least 2 Redis servers: 1 Master, 1 or more Slaves. +You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to +be each in a independent machine (see explanation above). + They should be configured the same way and with similar server specs, as -in a failover situation, any Slave can be elected as the new Master by +in a failover situation, any `Slave` can be elected as the new `Master` by the Sentinel servers. -In a minimal setup, the only required change for the slaves in `redis.conf` -is the addition of a `slaveof` line pointing to the initial master. -You can increase the security by defining a `requirepass` configuration in -the master, and `masterauth` in slaves. +With Sentinel, you must define a password to protect the access as both +Sentinel instances and other redis instances should be able to talk to +each other over the network. ---- +You'll need to define both `requirepass` and `masterauth` in all +nodes because they can be re-configured at any time by the Sentinels +during a failover, and change it's status as `Master` or `Slave`. -**Configuring your own Redis server** +Initial `Slave` nodes will have in `redis.conf` an additional `slaveof` line +pointing to the initial `Master`. -1. Add to the slaves' `redis.conf`: +#### Source install - ```conf - # IP and port of the master Redis server - slaveof 10.10.10.10 6379 - ``` +**Master Redis instance** -1. Optionally, set up password authentication for increased security. - Add the following to master's `redis.conf`: +You need to make the following changes in `redis.conf`: - ```conf - # Optional password authentication for increased security - requirepass "" - ``` +1. Define a `bind` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access: -1. Then add this line to all the slave servers' `redis.conf`: + ```conf + # By default, if no "bind" configuration directive is specified, Redis listens + # for connections from all the network interfaces available on the server. + # It is possible to listen to just one or multiple selected interfaces using + # the "bind" configuration directive, followed by one or more IP addresses. + # + # Examples: + # + # bind 192.168.1.100 10.0.0.1 + # bind 127.0.0.1 ::1 + bind 0.0.0.0 # This will bind to all interfaces + ``` + +1. Define a `port` to force redis to listin on TCP so other machines can + connect to it: + + ```conf + # Accept connections on the specified port, default is 6379 (IANA #815344). + # If port 0 is specified Redis will not listen on a TCP socket. + port 6379 + ``` + +1. Set up password authentication (use the same password in all nodes) ```conf - masterauth "" + requirepass "redis-password-goes-here" + masterauth "redis-password-goes-here" ``` 1. Restart the Redis services for the changes to take effect. ---- +**Slave Redis instance** -**Using Redis via Omnibus** +1. Follow same instructions from master with the extra change in `redis.conf`: -1. Edit `/etc/gitlab/gitlab.rb` of a master Redis machine (usualy a single machine): + ```conf + # IP and port of the master Redis server + slaveof 10.10.10.10 6379 + ``` - ```ruby - ## Redis TCP support (will disable UNIX socket transport) - redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one - redis['port'] = 6379 +1. Restart the Redis services for the changes to take effect. - ## Master redis instance - redis['password'] = 'redis-password-goes-here' - ``` +#### Omnibus Install -1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines): +You need to install the omnibus package in 3 different and independent machines. +We will elect one as the initial `Master` and the other 2 as `Slaves`. - ```ruby - ## Redis TCP support (will disable UNIX socket transport) - redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one - redis['port'] = 6379 +If you are migrating from a single machine install, you may want to setup the +machines as Slaves, pointing to the original machine as `Master`, to migrate +the data first, and than switch to this setup. - ## Slave redis instance - redis['master'] = false - redis['master_ip'] = '10.10.10.10' # IP of master Redis server - redis['master_port'] = 6379 # Port of master Redis server - redis['master_password'] = "redis-password-goes-here" - ``` +To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: + +```ruby +redis['enable'] = false +``` + +**Master Redis instances** + +You need to make the following changes in `/etc/gitlab/gitlab.rb`: + +1. Define a `redis['bind']` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access. +1. Define a `redis['port']` to force redis to listin on TCP so other machines can + connect to it. +1. Set up password authentication with `redis['master_password']` (use the same + password in all nodes). -1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` +```ruby +## Redis TCP support (will disable UNIX socket transport) +redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis['port'] = 6379 +redis['requirepass'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' +``` + +Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` + +**Slave Redis instances** + +You need to make the same changes listed for the `Master` instance, +with an additional `Slave` section as in the example below: + +```ruby +## Redis TCP support (will disable UNIX socket transport) +redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis['port'] = 6379 +redis['requirepass'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' + +## Slave redis instance +redis['master'] = false +redis['master_ip'] = '10.10.10.10' # IP of master Redis server +redis['master_port'] = 6379 # Port of master Redis server +``` + +Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` + +#### Troubleshooting Replication + +You can check if everything is correct by connecting to each server using +`redis-cli` application, and sending the `INFO` command. + +If authentication was correctly defined, it should fail with: +`NOAUTH Authentication required` error. Try to authenticate with the +previous defined password with `AUTH redis-password-goes-here` and +try the `INFO` command again. + +Look for the `# Replication` section where you should see some important +information like the `role` of the server. + +When connected to a `master` redis, you will see the number of connected +`slaves`, and a list of each with connection details. + +When it's a `slave`, you will see details of the master connection and if +its `up` or `down`. --- Now that the Redis servers are all set up, let's configure the Sentinel servers. -### Sentinel setup +If you are not sure if your Redis servers are working and replicating +correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) +and fix it before proceeding with Sentinel setup. + +### Sentinel + +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine. You can install them in the same +machines you installed the other `3` Redis servers. + +This number is required for the consensus algorithm to be effective +in the case of a failure. You should always have and `odd` number +of Sentinel nodes provisioned. -We provide an automated way to setup and run the Sentinel daemon -with GitLab EE. +Here is a simple explanation on how Sentinel handles a failover: -See the instructions below how to setup it by yourself. +When a number of Sentinels (`quorum` value) agree the fact the `master` is +not reachable, the **majority** of the sentinels must elect a temporary +Sentinel `leader`, that will be responsible to start the failover proceedings. -Here is an example configuration file (`sentinel.conf`) for a Sentinel node: +As an example, for a cluster of `3` Sentinels, at least `2` must agree on a +`leader`. If you have total of `5` at least `3` must agree on the leader. + +The `quorum` is only used to detect failure, not to elect the `leader`. + +Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments) +also lists different network topologies and warns againts situations like +network partition and how it can affect the state of the HA solution. Make +sure you read it carefully and understand the implications in your current +setup. + +To make Sentinel setup easier, ee provide an [automated way to setup and run](#sentinel-setup-ee-only) +the Sentinel daemon with GitLab EE. + +#### Sentinel setup (Community Edition) + +For GitLab CE, you need to install, configure, execute and monitor Sentinel +by yourself. + +Here is an example configuration file (`sentinel.conf`) for a minimal Sentinel +node: ```conf -port 26379 -sentinel monitor gitlab-redis 10.0.0.1 6379 1 +bind 0.0.0.0 # bind to all interfaces or change to a specific IP +port 26379 # default sentinel port +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 sentinel down-after-milliseconds gitlab-redis 10000 sentinel config-epoch gitlab-redis 0 sentinel leader-epoch gitlab-redis 0 ``` +#### Sentinel setup (EE Only) + +To setup sentinel, you must edit `/etc/gitlab/gitlab.rb` file. +This is a minimal configuration required to run the daemon: + +```ruby +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +redis['master_port'] = 6379 # port of the initial master redis instance +redis['master_password'] = 'your-secure-password-here' # the same value defined in redis['password'] in the master instance + +sentinel['enable'] = true +# sentinel['port'] = 26379 + +## Quorum must reflect the amount of voting sentinels it take to start a failover. +sentinel['quorum'] = 2 + +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 + +# sentinel['failover_timeout'] = 60000 +``` + +When you install Sentinel in a separate machine, you need to control which +other services will be running in it. Take a look at the following variables +and enable or disable whenever it fits your strategy: + +```ruby +# Enabled Redis and Sentinel services +redis['enable'] = true +sentinel['enable'] = true + +# Disabled all other services +redis['enable'] = false +bootstrap['enable'] = false +nginx['enable'] = false +unicorn['enable'] = false +sidekiq['enable'] = false +postgresql['enable'] = false +gitlab_workhorse['enable'] = false +gitlab_rails['enable'] = false +mailroom['enable'] = false +``` + +Remember that enabling a new service may also require additional configuration +params (like `redis` for example). + --- The final part is to inform the main GitLab application server of the Redis @@ -243,7 +439,7 @@ or `gitlab-rails['redis_*']` in Omnibus): ```conf # sentinel.conf: -sentinel monitor gitlab-redis 10.10.10.10 6379 1 +sentinel monitor gitlab-redis 10.10.10.10 6379 2 sentinel down-after-milliseconds gitlab-redis 10000 sentinel config-epoch gitlab-redis 0 sentinel leader-epoch gitlab-redis 0 @@ -276,7 +472,7 @@ To make sure your configuration is correct: sudo gitlab-rails console # For source installations - sudo -u git rails console RAILS_ENV=production + sudo -u git rails console production ``` 1. Run in the console: -- cgit v1.2.1 From 3242ea6d40442f19c3932715521864ac5eb3700b Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Tue, 18 Oct 2016 19:45:57 +0200 Subject: Improved Redis HA and Sentinel documentation. --- doc/administration/high_availability/redis.md | 327 +++++++++++++++++--------- 1 file changed, 210 insertions(+), 117 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 840f5896bd7..29bf2c54a08 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,7 +1,7 @@ # Configuring Redis for GitLab HA You can choose to install and manage Redis yourself, or you can use the one -that comes bundled with GitLab Omnibus packages. +that comes bundled with Omnibus GitLab packages. > **Note:** Redis does not require authentication by default. See [Redis Security](http://redis.io/topics/security) documentation for more @@ -15,17 +15,22 @@ that comes bundled with GitLab Omnibus packages. - [Configure your own Redis server](#configure-your-own-redis-server) - [Configure Redis using Omnibus](#configure-redis-using-omnibus) - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) +- [Redis Sentinel support](#redis-sentinel-support) - [Redis setup](#redis-setup) - - [Source install](#source-install) - - [Omnibus Install](#omnibus-install) - - [Troubleshooting Replication](#troubleshooting-replication) - - [Sentinel](#sentinel) - - [Sentinel setup (Community Edition)](#sentinel-setup-community-edition) - - [Sentinel setup (EE Only)](#sentinel-setup-ee-only) + - [Existing single-machine installation](#existing-single-machine-installation) + - [Installation from source](#installation-from-source) + - [Omnibus packages](#omnibus-packages) + - [Configuring Sentinel](#configuring-sentinel) + - [How sentinel handles a failover](#how-sentinel-handles-a-failover) + - [Sentinel setup](#sentinel-setup) + - [Community Edition](#community-edition) + - [Enterprise Edition](#enterprise-edition) - [GitLab setup](#gitlab-setup) - - [Sentinel troubleshooting](#sentinel-troubleshooting) +- [Troubleshooting](#troubleshooting) + - [Redis replication](#redis-replication) + - [Sentinel](#sentinel) - [Omnibus install](#omnibus-install) - - [Source install](#source-install-1) + - [Source install](#source-install) @@ -41,7 +46,7 @@ If you don't want to bother setting up your own Redis server, you can use the one bundled with Omnibus. In this case, you should disable all services except Redis. -1. Download/install GitLab Omnibus using **steps 1 and 2** from +1. Download/install Omnibus GitLab using **steps 1 and 2** from [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other steps on the download page. 1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. @@ -70,7 +75,7 @@ Redis. redis['password'] = 'redis-password-goes-here' ``` -1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. +1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis. > **Note**: This `reconfigure` step will result in some errors. That's OK - don't be alarmed. @@ -81,7 +86,12 @@ Redis. ## Experimental Redis Sentinel support -> [Introduced][ce-1877] in GitLab 8.11, improved in 8.13. + > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. + Starting with 8.13, Redis Sentinel is no longer experimental. + If you used with versions `< 8.13` before, please check the updated + documentation below. + +## Redis Sentinel support Since GitLab 8.11, you can configure a list of Redis Sentinel servers that will monitor a group of Redis servers to provide you with a standard failover @@ -100,20 +110,19 @@ keep servers online with minimal to no downtime: data-partitioning). - Can be queried by clients to always connect to the correct master server. -There is currently one exception to the Sentinel support: `mail_room`, the -component that processes incoming emails. It doesn't support Sentinel yet, but -we hope to integrate a future release that does support it soon. - The configuration consists of three parts: - Setup Redis Master and Slave nodes - Setup Sentinel nodes - Setup GitLab -> **IMPORTANT**: You need at least 3 independent machines: physical, or VMs -running into distinct physical machines. If you fail to provision the -machines in that specific way, any issue with the shared environment can -bring your entire setup down. +### Prerequisites + +You need at least `3` independent machines: physical, or VMs running into +distinct physical machines. + +If you fail to provision the machines in that specific way, any issue with +the shared environment can bring your entire setup down. Read carefully how to configure those components below. @@ -131,15 +140,35 @@ Sentinel instances and other redis instances should be able to talk to each other over the network. You'll need to define both `requirepass` and `masterauth` in all -nodes because they can be re-configured at any time by the Sentinels -during a failover, and change it's status as `Master` or `Slave`. +nodes. At any time during a failover the Sentinels can reconfigure a node +and change it's status from `Master` to `Slave` and vice versa. -Initial `Slave` nodes will have in `redis.conf` an additional `slaveof` line +Initial `Slave` nodes require an additional `slaveof` setting in `redis.conf` pointing to the initial `Master`. -#### Source install +#### Existing single-machine installation + +If you already have a single-machine GitLab install running, you will need to +replicate from this machine first, before de-activating the Redis instance +inside it. + +Your single-machine install will be the initial `Master`, and the `3` others +should be configured as `Slave` pointing to this machine. + +After replication catchs-up, you will need to stop services in the +single-machine install, to rotate the `Master` to one of the new nodes. + +Make the required changes in configuration and restart the new nodes again. + +To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: -**Master Redis instance** +```ruby +redis['enable'] = false +``` + +#### Installation from source + +**Configuring Master Redis instance** You need to make the following changes in `redis.conf`: @@ -178,9 +207,9 @@ You need to make the following changes in `redis.conf`: 1. Restart the Redis services for the changes to take effect. -**Slave Redis instance** +**Configuring Slave Redis instance** -1. Follow same instructions from master with the extra change in `redis.conf`: +1. Follow same instructions from master, with the extra change in `redis.conf`: ```conf # IP and port of the master Redis server @@ -189,33 +218,24 @@ You need to make the following changes in `redis.conf`: 1. Restart the Redis services for the changes to take effect. -#### Omnibus Install - -You need to install the omnibus package in 3 different and independent machines. -We will elect one as the initial `Master` and the other 2 as `Slaves`. +#### Omnibus packages -If you are migrating from a single machine install, you may want to setup the -machines as Slaves, pointing to the original machine as `Master`, to migrate -the data first, and than switch to this setup. - -To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: - -```ruby -redis['enable'] = false -``` +You need to install the Omnibus GitLab package in `3` independent machines. -**Master Redis instances** +**Configuring Master Redis instance** -You need to make the following changes in `/etc/gitlab/gitlab.rb`: +You will need to configure the following: 1. Define a `redis['bind']` address pointing to a local IP that your other machines can reach you. If you really need to bind to an external acessible IP, make sure you add extra firewall rules to prevent unauthorized access. -1. Define a `redis['port']` to force redis to listin on TCP so other machines can - connect to it. -1. Set up password authentication with `redis['master_password']` (use the same +1. Define a `redis['port']` so redis can listen for TCP requests which will + allow other machines to connect to it. +1. Set up a password authentication with `redis['master_password']` (use the same password in all nodes). +In `/etc/gitlab/gitlab.rb`: + ```ruby ## Redis TCP support (will disable UNIX socket transport) redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one @@ -224,15 +244,14 @@ redis['requirepass'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' ``` -Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` +Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -**Slave Redis instances** +**Configuring Slave Redis instances** You need to make the same changes listed for the `Master` instance, with an additional `Slave` section as in the example below: ```ruby -## Redis TCP support (will disable UNIX socket transport) redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one redis['port'] = 6379 redis['requirepass'] = 'redis-password-goes-here' @@ -244,26 +263,7 @@ redis['master_ip'] = '10.10.10.10' # IP of master Redis server redis['master_port'] = 6379 # Port of master Redis server ``` -Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` - -#### Troubleshooting Replication - -You can check if everything is correct by connecting to each server using -`redis-cli` application, and sending the `INFO` command. - -If authentication was correctly defined, it should fail with: -`NOAUTH Authentication required` error. Try to authenticate with the -previous defined password with `AUTH redis-password-goes-here` and -try the `INFO` command again. - -Look for the `# Replication` section where you should see some important -information like the `role` of the server. - -When connected to a `master` redis, you will see the number of connected -`slaves`, and a list of each with connection details. - -When it's a `slave`, you will see details of the master connection and if -its `up` or `down`. +Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` --- @@ -274,26 +274,29 @@ If you are not sure if your Redis servers are working and replicating correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) and fix it before proceeding with Sentinel setup. -### Sentinel +### Configuring Sentinel You must have at least `3` Redis Sentinel servers, and they need to -be each in a independent machine. You can install them in the same -machines you installed the other `3` Redis servers. +be each in a independent machine. You can configure them in the same +machines where you've configured the other Redis servers. This number is required for the consensus algorithm to be effective -in the case of a failure. You should always have and `odd` number -of Sentinel nodes provisioned. +in the case of a failure. **You should always have and `odd` number +of Sentinel nodes provisioned**. + +#### How sentinel handles a failover -Here is a simple explanation on how Sentinel handles a failover: +If (`quorum` value of) Sentinels agree the fact the `master` is not reachable, +Sentinels will try to elect a temporary `Leader`. The **Majority** of the +Sentinels must agree to start a failover. -When a number of Sentinels (`quorum` value) agree the fact the `master` is -not reachable, the **majority** of the sentinels must elect a temporary -Sentinel `leader`, that will be responsible to start the failover proceedings. +If you don't have the **Majority** of the Sentinels online (for example if you +are under a network partitioning), a failover **will not be started**. -As an example, for a cluster of `3` Sentinels, at least `2` must agree on a -`leader`. If you have total of `5` at least `3` must agree on the leader. +For example, for a cluster of `3` Sentinels, at least `2` must agree on a +`Leader`. If you have total of `5` at least `3` must agree on a `Leader`. -The `quorum` is only used to detect failure, not to elect the `leader`. +The `quorum` is only used to detect failure, not to elect the `Leader`. Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments) also lists different network topologies and warns againts situations like @@ -301,16 +304,16 @@ network partition and how it can affect the state of the HA solution. Make sure you read it carefully and understand the implications in your current setup. -To make Sentinel setup easier, ee provide an [automated way to setup and run](#sentinel-setup-ee-only) -the Sentinel daemon with GitLab EE. +GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-setup-ee-only) the Sentinel daemon. -#### Sentinel setup (Community Edition) +#### Sentinel setup -For GitLab CE, you need to install, configure, execute and monitor Sentinel -by yourself. +##### Community Edition +With GitLab Community Edition, you need to install, configure, execute and +monitor Sentinel from source. Omnibus GitLab Community Edition package does +not support Sentinel configuration. -Here is an example configuration file (`sentinel.conf`) for a minimal Sentinel -node: +A minimal configuration file (`sentinel.conf`) should contain the following: ```conf bind 0.0.0.0 # bind to all interfaces or change to a specific IP @@ -322,35 +325,17 @@ sentinel config-epoch gitlab-redis 0 sentinel leader-epoch gitlab-redis 0 ``` -#### Sentinel setup (EE Only) +##### Enterprise Edition -To setup sentinel, you must edit `/etc/gitlab/gitlab.rb` file. -This is a minimal configuration required to run the daemon: +To setup sentinel, you edit `/etc/gitlab/gitlab.rb` file: ```ruby -redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node -redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance -redis['master_port'] = 6379 # port of the initial master redis instance -redis['master_password'] = 'your-secure-password-here' # the same value defined in redis['password'] in the master instance -sentinel['enable'] = true -# sentinel['port'] = 26379 +## When you install Sentinel in a separate machine, you need to control which +## other services will be running in it. Take a look at the following variables +## and enable or disable whenever it fits your strategy: -## Quorum must reflect the amount of voting sentinels it take to start a failover. -sentinel['quorum'] = 2 - -## Consider unresponsive server down after x amount of ms. -# sentinel['down_after_milliseconds'] = 10000 - -# sentinel['failover_timeout'] = 60000 -``` - -When you install Sentinel in a separate machine, you need to control which -other services will be running in it. Take a look at the following variables -and enable or disable whenever it fits your strategy: - -```ruby -# Enabled Redis and Sentinel services +## Enabled Redis and Sentinel services redis['enable'] = true sentinel['enable'] = true @@ -364,10 +349,53 @@ postgresql['enable'] = false gitlab_workhorse['enable'] = false gitlab_rails['enable'] = false mailroom['enable'] = false -``` -Remember that enabling a new service may also require additional configuration -params (like `redis` for example). +## Configure Redis +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +redis['master_port'] = 6379 # port of the initial master redis instance +redis['master_password'] = 'your-secure-password-here' # the same value defined in redis['password'] in the master instance + +## Configure Sentinel +# sentinel['port'] = 26379 # uncomment to change default port + +## Quorum must reflect the amount of voting sentinels it take to start a failover. +## Value must NOT be greater then the ammount of sentinels. +## +## The quorum can be used to tune Sentinel in two ways: +## 1. If a the quorum is set to a value smaller than the majority of Sentinels +## we deploy, we are basically making Sentinel more sensible to master failures, +## triggering a failover as soon as even just a minority of Sentinels is no longer +## able to talk with the master. +## 1. If a quorum is set to a value greater than the majority of Sentinels, we are +## making Sentinel able to failover only when there are a very large number (larger +## than majority) of well connected Sentinels which agree about the master being down.s +sentinel['quorum'] = 2 + +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 + +## Specifies the failover timeout in milliseconds. It is used in many ways: +## +## - The time needed to re-start a failover after a previous failover was +## already tried against the same master by a given Sentinel, is two +## times the failover timeout. +## +## - The time needed for a slave replicating to a wrong master according +## to a Sentinel current configuration, to be forced to replicate +## with the right master, is exactly the failover timeout (counting since +## the moment a Sentinel detected the misconfiguration). +## +## - The time needed to cancel a failover that is already in progress but +## did not produced any configuration change (SLAVEOF NO ONE yet not +## acknowledged by the promoted slave). +## +## - The maximum time a failover in progress waits for all the slaves to be +## reconfigured as slaves of the new master. However even after this time +## the slaves will be reconfigured by the Sentinels anyway, but not with +## the exact parallel-syncs progression as specified. +# sentinel['failover_timeout'] = 60000 +``` --- @@ -409,9 +437,74 @@ The following steps should be performed in the [GitLab application server](gitla 1. [Reconfigure] the GitLab for the changes to take effect. -### Sentinel troubleshooting +## Troubleshooting + +There are a lot of moving parts that needs to be taken care carefully +in order for the HA setup to work as expected. + +Before proceeding with the troubleshooting below, check your firewall +rules: +- Redis machines + - Accept TCP connection in `6379` + - Connect to the other Redis machines via TCP in `6379` +- Sentinel machines + - Accept TCP connection in `26379` + - Connect to other Sentinel machines via TCP in `26379` + - Connect to the Redis machines via TCP in `6379` + +### Redis replication + +You can check if everything is correct by connecting to each server using +`redis-cli` application, and sending the `INFO` command. + +If authentication was correctly defined, it should fail with: +`NOAUTH Authentication required` error. Try to authenticate with the +previous defined password with `AUTH redis-password-goes-here` and +try the `INFO` command again. + +Look for the `# Replication` section where you should see some important +information like the `role` of the server. + +When connected to a `master` redis, you will see the number of connected +`slaves`, and a list of each with connection details: + +``` +# Replication +role:master +connected_slaves:1 +slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1 +master_repl_offset:208037658 +repl_backlog_active:1 +repl_backlog_size:1048576 +repl_backlog_first_byte_offset:206989083 +repl_backlog_histlen:1048576 +``` + +When it's a `slave`, you will see details of the master connection and if +its `up` or `down`: + +``` +# Replication +role:slave +master_host:10.133.1.58 +master_port:6379 +master_link_status:up +master_last_io_seconds_ago:1 +master_sync_in_progress:0 +slave_repl_offset:208096498 +slave_priority:100 +slave_read_only:1 +connected_slaves:0 +master_repl_offset:0 +repl_backlog_active:0 +repl_backlog_size:1048576 +repl_backlog_first_byte_offset:0 +repl_backlog_histlen:0 +``` + +### Sentinel -#### Omnibus install +#### Omnibus GitLab If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related @@ -424,7 +517,7 @@ The way the redis connector `redis-rb` works with sentinel is a bit non-intuitive. We try to hide the complexity in omnibus, but it still requires a few extra configs. -#### Source install +#### Install from Source If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related -- cgit v1.2.1 From 0ca14544df6f8053a998009043290a4242d0362c Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Tue, 18 Oct 2016 19:52:19 +0200 Subject: Reduce the ammount of lines to disable services and update TOC --- doc/administration/high_availability/redis.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 29bf2c54a08..aa334a8a0f6 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -16,6 +16,7 @@ that comes bundled with Omnibus GitLab packages. - [Configure Redis using Omnibus](#configure-redis-using-omnibus) - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) - [Redis Sentinel support](#redis-sentinel-support) + - [Prerequisites](#prerequisites) - [Redis setup](#redis-setup) - [Existing single-machine installation](#existing-single-machine-installation) - [Installation from source](#installation-from-source) @@ -29,8 +30,8 @@ that comes bundled with Omnibus GitLab packages. - [Troubleshooting](#troubleshooting) - [Redis replication](#redis-replication) - [Sentinel](#sentinel) - - [Omnibus install](#omnibus-install) - - [Source install](#source-install) + - [Omnibus GitLab](#omnibus-gitlab) + - [Install from Source](#install-from-source) @@ -60,11 +61,8 @@ Redis. redis['enable'] = true bootstrap['enable'] = false nginx['enable'] = false - unicorn['enable'] = false - sidekiq['enable'] = false postgresql['enable'] = false gitlab_rails['enable'] = false - gitlab_workhorse['enable'] = false mailroom['enable'] = false # Redis configuration @@ -340,13 +338,9 @@ redis['enable'] = true sentinel['enable'] = true # Disabled all other services -redis['enable'] = false bootstrap['enable'] = false nginx['enable'] = false -unicorn['enable'] = false -sidekiq['enable'] = false postgresql['enable'] = false -gitlab_workhorse['enable'] = false gitlab_rails['enable'] = false mailroom['enable'] = false -- cgit v1.2.1 From 1dcbff1c4b07a0893f6d4a676895e8a01b8cfb14 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 20 Oct 2016 18:11:00 +0200 Subject: Few more fixes to Sentinel documentation to address MR feedback --- doc/administration/high_availability/redis.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index aa334a8a0f6..d60852814bb 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -63,13 +63,10 @@ Redis. nginx['enable'] = false postgresql['enable'] = false gitlab_rails['enable'] = false - mailroom['enable'] = false # Redis configuration redis['port'] = 6379 redis['bind'] = '0.0.0.0' - - # If you wish to use Redis authentication (recommended) redis['password'] = 'redis-password-goes-here' ``` @@ -85,8 +82,8 @@ Redis. ## Experimental Redis Sentinel support > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. - Starting with 8.13, Redis Sentinel is no longer experimental. - If you used with versions `< 8.13` before, please check the updated + Starting with 8.14, Redis Sentinel is no longer experimental. + If you used with versions `< 8.14` before, please check the updated documentation below. ## Redis Sentinel support @@ -279,7 +276,7 @@ be each in a independent machine. You can configure them in the same machines where you've configured the other Redis servers. This number is required for the consensus algorithm to be effective -in the case of a failure. **You should always have and `odd` number +in the case of a failure. **You should always have an `odd` number of Sentinel nodes provisioned**. #### How sentinel handles a failover @@ -400,7 +397,7 @@ master and the new sentinels servers. You can enable or disable sentinel support at any time in new or existing installations. From the GitLab application perspective, all it requires is -the correct credentials for the master Redis and for a few Sentinel nodes. +the correct credentials for the master Redis and for all Sentinel nodes. It doesn't require a list of all Sentinel nodes, as in case of a failure, the application will need to query only one of them. -- cgit v1.2.1 From 92e603727fd93ab862f7c6b5b46dfa2b1dd3a44e Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Sat, 29 Oct 2016 00:40:28 +0200 Subject: Fixed documentation and added redis/sentinel roles instruction --- doc/administration/high_availability/redis.md | 68 +++++++++++++++++++++------ 1 file changed, 54 insertions(+), 14 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index d60852814bb..bfad3047385 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -235,7 +235,7 @@ In `/etc/gitlab/gitlab.rb`: ## Redis TCP support (will disable UNIX socket transport) redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one redis['port'] = 6379 -redis['requirepass'] = 'redis-password-goes-here' +redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' ``` @@ -249,7 +249,7 @@ with an additional `Slave` section as in the example below: ```ruby redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one redis['port'] = 6379 -redis['requirepass'] = 'redis-password-goes-here' +redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' ## Slave redis instance @@ -327,27 +327,25 @@ To setup sentinel, you edit `/etc/gitlab/gitlab.rb` file: ```ruby ## When you install Sentinel in a separate machine, you need to control which -## other services will be running in it. Take a look at the following variables -## and enable or disable whenever it fits your strategy: +## other services will be running in it. +## We've simplified the choice using special "roles" settings: -## Enabled Redis and Sentinel services -redis['enable'] = true -sentinel['enable'] = true +## Enabled Sentinel and Redis Master services +redis_sentinel_role['enable'] = true +redis_master_role['enable'] = true -# Disabled all other services -bootstrap['enable'] = false -nginx['enable'] = false -postgresql['enable'] = false -gitlab_rails['enable'] = false -mailroom['enable'] = false +## Enabled Sentinel and Redis Slave services +redis_sentinel_role['enable'] = true +redis_master_role['enable'] = true ## Configure Redis redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance redis['master_port'] = 6379 # port of the initial master redis instance -redis['master_password'] = 'your-secure-password-here' # the same value defined in redis['password'] in the master instance +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel +sentinel['bind'] = '0.0.0.0' # or specify an IP to bind to a single one # sentinel['port'] = 26379 # uncomment to change default port ## Quorum must reflect the amount of voting sentinels it take to start a failover. @@ -388,6 +386,48 @@ sentinel['quorum'] = 2 # sentinel['failover_timeout'] = 60000 ``` +In the example above we've used `redis_sentinel_role` and `redis_master_role` +which simplify the ammount of configuration changes. + +If you want more control, here is what each one sets for you automatically +when enabled: + +```ruby +## Redis Sentinel Role +redis_sentinel_role['enable'] = true + +# When Sentinel Role is enabled, the following services are enabled/disabled: +sentinel['enable'] = true + +# This others are disabled: +redis['enable'] = false +bootstrap['enable'] = false +nginx['enable'] = false +postgresql['enable'] = false +gitlab_rails['enable'] = false +mailroom['enable'] = false + +## Redis master/slave Role: +redis_master_role['enable'] = true # enable only one of them +redis_slave_role['enable'] = true # enable only one of them + +# When Redis Master or Slave role are enabled, the following services are enabled/disabled: +# (Note that if redis and sentinel roles are combined both services will be enabled) + +# When Sentinel Role is enabled, the following services are enabled/disabled: +redis['enable'] = true + +# This others are disabled: +sentinel['enable'] = false +bootstrap['enable'] = false +nginx['enable'] = false +postgresql['enable'] = false +gitlab_rails['enable'] = false +mailroom['enable'] = false + +# Redis Slave role also change this setting from default 'true' to 'false': +redis['master'] = false +``` --- The final part is to inform the main GitLab application server of the Redis -- cgit v1.2.1 From 494c2785fdc6a2f67edf20cbfc2106ffdba3ef28 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Tue, 8 Nov 2016 06:24:43 +0100 Subject: Fixed some documentation and moved Source install specific to other file. --- doc/administration/high_availability/redis.md | 135 +--------- .../high_availability/redis_source.md | 286 +++++++++++++++++++++ 2 files changed, 300 insertions(+), 121 deletions(-) create mode 100644 doc/administration/high_availability/redis_source.md (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index bfad3047385..f49bf1d4ab5 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -19,7 +19,6 @@ that comes bundled with Omnibus GitLab packages. - [Prerequisites](#prerequisites) - [Redis setup](#redis-setup) - [Existing single-machine installation](#existing-single-machine-installation) - - [Installation from source](#installation-from-source) - [Omnibus packages](#omnibus-packages) - [Configuring Sentinel](#configuring-sentinel) - [How sentinel handles a failover](#how-sentinel-handles-a-failover) @@ -31,7 +30,6 @@ that comes bundled with Omnibus GitLab packages. - [Redis replication](#redis-replication) - [Sentinel](#sentinel) - [Omnibus GitLab](#omnibus-gitlab) - - [Install from Source](#install-from-source) @@ -161,58 +159,6 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: redis['enable'] = false ``` -#### Installation from source - -**Configuring Master Redis instance** - -You need to make the following changes in `redis.conf`: - -1. Define a `bind` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external acessible IP, make - sure you add extra firewall rules to prevent unauthorized access: - - ```conf - # By default, if no "bind" configuration directive is specified, Redis listens - # for connections from all the network interfaces available on the server. - # It is possible to listen to just one or multiple selected interfaces using - # the "bind" configuration directive, followed by one or more IP addresses. - # - # Examples: - # - # bind 192.168.1.100 10.0.0.1 - # bind 127.0.0.1 ::1 - bind 0.0.0.0 # This will bind to all interfaces - ``` - -1. Define a `port` to force redis to listin on TCP so other machines can - connect to it: - - ```conf - # Accept connections on the specified port, default is 6379 (IANA #815344). - # If port 0 is specified Redis will not listen on a TCP socket. - port 6379 - ``` - -1. Set up password authentication (use the same password in all nodes) - - ```conf - requirepass "redis-password-goes-here" - masterauth "redis-password-goes-here" - ``` - -1. Restart the Redis services for the changes to take effect. - -**Configuring Slave Redis instance** - -1. Follow same instructions from master, with the extra change in `redis.conf`: - - ```conf - # IP and port of the master Redis server - slaveof 10.10.10.10 6379 - ``` - -1. Restart the Redis services for the changes to take effect. - #### Omnibus packages You need to install the Omnibus GitLab package in `3` independent machines. @@ -304,25 +250,16 @@ GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-se #### Sentinel setup ##### Community Edition + With GitLab Community Edition, you need to install, configure, execute and monitor Sentinel from source. Omnibus GitLab Community Edition package does not support Sentinel configuration. -A minimal configuration file (`sentinel.conf`) should contain the following: - -```conf -bind 0.0.0.0 # bind to all interfaces or change to a specific IP -port 26379 # default sentinel port -sentinel auth-pass gitlab-redis redis-password-goes-here -sentinel monitor gitlab-redis 10.0.0.1 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel config-epoch gitlab-redis 0 -sentinel leader-epoch gitlab-redis 0 -``` +See documentation for Source Install [here](redis_source.md). ##### Enterprise Edition -To setup sentinel, you edit `/etc/gitlab/gitlab.rb` file: +To setup sentinel, edit `/etc/gitlab/gitlab.rb` file: ```ruby @@ -336,7 +273,7 @@ redis_master_role['enable'] = true ## Enabled Sentinel and Redis Slave services redis_sentinel_role['enable'] = true -redis_master_role['enable'] = true +redis_slave_role['enable'] = true ## Configure Redis redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node @@ -345,7 +282,7 @@ redis['master_port'] = 6379 # port of the initial master redis instance redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel -sentinel['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one # sentinel['port'] = 26379 # uncomment to change default port ## Quorum must reflect the amount of voting sentinels it take to start a failover. @@ -435,24 +372,16 @@ master and the new sentinels servers. ### GitLab setup -You can enable or disable sentinel support at any time in new or existing +You can enable or disable Sentinel support at any time in new or existing installations. From the GitLab application perspective, all it requires is -the correct credentials for the master Redis and for all Sentinel nodes. +the correct credentials for the Sentinel nodes. -It doesn't require a list of all Sentinel nodes, as in case of a failure, -the application will need to query only one of them. +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at one of listed ones. >**Note:** -The following steps should be performed in the [GitLab application server](gitlab.md). - -**For source based installations** - -1. Edit `/home/git/gitlab/config/resque.yml` following the example in - `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels - line, changing to the correct server credentials. -1. Restart GitLab for the changes to take effect. - -**For Omnibus installations** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA setup. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: @@ -466,7 +395,7 @@ The following steps should be performed in the [GitLab application server](gitla ] ``` -1. [Reconfigure] the GitLab for the changes to take effect. +1. [Reconfigure] GitLab for the changes to take effect. ## Troubleshooting @@ -548,42 +477,6 @@ The way the redis connector `redis-rb` works with sentinel is a bit non-intuitive. We try to hide the complexity in omnibus, but it still requires a few extra configs. -#### Install from Source - -If you get an error like: `Redis::CannotConnectError: No sentinels available.`, -there may be something wrong with your configuration files or it can be related -to [this issue][gh-531]. - -It's a bit non-intuitive the way you have to config `resque.yml` and -`sentinel.conf`, otherwise `redis-rb` will not work properly. - -The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) -**must** be used as the hostname in GitLab (`resque.yml` for source installations -or `gitlab-rails['redis_*']` in Omnibus): - -```conf -# sentinel.conf: -sentinel monitor gitlab-redis 10.10.10.10 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel config-epoch gitlab-redis 0 -sentinel leader-epoch gitlab-redis 0 -``` - -```yaml -# resque.yaml -production: - url: redis://:myredispassword@gitlab-redis/ - sentinels: - - - host: slave1.example.com # or use ip - port: 26380 # point to sentinel, not to redis port - - - host: slave2.exampl.com # or use ip - port: 26381 # point to sentinel, not to redis port -``` - -When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) - --- To make sure your configuration is correct: @@ -611,8 +504,8 @@ To make sure your configuration is correct: 1. To simulate a failover on master Redis, SSH into the Redis server and run: ```bash - # port must match your master redis port - redis-cli -h localhost -p 6379 DEBUG sleep 60 + # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one + redis-cli -h localhost -p 6379 DEBUG sleep 20 ``` 1. Then back in the Rails console from the first step, run: diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md new file mode 100644 index 00000000000..7e8c8c2b4b9 --- /dev/null +++ b/doc/administration/high_availability/redis_source.md @@ -0,0 +1,286 @@ +# Configuring Redis for GitLab HA (Source Install) + +We highly recommend that you use Omnibus GitLab packages, as we can optimize +required packages specifically for GitLab, and we will take care of upgrading +to the latest supported version. + +If you are building packages for a specific distro, or trying to build some +internal automation, you can check this documentation to learn about the +minimal setup, required changes, etc. + +If you want to see the documentation for Omnibus GitLab Install, please [read it here](redis.md). + + + +**Table of Contents** + +- [Configure your own Redis server](#configure-your-own-redis-server) + - [Configuring Master Redis instance](#configuring-master-redis-instance) + - [Configuring Slave Redis instances](#configuring-slave-redis-instances) + - [Configuring Redis Sentinel instances](#configuring-redis-sentinel-instances) +- [GitLab setup](#gitlab-setup) +- [Example configurations](#example-configurations) + - [Configuring Redis Master](#configuring-redis-master) + - [Configuring Redis Slaves](#configuring-redis-slaves) + - [Configuring Redis Sentinel](#configuring-redis-sentinel) +- [Troubleshooting](#troubleshooting) + + + +## Configure your own Redis server + +Redis server must be configured to use TCP connection instead of socket, +and since Redis `3.2`, you must define a password to receive external +connections (`requirepass`). + +You will also need to define equal password for slave password definition +(`masterauth`), in the same instance, if you are using Redis with Sentinel. + +To configure Redis to use TCP connection you need to define both +`bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the +ip of the desired interface (for ex. one from an internal network). + + +### Configuring Master Redis instance + +You need to make the following changes in `redis.conf`: + +1. Define a `bind` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access: + +1. Define a `port` to force redis to listen on TCP so other machines can + connect to it (default port is `6379`). + +1. Set up password authentication (use the same password in all nodes). + The password should be defined equal for both `requirepass` and `masterauth` + when setting up Redis to use with Sentinel. + +1. Restart the Redis services for the changes to take effect. + +See [example configuration](#configuring-redis-master) below. + +### Configuring Slave Redis instances + +1. Follow same instructions for Redis Master + +1. Define `slaveof` pointing to the Redis master instance with **IP** and **port**. + +1. Restart the Redis services for the changes to take effect. + +See [example configuration](#configuring-redis-slaves) below. + +### Configuring Redis Sentinel instances + +Sentinel is a special type of Redis server. It inherits most of the basic +configuration options you can define in `redis.conf`, with specific ones +starting with `sentinel` prefix. + +You will need to define the initial configs to enable connectivity: + +1. Define a `bind` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access: + +1. Define a `port` to force sentinel to listen on TCP so other machines can + connect to it (default port is `26379`). + +And the sentinel specific ones: + +1. Define with `sentinel auth-pass` the same shared password you have + defined for both Redis **Master** and **Slaves** instances. + +1. Define with `sentinel monitor` the **IP** and **port** of the Redis + **Master** node, and the **quorum** required to start a failover. + If you need more information to understand about quorum, please + read the detailed explanation in the [HA documentation for Omnibus Installs](redis.md). + +1. Define with `sentinel down-after-milliseconds` the ammount in `ms` of time + that an unresponsive server will be considered down. + +1. Define a value for `sentinel failover_timeout` in `ms`. This has multiple + meanings: + + * The time needed to re-start a failover after a previous failover was + already tried against the same master by a given Sentinel, is two + times the failover timeout. + + * The time needed for a slave replicating to a wrong master according + to a Sentinel current configuration, to be forced to replicate + with the right master, is exactly the failover timeout (counting since + the moment a Sentinel detected the misconfiguration). + + * The time needed to cancel a failover that is already in progress but + did not produced any configuration change (SLAVEOF NO ONE yet not + acknowledged by the promoted slave). + + * The maximum time a failover in progress waits for all the slaves to be + reconfigured as slaves of the new master. However even after this time + the slaves will be reconfigured by the Sentinels anyway, but not with + the exact parallel-syncs progression as specified. + +See [example configuration](#configuring-redis-sentinel) below. + +## GitLab setup + +You can enable or disable Sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Sentinel nodes. + +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at one of listed ones. + +>**Note:** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA setup. + +1. Edit `/home/git/gitlab/config/resque.yml` following the example in + `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels + lines, pointing to the correct server credentials. + +1. Restart GitLab for the changes to take effect. + +## Example configurations + +In this example we consider that all servers have an internal network +interface with IPs in the `10.0.0.x` range, and that they can connect +to each other using these IPs. + +In a real world usage, you would also setup firewall rules to prevent +unauthorized access from other machines, and block traffic from the +outside (Internet). + +We will use the same `3` nodes with **Redis** + **Sentinel** topology +discussed in the [Configuring Redis for GitLab HA](redis.md) documentation. + +Here is a list and description of each **machine** and the assined **ip**: + +* `10.0.0.1`: Redis Master + Sentinel 1 +* `10.0.0.2`: Redis Slave 1 + Sentinel 2 +* `10.0.0.2`: Redis Slave 2 + Sentinel 3 + +Please note that after the initial configuration, if a failover is initiated +by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** +will change permanently (including in `redis.conf`) from one node to the other, +until a new failover is initiated again. + +The same thing will happen with `sentinel.conf` that will be overriten after the +initial execution, after any new sentinel node starts watching the **Master**, +or a failover promotes a different **Master** node. + +### Configuring Redis Master + +`redis.conf`: + +```conf +bind 10.0.0.1 +port 6379 +requirepass redis-password-goes-here +masterauth redis-password-goes-here +``` + +### Configuring Redis Slaves + +**Slave 1 - `redis.conf`:** + +```conf +bind 10.0.0.2 +port 6379 +requirepass redis-password-goes-here +masterauth redis-password-goes-here + +# IP and port of the master Redis server +slaveof 10.0.0.1 6379 +``` + +**Slave 2 - `redis.conf`:** + +```conf +bind 10.0.0.3 +port 6379 +requirepass redis-password-goes-here +masterauth redis-password-goes-here + +# IP and port of the master Redis server +slaveof 10.0.0.1 6379 +``` + +### Configuring Redis Sentinel + +For this example, **Sentinel 1** will be configured in the same machine as the +**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the +**Slave 1** and **Slave 2** respectively. + +Sentinel 1 - `sentinel.conf` + +```conf +bind 10.0.0.1 +port 26379 +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel failover_timeout 30000 +``` + +Sentinel 2 - `sentinel.conf` + +```conf +bind 10.0.0.2 +port 26379 +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel failover_timeout 30000 +``` + +Sentinel 3 - `sentinel.conf` + +```conf +bind 10.0.0.3 +port 26379 +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel failover_timeout 30000 +``` + +## Troubleshooting + +We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained in the documentation for Omnibus +Install. Here we will list only the things that are specific to a **Source** install. + +If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this issue][gh-531]. + +It's a bit non-intuitive the way you have to config `resque.yml` and +`sentinel.conf`, otherwise `redis-rb` will not work properly. + +The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) +**must** be used as the hostname in GitLab (`resque.yml`): + +```conf +# sentinel.conf: +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 +``` + +```yaml +# resque.yaml +production: + url: redis://:myredispassword@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port +``` + +When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) -- cgit v1.2.1 From 1afab9eb79c87f32c7b899e58bc9a0ea8a113594 Mon Sep 17 00:00:00 2001 From: Toon Claes Date: Wed, 21 Sep 2016 16:15:12 +0200 Subject: Add button to delete all merged branches It adds a button to the branches page that the user can use to delete all the branches that are already merged. This can be used to clean up all the branches that were forgotten to delete while merging MRs. Fixes #21076. --- doc/api/branches.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) (limited to 'doc') diff --git a/doc/api/branches.md b/doc/api/branches.md index 0b5f7778fc7..f68eeb9f86b 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -240,3 +240,21 @@ Example response: "branch_name": "newbranch" } ``` + +## Delete merged branches + +Will delete all branches that are merged into the project's default branch. + +``` +DELETE /projects/:id/repository/merged_branches +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | + +It returns `200` to indicate deletion of all merged branches was started. + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/5/repository/merged_branches" +``` -- cgit v1.2.1 From 1e4f86744627fbf0d0190c7d76f30cd77480db37 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 10 Nov 2016 07:07:30 +0100 Subject: Remade documentation for Redis HA with Omnibus --- doc/administration/high_availability/redis.md | 648 +++++++++++++++------ .../high_availability/redis_source.md | 14 +- 2 files changed, 480 insertions(+), 182 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index f49bf1d4ab5..6b16f9e791f 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,9 +1,14 @@ # Configuring Redis for GitLab HA -You can choose to install and manage Redis yourself, or you can use the one +High Availability with Redis is possible using a **Master** x **Slave** +topology with **Sentinel** service to watch and automatically start +failover proceedings. + +You can choose to install and manage Redis and Sentinel yourself, use +a hosted, managed clouse solution or you can use or you can use the one that comes bundled with Omnibus GitLab packages. -> **Note:** Redis does not require authentication by default. See +> **Note:** Redis requires authentication for High Availability. See [Redis Security](http://redis.io/topics/security) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. @@ -12,20 +17,26 @@ that comes bundled with Omnibus GitLab packages. **Table of Contents** -- [Configure your own Redis server](#configure-your-own-redis-server) -- [Configure Redis using Omnibus](#configure-redis-using-omnibus) -- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) -- [Redis Sentinel support](#redis-sentinel-support) +- [Using an external Redis server](#using-an-external-redis-server) +- [High Availability with Sentinel](#high-availability-with-sentinel) - [Prerequisites](#prerequisites) - [Redis setup](#redis-setup) - - [Existing single-machine installation](#existing-single-machine-installation) - - [Omnibus packages](#omnibus-packages) - - [Configuring Sentinel](#configuring-sentinel) - - [How sentinel handles a failover](#how-sentinel-handles-a-failover) - - [Sentinel setup](#sentinel-setup) + - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) + - [Sentinel setup](#sentinel-setup) + - [Recommended setup](#recommended-setup) +- [Configuring instances using Omnibus](#configuring-instances-using-omnibus) + - [Existing single-machine installation](#existing-single-machine-installation) + - [Configuring Master Redis instance](#configuring-master-redis-instance) + - [Configuring Slave Redis instances](#configuring-slave-redis-instances) + - [Configuring Sentinel instances](#configuring-sentinel-instances) - [Community Edition](#community-edition) - [Enterprise Edition](#enterprise-edition) - [GitLab setup](#gitlab-setup) +- [Example Configurations](#example-configurations) + - [Configuration for Redis Master](#configuration-for-redis-master) + - [Configuration for Redis Slave](#configuration-for-redis-slave) + - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) + - [Control running services](#control-running-services) - [Troubleshooting](#troubleshooting) - [Redis replication](#redis-replication) - [Sentinel](#sentinel) @@ -33,75 +44,57 @@ that comes bundled with Omnibus GitLab packages. -## Configure your own Redis server +## Using an external Redis server If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers a managed ElastiCache service that runs Redis. -## Configure Redis using Omnibus +Managed services can provide High Availability using their own proprietary +technology and provide a transparent proxy, which means that GitLab doesn't +need any additional change, or will use Sentinel and manage it for you. -If you don't want to bother setting up your own Redis server, you can use the -one bundled with Omnibus. In this case, you should disable all services except -Redis. +If your provider, uses Sentinel method, see [GitLab Setup](#gitlab-setup) +to understant where you need to provide the list of servers and credentials. -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL: +If you want to setup Redis by yourself, without using Omnibus, you can +read our documentation: [Configuring Redis for GitLab HA (Source Install)](redis_source.md) - ```ruby - external_url 'https://gitlab.example.com' - - # Disable all services except Redis - redis['enable'] = true - bootstrap['enable'] = false - nginx['enable'] = false - postgresql['enable'] = false - gitlab_rails['enable'] = false - - # Redis configuration - redis['port'] = 6379 - redis['bind'] = '0.0.0.0' - redis['password'] = 'redis-password-goes-here' - ``` +## High Availability with Sentinel -1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis. +> Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that +will monitor a group of Redis servers to provide failover support. - > **Note**: This `reconfigure` step will result in some errors. - That's OK - don't be alarmed. +> With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and +improved the way you use and configure it. -1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. +High Availability with Redis requires a few things: -## Experimental Redis Sentinel support +- Multiple Redis instances +- Run Redis in a **Master** x **Slave** topology +- Multiple Sentinel instances +- Application support and visiblity to all Sentinel and Redis instances - > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. - Starting with 8.14, Redis Sentinel is no longer experimental. - If you used with versions `< 8.14` before, please check the updated - documentation below. +Redis Sentinel can handle the most important tasks in a HA environment to help +keep servers online with minimal to no downtime: -## Redis Sentinel support +- Monitors **Master** and **Slaves** instances to see if they are available +- Promote a **Slave** to **Master** when the **Master** fails. +- Demote a **Master** to **Slave** when failed **Master** comes back online (to prevent + data-partitioning). +- Can be queried by clients to always connect to the current **Master** server. -Since GitLab 8.11, you can configure a list of Redis Sentinel servers that -will monitor a group of Redis servers to provide you with a standard failover -support. +When a **Master** fails to respond, it's the client responsability to handle timeout +and reconnect (querying a **Sentinel** for a new **Master**). To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as -failing to configure it correctly can lead to data loss. +failing to configure it correctly can lead to data loss, or can bring your +whole cluster down, invalidating the failover effort. -Redis Sentinel can handle the most important tasks in a HA environment to help -keep servers online with minimal to no downtime: - -- Monitors master and slave instances to see if they are available -- Promote a slave to master when the master fails. -- Demote a master to slave when failed master comes back online (to prevent - data-partitioning). -- Can be queried by clients to always connect to the correct master server. +This documentation will provide you with a minimal and a recommended topology +that can resist to some levels of failure. Usually the more Redis and Sentinel +instances you have provisioned, the better will be your availability. The configuration consists of three parts: @@ -112,44 +105,191 @@ The configuration consists of three parts: ### Prerequisites You need at least `3` independent machines: physical, or VMs running into -distinct physical machines. +distinct physical machines. They must be believed to fail in an +independent way. If you fail to provision the machines in that specific way, any issue with the shared environment can bring your entire setup down. -Read carefully how to configure those components below. +You also need to take in consideration the underlying network topology, +making sure you have redundant connectivity between Redis / Sentinel and +GitLab instances, otherwhise the networks will become a single point of +failure. + +Read carefully how to configure the components below. ### Redis setup -You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to -be each in a independent machine (see explanation above). +You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they +need to be each in a independent machine (see explanation above). -They should be configured the same way and with similar server specs, as -in a failover situation, any `Slave` can be elected as the new `Master` by +You can have additional Redis nodes, that will help survive a situation +where more nodes goes down. Whenever there is only `2` nodes online, a failover +will not be initiated. + +As an example, if you have `6` Redis nodes, a maximum of `3` can be +simultaneously down. + +Please note that there are different requirements for Sentinel nodes. +If you host them in the same Redis machines, you may need to take +that restrictions into consideration when calculating the ammount of +nodes to be provisioned. See [Sentinel setup](#sentinel-setup) +documentation for more information. + +All Redis nodes should be configured the same way and with similar server specs, as +in a failover situation, any **Slave** can be promoted as the new **Master** by the Sentinel servers. -With Sentinel, you must define a password to protect the access as both -Sentinel instances and other redis instances should be able to talk to +The replication requires authentication, so you need to define a password to +protect all Redis nodes and the Sentinels. They will all share the same +password, and all instances must be able to talk to each other over the network. -You'll need to define both `requirepass` and `masterauth` in all -nodes. At any time during a failover the Sentinels can reconfigure a node -and change it's status from `Master` to `Slave` and vice versa. +Redis nodes will need the same password defined in `redis['password']` and +`redis['master_password']`, no matter if **Master** or **Slave**. At any time +during a failover the Sentinels can reconfigure a node and change it's status +from **Master** to **Slave** and vice versa. + +Initial **Slave** nodes requires `redis['master']` defined to `false` and +`redis['master_ip']` pointing to the initial **Master**. If you use the +simplified configuration by enabling `redis_slave_role['enable']`, you +just need to fill in the `redis['master_ip']`. + +This values doesn't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the Sentinels, and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +### Experimental Redis Sentinel support + + > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. + Starting with 8.14, Redis Sentinel is no longer experimental. + If you used with versions `< 8.14` before, please check the updated + documentation here. + +### Sentinel setup + +Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel +detects that a Redis node is not responding, it will announce that to the +other sentinels. You have to reach the **quorum**, the minimum ammount of +sentinels that agrees that a node is down, to be able to start a failover. + +Whenver the **quorum** is met, you need the **majority** of all known +Sentinel nodes to be available and reachable, to elect the Sentinel **leader** +who will take all the decisions to restore the service availability by: + +- Promoting a new **Master** +- Reconfiguring the other **Slaves** and make them point to the new **Master** +- Announce the new **Master** to every other Sentinel peer +- Reconfigure the old **Master** and demote to **Slave** when it comes back online + +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine (that are believed to fail independently). + +You can configure them in the same machines where you've configured the other +Redis servers, but understand that if a whole node goes down, you loose both +a Sentinel and a Redis instance. -Initial `Slave` nodes require an additional `slaveof` setting in `redis.conf` -pointing to the initial `Master`. +The number of sentinels should ideally always be an **odd** number, for the +consensus algorithm to be effective in the case of a failure. -#### Existing single-machine installation +In a `3` nodes topology, you can only afford `1` Sentinel node going down. +Whenever the **majority** of the Sentinels goes down, the network partition +protection prevents destructive actions and a failover **will not be started**. + +Here are some examples: + +- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin. +- With `7` sentinels, a maximum of `3` nodes can go down. + +The **Leader** election can sometimes fail the voting round when **consensus**, +is not achieved (see the odd number of nodes requirement above). In that case, +a new attempt will be made after the amount of time defined in +`sentinel['failover_timeout']` (in milliseconds). + +The `failover_time` variable have a lot of different usages, according to +official documentation: + +- The time needed to re-start a failover after a previous failover was + already tried against the same master by a given Sentinel, is two + times the failover timeout. + +- The time needed for a slave replicating to a wrong master according + to a Sentinel current configuration, to be forced to replicate + with the right master, is exactly the failover timeout (counting since + the moment a Sentinel detected the misconfiguration). + +- The time needed to cancel a failover that is already in progress but + did not produced any configuration change (SLAVEOF NO ONE yet not + acknowledged by the promoted slave). + +- The maximum time a failover in progress waits for all the slaves to be + reconfigured as slaves of the new master. However even after this time + the slaves will be reconfigured by the Sentinels anyway, but not with + the exact parallel-syncs progression as specified. + +### Recommended setup + +For a minimal setup, you will install the Omnibus GitLab package in `3` +independent machines, both with **Redis** and **Sentinel**: + +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel + +Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) +before, to understant how and why the ammount of nodes came from. + +For a recommended setup, that can resist more failures, you will install +the Omnibus GitLab package in `5` independent machines, both with +**Redis** and **Sentinel**: + +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel + +## Configuring instances using Omnibus + +This is a summary of what are we going to do: + +1. Provision the required number of instances specified previously + - You can opt to install Redis and Sentinel in the same machine or each in + independent ones. + - Don't install Redis and Sentinel in the same machines your GitLab instance + is running on. + - All machines must be able to talk to each other and accept incomming + connection over Redis (`6379`) and Sentinel (`26379`) ports. + - GitLab machines must be able to access these machines and with the same + permissions. + - Protected them from indiscriminated access from external networks (Internet), + to harden the security. + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/downloads) in each node. + - Do not complete other steps on the download page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + +1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations + from running on upgrade. Only the primary GitLab application server should + handle migrations. + +1. Create/edit `/etc/gitlab/gitlab.rb` and make the changes based on the + [Example Configurations](#example-configurations). + +### Existing single-machine installation If you already have a single-machine GitLab install running, you will need to replicate from this machine first, before de-activating the Redis instance inside it. -Your single-machine install will be the initial `Master`, and the `3` others -should be configured as `Slave` pointing to this machine. +Your single-machine install will be the initial **Master**, and the `3` others +should be configured as **Slave** pointing to this machine. After replication catchs-up, you will need to stop services in the -single-machine install, to rotate the `Master` to one of the new nodes. +single-machine install, to rotate the **Master** to one of the new nodes. Make the required changes in configuration and restart the new nodes again. @@ -159,130 +299,317 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: redis['enable'] = false ``` -#### Omnibus packages +If you fail to replicate first, you may loose data (unprocessed background jobs). + +### Configuring Master Redis instance + +You will need to configure the following in `/etc/gitlab/gitlab.rb`: + +1. Define `redis_master_role['enable']` to `true`, to disable other services + in the machine (you can still enable Sentinel) + +1. Define a `redis['bind']` address pointing to a local IP that your other machines + can reach you. + - If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access. + - You can also set bind to `0.0.0.0` which listen in all interfaces. + +1. Define a `redis['port']` so redis can listen for TCP requests which will + allow other machines to connect to it. + +1. Set up a password authentication with `redis['password']` and + `redis['master_password']` (use the same password in all nodes). + +Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -You need to install the Omnibus GitLab package in `3` independent machines. +### Configuring Slave Redis instances -**Configuring Master Redis instance** +You will need to configure the following in `/etc/gitlab/gitlab.rb`: -You will need to configure the following: +1. Define `redis_slaves_role['enable']` to `true`, to disable other services + in the machine (you can still enable Sentinel) + - This will also set automatically `redis['master'] = false`. 1. Define a `redis['bind']` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external acessible IP, make + can reach you. + - If you really need to bind to an external acessible IP, make sure you add extra firewall rules to prevent unauthorized access. + - You can also set bind to `0.0.0.0` which listen in all interfaces. + 1. Define a `redis['port']` so redis can listen for TCP requests which will allow other machines to connect to it. -1. Set up a password authentication with `redis['master_password']` (use the same - password in all nodes). + +1. Set up a password authentication with `redis['password']` and + `redis['master_password']` (use the same password in all nodes). + +1. Define `redis['master_ip']` with the IP of the **Master** Redis. + +1. Define `redis['master_port']` with the port of the **Master** Redis (default to `6379`). + +### Configuring Sentinel instances + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. + +If you are not sure if your Redis servers are working and replicating +correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) +and fix it before proceeding with Sentinel setup. + +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine. You can configure them in the same +machines where you've configured the other Redis servers. + +##### Community Edition + +With GitLab Community Edition, you need to install, configure, execute and +monitor Sentinel from source. Omnibus GitLab Community Edition package does +not support Sentinel configuration. + +See [documentation for Source Install](redis_source.md). + +##### Enterprise Edition + +With GitLab Enterprise Edition, you can use Omnibus package to setup multiple +machines with Sentinel daemon. + +See [example configuration](#configuration-for-sentinel-ee-only) below. + +### GitLab setup + +The final part is to inform the main GitLab application server of the Redis +Sentinels servers and authentication credentials. + +You can enable or disable Sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Sentinel nodes. + +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at least one of listeds. + +>**Note:** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA setup. + +1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: + + - `redis['master_name']` - this is the `master-group-name` from sentinel (default: `gitlab-redis`) + - `redis['master_password']` - the same password you've defined before for Redis and Sentinels + - `gitlab_rails['redis_sentinels']` - a list of sentinels with `host` and `port` + +1. [Reconfigure] GitLab for the changes to take effect. + +See [example configuration](#configuration-for-gitlab) below. + +## Example Configurations + +In this example we consider that all servers have an internal network +interface with IPs in the `10.0.0.x` range, and that they can connect +to each other using these IPs. + +In a real world usage, you would also setup firewall rules to prevent +unauthorized access from other machines, and block traffic from the +outside (Internet). + +We will use the same `3` nodes with **Redis** + **Sentinel** topology +discussed in [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) +documentation. + +Here is a list and description of each **machine** and the assigned **IP**: + +* `10.0.0.1`: Redis Master + Sentinel 1 +* `10.0.0.2`: Redis Slave 1 + Sentinel 2 +* `10.0.0.2`: Redis Slave 2 + Sentinel 3 + +Please note that after the initial configuration, if a failover is initiated +by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** +will change permanently (including in `redis.conf`) from one node to the other, +until a new failover is initiated again. + +The same thing will happen with `sentinel.conf` that will be overriten after the +initial execution, after any new sentinel node starts watching the **Master**, +or a failover promotes a different **Master** node. + +### Configuration for Redis Master + +**Example configation for Redis Master:** In `/etc/gitlab/gitlab.rb`: ```ruby -## Redis TCP support (will disable UNIX socket transport) -redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis_master_role['enable'] = true + +redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' ``` - Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -**Configuring Slave Redis instances** +### Configuration for Redis Slave + +**Example configation for Slave 1:** -You need to make the same changes listed for the `Master` instance, -with an additional `Slave` section as in the example below: +In `/etc/gitlab/gitlab.rb`: ```ruby -redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis_slave_role['enable'] = true + +redis['bind'] = '10.0.0.2' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' -## Slave redis instance -redis['master'] = false -redis['master_ip'] = '10.10.10.10' # IP of master Redis server -redis['master_port'] = 6379 # Port of master Redis server +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default ``` Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` ---- - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. +**Example configation for Slave 2:** -If you are not sure if your Redis servers are working and replicating -correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) -and fix it before proceeding with Sentinel setup. +In `/etc/gitlab/gitlab.rb`: -### Configuring Sentinel +```ruby +redis_slave_role['enable'] = true -You must have at least `3` Redis Sentinel servers, and they need to -be each in a independent machine. You can configure them in the same -machines where you've configured the other Redis servers. +redis['bind'] = '10.0.0.3' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' -This number is required for the consensus algorithm to be effective -in the case of a failure. **You should always have an `odd` number -of Sentinel nodes provisioned**. +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default +``` -#### How sentinel handles a failover +Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -If (`quorum` value of) Sentinels agree the fact the `master` is not reachable, -Sentinels will try to elect a temporary `Leader`. The **Majority** of the -Sentinels must agree to start a failover. +### Configuration for Sentinel (EE only) -If you don't have the **Majority** of the Sentinels online (for example if you -are under a network partitioning), a failover **will not be started**. +Please note that some of the variables are already configured previously +as they are required for Redis replication. -For example, for a cluster of `3` Sentinels, at least `2` must agree on a -`Leader`. If you have total of `5` at least `3` must agree on a `Leader`. +**Example configation for Sentinel 1:** -The `quorum` is only used to detect failure, not to elect the `Leader`. +In `/etc/gitlab/gitlab.rb`: -Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments) -also lists different network topologies and warns againts situations like -network partition and how it can affect the state of the HA solution. Make -sure you read it carefully and understand the implications in your current -setup. +```ruby +redis_sentinel_role['enable'] = true -GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-setup-ee-only) the Sentinel daemon. +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance -#### Sentinel setup +## Configure Sentinel +sentinel['bind'] = '10.0.0.1' +# sentinel['port'] = 26379 # uncomment to change default port -##### Community Edition +## Quorum must reflect the amount of voting sentinels it take to start a failover. +## Value must NOT be greater then the ammount of sentinels. +## +## The quorum can be used to tune Sentinel in two ways: +## 1. If a the quorum is set to a value smaller than the majority of Sentinels +## we deploy, we are basically making Sentinel more sensible to master failures, +## triggering a failover as soon as even just a minority of Sentinels is no longer +## able to talk with the master. +## 1. If a quorum is set to a value greater than the majority of Sentinels, we are +## making Sentinel able to failover only when there are a very large number (larger +## than majority) of well connected Sentinels which agree about the master being down.s +sentinel['quorum'] = 2 -With GitLab Community Edition, you need to install, configure, execute and -monitor Sentinel from source. Omnibus GitLab Community Edition package does -not support Sentinel configuration. +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 -See documentation for Source Install [here](redis_source.md). +## Specifies the failover timeout in milliseconds. It is used in many ways: +## +## - The time needed to re-start a failover after a previous failover was +## already tried against the same master by a given Sentinel, is two +## times the failover timeout. +## +## - The time needed for a slave replicating to a wrong master according +## to a Sentinel current configuration, to be forced to replicate +## with the right master, is exactly the failover timeout (counting since +## the moment a Sentinel detected the misconfiguration). +## +## - The time needed to cancel a failover that is already in progress but +## did not produced any configuration change (SLAVEOF NO ONE yet not +## acknowledged by the promoted slave). +## +## - The maximum time a failover in progress waits for all the slaves to be +## reconfigured as slaves of the new master. However even after this time +## the slaves will be reconfigured by the Sentinels anyway, but not with +## the exact parallel-syncs progression as specified. +# sentinel['failover_timeout'] = 60000 +``` -##### Enterprise Edition +**Example configation for Sentinel 2:** -To setup sentinel, edit `/etc/gitlab/gitlab.rb` file: +In `/etc/gitlab/gitlab.rb`: ```ruby +redis_sentinel_role['enable'] = true -## When you install Sentinel in a separate machine, you need to control which -## other services will be running in it. -## We've simplified the choice using special "roles" settings: +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance -## Enabled Sentinel and Redis Master services -redis_sentinel_role['enable'] = true -redis_master_role['enable'] = true +## Configure Sentinel +sentinel['bind'] = '10.0.0.2' +# sentinel['port'] = 26379 # uncomment to change default port + +## Quorum must reflect the amount of voting sentinels it take to start a failover. +## Value must NOT be greater then the ammount of sentinels. +## +## The quorum can be used to tune Sentinel in two ways: +## 1. If a the quorum is set to a value smaller than the majority of Sentinels +## we deploy, we are basically making Sentinel more sensible to master failures, +## triggering a failover as soon as even just a minority of Sentinels is no longer +## able to talk with the master. +## 1. If a quorum is set to a value greater than the majority of Sentinels, we are +## making Sentinel able to failover only when there are a very large number (larger +## than majority) of well connected Sentinels which agree about the master being down.s +sentinel['quorum'] = 2 -## Enabled Sentinel and Redis Slave services +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 + +## Specifies the failover timeout in milliseconds. It is used in many ways: +## +## - The time needed to re-start a failover after a previous failover was +## already tried against the same master by a given Sentinel, is two +## times the failover timeout. +## +## - The time needed for a slave replicating to a wrong master according +## to a Sentinel current configuration, to be forced to replicate +## with the right master, is exactly the failover timeout (counting since +## the moment a Sentinel detected the misconfiguration). +## +## - The time needed to cancel a failover that is already in progress but +## did not produced any configuration change (SLAVEOF NO ONE yet not +## acknowledged by the promoted slave). +## +## - The maximum time a failover in progress waits for all the slaves to be +## reconfigured as slaves of the new master. However even after this time +## the slaves will be reconfigured by the Sentinels anyway, but not with +## the exact parallel-syncs progression as specified. +# sentinel['failover_timeout'] = 60000 +``` + +**Example configation for Sentinel 3:** + +In `/etc/gitlab/gitlab.rb`: + +```ruby redis_sentinel_role['enable'] = true -redis_slave_role['enable'] = true -## Configure Redis redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance -redis['master_port'] = 6379 # port of the initial master redis instance +#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel -# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one +sentinel['bind'] = '10.0.0.3' # sentinel['port'] = 26379 # uncomment to change default port ## Quorum must reflect the amount of voting sentinels it take to start a failover. @@ -323,6 +650,8 @@ sentinel['quorum'] = 2 # sentinel['failover_timeout'] = 60000 ``` +### Control running services + In the example above we've used `redis_sentinel_role` and `redis_master_role` which simplify the ammount of configuration changes. @@ -365,37 +694,6 @@ mailroom['enable'] = false # Redis Slave role also change this setting from default 'true' to 'false': redis['master'] = false ``` ---- - -The final part is to inform the main GitLab application server of the Redis -master and the new sentinels servers. - -### GitLab setup - -You can enable or disable Sentinel support at any time in new or existing -installations. From the GitLab application perspective, all it requires is -the correct credentials for the Sentinel nodes. - -While it doesn't require a list of all Sentinel nodes, in case of a failure, -it needs to access at one of listed ones. - ->**Note:** -The following steps should be performed in the [GitLab application server](gitlab.md) -which ideally should not have Redis or Sentinels in the same machine for a HA setup. - -1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: - - ```ruby - redis['master_name'] = "gitlab-redis" - redis['master_password'] = 'redis-password-goes-here' - gitlab_rails['redis_sentinels'] = [ - {'host' => '10.10.10.1', 'port' => 26379}, - {'host' => '10.10.10.2', 'port' => 26379}, - {'host' => '10.10.10.3', 'port' => 26379} - ] - ``` - -1. [Reconfigure] GitLab for the changes to take effect. ## Troubleshooting diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 7e8c8c2b4b9..ab5bb13f070 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -153,7 +153,7 @@ outside (Internet). We will use the same `3` nodes with **Redis** + **Sentinel** topology discussed in the [Configuring Redis for GitLab HA](redis.md) documentation. -Here is a list and description of each **machine** and the assined **ip**: +Here is a list and description of each **machine** and the assigned **IP**: * `10.0.0.1`: Redis Master + Sentinel 1 * `10.0.0.2`: Redis Slave 1 + Sentinel 2 @@ -170,7 +170,7 @@ or a failover promotes a different **Master** node. ### Configuring Redis Master -`redis.conf`: +**Example configation for Redis Master - `redis.conf`:** ```conf bind 10.0.0.1 @@ -181,7 +181,7 @@ masterauth redis-password-goes-here ### Configuring Redis Slaves -**Slave 1 - `redis.conf`:** +**Example configation for Slave 1 - `redis.conf`:** ```conf bind 10.0.0.2 @@ -193,7 +193,7 @@ masterauth redis-password-goes-here slaveof 10.0.0.1 6379 ``` -**Slave 2 - `redis.conf`:** +**Example configation for Slave 2 - `redis.conf`:** ```conf bind 10.0.0.3 @@ -211,7 +211,7 @@ For this example, **Sentinel 1** will be configured in the same machine as the **Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the **Slave 1** and **Slave 2** respectively. -Sentinel 1 - `sentinel.conf` +**Example configation for Sentinel 1 - `sentinel.conf`:** ```conf bind 10.0.0.1 @@ -222,7 +222,7 @@ sentinel down-after-milliseconds gitlab-redis 10000 sentinel failover_timeout 30000 ``` -Sentinel 2 - `sentinel.conf` +**Example configation for Sentinel 2 - `sentinel.conf`:** ```conf bind 10.0.0.2 @@ -233,7 +233,7 @@ sentinel down-after-milliseconds gitlab-redis 10000 sentinel failover_timeout 30000 ``` -Sentinel 3 - `sentinel.conf` +**Example configation for Sentinel 3 - `sentinel.conf`:** ```conf bind 10.0.0.3 -- cgit v1.2.1 From 646f81afac23b74b604c77af61254f90e205505f Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 10 Nov 2016 13:14:22 +0100 Subject: Fix typos of Redis sentinel docs --- doc/administration/high_availability/redis.md | 44 +++++++++++++-------------- 1 file changed, 22 insertions(+), 22 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 6b16f9e791f..ca773ea20aa 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -5,7 +5,7 @@ topology with **Sentinel** service to watch and automatically start failover proceedings. You can choose to install and manage Redis and Sentinel yourself, use -a hosted, managed clouse solution or you can use or you can use the one +a hosted, managed cloud solution or you can use or you can use the one that comes bundled with Omnibus GitLab packages. > **Note:** Redis requires authentication for High Availability. See @@ -55,10 +55,10 @@ technology and provide a transparent proxy, which means that GitLab doesn't need any additional change, or will use Sentinel and manage it for you. If your provider, uses Sentinel method, see [GitLab Setup](#gitlab-setup) -to understant where you need to provide the list of servers and credentials. +to understand where you need to provide the list of servers and credentials. If you want to setup Redis by yourself, without using Omnibus, you can -read our documentation: [Configuring Redis for GitLab HA (Source Install)](redis_source.md) +read our documentation: [Configuring Redis for GitLab HA (source install)](redis_source.md). ## High Availability with Sentinel @@ -73,19 +73,19 @@ High Availability with Redis requires a few things: - Multiple Redis instances - Run Redis in a **Master** x **Slave** topology - Multiple Sentinel instances -- Application support and visiblity to all Sentinel and Redis instances +- Application support and visibility to all Sentinel and Redis instances Redis Sentinel can handle the most important tasks in a HA environment to help keep servers online with minimal to no downtime: - Monitors **Master** and **Slaves** instances to see if they are available -- Promote a **Slave** to **Master** when the **Master** fails. +- Promote a **Slave** to **Master** when the **Master** fails - Demote a **Master** to **Slave** when failed **Master** comes back online (to prevent - data-partitioning). -- Can be queried by clients to always connect to the current **Master** server. + data-partitioning) +- Can be queried by clients to always connect to the current **Master** server -When a **Master** fails to respond, it's the client responsability to handle timeout -and reconnect (querying a **Sentinel** for a new **Master**). +When a **Master** fails to respond, it's the client's responsibility to handle +timeout and reconnect (querying a **Sentinel** for a new **Master**). To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as @@ -113,7 +113,7 @@ the shared environment can bring your entire setup down. You also need to take in consideration the underlying network topology, making sure you have redundant connectivity between Redis / Sentinel and -GitLab instances, otherwhise the networks will become a single point of +GitLab instances, otherwise the networks will become a single point of failure. Read carefully how to configure the components below. @@ -132,7 +132,7 @@ simultaneously down. Please note that there are different requirements for Sentinel nodes. If you host them in the same Redis machines, you may need to take -that restrictions into consideration when calculating the ammount of +that restrictions into consideration when calculating the amount of nodes to be provisioned. See [Sentinel setup](#sentinel-setup) documentation for more information. @@ -171,10 +171,10 @@ the same Sentinels. Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel detects that a Redis node is not responding, it will announce that to the -other sentinels. You have to reach the **quorum**, the minimum ammount of +other sentinels. You have to reach the **quorum**, the minimum amount of sentinels that agrees that a node is down, to be able to start a failover. -Whenver the **quorum** is met, you need the **majority** of all known +Whenever the **quorum** is met, you need the **majority** of all known Sentinel nodes to be available and reachable, to elect the Sentinel **leader** who will take all the decisions to restore the service availability by: @@ -238,7 +238,7 @@ independent machines, both with **Redis** and **Sentinel**: - Redis Slave + Sentinel Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) -before, to understant how and why the ammount of nodes came from. +before, to understand how and why the amount of nodes came from. For a recommended setup, that can resist more failures, you will install the Omnibus GitLab package in `5` independent machines, both with @@ -259,11 +259,11 @@ This is a summary of what are we going to do: independent ones. - Don't install Redis and Sentinel in the same machines your GitLab instance is running on. - - All machines must be able to talk to each other and accept incomming + - All machines must be able to talk to each other and accept incoming connection over Redis (`6379`) and Sentinel (`26379`) ports. - GitLab machines must be able to access these machines and with the same permissions. - - Protected them from indiscriminated access from external networks (Internet), + - Protected them from indiscriminating access from external networks (Internet), to harden the security. 1. Download/install Omnibus GitLab using **steps 1 and 2** from @@ -288,7 +288,7 @@ inside it. Your single-machine install will be the initial **Master**, and the `3` others should be configured as **Slave** pointing to this machine. -After replication catchs-up, you will need to stop services in the +After replication catches up, you will need to stop services in the single-machine install, to rotate the **Master** to one of the new nodes. Make the required changes in configuration and restart the new nodes again. @@ -310,7 +310,7 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`: 1. Define a `redis['bind']` address pointing to a local IP that your other machines can reach you. - - If you really need to bind to an external acessible IP, make + - If you really need to bind to an external accessible IP, make sure you add extra firewall rules to prevent unauthorized access. - You can also set bind to `0.0.0.0` which listen in all interfaces. @@ -332,7 +332,7 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`: 1. Define a `redis['bind']` address pointing to a local IP that your other machines can reach you. - - If you really need to bind to an external acessible IP, make + - If you really need to bind to an external accessible IP, make sure you add extra firewall rules to prevent unauthorized access. - You can also set bind to `0.0.0.0` which listen in all interfaces. @@ -384,7 +384,7 @@ installations. From the GitLab application perspective, all it requires is the correct credentials for the Sentinel nodes. While it doesn't require a list of all Sentinel nodes, in case of a failure, -it needs to access at least one of listeds. +it needs to access at least one of the listed. >**Note:** The following steps should be performed in the [GitLab application server](gitlab.md) @@ -425,7 +425,7 @@ by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** will change permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. -The same thing will happen with `sentinel.conf` that will be overriten after the +The same thing will happen with `sentinel.conf` that will be overridden after the initial execution, after any new sentinel node starts watching the **Master**, or a failover promotes a different **Master** node. @@ -653,7 +653,7 @@ sentinel['quorum'] = 2 ### Control running services In the example above we've used `redis_sentinel_role` and `redis_master_role` -which simplify the ammount of configuration changes. +which simplify the amount of configuration changes. If you want more control, here is what each one sets for you automatically when enabled: -- cgit v1.2.1 From c43b540095b6de03e679dc4dd56df97f08a6aed0 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 10 Nov 2016 13:21:34 +0100 Subject: Move experimental heading at the bottom under changelog --- doc/administration/high_availability/redis.md | 59 +++++++++++++++------------ 1 file changed, 33 insertions(+), 26 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index ca773ea20aa..3837a8a0a88 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -19,28 +19,29 @@ that comes bundled with Omnibus GitLab packages. - [Using an external Redis server](#using-an-external-redis-server) - [High Availability with Sentinel](#high-availability-with-sentinel) - - [Prerequisites](#prerequisites) - - [Redis setup](#redis-setup) - - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) - - [Sentinel setup](#sentinel-setup) - - [Recommended setup](#recommended-setup) + - [Prerequisites](#prerequisites) + - [Redis setup](#redis-setup) + - [Sentinel setup](#sentinel-setup) + - [Recommended setup](#recommended-setup) - [Configuring instances using Omnibus](#configuring-instances-using-omnibus) - - [Existing single-machine installation](#existing-single-machine-installation) - - [Configuring Master Redis instance](#configuring-master-redis-instance) - - [Configuring Slave Redis instances](#configuring-slave-redis-instances) - - [Configuring Sentinel instances](#configuring-sentinel-instances) - - [Community Edition](#community-edition) - - [Enterprise Edition](#enterprise-edition) - - [GitLab setup](#gitlab-setup) + - [Existing single-machine installation](#existing-single-machine-installation) + - [Configuring Master Redis instance](#configuring-master-redis-instance) + - [Configuring Slave Redis instances](#configuring-slave-redis-instances) + - [Configuring Sentinel instances](#configuring-sentinel-instances) + - [Community Edition](#community-edition) + - [Enterprise Edition](#enterprise-edition) + - [GitLab setup](#gitlab-setup) - [Example Configurations](#example-configurations) - - [Configuration for Redis Master](#configuration-for-redis-master) - - [Configuration for Redis Slave](#configuration-for-redis-slave) - - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) - - [Control running services](#control-running-services) + - [Configuration for Redis Master](#configuration-for-redis-master) + - [Configuration for Redis Slave](#configuration-for-redis-slave) + - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) + - [Control running services](#control-running-services) - [Troubleshooting](#troubleshooting) - - [Redis replication](#redis-replication) - - [Sentinel](#sentinel) - - [Omnibus GitLab](#omnibus-gitlab) + - [Redis replication](#redis-replication) + - [Sentinel](#sentinel) + - [Omnibus GitLab](#omnibus-gitlab) +- [Changelog](#changelog) + - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) @@ -160,13 +161,6 @@ a failover, as the nodes will be managed by the Sentinels, and even after a `gitlab-ctl reconfigure`, they will get their configuration restored by the same Sentinels. -### Experimental Redis Sentinel support - - > Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. - Starting with 8.14, Redis Sentinel is no longer experimental. - If you used with versions `< 8.14` before, please check the updated - documentation here. - ### Sentinel setup Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel @@ -815,6 +809,19 @@ To make sure your configuration is correct: You should see a different port after a few seconds delay (the failover/reconnect time). + +## Changelog + +Changes to Redis HA over time. + +### Experimental Redis Sentinel support + +> +Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. +Starting with 8.14, Redis Sentinel is no longer experimental. +If you used with versions `< 8.14` before, please check the updated +documentation here. + --- Read more on high-availability configuration: -- cgit v1.2.1 From fb0dff6ea848f58a068030f93e8728ce0143c592 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 10 Nov 2016 15:51:13 +0100 Subject: Move some things over --- doc/administration/high_availability/redis.md | 195 +++++++++++++++----------- 1 file changed, 112 insertions(+), 83 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 3837a8a0a88..5d4dfe22779 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,73 +1,105 @@ # Configuring Redis for GitLab HA -High Availability with Redis is possible using a **Master** x **Slave** -topology with **Sentinel** service to watch and automatically start -failover proceedings. +High Availability with [Redis] is possible using a **Master** x **Slave** +topology with a [Redis Sentinel][sentinel] service to watch and automatically +start failover proceedings. You can choose to install and manage Redis and Sentinel yourself, use -a hosted, managed cloud solution or you can use or you can use the one -that comes bundled with Omnibus GitLab packages. +a hosted cloud solution or you can use the one that comes bundled with +Omnibus GitLab packages. -> **Note:** Redis requires authentication for High Availability. See +> **Notes:** +- Redis requires authentication for High Availability. See [Redis Security](http://redis.io/topics/security) documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. +- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation + before configuring Redis HA with GitLab to fully understand the topology and + architecture. **Table of Contents** -- [Using an external Redis server](#using-an-external-redis-server) -- [High Availability with Sentinel](#high-availability-with-sentinel) - - [Prerequisites](#prerequisites) - - [Redis setup](#redis-setup) - - [Sentinel setup](#sentinel-setup) - - [Recommended setup](#recommended-setup) +- [Overview](#overview) + - [Available setups](#available-setups) + - [Using a non-Omnibus external Redis server](#using-a-non-omnibus-external-redis-server) + - [High Availability with Sentinel](#high-availability-with-sentinel) + - [Prerequisites](#prerequisites) + - [Recommended setup](#recommended-setup) +- [Redis HA configuration](#redis-ha-configuration) + - [Redis setup](#redis-setup) + - [Sentinel setup](#sentinel-setup) - [Configuring instances using Omnibus](#configuring-instances-using-omnibus) - - [Existing single-machine installation](#existing-single-machine-installation) - - [Configuring Master Redis instance](#configuring-master-redis-instance) - - [Configuring Slave Redis instances](#configuring-slave-redis-instances) - - [Configuring Sentinel instances](#configuring-sentinel-instances) - - [Community Edition](#community-edition) - - [Enterprise Edition](#enterprise-edition) - - [GitLab setup](#gitlab-setup) -- [Example Configurations](#example-configurations) - - [Configuration for Redis Master](#configuration-for-redis-master) - - [Configuration for Redis Slave](#configuration-for-redis-slave) - - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) - - [Control running services](#control-running-services) + - [Existing single-machine installation](#existing-single-machine-installation) + - [Configuring Master Redis instance](#configuring-master-redis-instance) + - [Configuring Slave Redis instances](#configuring-slave-redis-instances) + - [Configuring Sentinel instances](#configuring-sentinel-instances) + - [Community Edition](#community-edition) + - [Enterprise Edition](#enterprise-edition) + - [GitLab setup](#gitlab-setup) +- [Minimal example configuration with 1 master, 2 slaves and 3 sentinels](#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels) + - [Configuration for Redis Master](#configuration-for-redis-master) + - [Configuration for Redis Slave](#configuration-for-redis-slave) + - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) + - [Control running services](#control-running-services) - [Troubleshooting](#troubleshooting) - - [Redis replication](#redis-replication) - - [Sentinel](#sentinel) - - [Omnibus GitLab](#omnibus-gitlab) + - [Redis replication](#redis-replication) + - [Sentinel](#sentinel) + - [Omnibus GitLab](#omnibus-gitlab) - [Changelog](#changelog) - - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) + - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) -## Using an external Redis server + +## Overview + +Before diving into the details of setting up Redis and Redis Sentinel for HA, +make sure you read this section to better understand the underline architecture. + +### Available setups + +Based on your infrastructure setup, there are multiple ways to setup Redis HA +with GitLab. Omnibus GitLab packages have Redis and Redis Sentinel bundled with +them to save you the hassle to install it yourself. Pick the one that suits your +needs. + +- **Installations from source:** You need to install Redis and Redis Sentinel + yourself. Use the [Redis HA source install](redis_source.md) guide. +- **Omnibus Community Edition (CE):** Redis is bundled so you can use the + package with only the Redis service enabled (works for both master and slave + setups). +- **Omnibus Enterprise Edition (EE):** Both Redis and Redis Sentinel are bundled + in the Omnibus package so you can use only them to setup the whole Redis HA + infrastructure (master, slave and Sentinel). + +Note that with the Omnibus packages (both CE and EE), you can also use an +[external Redis server](#using-a-non-omnibus-external-redis-server). + +### Using a non-Omnibus external Redis server If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers a managed ElastiCache service that runs Redis. Managed services can provide High Availability using their own proprietary -technology and provide a transparent proxy, which means that GitLab doesn't -need any additional change, or will use Sentinel and manage it for you. +technology and provide a transparent proxy (which means that GitLab doesn't +need any additional change) or they will use Sentinel and manage it for you. -If your provider, uses Sentinel method, see [GitLab Setup](#gitlab-setup) +If your provider, uses Sentinel, see [GitLab Setup](#gitlab-setup) to understand where you need to provide the list of servers and credentials. If you want to setup Redis by yourself, without using Omnibus, you can -read our documentation: [Configuring Redis for GitLab HA (source install)](redis_source.md). - -## High Availability with Sentinel +read the documentation on [configuring Redis HA for source installs](redis_source.md). -> Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that -will monitor a group of Redis servers to provide failover support. +### High Availability with Sentinel -> With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and -improved the way you use and configure it. +> +- Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that + will monitor a group of Redis servers to provide failover support. +- With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and + improved the way you use and configure it. High Availability with Redis requires a few things: @@ -77,12 +109,12 @@ High Availability with Redis requires a few things: - Application support and visibility to all Sentinel and Redis instances Redis Sentinel can handle the most important tasks in a HA environment to help -keep servers online with minimal to no downtime: +keep servers online with minimal to no downtime. Redis Sentinel: - Monitors **Master** and **Slaves** instances to see if they are available -- Promote a **Slave** to **Master** when the **Master** fails -- Demote a **Master** to **Slave** when failed **Master** comes back online (to prevent - data-partitioning) +- Promotes a **Slave** to **Master** when the **Master** fails +- Demotes a **Master** to **Slave** when failed **Master** comes back online + (to prevent data-partitioning) - Can be queried by clients to always connect to the current **Master** server When a **Master** fails to respond, it's the client's responsibility to handle @@ -93,24 +125,16 @@ the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as failing to configure it correctly can lead to data loss, or can bring your whole cluster down, invalidating the failover effort. -This documentation will provide you with a minimal and a recommended topology -that can resist to some levels of failure. Usually the more Redis and Sentinel -instances you have provisioned, the better will be your availability. - -The configuration consists of three parts: - -- Setup Redis Master and Slave nodes -- Setup Sentinel nodes -- Setup GitLab - ### Prerequisites You need at least `3` independent machines: physical, or VMs running into -distinct physical machines. They must be believed to fail in an -independent way. +distinct physical machines. It is essential that all master and Redis slaves +run in different machines. If you fail to provision the machines in that +specific way, any issue with the shared environment can bring your entire setup +down. -If you fail to provision the machines in that specific way, any issue with -the shared environment can bring your entire setup down. +It is OK to run a Sentinel along with a master or slave Redis instance. +No more than one though. You also need to take in consideration the underlying network topology, making sure you have redundant connectivity between Redis / Sentinel and @@ -119,6 +143,30 @@ failure. Read carefully how to configure the components below. +### Recommended setup + +For a minimal setup, you will install the Omnibus GitLab package in `3` +independent machines, both with **Redis** and **Sentinel**: + +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel + +Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) +before, to understand how and why the amount of nodes came from. + +For a recommended setup, that can resist more failures, you will install +the Omnibus GitLab package in `5` independent machines, both with +**Redis** and **Sentinel**: + +- Redis Master + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel +- Redis Slave + Sentinel + +## Redis HA configuration + ### Redis setup You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they @@ -156,14 +204,14 @@ Initial **Slave** nodes requires `redis['master']` defined to `false` and simplified configuration by enabling `redis_slave_role['enable']`, you just need to fill in the `redis['master_ip']`. -This values doesn't have to be changed again in `/etc/gitlab/gitlab.rb` after +This values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the Sentinels, and even after a `gitlab-ctl reconfigure`, they will get their configuration restored by the same Sentinels. ### Sentinel setup -Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel +Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel detects that a Redis node is not responding, it will announce that to the other sentinels. You have to reach the **quorum**, the minimum amount of sentinels that agrees that a node is down, to be able to start a failover. @@ -222,33 +270,12 @@ official documentation: the slaves will be reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. -### Recommended setup - -For a minimal setup, you will install the Omnibus GitLab package in `3` -independent machines, both with **Redis** and **Sentinel**: - -- Redis Master + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel - -Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) -before, to understand how and why the amount of nodes came from. - -For a recommended setup, that can resist more failures, you will install -the Omnibus GitLab package in `5` independent machines, both with -**Redis** and **Sentinel**: - -- Redis Master + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel - ## Configuring instances using Omnibus This is a summary of what are we going to do: 1. Provision the required number of instances specified previously + - You can opt to install Redis and Sentinel in the same machine or each in independent ones. - Don't install Redis and Sentinel in the same machines your GitLab instance @@ -257,7 +284,7 @@ This is a summary of what are we going to do: connection over Redis (`6379`) and Sentinel (`26379`) ports. - GitLab machines must be able to access these machines and with the same permissions. - - Protected them from indiscriminating access from external networks (Internet), + - Protect them from access from external networks (Internet), to harden the security. 1. Download/install Omnibus GitLab using **steps 1 and 2** from @@ -394,7 +421,7 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se See [example configuration](#configuration-for-gitlab) below. -## Example Configurations +## Minimal example configuration with 1 master, 2 slaves and 3 sentinels In this example we consider that all servers have an internal network interface with IPs in the `10.0.0.x` range, and that they can connect @@ -836,3 +863,5 @@ Read more on high-availability configuration: [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [gh-531]: https://github.com/redis/redis-rb/issues/531 [gh-534]: https://github.com/redis/redis-rb/issues/534 +[redis]: http://redis.io/ +[sentinel]: http://redis.io/topics/sentinel -- cgit v1.2.1 From 1d3ada80ad6cb9a4927512fdf4018907bf3098a6 Mon Sep 17 00:00:00 2001 From: Patricio Cano Date: Mon, 7 Nov 2016 15:44:55 -0600 Subject: Added documentation and CHANGELOG item. --- doc/administration/operations.md | 1 + .../operations/img/sidekiq_job_throttling.png | Bin 0 -> 114784 bytes .../operations/sidekiq_job_throttling.md | 32 +++++++++++++++++++++ 3 files changed, 33 insertions(+) create mode 100644 doc/administration/operations/img/sidekiq_job_throttling.png create mode 100644 doc/administration/operations/sidekiq_job_throttling.md (limited to 'doc') diff --git a/doc/administration/operations.md b/doc/administration/operations.md index 4b582d16b64..0daceb98d99 100644 --- a/doc/administration/operations.md +++ b/doc/administration/operations.md @@ -1,6 +1,7 @@ # GitLab operations - [Sidekiq MemoryKiller](operations/sidekiq_memory_killer.md) +- [Sidekiq Job throttling](operations/sidekiq_job_throttling.md) - [Cleaning up Redis sessions](operations/cleaning_up_redis_sessions.md) - [Understanding Unicorn and unicorn-worker-killer](operations/unicorn.md) - [Moving repositories to a new location](operations/moving_repositories.md) diff --git a/doc/administration/operations/img/sidekiq_job_throttling.png b/doc/administration/operations/img/sidekiq_job_throttling.png new file mode 100644 index 00000000000..7f29a4d3c46 Binary files /dev/null and b/doc/administration/operations/img/sidekiq_job_throttling.png differ diff --git a/doc/administration/operations/sidekiq_job_throttling.md b/doc/administration/operations/sidekiq_job_throttling.md new file mode 100644 index 00000000000..33cedee7ebd --- /dev/null +++ b/doc/administration/operations/sidekiq_job_throttling.md @@ -0,0 +1,32 @@ +# Sidekiq Job throttling + +> Note: Introduced with GitLab 8.14 + +When your GitLab installation needs to handle tens of thousands of background +jobs, it can be convenient to prioritize queues that need to be executed +immediately, e.g. user initiated actions like merging a Merge Request. + +In order to accomplish this, you can limit the amount of workers that certain +slow running queues get can have available. This is what we call Sidekiq Job +Throttling. Depending on your infrastructure, you might have different slow +running queues, which is why you can choose which queues to throttle and by +how much you want to throttle them. + +These settings are available in the Application Settings of your GitLab +installation. + +![Sidekiq Job Throttling](img/sidekiq_job_throttling.png) + +The throttle factor determines the maximum number of workers a queue can run on. +This value gets multiplied by `:concurrency` value set in the Sidekiq settings +and rounded up to the closest full integer. + +So, for example, you set the `:concurrency` to 25 and the `Throttling factor` to +0.1, the maximum workers assigned to the selected queues would be 3. + +``` +limit = (factor * Sidekiq.options[:concurrency]).ceil +``` + +After enabling the job throttling, you will need to restart your GitLab +instance, in order for the changes to take effect. \ No newline at end of file -- cgit v1.2.1 From b575b00402da92d15f8edcaab702039e6e92eddc Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 10 Nov 2016 19:00:06 +0100 Subject: Rearrange sections in Sentinel docs [ci skip] --- doc/administration/high_availability/redis.md | 343 ++++++++++++++------------ 1 file changed, 181 insertions(+), 162 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 5d4dfe22779..3eaa0ffdcec 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -16,82 +16,67 @@ Omnibus GitLab packages. - You are highly encouraged to read the [Redis Sentinel][sentinel] documentation before configuring Redis HA with GitLab to fully understand the topology and architecture. +- This is the documentation for the Omnibus packages. For installations from + source, follow the [Redis HA source install](redis_source.md) guide. +- Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the + Omnibus Community Edition and installations from source, follow the + [Redis HA source install](redis_source.md) guide. **Table of Contents** - [Overview](#overview) - - [Available setups](#available-setups) - - [Using a non-Omnibus external Redis server](#using-a-non-omnibus-external-redis-server) - - [High Availability with Sentinel](#high-availability-with-sentinel) - [Prerequisites](#prerequisites) + - [High Availability with Sentinel](#high-availability-with-sentinel) - [Recommended setup](#recommended-setup) + - [Available configuration setups](#available-configuration-setups) + - [Using a non-Omnibus external Redis server](#using-a-non-omnibus-external-redis-server) + - [Redis setup overview](#redis-setup-overview) + - [Sentinel setup overview](#sentinel-setup-overview) - [Redis HA configuration](#redis-ha-configuration) - - [Redis setup](#redis-setup) - - [Sentinel setup](#sentinel-setup) -- [Configuring instances using Omnibus](#configuring-instances-using-omnibus) - - [Existing single-machine installation](#existing-single-machine-installation) - - [Configuring Master Redis instance](#configuring-master-redis-instance) - - [Configuring Slave Redis instances](#configuring-slave-redis-instances) - - [Configuring Sentinel instances](#configuring-sentinel-instances) - - [Community Edition](#community-edition) - - [Enterprise Edition](#enterprise-edition) - - [GitLab setup](#gitlab-setup) + - [Configuring the Master Redis instance](#configuring-the-master-redis-instance) + - [Configuring the Slave Redis instances](#configuring-the-slave-redis-instances) + - [Configuring the Sentinel instances](#configuring-the-sentinel-instances) + - [Configuring the GitLab application](#configuring-the-gitlab-application) +- [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha) - [Minimal example configuration with 1 master, 2 slaves and 3 sentinels](#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels) - - [Configuration for Redis Master](#configuration-for-redis-master) - - [Configuration for Redis Slave](#configuration-for-redis-slave) - - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) + - [Configuration for Redis master](#configuration-for-redis-master) + - [Configuration for Redis slaves](#configuration-for-redis-slaves) + - [Configuration for Sentinels](#configuration-for-sentinels) +- [Advanced configuration](#advanced-configuration) - [Control running services](#control-running-services) - [Troubleshooting](#troubleshooting) - - [Redis replication](#redis-replication) - - [Sentinel](#sentinel) - - [Omnibus GitLab](#omnibus-gitlab) + - [Troubleshooting Redis replication](#troubleshooting-redis-replication) + - [Troubleshooting Sentinel](#troubleshooting-sentinel) - [Changelog](#changelog) - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) - ## Overview Before diving into the details of setting up Redis and Redis Sentinel for HA, -make sure you read this section to better understand the underline architecture. - -### Available setups - -Based on your infrastructure setup, there are multiple ways to setup Redis HA -with GitLab. Omnibus GitLab packages have Redis and Redis Sentinel bundled with -them to save you the hassle to install it yourself. Pick the one that suits your -needs. - -- **Installations from source:** You need to install Redis and Redis Sentinel - yourself. Use the [Redis HA source install](redis_source.md) guide. -- **Omnibus Community Edition (CE):** Redis is bundled so you can use the - package with only the Redis service enabled (works for both master and slave - setups). -- **Omnibus Enterprise Edition (EE):** Both Redis and Redis Sentinel are bundled - in the Omnibus package so you can use only them to setup the whole Redis HA - infrastructure (master, slave and Sentinel). - -Note that with the Omnibus packages (both CE and EE), you can also use an -[external Redis server](#using-a-non-omnibus-external-redis-server). +make sure you read this Overview section to better understand how the components +are tied together. -### Using a non-Omnibus external Redis server +### Prerequisites -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for Redis. For example, AWS offers a managed ElastiCache service -that runs Redis. +You need at least `3` independent machines: physical, or VMs running into +distinct physical machines. It is essential that all master and slaves Redis +instances run in different machines. If you fail to provision the machines in +that specific way, any issue with the shared environment can bring your entire +setup down. -Managed services can provide High Availability using their own proprietary -technology and provide a transparent proxy (which means that GitLab doesn't -need any additional change) or they will use Sentinel and manage it for you. +It is OK to run a Sentinel along with a master or slave Redis instance. +No more than one Sentinel in the same machine though. -If your provider, uses Sentinel, see [GitLab Setup](#gitlab-setup) -to understand where you need to provide the list of servers and credentials. +You also need to take in consideration the underlying network topology, +making sure you have redundant connectivity between Redis / Sentinel and +GitLab instances, otherwise the networks will become a single point of +failure. -If you want to setup Redis by yourself, without using Omnibus, you can -read the documentation on [configuring Redis HA for source installs](redis_source.md). +Read carefully how to configure the components below. ### High Availability with Sentinel @@ -108,12 +93,12 @@ High Availability with Redis requires a few things: - Multiple Sentinel instances - Application support and visibility to all Sentinel and Redis instances -Redis Sentinel can handle the most important tasks in a HA environment to help -keep servers online with minimal to no downtime. Redis Sentinel: +Redis Sentinel can handle the most important tasks in a HA environment and that's +to help keep servers online with minimal to no downtime. Redis Sentinel: - Monitors **Master** and **Slaves** instances to see if they are available - Promotes a **Slave** to **Master** when the **Master** fails -- Demotes a **Master** to **Slave** when failed **Master** comes back online +- Demotes a **Master** to **Slave** when the failed **Master** comes back online (to prevent data-partitioning) - Can be queried by clients to always connect to the current **Master** server @@ -122,41 +107,23 @@ timeout and reconnect (querying a **Sentinel** for a new **Master**). To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as -failing to configure it correctly can lead to data loss, or can bring your +failing to configure it correctly can lead to data loss or can bring your whole cluster down, invalidating the failover effort. -### Prerequisites - -You need at least `3` independent machines: physical, or VMs running into -distinct physical machines. It is essential that all master and Redis slaves -run in different machines. If you fail to provision the machines in that -specific way, any issue with the shared environment can bring your entire setup -down. - -It is OK to run a Sentinel along with a master or slave Redis instance. -No more than one though. - -You also need to take in consideration the underlying network topology, -making sure you have redundant connectivity between Redis / Sentinel and -GitLab instances, otherwise the networks will become a single point of -failure. - -Read carefully how to configure the components below. - ### Recommended setup For a minimal setup, you will install the Omnibus GitLab package in `3` -independent machines, both with **Redis** and **Sentinel**: +**independent** machines, both with **Redis** and **Sentinel**: - Redis Master + Sentinel - Redis Slave + Sentinel - Redis Slave + Sentinel -Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) -before, to understand how and why the amount of nodes came from. +If you are not sure or don't understand why and where the amount of nodes come +from, read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup). -For a recommended setup, that can resist more failures, you will install -the Omnibus GitLab package in `5` independent machines, both with +For a recommended setup that can resist more failures, you will install +the Omnibus GitLab package in `5` **independent** machines, both with **Redis** and **Sentinel**: - Redis Master + Sentinel @@ -165,9 +132,42 @@ the Omnibus GitLab package in `5` independent machines, both with - Redis Slave + Sentinel - Redis Slave + Sentinel -## Redis HA configuration +### Available configuration setups + +Based on your infrastructure setup and how you have installed GitLab, there are +multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or +Redis Sentinel bundled with them to save you the hassle to install them yourself. +Pick the one that suits your needs. + +- [Installations from source][source]: You need to install Redis and Sentinel + yourself. Use the [Redis HA source install](redis_source.md) guide. +- [Omnibus package Community Edition (CE)][ce]: Redis is bundled, so you can use the + package with only the Redis service enabled (works for both master and slave + setups). +- [Omnibus package Enterprise Edition (EE)][ee]: Both Redis and Sentinel are bundled, + so you can use the EE package to setup the whole Redis HA infrastructure + (master, slave and Sentinel). + +Note that if you have installed GitLab using the Omnibus packages (both CE and EE), +you can also use an [external Redis server](#using-a-non-omnibus-external-redis-server). + +### Using a non-Omnibus external Redis server + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for Redis. For example, AWS offers a managed ElastiCache service +that runs Redis. + +Managed services can provide High Availability using their own proprietary +technology and provide a transparent proxy (which means that GitLab doesn't +need any additional change) or they will use Sentinel and manage it for you. + +If your provider uses Sentinel, see [GitLab Setup](#gitlab-setup) +to understand where you need to provide the list of servers and credentials. + +If you want to setup Redis by yourself, without using Omnibus, you can +read the documentation on [configuring Redis HA for source installs](redis_source.md). -### Redis setup +### Redis setup overview You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to be each in a independent machine (see explanation above). @@ -194,22 +194,7 @@ protect all Redis nodes and the Sentinels. They will all share the same password, and all instances must be able to talk to each other over the network. -Redis nodes will need the same password defined in `redis['password']` and -`redis['master_password']`, no matter if **Master** or **Slave**. At any time -during a failover the Sentinels can reconfigure a node and change it's status -from **Master** to **Slave** and vice versa. - -Initial **Slave** nodes requires `redis['master']` defined to `false` and -`redis['master_ip']` pointing to the initial **Master**. If you use the -simplified configuration by enabling `redis_slave_role['enable']`, you -just need to fill in the `redis['master_ip']`. - -This values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the Sentinels, and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -### Sentinel setup +### Sentinel setup overview Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel detects that a Redis node is not responding, it will announce that to the @@ -249,8 +234,11 @@ is not achieved (see the odd number of nodes requirement above). In that case, a new attempt will be made after the amount of time defined in `sentinel['failover_timeout']` (in milliseconds). -The `failover_time` variable have a lot of different usages, according to -official documentation: +>**Note:** +We will see where `sentinel['failover_timeout']` is defined later. + +The `failover_timeout` variable has a lot of different use cases, according to +the official documentation: - The time needed to re-start a failover after a previous failover was already tried against the same master by a given Sentinel, is two @@ -270,18 +258,29 @@ official documentation: the slaves will be reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. -## Configuring instances using Omnibus +## Redis HA configuration + +This is the section where we install and setup the new Redis instances. -This is a summary of what are we going to do: +>**Notes:** +- We assume that you install GitLab and all HA components from scratch. If you + already have it installed and running, read how to + [switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha). +- Redis nodes (both master and slaves) will need the same password defined in + `redis['password']` and `redis['master_password']`. At any time during a + failover the Sentinels can reconfigure a node and change its status + from master to slave and vice versa. -1. Provision the required number of instances specified previously +A summary of what are we going to do: +1. Provision the required number of instances specified previously: - You can opt to install Redis and Sentinel in the same machine or each in independent ones. - - Don't install Redis and Sentinel in the same machines your GitLab instance + - Don't install Redis and Sentinel in the same machines your GitLab application is running on. - All machines must be able to talk to each other and accept incoming - connection over Redis (`6379`) and Sentinel (`26379`) ports. + connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you + change the default ports). - GitLab machines must be able to access these machines and with the same permissions. - Protect them from access from external networks (Internet), @@ -297,32 +296,10 @@ This is a summary of what are we going to do: from running on upgrade. Only the primary GitLab application server should handle migrations. -1. Create/edit `/etc/gitlab/gitlab.rb` and make the changes based on the +1. Edit `/etc/gitlab/gitlab.rb` and make the changes based on the [Example Configurations](#example-configurations). -### Existing single-machine installation - -If you already have a single-machine GitLab install running, you will need to -replicate from this machine first, before de-activating the Redis instance -inside it. - -Your single-machine install will be the initial **Master**, and the `3` others -should be configured as **Slave** pointing to this machine. - -After replication catches up, you will need to stop services in the -single-machine install, to rotate the **Master** to one of the new nodes. - -Make the required changes in configuration and restart the new nodes again. - -To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: - -```ruby -redis['enable'] = false -``` - -If you fail to replicate first, you may loose data (unprocessed background jobs). - -### Configuring Master Redis instance +### Configuring the Master Redis instance You will need to configure the following in `/etc/gitlab/gitlab.rb`: @@ -341,9 +318,9 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`: 1. Set up a password authentication with `redis['password']` and `redis['master_password']` (use the same password in all nodes). -Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` +1. [Reconfigure Omnibus GitLab][reocnfigure] for the changes to take effect. -### Configuring Slave Redis instances +### Configuring the Slave Redis instances You will need to configure the following in `/etc/gitlab/gitlab.rb`: @@ -367,7 +344,22 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`: 1. Define `redis['master_port']` with the port of the **Master** Redis (default to `6379`). -### Configuring Sentinel instances +Initial **Slave** nodes require `redis['master']` defined to `false` and +`redis['master_ip']` pointing to the initial **Master**. If you use the +simplified configuration by enabling `redis_slave_role['enable']`, you +just need to fill in the `redis['master_ip']`. + +This values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the Sentinels, and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +### Configuring the Sentinel instances + +>**Note:** +- Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the + Omnibus Community Edition and installations from source, follow the + [Redis HA source install](redis_source.md) guide. Now that the Redis servers are all set up, let's configure the Sentinel servers. @@ -380,22 +372,12 @@ You must have at least `3` Redis Sentinel servers, and they need to be each in a independent machine. You can configure them in the same machines where you've configured the other Redis servers. -##### Community Edition - -With GitLab Community Edition, you need to install, configure, execute and -monitor Sentinel from source. Omnibus GitLab Community Edition package does -not support Sentinel configuration. - -See [documentation for Source Install](redis_source.md). - -##### Enterprise Edition - -With GitLab Enterprise Edition, you can use Omnibus package to setup multiple -machines with Sentinel daemon. +With GitLab Enterprise Edition, you can use the Omnibus package to setup multiple +machines with the Sentinel daemon. See [example configuration](#configuration-for-sentinel-ee-only) below. -### GitLab setup +### Configuring the GitLab application The final part is to inform the main GitLab application server of the Redis Sentinels servers and authentication credentials. @@ -409,7 +391,7 @@ it needs to access at least one of the listed. >**Note:** The following steps should be performed in the [GitLab application server](gitlab.md) -which ideally should not have Redis or Sentinels in the same machine for a HA setup. +which ideally should not have Redis or Sentinels on it for a HA setup. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: @@ -417,10 +399,32 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se - `redis['master_password']` - the same password you've defined before for Redis and Sentinels - `gitlab_rails['redis_sentinels']` - a list of sentinels with `host` and `port` -1. [Reconfigure] GitLab for the changes to take effect. +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. See [example configuration](#configuration-for-gitlab) below. +## Switching from an existing single-machine installation to Redis HA + +If you already have a single-machine GitLab install running, you will need to +replicate from this machine first, before de-activating the Redis instance +inside it. + +Your single-machine install will be the initial **Master**, and the `3` others +should be configured as **Slave** pointing to this machine. + +After replication catches up, you will need to stop services in the +single-machine install, to rotate the **Master** to one of the new nodes. + +Make the required changes in configuration and restart the new nodes again. + +To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: + +```ruby +redis['enable'] = false +``` + +If you fail to replicate first, you may loose data (unprocessed background jobs). + ## Minimal example configuration with 1 master, 2 slaves and 3 sentinels In this example we consider that all servers have an internal network @@ -428,7 +432,7 @@ interface with IPs in the `10.0.0.x` range, and that they can connect to each other using these IPs. In a real world usage, you would also setup firewall rules to prevent -unauthorized access from other machines, and block traffic from the +unauthorized access from other machines and block traffic from the outside (Internet). We will use the same `3` nodes with **Redis** + **Sentinel** topology @@ -450,7 +454,7 @@ The same thing will happen with `sentinel.conf` that will be overridden after th initial execution, after any new sentinel node starts watching the **Master**, or a failover promotes a different **Master** node. -### Configuration for Redis Master +### Configuration for Redis master **Example configation for Redis Master:** @@ -464,9 +468,10 @@ redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' redis['master_password'] = 'redis-password-goes-here' ``` -Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` -### Configuration for Redis Slave +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +### Configuration for Redis slaves **Example configation for Slave 1:** @@ -502,9 +507,14 @@ redis['master_ip'] = '10.0.0.1' # IP of master Redis server #redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default ``` -Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +### Configuration for Sentinels -### Configuration for Sentinel (EE only) +>**Note:** +Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the +Omnibus Community Edition and installations from source, follow the +[Redis HA source install](redis_source.md) guide. Please note that some of the variables are already configured previously as they are required for Redis replication. @@ -517,9 +527,9 @@ In `/etc/gitlab/gitlab.rb`: redis_sentinel_role['enable'] = true redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance #redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default -redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel sentinel['bind'] = '10.0.0.1' @@ -571,9 +581,9 @@ In `/etc/gitlab/gitlab.rb`: redis_sentinel_role['enable'] = true redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance #redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default -redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel sentinel['bind'] = '10.0.0.2' @@ -625,9 +635,9 @@ In `/etc/gitlab/gitlab.rb`: redis_sentinel_role['enable'] = true redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance #redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default -redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel sentinel['bind'] = '10.0.0.3' @@ -671,10 +681,15 @@ sentinel['quorum'] = 2 # sentinel['failover_timeout'] = 60000 ``` +## Advanced configuration + +Omnibus GitLab configures some things behind the curtains to make the sysadmins' +lives easier. If you want to know what happens underneath keep reading. + ### Control running services -In the example above we've used `redis_sentinel_role` and `redis_master_role` -which simplify the amount of configuration changes. +In the previous example above we've used `redis_sentinel_role` and +`redis_master_role` which simplify the amount of configuration changes. If you want more control, here is what each one sets for you automatically when enabled: @@ -716,13 +731,15 @@ mailroom['enable'] = false redis['master'] = false ``` +You can find the relevant attributes defined in [gitlab_rails.rb][omnifile]. + ## Troubleshooting There are a lot of moving parts that needs to be taken care carefully in order for the HA setup to work as expected. -Before proceeding with the troubleshooting below, check your firewall -rules: +Before proceeding with the troubleshooting below, check your firewall rules: + - Redis machines - Accept TCP connection in `6379` - Connect to the other Redis machines via TCP in `6379` @@ -731,7 +748,7 @@ rules: - Connect to other Sentinel machines via TCP in `26379` - Connect to the Redis machines via TCP in `6379` -### Redis replication +### Troubleshooting Redis replication You can check if everything is correct by connecting to each server using `redis-cli` application, and sending the `INFO` command. @@ -781,9 +798,7 @@ repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 ``` -### Sentinel - -#### Omnibus GitLab +### Troubleshooting Sentinel If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related @@ -865,3 +880,7 @@ Read more on high-availability configuration: [gh-534]: https://github.com/redis/redis-rb/issues/534 [redis]: http://redis.io/ [sentinel]: http://redis.io/topics/sentinel +[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/82b7345b150f072c8673c79738ce893f92d0d652/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L134-159 +[source]: ../../install/installation.md +[ce]: https://about.gitlab.com/downloads +[ee]: https://about.gitlab.com/downloads-ee -- cgit v1.2.1 From 11510bf729e04ea6f4a713820582b490266505d2 Mon Sep 17 00:00:00 2001 From: awhildy Date: Fri, 21 Oct 2016 14:25:09 -0700 Subject: [ci skip] Establish basic structure for ux_guide README.md Block out pages needed for ux_guide Add resources stub to ux_guide home Fill out principles and basics Add TOC to basics Move all of UI guide to new UX guide structure Add first level structure on ux-guide pages Add more details to buttons Add button images. Update link on development Renamed surfaces to templates. Add tooltip details Update typography and icons on Basics page Add images for color. First draft of voice and tone Delete findings page Refine pages. Fill out Surfaces pages Clean up layout on basics, surfaces, features. Add anchorlinks and counts to components Fill out components page Add item title and system info block Fill out Features page Switch tooltip placement image --- doc/development/README.md | 2 +- doc/development/ux_guide/README.md | 59 +++++ doc/development/ux_guide/basics.md | 112 ++++++++ doc/development/ux_guide/components.md | 282 +++++++++++++++++++++ doc/development/ux_guide/features.md | 63 +++++ doc/development/ux_guide/img/button-primary.png | Bin 0 -> 8410 bytes doc/development/ux_guide/img/button-secondary.png | Bin 0 -> 11160 bytes doc/development/ux_guide/img/color-blue.png | Bin 0 -> 3865 bytes doc/development/ux_guide/img/color-green.png | Bin 0 -> 4127 bytes doc/development/ux_guide/img/color-grey.png | Bin 0 -> 2384 bytes doc/development/ux_guide/img/color-orange.png | Bin 0 -> 4698 bytes doc/development/ux_guide/img/color-red.png | Bin 0 -> 3669 bytes doc/development/ux_guide/img/components-alerts.png | Bin 0 -> 46785 bytes .../ux_guide/img/components-anchorlinks.png | Bin 0 -> 36456 bytes .../ux_guide/img/components-contentblock.png | Bin 0 -> 19841 bytes doc/development/ux_guide/img/components-counts.png | Bin 0 -> 2438 bytes .../ux_guide/img/components-coverblock.png | Bin 0 -> 15757 bytes .../ux_guide/img/components-dateexact.png | Bin 0 -> 5609 bytes .../ux_guide/img/components-daterelative.png | Bin 0 -> 5843 bytes .../ux_guide/img/components-dropdown.png | Bin 0 -> 60448 bytes .../ux_guide/img/components-fileholder.png | Bin 0 -> 4953 bytes .../ux_guide/img/components-horizontalform.png | Bin 0 -> 5708 bytes .../ux_guide/img/components-listinsidepanel.png | Bin 0 -> 3962 bytes .../ux_guide/img/components-listwithavatar.png | Bin 0 -> 7952 bytes .../ux_guide/img/components-listwithhover.png | Bin 0 -> 3313 bytes doc/development/ux_guide/img/components-panels.png | Bin 0 -> 32886 bytes .../ux_guide/img/components-referencehover.png | Bin 0 -> 11519 bytes .../ux_guide/img/components-referenceissues.png | Bin 0 -> 14587 bytes .../ux_guide/img/components-referencelabels.png | Bin 0 -> 4643 bytes .../ux_guide/img/components-referencemilestone.png | Bin 0 -> 2468 bytes .../ux_guide/img/components-referencemrs.png | Bin 0 -> 12646 bytes .../ux_guide/img/components-referencepeople.png | Bin 0 -> 7214 bytes .../ux_guide/img/components-rowcontentblock.png | Bin 0 -> 19730 bytes .../ux_guide/img/components-simplelist.png | Bin 0 -> 3078 bytes doc/development/ux_guide/img/components-table.png | Bin 0 -> 7668 bytes .../ux_guide/img/components-verticalform.png | Bin 0 -> 6541 bytes .../ux_guide/img/features-contextualnav.png | Bin 0 -> 8051 bytes .../ux_guide/img/features-emptystates.png | Bin 0 -> 114540 bytes doc/development/ux_guide/img/features-filters.png | Bin 0 -> 4529 bytes .../ux_guide/img/features-globalnav.png | Bin 0 -> 8953 bytes doc/development/ux_guide/img/icon-add.png | Bin 0 -> 155 bytes doc/development/ux_guide/img/icon-close.png | Bin 0 -> 225 bytes doc/development/ux_guide/img/icon-edit.png | Bin 0 -> 231 bytes doc/development/ux_guide/img/icon-notification.png | Bin 0 -> 259 bytes doc/development/ux_guide/img/icon-rss.png | Bin 0 -> 307 bytes doc/development/ux_guide/img/icon-subscribe.png | Bin 0 -> 348 bytes doc/development/ux_guide/img/icon-trash.png | Bin 0 -> 380 bytes .../ux_guide/img/monospacefont-sample.png | Bin 0 -> 14282 bytes .../ux_guide/img/sourcesanspro-sample.png | Bin 0 -> 10948 bytes .../ux_guide/img/surfaces-contentitemtitle.png | Bin 0 -> 7463 bytes doc/development/ux_guide/img/surfaces-header.png | Bin 0 -> 6103 bytes .../img/surfaces-systeminformationblock.png | Bin 0 -> 15412 bytes doc/development/ux_guide/img/surfaces-ux.png | Bin 0 -> 7673 bytes doc/development/ux_guide/img/tooltip-placement.png | Bin 0 -> 2645 bytes doc/development/ux_guide/img/tooltip-usage.png | Bin 0 -> 9160 bytes doc/development/ux_guide/principles.md | 29 +++ doc/development/ux_guide/resources.md | 13 + doc/development/ux_guide/surfaces.md | 53 ++++ doc/development/ux_guide/tips.md | 46 ++++ doc/development/ux_guide/users.md | 18 ++ 60 files changed, 676 insertions(+), 1 deletion(-) create mode 100644 doc/development/ux_guide/README.md create mode 100644 doc/development/ux_guide/basics.md create mode 100644 doc/development/ux_guide/components.md create mode 100644 doc/development/ux_guide/features.md create mode 100644 doc/development/ux_guide/img/button-primary.png create mode 100644 doc/development/ux_guide/img/button-secondary.png create mode 100644 doc/development/ux_guide/img/color-blue.png create mode 100644 doc/development/ux_guide/img/color-green.png create mode 100644 doc/development/ux_guide/img/color-grey.png create mode 100644 doc/development/ux_guide/img/color-orange.png create mode 100644 doc/development/ux_guide/img/color-red.png create mode 100644 doc/development/ux_guide/img/components-alerts.png create mode 100644 doc/development/ux_guide/img/components-anchorlinks.png create mode 100644 doc/development/ux_guide/img/components-contentblock.png create mode 100644 doc/development/ux_guide/img/components-counts.png create mode 100644 doc/development/ux_guide/img/components-coverblock.png create mode 100644 doc/development/ux_guide/img/components-dateexact.png create mode 100644 doc/development/ux_guide/img/components-daterelative.png create mode 100644 doc/development/ux_guide/img/components-dropdown.png create mode 100644 doc/development/ux_guide/img/components-fileholder.png create mode 100644 doc/development/ux_guide/img/components-horizontalform.png create mode 100644 doc/development/ux_guide/img/components-listinsidepanel.png create mode 100644 doc/development/ux_guide/img/components-listwithavatar.png create mode 100644 doc/development/ux_guide/img/components-listwithhover.png create mode 100644 doc/development/ux_guide/img/components-panels.png create mode 100644 doc/development/ux_guide/img/components-referencehover.png create mode 100644 doc/development/ux_guide/img/components-referenceissues.png create mode 100644 doc/development/ux_guide/img/components-referencelabels.png create mode 100644 doc/development/ux_guide/img/components-referencemilestone.png create mode 100644 doc/development/ux_guide/img/components-referencemrs.png create mode 100644 doc/development/ux_guide/img/components-referencepeople.png create mode 100644 doc/development/ux_guide/img/components-rowcontentblock.png create mode 100644 doc/development/ux_guide/img/components-simplelist.png create mode 100644 doc/development/ux_guide/img/components-table.png create mode 100644 doc/development/ux_guide/img/components-verticalform.png create mode 100644 doc/development/ux_guide/img/features-contextualnav.png create mode 100644 doc/development/ux_guide/img/features-emptystates.png create mode 100644 doc/development/ux_guide/img/features-filters.png create mode 100644 doc/development/ux_guide/img/features-globalnav.png create mode 100644 doc/development/ux_guide/img/icon-add.png create mode 100644 doc/development/ux_guide/img/icon-close.png create mode 100644 doc/development/ux_guide/img/icon-edit.png create mode 100644 doc/development/ux_guide/img/icon-notification.png create mode 100644 doc/development/ux_guide/img/icon-rss.png create mode 100644 doc/development/ux_guide/img/icon-subscribe.png create mode 100644 doc/development/ux_guide/img/icon-trash.png create mode 100644 doc/development/ux_guide/img/monospacefont-sample.png create mode 100644 doc/development/ux_guide/img/sourcesanspro-sample.png create mode 100644 doc/development/ux_guide/img/surfaces-contentitemtitle.png create mode 100644 doc/development/ux_guide/img/surfaces-header.png create mode 100644 doc/development/ux_guide/img/surfaces-systeminformationblock.png create mode 100644 doc/development/ux_guide/img/surfaces-ux.png create mode 100644 doc/development/ux_guide/img/tooltip-placement.png create mode 100644 doc/development/ux_guide/img/tooltip-usage.png create mode 100644 doc/development/ux_guide/principles.md create mode 100644 doc/development/ux_guide/resources.md create mode 100644 doc/development/ux_guide/surfaces.md create mode 100644 doc/development/ux_guide/tips.md create mode 100644 doc/development/ux_guide/users.md (limited to 'doc') diff --git a/doc/development/README.md b/doc/development/README.md index bf1f054b7d5..87306b1501d 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -14,7 +14,7 @@ contributing to documentation. - [SQL Migration Style Guide](migration_style_guide.md) for creating safe SQL migrations - [Testing standards and style guidelines](testing.md) -- [UI guide](ui_guide.md) for building GitLab with existing CSS styles and elements +- [UX guide](ux_guide/README.md) for building GitLab with existing CSS styles and elements - [Frontend guidelines](frontend.md) - [SQL guidelines](sql.md) for working with SQL queries - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers diff --git a/doc/development/ux_guide/README.md b/doc/development/ux_guide/README.md new file mode 100644 index 00000000000..e17a4559069 --- /dev/null +++ b/doc/development/ux_guide/README.md @@ -0,0 +1,59 @@ +# GitLab UX Guide + +The goal of this guide is to provide standards, principles and in-depth information to design beautiful and effective GitLab features. This will be a living document, and we welcome contributions, feedback and suggestions. + +## Design + +--- + +### [Principles](principles.md) +These guiding principles set a solid foundation for our design system, and should remain relatively stable over multiple releases. They should be referenced as new design patterns are created. + +--- + +### [Basics](basics.md) +The basic ingredients of our experience establish our personality and feel. This section includes details about typography, color, and motion. + +--- + +### [Components](components.md) +Components are the controls that make up the GitLab experience, including guidance around buttons, links, dropdowns, etc. + +--- + +### [Surfaces](surfaces.md) +The GitLab experience is broken apart into several surfaces. Each of these surfaces is designated for a specific scope or type of content. Examples include the header, global menu, side pane, etc. + +--- + +### [Features](features.md) +The previous building blocks are combined into complete features in the GitLab UX. Examples include our navigation, filters, search results, and empty states. + +--- + +
+ +## Research + +--- + +### [Users](users.md) +How we think about the variety of users of GitLab, from small to large teams, comparing opensource usage to enterprise, etc. + +--- + +
+ +## Other + +--- + +### [Tips for designers](tips.md) +Tips for exporting assets, and other guidance. + +--- + +### [Resources](resources.md) +Resources for GitLab UX + +--- diff --git a/doc/development/ux_guide/basics.md b/doc/development/ux_guide/basics.md new file mode 100644 index 00000000000..7e7cb103694 --- /dev/null +++ b/doc/development/ux_guide/basics.md @@ -0,0 +1,112 @@ +# Basics + +## Contents +* [Responsive](#responsive) +* [Typography](#typography) +* [Icons](#icons) +* [Color](#color) +* [Motion](#motion) +* [Voice and tone](#voice-and-tone) + +--- + +
+ +## Responsive +GitLab is a responsive experience that works well across all screen sizes, from mobile devices to large monitors. In order to provide a great user experience, the core functionality (browsing files, creating issues, writing comments, etc.) must be available at all resolutions. However, due to size limitations, some secondary functionality may be hidden on smaller screens. Please keep this functionality limited to rare actions that aren't expected to be needed on small devices. + +--- + +
+ +## Typography +### Primary typeface +GitLab's main typeface used throughout the UI is **Source Sans Pro**. We support both the bold and regular weight. + +![Source Sans Pro sample](img/sourcesanspro-sample.png) + + +### Monospace typeface +This is the typeface used for code blocks. GitLab uses the OS default font. +- **Menlo** (Mac) +- **Consolas** (Windows) +- **Liberation Mono** (Linux) + +![Monospace font sample](img/monospacefont-sample.png) + +--- + +
+ +## Icons +GitLab uses Font Awesome icons throughout our interface. + +![Trash icon](img/icon-trash.png) +The trash icon is used for destructive actions that deletes information. + +![Edit icon](img/icon-edit.png) +The pencil icon is used for editing content such as comments. + +![Notification icon](img/icon-notification.png) +The bell icon is for notifications, such as Todos. + +![Subscribe icon](img/icon-subscribe.png) +The eye icon is for subscribing to updates. For example, you can subscribe to a label and get updated on issues with that label. + +![RSS icon](img/icon-rss.png) +The standard RSS icon is used for linking to RSS/atom feeds. + +![Close icon](img/icon-close.png) +An 'x' is used for closing UI elements such as dropdowns. + +![Add icon](img/icon-add.png) +A plus is used when creating new objects, such as issues, projects, etc. + +>>> +TODO: update this section, add more general guidance to icon usage and personality, etc. +>>> + +--- + +
+ +## Color + +![Blue](img/color-blue.png) +Blue is used to highlight primary active elements (such as current tab), as well as other organization and managing commands. + +![Green](img/color-green.png) +Green is for actions that create new objects. + +![Orange](img/color-orange.png) +Orange is used for warnings + +![Red](img/color-red.png) +Red is reserved for delete and other destructive commands + +![Grey](img/color-grey.png) +Grey, and white (depending on context) is used for netral, secondary elements + +>>> +TODO: Establish a perspective for color in terms of our personality and rationalize with Marketing usage. +>>> + +--- + +
+ +## Motion + +Motion is a tool to help convey important relationships, changes or transitions between elements. It should be used sparingly and intentionally, highlighting the right elements at the right moment. + +>>> +TODO: Determine a more concrete perspective on motion, create consistent easing/timing curves to follow. +>>> + +--- + +
+ +## Voice and tone + +The copy for GitLab is clear and direct. We strike a clear balance between professional and friendly. We can empathesize with users (such as celebrating completing all Todos), and remain respectful of the importance of the work. We are that trusted, friendly coworker that is helpful and understanding. \ No newline at end of file diff --git a/doc/development/ux_guide/components.md b/doc/development/ux_guide/components.md new file mode 100644 index 00000000000..6b21566265c --- /dev/null +++ b/doc/development/ux_guide/components.md @@ -0,0 +1,282 @@ +# Components + +## Contents +* [Tooltips](#tooltips) +* [Anchor links](#anchor-links) +* [Buttons](#buttons) +* [Dropdowns](#dropdowns) +* [Counts](#counts) +* [Lists](#lists) +* [Tables](#tables) +* [Blocks](#blocks) +* [Panels](#panels) +* [Alerts](#alerts) +* [Forms](#forms) +* [File holders](#file-holders) +* [Data formats](#data-formats) + +--- + +
+ +## Tooltips + +### Usage +A tooltip should only be added if additional information is required. + +![Tooltip usage](img/tooltip-usage.png) + +### Placement +By default, tooltips should be placed below the element that they refer to. However, if there is not enough space in the viewpoint, the tooltip should be moved to the side as needed. + +![Tooltip placement location](img/tooltip-placement.png) + +--- + +
+ +## Anchor links + +Anchor links are used for navigational actions and lone, secondary commands (such as 'Reset filters' on the Issues List) when deemed appropriate by the UX team. + +### States + +#### Rest + +Primary links are blue in their rest state. Secondary links (such as the time stamp on comments) are a neutral gray color in rest. Details on the main GitLab navigation links can be found on the [features](features.md#navigation) page. + +#### Hover + +An underline should always be added on hover. A gray link becomes blue on hover. + +#### Focus + +The focus state should match the hover state. + +![Anchor link states ](img/components-anchorlinks.png) + +--- + +
+ +## Buttons + +Buttons communicate the command that will occur when the user clicks on them. + +### Types + +#### Primary +Primary buttons communicate the main call to action. There should only be one call to action in any given experience. Visually, primary buttons are conveyed with a full background fill + +![Primary button example](img/button-primary.png) + +#### Secondary +Secondary buttons are for alternative commands. They should be conveyed by a button with an stroke, and no background fill. + +![Secondary button example](img/button-secondary.png) + +### Icon and text treatment +Text should be in sentence case, where only the first word is capitalized. "Create issue" is correct, not "Create Issue". Buttons should only contain an icon or a text, not both. + +>>> +TODO: Rationalize this. Ensure that we still believe this. +>>> + +### Colors +Follow the color guidance on the [basics](basics.md#color) page. The default color treatment is the white/grey button. + +--- + +
+ +## Dropdowns + +Dropdowns are used to allow users to choose one (or many) options from a list of options. If this list of options is more 20, there should generally be a way to search through and filter the options (see the complex filter dropdowns below.) + +>>> +TODO: Will update this section when the new filters UI is implemented. +>>> + +![Dropdown states](img/components-dropdown.png) + + + +--- + +
+ +## Counts + +A count element is used in navigation contexts where it is helpful to indicate the count, or number of items, in a list. Always use the [`number_with_delimiter`][number_with_delimiter] helper to display counts in the UI. + +![Counts example](img/components-counts.png) + +[number_with_delimiter]: http://api.rubyonrails.org/classes/ActionView/Helpers/NumberHelper.html#method-i-number_with_delimiter + +--- + +
+ +## Lists + +Lists are used where ever there is a single column of information to display. Ths [issues list](https://gitlab.com/gitlab-org/gitlab-ce/issues) is an example of a important list in the GitLab UI. + +### Types + +Simple list using .content-list + +![Simple list](img/components-simplelist.png) + +List with avatar, title and description using .content-list + +![List with avatar](img/components-listwithavatar.png) + +List with hover effect .well-list + +![List with hover effect](img/components-listwithhover.png) + +List inside panel + +![List inside panel](img/components-listinsidepanel.png) + +--- + +
+ +## Tables + +When the information is too complex for a list, with multiple columns of information, a table can be used. For example, the [pipelines page](https://gitlab.com/gitlab-org/gitlab-ce/pipelines) uses a table. + +![Table](img/components-table.png) + +--- + +
+ +## Blocks + +Blocks are a way to group related information. + +### Types + +#### Content blocks + +Content blocks (`.content-block`) are the basic grouping of content. They are commonly used in [lists](#lists), and are separated by a botton border. + +![Content block](img/components-contentblock.png) + +#### Row content blocks + +A background color can be added to this blocks. For example, items in the [issue list](https://gitlab.com/gitlab-org/gitlab-ce/issues) have a green background if they were created recently. Below is an example of a gray content block with side padding using `.row-content-block`. + +![Row content block](img/components-rowcontentblock.png) + +#### Cover blocks +Cover blocks are generally used to create a heading element for a page, such as a new project, or a user profile page. Below is a cover block (`.cover-block`) for the profile page with an avatar, name and description. + +![Cover block](img/components-coverblock.png) + +--- + +
+ +## Panels + +>>> +TODO: Catalog how we are currently using panels and rationalize how they relate to alerts +>>> + +![Panels](img/components-panels.png) + +--- + +
+ +## Alerts + +>>> +TODO: Catalog how we are currently using alerts +>>> + +![Alerts](img/components-alerts.png) + +--- + +
+ +## Forms + +There are two options shown below regarding the positioning of labels in forms. Both are options to consider based on context and available size. However, it is important to have a consistent treatment of labels in the same form. + +### Types + +#### Labels stack vertically + +Form (`form`) with label rendered above input. + +![Vertical form](img/components-verticalform.png) + +#### Labels side-by-side + +Horizontal form (`form.horizontal-form`) with label rendered inline with input. + +![Horizontal form](img/components-horizontalform.png) + +--- + +
+ +## File holders +A file holder (`.file-holder`) is used to show the contents of a file inline on a page of GitLab. + +![File Holder component](img/components-fileholder.png) + +--- + +
+ +## Data formats + +### Dates + +#### Exact + +Format for exacts dates should be ‘Mon DD, YYYY’, such as the examples below. + +![Exact date](img/components-dateexact.png) + +#### Relative + +This format relates how long since an action has occurred. The exact date can be shown as a tooltip on hover. + +![Relative date](img/components-daterelative.png) + +### References + +Referencing GitLab items depends on a symbol for each type of item. Typing that symbol will invoke a dropdown that allows you to search for and autocomplete the item you were looking for. References are shown as [links](#links) in context, and hovering on them shows the full title or name of the item. + +![Hovering on a reference](img/components-referencehover.png) + +#### `%` Milestones + +![Milestone reference](img/components-referencemilestone.png) + +#### `#` Issues + +![Issue reference](img/components-referenceissues.png) + +#### `!` Merge Requests + +![Merge request reference](img/components-referencemrs.png) + +#### `~` Labels + +![Labels reference](img/components-referencelabels.png) + +#### `@` People + +![People reference](img/components-referencepeople.png) + +>>> +Open issue: Some commit references use monospace fonts, but others don't. Need to standardize this. +>>> diff --git a/doc/development/ux_guide/features.md b/doc/development/ux_guide/features.md new file mode 100644 index 00000000000..00700af72aa --- /dev/null +++ b/doc/development/ux_guide/features.md @@ -0,0 +1,63 @@ +# Features + +## Contents +* [Navigation](#navigation) +* [Filtering](#filtering) +* [Search results](#search-results) +* [Conversations](#conversations) +* [Empty states](#empty-states) + +--- + +## Navigation + +### Global navigation + +The global navigation is accessible via the menu button on the top left of the screen, and can be pinned to keep it open. It contains a consistent list of pages that allow you to view content that is across GitLab. For example, you can view your todos, issues and merge requests across projects and groups. + +![Global nav](img/features-globalnav.png) + + +### Contextual navigation + +The navigation in the header is contextual to each page. These options change depending on if you are looking at a project, group, or settings page. There should be no more than 10 items on a level in the contextual navigation, allowing it to comfortably fit on a typical laptop screen. There can be up to too levels of navigation. Each sub nav group should be a self-contained group of functionality. For example, everything related to the issue tracker should be under the 'Issue' tab, while everything relating to the wiki will be grouped under the 'Wiki' tab. The names used for each section should be short and easy to remember, ideally 1-2 words in length. + +![Contextual nav](img/features-contextualnav.png) + +### Information architecture + +The [GitLab Product Map](https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png) shows a visual representation of the information architecture for GitLab. + +--- + +
+ +## Filtering + +Today, lists are filtered by a series of dropdowns. Some of these dropdowns allow multiselect (labels), while others allow you to filter to one option (milestones). However, we are currently implementing a [new model](https://gitlab.com/gitlab-org/gitlab-ce/issues/21747) for this, and will update the guide when it is ready. + +![Filters](img/features-filters.png) + +--- + +
+ +## Search results + +### Global search + +[Global search](https://gitlab.com/search?group_id=&project_id=13083&repository_ref=&scope=issues&search=mobile) allows you to search across items in a project, or even across multiple projects. You can switch tabs to filter on type of object, or filter by group. + +### List search + +There are several core lists in the GitLab experience, such as the Issue list and the Merge Request list. You are also able to [filter and search these lists](https://gitlab.com/gitlab-org/gitlab-ce/issues?utf8=%E2%9C%93&search=mobile). This UI will be updated with the [new filtering model](https://gitlab.com/gitlab-org/gitlab-ce/issues/21747). + +--- + +
+ +## Empty states + +Empty states need to be considered in the design of features. They are vital to helping onboard new users, making the experience feel more approachable and understandable. Empty states should feel inviting and provide just enough information to get people started. There should be a single call to action and a clear explanation of what to use the feature for. + +![Empty states](img/features-emptystates.png) diff --git a/doc/development/ux_guide/img/button-primary.png b/doc/development/ux_guide/img/button-primary.png new file mode 100644 index 00000000000..f4c673f5b88 Binary files /dev/null and b/doc/development/ux_guide/img/button-primary.png differ diff --git a/doc/development/ux_guide/img/button-secondary.png b/doc/development/ux_guide/img/button-secondary.png new file mode 100644 index 00000000000..57fa65b247c Binary files /dev/null and b/doc/development/ux_guide/img/button-secondary.png differ diff --git a/doc/development/ux_guide/img/color-blue.png b/doc/development/ux_guide/img/color-blue.png new file mode 100644 index 00000000000..6449613eb16 Binary files /dev/null and b/doc/development/ux_guide/img/color-blue.png differ diff --git a/doc/development/ux_guide/img/color-green.png b/doc/development/ux_guide/img/color-green.png new file mode 100644 index 00000000000..15475b36f02 Binary files /dev/null and b/doc/development/ux_guide/img/color-green.png differ diff --git a/doc/development/ux_guide/img/color-grey.png b/doc/development/ux_guide/img/color-grey.png new file mode 100644 index 00000000000..58c474d5ce9 Binary files /dev/null and b/doc/development/ux_guide/img/color-grey.png differ diff --git a/doc/development/ux_guide/img/color-orange.png b/doc/development/ux_guide/img/color-orange.png new file mode 100644 index 00000000000..f4fc09b2d9b Binary files /dev/null and b/doc/development/ux_guide/img/color-orange.png differ diff --git a/doc/development/ux_guide/img/color-red.png b/doc/development/ux_guide/img/color-red.png new file mode 100644 index 00000000000..6fbbf0a885d Binary files /dev/null and b/doc/development/ux_guide/img/color-red.png differ diff --git a/doc/development/ux_guide/img/components-alerts.png b/doc/development/ux_guide/img/components-alerts.png new file mode 100644 index 00000000000..0b2ecc16a5f Binary files /dev/null and b/doc/development/ux_guide/img/components-alerts.png differ diff --git a/doc/development/ux_guide/img/components-anchorlinks.png b/doc/development/ux_guide/img/components-anchorlinks.png new file mode 100644 index 00000000000..950f348277d Binary files /dev/null and b/doc/development/ux_guide/img/components-anchorlinks.png differ diff --git a/doc/development/ux_guide/img/components-contentblock.png b/doc/development/ux_guide/img/components-contentblock.png new file mode 100644 index 00000000000..31fc1eec9df Binary files /dev/null and b/doc/development/ux_guide/img/components-contentblock.png differ diff --git a/doc/development/ux_guide/img/components-counts.png b/doc/development/ux_guide/img/components-counts.png new file mode 100644 index 00000000000..19280e988a0 Binary files /dev/null and b/doc/development/ux_guide/img/components-counts.png differ diff --git a/doc/development/ux_guide/img/components-coverblock.png b/doc/development/ux_guide/img/components-coverblock.png new file mode 100644 index 00000000000..c8f1f87a108 Binary files /dev/null and b/doc/development/ux_guide/img/components-coverblock.png differ diff --git a/doc/development/ux_guide/img/components-dateexact.png b/doc/development/ux_guide/img/components-dateexact.png new file mode 100644 index 00000000000..8c0c5c1be40 Binary files /dev/null and b/doc/development/ux_guide/img/components-dateexact.png differ diff --git a/doc/development/ux_guide/img/components-daterelative.png b/doc/development/ux_guide/img/components-daterelative.png new file mode 100644 index 00000000000..1dc6d89e4ef Binary files /dev/null and b/doc/development/ux_guide/img/components-daterelative.png differ diff --git a/doc/development/ux_guide/img/components-dropdown.png b/doc/development/ux_guide/img/components-dropdown.png new file mode 100644 index 00000000000..5770a393b37 Binary files /dev/null and b/doc/development/ux_guide/img/components-dropdown.png differ diff --git a/doc/development/ux_guide/img/components-fileholder.png b/doc/development/ux_guide/img/components-fileholder.png new file mode 100644 index 00000000000..4b8962905d6 Binary files /dev/null and b/doc/development/ux_guide/img/components-fileholder.png differ diff --git a/doc/development/ux_guide/img/components-horizontalform.png b/doc/development/ux_guide/img/components-horizontalform.png new file mode 100644 index 00000000000..92e28cf9afc Binary files /dev/null and b/doc/development/ux_guide/img/components-horizontalform.png differ diff --git a/doc/development/ux_guide/img/components-listinsidepanel.png b/doc/development/ux_guide/img/components-listinsidepanel.png new file mode 100644 index 00000000000..30ceb3eaa08 Binary files /dev/null and b/doc/development/ux_guide/img/components-listinsidepanel.png differ diff --git a/doc/development/ux_guide/img/components-listwithavatar.png b/doc/development/ux_guide/img/components-listwithavatar.png new file mode 100644 index 00000000000..d3cb0ebc02b Binary files /dev/null and b/doc/development/ux_guide/img/components-listwithavatar.png differ diff --git a/doc/development/ux_guide/img/components-listwithhover.png b/doc/development/ux_guide/img/components-listwithhover.png new file mode 100644 index 00000000000..1484ecba6a0 Binary files /dev/null and b/doc/development/ux_guide/img/components-listwithhover.png differ diff --git a/doc/development/ux_guide/img/components-panels.png b/doc/development/ux_guide/img/components-panels.png new file mode 100644 index 00000000000..6e71d0ad9c9 Binary files /dev/null and b/doc/development/ux_guide/img/components-panels.png differ diff --git a/doc/development/ux_guide/img/components-referencehover.png b/doc/development/ux_guide/img/components-referencehover.png new file mode 100644 index 00000000000..e9fb27e2aa9 Binary files /dev/null and b/doc/development/ux_guide/img/components-referencehover.png differ diff --git a/doc/development/ux_guide/img/components-referenceissues.png b/doc/development/ux_guide/img/components-referenceissues.png new file mode 100644 index 00000000000..caf9477db38 Binary files /dev/null and b/doc/development/ux_guide/img/components-referenceissues.png differ diff --git a/doc/development/ux_guide/img/components-referencelabels.png b/doc/development/ux_guide/img/components-referencelabels.png new file mode 100644 index 00000000000..a122b45d1f1 Binary files /dev/null and b/doc/development/ux_guide/img/components-referencelabels.png differ diff --git a/doc/development/ux_guide/img/components-referencemilestone.png b/doc/development/ux_guide/img/components-referencemilestone.png new file mode 100644 index 00000000000..5aa9ecd1a78 Binary files /dev/null and b/doc/development/ux_guide/img/components-referencemilestone.png differ diff --git a/doc/development/ux_guide/img/components-referencemrs.png b/doc/development/ux_guide/img/components-referencemrs.png new file mode 100644 index 00000000000..6280243859a Binary files /dev/null and b/doc/development/ux_guide/img/components-referencemrs.png differ diff --git a/doc/development/ux_guide/img/components-referencepeople.png b/doc/development/ux_guide/img/components-referencepeople.png new file mode 100644 index 00000000000..99772a539cf Binary files /dev/null and b/doc/development/ux_guide/img/components-referencepeople.png differ diff --git a/doc/development/ux_guide/img/components-rowcontentblock.png b/doc/development/ux_guide/img/components-rowcontentblock.png new file mode 100644 index 00000000000..1c2d7096955 Binary files /dev/null and b/doc/development/ux_guide/img/components-rowcontentblock.png differ diff --git a/doc/development/ux_guide/img/components-simplelist.png b/doc/development/ux_guide/img/components-simplelist.png new file mode 100644 index 00000000000..892f507cfc2 Binary files /dev/null and b/doc/development/ux_guide/img/components-simplelist.png differ diff --git a/doc/development/ux_guide/img/components-table.png b/doc/development/ux_guide/img/components-table.png new file mode 100644 index 00000000000..7e964c885cf Binary files /dev/null and b/doc/development/ux_guide/img/components-table.png differ diff --git a/doc/development/ux_guide/img/components-verticalform.png b/doc/development/ux_guide/img/components-verticalform.png new file mode 100644 index 00000000000..38863ad3c1c Binary files /dev/null and b/doc/development/ux_guide/img/components-verticalform.png differ diff --git a/doc/development/ux_guide/img/features-contextualnav.png b/doc/development/ux_guide/img/features-contextualnav.png new file mode 100644 index 00000000000..df157f54c84 Binary files /dev/null and b/doc/development/ux_guide/img/features-contextualnav.png differ diff --git a/doc/development/ux_guide/img/features-emptystates.png b/doc/development/ux_guide/img/features-emptystates.png new file mode 100644 index 00000000000..3befc14588e Binary files /dev/null and b/doc/development/ux_guide/img/features-emptystates.png differ diff --git a/doc/development/ux_guide/img/features-filters.png b/doc/development/ux_guide/img/features-filters.png new file mode 100644 index 00000000000..281e55d590c Binary files /dev/null and b/doc/development/ux_guide/img/features-filters.png differ diff --git a/doc/development/ux_guide/img/features-globalnav.png b/doc/development/ux_guide/img/features-globalnav.png new file mode 100644 index 00000000000..3c0db2247ca Binary files /dev/null and b/doc/development/ux_guide/img/features-globalnav.png differ diff --git a/doc/development/ux_guide/img/icon-add.png b/doc/development/ux_guide/img/icon-add.png new file mode 100644 index 00000000000..0d4c1a7692a Binary files /dev/null and b/doc/development/ux_guide/img/icon-add.png differ diff --git a/doc/development/ux_guide/img/icon-close.png b/doc/development/ux_guide/img/icon-close.png new file mode 100644 index 00000000000..88d2b3b0c6d Binary files /dev/null and b/doc/development/ux_guide/img/icon-close.png differ diff --git a/doc/development/ux_guide/img/icon-edit.png b/doc/development/ux_guide/img/icon-edit.png new file mode 100644 index 00000000000..f73be7a10fb Binary files /dev/null and b/doc/development/ux_guide/img/icon-edit.png differ diff --git a/doc/development/ux_guide/img/icon-notification.png b/doc/development/ux_guide/img/icon-notification.png new file mode 100644 index 00000000000..4758632edd7 Binary files /dev/null and b/doc/development/ux_guide/img/icon-notification.png differ diff --git a/doc/development/ux_guide/img/icon-rss.png b/doc/development/ux_guide/img/icon-rss.png new file mode 100644 index 00000000000..c7ac9fb1349 Binary files /dev/null and b/doc/development/ux_guide/img/icon-rss.png differ diff --git a/doc/development/ux_guide/img/icon-subscribe.png b/doc/development/ux_guide/img/icon-subscribe.png new file mode 100644 index 00000000000..5cb277bfd5d Binary files /dev/null and b/doc/development/ux_guide/img/icon-subscribe.png differ diff --git a/doc/development/ux_guide/img/icon-trash.png b/doc/development/ux_guide/img/icon-trash.png new file mode 100644 index 00000000000..357289a6fff Binary files /dev/null and b/doc/development/ux_guide/img/icon-trash.png differ diff --git a/doc/development/ux_guide/img/monospacefont-sample.png b/doc/development/ux_guide/img/monospacefont-sample.png new file mode 100644 index 00000000000..1cd290b713c Binary files /dev/null and b/doc/development/ux_guide/img/monospacefont-sample.png differ diff --git a/doc/development/ux_guide/img/sourcesanspro-sample.png b/doc/development/ux_guide/img/sourcesanspro-sample.png new file mode 100644 index 00000000000..f7ecf0c7c66 Binary files /dev/null and b/doc/development/ux_guide/img/sourcesanspro-sample.png differ diff --git a/doc/development/ux_guide/img/surfaces-contentitemtitle.png b/doc/development/ux_guide/img/surfaces-contentitemtitle.png new file mode 100644 index 00000000000..2eb926c1c43 Binary files /dev/null and b/doc/development/ux_guide/img/surfaces-contentitemtitle.png differ diff --git a/doc/development/ux_guide/img/surfaces-header.png b/doc/development/ux_guide/img/surfaces-header.png new file mode 100644 index 00000000000..ab44d4de696 Binary files /dev/null and b/doc/development/ux_guide/img/surfaces-header.png differ diff --git a/doc/development/ux_guide/img/surfaces-systeminformationblock.png b/doc/development/ux_guide/img/surfaces-systeminformationblock.png new file mode 100644 index 00000000000..5d91e993e24 Binary files /dev/null and b/doc/development/ux_guide/img/surfaces-systeminformationblock.png differ diff --git a/doc/development/ux_guide/img/surfaces-ux.png b/doc/development/ux_guide/img/surfaces-ux.png new file mode 100644 index 00000000000..e692c51e8c0 Binary files /dev/null and b/doc/development/ux_guide/img/surfaces-ux.png differ diff --git a/doc/development/ux_guide/img/tooltip-placement.png b/doc/development/ux_guide/img/tooltip-placement.png new file mode 100644 index 00000000000..29a61c8400a Binary files /dev/null and b/doc/development/ux_guide/img/tooltip-placement.png differ diff --git a/doc/development/ux_guide/img/tooltip-usage.png b/doc/development/ux_guide/img/tooltip-usage.png new file mode 100644 index 00000000000..e8e4c6ded91 Binary files /dev/null and b/doc/development/ux_guide/img/tooltip-usage.png differ diff --git a/doc/development/ux_guide/principles.md b/doc/development/ux_guide/principles.md new file mode 100644 index 00000000000..32520845e41 --- /dev/null +++ b/doc/development/ux_guide/principles.md @@ -0,0 +1,29 @@ +# Principles + +These are the guiding principles that we should strive for to establish a solid foundation for the GitLab experience. + +
+ +## Professional and productive +GitLab is a tool to support what people do, day in, day out. We need to respect the importance of their work, and avoid gimicky details. + +
+ +## Minimal and efficient +While work can get complicated, GitLab is about bringing a sharp focus, helping our customers know what matters now. + +
+ +## Immediately recognizable +When you look at any screen, you should know immediately that it is GitLab. Our personality is strong and consistent across product and marketing experiences. + +
+ +## Human and quirky +We need to build empathy with our users, understanding their state of mind, and connect with them at a human level. Quirkiness is part of our DNA, and we should embrace it in the right moments and contexts. + +
+ +>>> +TODO: Ensure these principles align well with the goals of the Marketing team +>>> \ No newline at end of file diff --git a/doc/development/ux_guide/resources.md b/doc/development/ux_guide/resources.md new file mode 100644 index 00000000000..2f760c94414 --- /dev/null +++ b/doc/development/ux_guide/resources.md @@ -0,0 +1,13 @@ +# Resources + +## GitLab UI development kit + +We created a page inside GitLab where you can check commonly used html and css elements. + +When you run GitLab instance locally - just visit http://localhost:3000/help/ui page to see UI examples +you can use during GitLab development. + +## Design repository + +All design files are stored in the [gitlab-design](https://gitlab.com/gitlab-org/gitlab-design) +repository and maintained by GitLab UX designers. \ No newline at end of file diff --git a/doc/development/ux_guide/surfaces.md b/doc/development/ux_guide/surfaces.md new file mode 100644 index 00000000000..d9e48a66185 --- /dev/null +++ b/doc/development/ux_guide/surfaces.md @@ -0,0 +1,53 @@ +# Surfaces + +## Contents +* [Header](#header) +* [Global menu](#global-menu) +* [Side pane](#side-pane) +* [Content area](#content-area) + +--- + +![Surfaces UX](img/surfaces-ux.png) + +## Global menu + +This menu is to navigate to pages that contain content global to GitLab. + +--- + +
+ +## Header + +The header contains 3 main elements: Project switching and searching, user account avatar and settings, and a contextual menu that changes based on the current page. + +![Surfaces Header](img/surfaces-header.png) + +--- + +
+ +## Side pane + +The side pane holds supporting information and meta data for the information in the content area. + +--- + +
+ +## Content area + +The main content of the page. The content area can include other surfaces. + +### Item title bar + +The item title bar contains the top level information to identify the item, such as the name, id and status. + +![Item title](img/surfaces-contentitemtitle.png) + +### Item system information + +The system information block contains relevant system controlled information. + +![Item system information](img/surfaces-systeminformationblock.png) \ No newline at end of file diff --git a/doc/development/ux_guide/tips.md b/doc/development/ux_guide/tips.md new file mode 100644 index 00000000000..190ce9a2ee9 --- /dev/null +++ b/doc/development/ux_guide/tips.md @@ -0,0 +1,46 @@ +# Tips + +## Contents +* [SVGs](#svgs) + +--- + +## SVGs + +When exporting SVGs, be sure to follow the following guidelines: + +1. Convert all strokes to outlines. +2. Use pathfinder tools to combine overlapping paths and create compound paths. +3. SVGs that are limited to one color should be exported without a fill color so the color can be set using CSS. +4. Ensure that exported SVGs have been run through an [SVG cleaner](https://github.com/RazrFalcon/SVGCleaner) to remove unused elements and attributes. + +You can open your svg in a text editor to ensure that it is clean. +Incorrect files will look like this: + +```xml + + + + Group + Created with Sketch. + + + + + + + + + + +``` + +Correct file will look like this: + +```xml + +``` + +>>> +TODO: Checkout [https://github.com/svg/svgo](https://github.com/svg/svgo) +>>> \ No newline at end of file diff --git a/doc/development/ux_guide/users.md b/doc/development/ux_guide/users.md new file mode 100644 index 00000000000..cab4190604e --- /dev/null +++ b/doc/development/ux_guide/users.md @@ -0,0 +1,18 @@ +# Users + +>>> +TODO: Create personas. Understand the similarities and differences across the below spectrums. +>>> + +## Users by organization + +- Enterprise +- Medium company +- Small company +- Open source communities + +## Users by role + +- Admin +- Manager +- Developer \ No newline at end of file -- cgit v1.2.1 From b2a4a7fec72908001820c912203b67579e7a9582 Mon Sep 17 00:00:00 2001 From: Chris Wilson Date: Fri, 11 Nov 2016 06:30:33 +0000 Subject: Use GitLab.com link, remove GitHub link --- doc/integration/shibboleth.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index 5210ce0de9a..eb9bbb67e7d 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -10,7 +10,7 @@ To enable the Shibboleth OmniAuth provider you must: 1. Configure Apache shibboleth module. Installation and configuration of module it self is out of scope of this document. Check https://wiki.shibboleth.net/ for more info. -1. You can find Apache config in gitlab-recipes (https://github.com/gitlabhq/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf) +1. You can find Apache config in gitlab-recipes (https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache) Following changes are needed to enable shibboleth: -- cgit v1.2.1 From cc2bf503dbeb23561302bb77ff824e0c30a560b6 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 11 Nov 2016 17:53:26 +0100 Subject: Add missing link to GH issue --- doc/administration/high_availability/redis_source.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index ab5bb13f070..20630b070c0 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -8,7 +8,8 @@ If you are building packages for a specific distro, or trying to build some internal automation, you can check this documentation to learn about the minimal setup, required changes, etc. -If you want to see the documentation for Omnibus GitLab Install, please [read it here](redis.md). +If you want to see the documentation for Omnibus GitLab Install, please +[read it here](redis.md). @@ -38,7 +39,7 @@ You will also need to define equal password for slave password definition To configure Redis to use TCP connection you need to define both `bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the -ip of the desired interface (for ex. one from an internal network). +IP of the desired interface (for ex. one from an internal network). ### Configuring Master Redis instance @@ -46,7 +47,7 @@ ip of the desired interface (for ex. one from an internal network). You need to make the following changes in `redis.conf`: 1. Define a `bind` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external acessible IP, make + can reach you. If you really need to bind to an external accessible IP, make sure you add extra firewall rules to prevent unauthorized access: 1. Define a `port` to force redis to listen on TCP so other machines can @@ -79,7 +80,7 @@ starting with `sentinel` prefix. You will need to define the initial configs to enable connectivity: 1. Define a `bind` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external acessible IP, make + can reach you. If you really need to bind to an external accessible IP, make sure you add extra firewall rules to prevent unauthorized access: 1. Define a `port` to force sentinel to listen on TCP so other machines can @@ -95,7 +96,7 @@ And the sentinel specific ones: If you need more information to understand about quorum, please read the detailed explanation in the [HA documentation for Omnibus Installs](redis.md). -1. Define with `sentinel down-after-milliseconds` the ammount in `ms` of time +1. Define with `sentinel down-after-milliseconds` the amount in `ms` of time that an unresponsive server will be considered down. 1. Define a value for `sentinel failover_timeout` in `ms`. This has multiple @@ -164,7 +165,7 @@ by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** will change permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. -The same thing will happen with `sentinel.conf` that will be overriten after the +The same thing will happen with `sentinel.conf` that will be overridden after the initial execution, after any new sentinel node starts watching the **Master**, or a failover promotes a different **Master** node. @@ -284,3 +285,5 @@ production: ``` When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) + +[gh-531]: https://github.com/redis/redis-rb/issues/531 -- cgit v1.2.1 From e840749b84ceb226e46ebdfb489c735e3370cff7 Mon Sep 17 00:00:00 2001 From: Patricio Cano Date: Thu, 10 Nov 2016 11:36:52 -0600 Subject: Refactored Sidekiq Throttler and updated documentation --- doc/administration/operations/sidekiq_job_throttling.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/doc/administration/operations/sidekiq_job_throttling.md b/doc/administration/operations/sidekiq_job_throttling.md index 33cedee7ebd..ddeaa22e288 100644 --- a/doc/administration/operations/sidekiq_job_throttling.md +++ b/doc/administration/operations/sidekiq_job_throttling.md @@ -3,14 +3,15 @@ > Note: Introduced with GitLab 8.14 When your GitLab installation needs to handle tens of thousands of background -jobs, it can be convenient to prioritize queues that need to be executed -immediately, e.g. user initiated actions like merging a Merge Request. +jobs, it can be convenient to throttle queues that do not need to be executed +immediately, e.g. long running jobs like Pipelines, thus allowing jobs that do +need to be executed immediately to have access to more resources. In order to accomplish this, you can limit the amount of workers that certain -slow running queues get can have available. This is what we call Sidekiq Job +slow running queues can have available. This is what we call Sidekiq Job Throttling. Depending on your infrastructure, you might have different slow -running queues, which is why you can choose which queues to throttle and by -how much you want to throttle them. +running queues, which is why you can choose which queues you want to throttle +and by how much you want to throttle them. These settings are available in the Application Settings of your GitLab installation. @@ -24,8 +25,8 @@ and rounded up to the closest full integer. So, for example, you set the `:concurrency` to 25 and the `Throttling factor` to 0.1, the maximum workers assigned to the selected queues would be 3. -``` -limit = (factor * Sidekiq.options[:concurrency]).ceil +```ruby +queue_limit = (factor * Sidekiq.options[:concurrency]).ceil ``` After enabling the job throttling, you will need to restart your GitLab -- cgit v1.2.1 From deef24a27dd09f7f42221e6900a1b96db0e4ad47 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 11 Nov 2016 18:20:41 +0100 Subject: Rename README.md to index.md --- doc/development/ux_guide/README.md | 59 -------------------------------------- doc/development/ux_guide/index.md | 53 ++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 59 deletions(-) delete mode 100644 doc/development/ux_guide/README.md create mode 100644 doc/development/ux_guide/index.md (limited to 'doc') diff --git a/doc/development/ux_guide/README.md b/doc/development/ux_guide/README.md deleted file mode 100644 index e17a4559069..00000000000 --- a/doc/development/ux_guide/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# GitLab UX Guide - -The goal of this guide is to provide standards, principles and in-depth information to design beautiful and effective GitLab features. This will be a living document, and we welcome contributions, feedback and suggestions. - -## Design - ---- - -### [Principles](principles.md) -These guiding principles set a solid foundation for our design system, and should remain relatively stable over multiple releases. They should be referenced as new design patterns are created. - ---- - -### [Basics](basics.md) -The basic ingredients of our experience establish our personality and feel. This section includes details about typography, color, and motion. - ---- - -### [Components](components.md) -Components are the controls that make up the GitLab experience, including guidance around buttons, links, dropdowns, etc. - ---- - -### [Surfaces](surfaces.md) -The GitLab experience is broken apart into several surfaces. Each of these surfaces is designated for a specific scope or type of content. Examples include the header, global menu, side pane, etc. - ---- - -### [Features](features.md) -The previous building blocks are combined into complete features in the GitLab UX. Examples include our navigation, filters, search results, and empty states. - ---- - -
- -## Research - ---- - -### [Users](users.md) -How we think about the variety of users of GitLab, from small to large teams, comparing opensource usage to enterprise, etc. - ---- - -
- -## Other - ---- - -### [Tips for designers](tips.md) -Tips for exporting assets, and other guidance. - ---- - -### [Resources](resources.md) -Resources for GitLab UX - ---- diff --git a/doc/development/ux_guide/index.md b/doc/development/ux_guide/index.md new file mode 100644 index 00000000000..1a61be4ed51 --- /dev/null +++ b/doc/development/ux_guide/index.md @@ -0,0 +1,53 @@ +# GitLab UX Guide + +The goal of this guide is to provide standards, principles and in-depth information to design beautiful and effective GitLab features. This will be a living document, and we welcome contributions, feedback and suggestions. + +## Design + +--- + +### [Principles](principles.md) +These guiding principles set a solid foundation for our design system, and should remain relatively stable over multiple releases. They should be referenced as new design patterns are created. + +--- + +### [Basics](basics.md) +The basic ingredients of our experience establish our personality and feel. This section includes details about typography, color, and motion. + +--- + +### [Components](components.md) +Components are the controls that make up the GitLab experience, including guidance around buttons, links, dropdowns, etc. + +--- + +### [Surfaces](surfaces.md) +The GitLab experience is broken apart into several surfaces. Each of these surfaces is designated for a specific scope or type of content. Examples include the header, global menu, side pane, etc. + +--- + +### [Features](features.md) +The previous building blocks are combined into complete features in the GitLab UX. Examples include our navigation, filters, search results, and empty states. + +--- + +## Research + +--- + +### [Users](users.md) +How we think about the variety of users of GitLab, from small to large teams, comparing opensource usage to enterprise, etc. + +--- + +## Other + +--- + +### [Tips for designers](tips.md) +Tips for exporting assets, and other guidance. + +--- + +### [Resources](resources.md) +Resources for GitLab UX -- cgit v1.2.1 From 406eda17d444e534bfa429d8c72a90c8f91a6f00 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 11 Nov 2016 18:20:56 +0100 Subject: Remove
and replace GFM blockquote with the Markdown general [ci skip] --- doc/development/ux_guide/basics.md | 26 ++++---------------------- doc/development/ux_guide/components.md | 30 +----------------------------- doc/development/ux_guide/features.md | 6 ------ doc/development/ux_guide/principles.md | 14 +------------- doc/development/ux_guide/surfaces.md | 8 +------- doc/development/ux_guide/tips.md | 4 +--- doc/development/ux_guide/users.md | 6 ++---- 7 files changed, 10 insertions(+), 84 deletions(-) (limited to 'doc') diff --git a/doc/development/ux_guide/basics.md b/doc/development/ux_guide/basics.md index 7e7cb103694..62ac56a6bce 100644 --- a/doc/development/ux_guide/basics.md +++ b/doc/development/ux_guide/basics.md @@ -10,15 +10,11 @@ --- -
- ## Responsive GitLab is a responsive experience that works well across all screen sizes, from mobile devices to large monitors. In order to provide a great user experience, the core functionality (browsing files, creating issues, writing comments, etc.) must be available at all resolutions. However, due to size limitations, some secondary functionality may be hidden on smaller screens. Please keep this functionality limited to rare actions that aren't expected to be needed on small devices. --- -
- ## Typography ### Primary typeface GitLab's main typeface used throughout the UI is **Source Sans Pro**. We support both the bold and regular weight. @@ -36,8 +32,6 @@ This is the typeface used for code blocks. GitLab uses the OS default font. --- -
- ## Icons GitLab uses Font Awesome icons throughout our interface. @@ -62,14 +56,10 @@ An 'x' is used for closing UI elements such as dropdowns. ![Add icon](img/icon-add.png) A plus is used when creating new objects, such as issues, projects, etc. ->>> -TODO: update this section, add more general guidance to icon usage and personality, etc. ->>> +> TODO: update this section, add more general guidance to icon usage and personality, etc. --- -
- ## Color ![Blue](img/color-blue.png) @@ -87,26 +77,18 @@ Red is reserved for delete and other destructive commands ![Grey](img/color-grey.png) Grey, and white (depending on context) is used for netral, secondary elements ->>> -TODO: Establish a perspective for color in terms of our personality and rationalize with Marketing usage. ->>> +> TODO: Establish a perspective for color in terms of our personality and rationalize with Marketing usage. --- -
- ## Motion Motion is a tool to help convey important relationships, changes or transitions between elements. It should be used sparingly and intentionally, highlighting the right elements at the right moment. ->>> -TODO: Determine a more concrete perspective on motion, create consistent easing/timing curves to follow. ->>> +> TODO: Determine a more concrete perspective on motion, create consistent easing/timing curves to follow. --- -
- ## Voice and tone -The copy for GitLab is clear and direct. We strike a clear balance between professional and friendly. We can empathesize with users (such as celebrating completing all Todos), and remain respectful of the importance of the work. We are that trusted, friendly coworker that is helpful and understanding. \ No newline at end of file +The copy for GitLab is clear and direct. We strike a clear balance between professional and friendly. We can empathesize with users (such as celebrating completing all Todos), and remain respectful of the importance of the work. We are that trusted, friendly coworker that is helpful and understanding. diff --git a/doc/development/ux_guide/components.md b/doc/development/ux_guide/components.md index 6b21566265c..764c3355714 100644 --- a/doc/development/ux_guide/components.md +++ b/doc/development/ux_guide/components.md @@ -17,8 +17,6 @@ --- -
- ## Tooltips ### Usage @@ -33,8 +31,6 @@ By default, tooltips should be placed below the element that they refer to. Howe --- -
- ## Anchor links Anchor links are used for navigational actions and lone, secondary commands (such as 'Reset filters' on the Issues List) when deemed appropriate by the UX team. @@ -57,8 +53,6 @@ The focus state should match the hover state. --- -
- ## Buttons Buttons communicate the command that will occur when the user clicks on them. @@ -87,8 +81,6 @@ Follow the color guidance on the [basics](basics.md#color) page. The default col --- -
- ## Dropdowns Dropdowns are used to allow users to choose one (or many) options from a list of options. If this list of options is more 20, there should generally be a way to search through and filter the options (see the complex filter dropdowns below.) @@ -103,8 +95,6 @@ TODO: Will update this section when the new filters UI is implemented. --- -
- ## Counts A count element is used in navigation contexts where it is helpful to indicate the count, or number of items, in a list. Always use the [`number_with_delimiter`][number_with_delimiter] helper to display counts in the UI. @@ -115,8 +105,6 @@ A count element is used in navigation contexts where it is helpful to indicate t --- -
- ## Lists Lists are used where ever there is a single column of information to display. Ths [issues list](https://gitlab.com/gitlab-org/gitlab-ce/issues) is an example of a important list in the GitLab UI. @@ -141,8 +129,6 @@ List inside panel --- -
- ## Tables When the information is too complex for a list, with multiple columns of information, a table can be used. For example, the [pipelines page](https://gitlab.com/gitlab-org/gitlab-ce/pipelines) uses a table. @@ -151,8 +137,6 @@ When the information is too complex for a list, with multiple columns of informa --- -
- ## Blocks Blocks are a way to group related information. @@ -178,8 +162,6 @@ Cover blocks are generally used to create a heading element for a page, such as --- -
- ## Panels >>> @@ -190,8 +172,6 @@ TODO: Catalog how we are currently using panels and rationalize how they relate --- -
- ## Alerts >>> @@ -202,8 +182,6 @@ TODO: Catalog how we are currently using alerts --- -
- ## Forms There are two options shown below regarding the positioning of labels in forms. Both are options to consider based on context and available size. However, it is important to have a consistent treatment of labels in the same form. @@ -224,8 +202,6 @@ Horizontal form (`form.horizontal-form`) with label rendered inline with input. --- -
- ## File holders A file holder (`.file-holder`) is used to show the contents of a file inline on a page of GitLab. @@ -233,8 +209,6 @@ A file holder (`.file-holder`) is used to show the contents of a file inline on --- -
- ## Data formats ### Dates @@ -277,6 +251,4 @@ Referencing GitLab items depends on a symbol for each type of item. Typing that ![People reference](img/components-referencepeople.png) ->>> -Open issue: Some commit references use monospace fonts, but others don't. Need to standardize this. ->>> +> TODO: Open issue: Some commit references use monospace fonts, but others don't. Need to standardize this. diff --git a/doc/development/ux_guide/features.md b/doc/development/ux_guide/features.md index 00700af72aa..9472995c68c 100644 --- a/doc/development/ux_guide/features.md +++ b/doc/development/ux_guide/features.md @@ -30,8 +30,6 @@ The [GitLab Product Map](https://gitlab.com/gitlab-org/gitlab-design/raw/master/ --- -
- ## Filtering Today, lists are filtered by a series of dropdowns. Some of these dropdowns allow multiselect (labels), while others allow you to filter to one option (milestones). However, we are currently implementing a [new model](https://gitlab.com/gitlab-org/gitlab-ce/issues/21747) for this, and will update the guide when it is ready. @@ -40,8 +38,6 @@ Today, lists are filtered by a series of dropdowns. Some of these dropdowns allo --- -
- ## Search results ### Global search @@ -54,8 +50,6 @@ There are several core lists in the GitLab experience, such as the Issue list an --- -
- ## Empty states Empty states need to be considered in the design of features. They are vital to helping onboard new users, making the experience feel more approachable and understandable. Empty states should feel inviting and provide just enough information to get people started. There should be a single call to action and a clear explanation of what to use the feature for. diff --git a/doc/development/ux_guide/principles.md b/doc/development/ux_guide/principles.md index 32520845e41..1a297cba2cc 100644 --- a/doc/development/ux_guide/principles.md +++ b/doc/development/ux_guide/principles.md @@ -2,28 +2,16 @@ These are the guiding principles that we should strive for to establish a solid foundation for the GitLab experience. -
- ## Professional and productive GitLab is a tool to support what people do, day in, day out. We need to respect the importance of their work, and avoid gimicky details. -
- ## Minimal and efficient While work can get complicated, GitLab is about bringing a sharp focus, helping our customers know what matters now. -
- ## Immediately recognizable When you look at any screen, you should know immediately that it is GitLab. Our personality is strong and consistent across product and marketing experiences. -
- ## Human and quirky We need to build empathy with our users, understanding their state of mind, and connect with them at a human level. Quirkiness is part of our DNA, and we should embrace it in the right moments and contexts. -
- ->>> -TODO: Ensure these principles align well with the goals of the Marketing team ->>> \ No newline at end of file +> TODO: Ensure these principles align well with the goals of the Marketing team diff --git a/doc/development/ux_guide/surfaces.md b/doc/development/ux_guide/surfaces.md index d9e48a66185..881d6aa4cd6 100644 --- a/doc/development/ux_guide/surfaces.md +++ b/doc/development/ux_guide/surfaces.md @@ -16,8 +16,6 @@ This menu is to navigate to pages that contain content global to GitLab. --- -
- ## Header The header contains 3 main elements: Project switching and searching, user account avatar and settings, and a contextual menu that changes based on the current page. @@ -26,16 +24,12 @@ The header contains 3 main elements: Project switching and searching, user accou --- -
- ## Side pane The side pane holds supporting information and meta data for the information in the content area. --- -
- ## Content area The main content of the page. The content area can include other surfaces. @@ -50,4 +44,4 @@ The item title bar contains the top level information to identify the item, such The system information block contains relevant system controlled information. -![Item system information](img/surfaces-systeminformationblock.png) \ No newline at end of file +![Item system information](img/surfaces-systeminformationblock.png) diff --git a/doc/development/ux_guide/tips.md b/doc/development/ux_guide/tips.md index 190ce9a2ee9..8348de4f8a2 100644 --- a/doc/development/ux_guide/tips.md +++ b/doc/development/ux_guide/tips.md @@ -41,6 +41,4 @@ Correct file will look like this: ``` ->>> -TODO: Checkout [https://github.com/svg/svgo](https://github.com/svg/svgo) ->>> \ No newline at end of file +> TODO: Checkout [https://github.com/svg/svgo](https://github.com/svg/svgo) diff --git a/doc/development/ux_guide/users.md b/doc/development/ux_guide/users.md index cab4190604e..717a902c424 100644 --- a/doc/development/ux_guide/users.md +++ b/doc/development/ux_guide/users.md @@ -1,8 +1,6 @@ # Users ->>> -TODO: Create personas. Understand the similarities and differences across the below spectrums. ->>> +> TODO: Create personas. Understand the similarities and differences across the below spectrums. ## Users by organization @@ -15,4 +13,4 @@ TODO: Create personas. Understand the similarities and differences across the be - Admin - Manager -- Developer \ No newline at end of file +- Developer -- cgit v1.2.1 From e86ef2fd03500eaae2abd86164c9a91d9cc79921 Mon Sep 17 00:00:00 2001 From: Ben Bodenmiller Date: Sat, 12 Nov 2016 00:54:24 +0000 Subject: fix Super User spacing --- doc/integration/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/integration/README.md b/doc/integration/README.md index c2fd299db07..77bea3d2ceb 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -44,7 +44,7 @@ This [resource](http://kb.kerio.com/product/kerio-connect/server-configuration/s has all the information you need to add a certificate to the main trusted chain. This [answer](http://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu) -at SuperUser also has relevant information. +at Super User also has relevant information. **Omnibus Trusted Chain** -- cgit v1.2.1 From bfbf23d788901692a490622b6562b383f6f020b6 Mon Sep 17 00:00:00 2001 From: Ben Bodenmiller Date: Sat, 12 Nov 2016 04:05:51 +0000 Subject: update redis server details --- doc/administration/high_availability/redis.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index bc424330656..8656b42f274 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -48,15 +48,15 @@ Redis. redis['password'] = 'Redis Password' ``` -1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. +1. Run `sudo touch /etc/gitlab/skip-auto-migrations` to prevent database migrations + from running on upgrade. Only the primary GitLab application server should + handle migrations. + +1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis. > **Note**: This `reconfigure` step will result in some errors. That's OK - don't be alarmed. -1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. - ## Experimental Redis Sentinel support > [Introduced][ce-1877] in GitLab 8.11. -- cgit v1.2.1 From af941732ed92ad44a7568964e2a9191d0c72164c Mon Sep 17 00:00:00 2001 From: Vincent Composieux Date: Fri, 11 Nov 2016 12:27:23 +0100 Subject: Added ability to put emojis into repository name Added ability to put emojis into repository name --- doc/gitlab-basics/create-project.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index 3f45a631b3a..1c549844ee1 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -14,8 +14,8 @@ There are two ways to create a new project in GitLab. 1. Fill out the information: - 1. "Project name" is the name of your project (you can't use spaces, but you - can use hyphens or underscores). + 1. "Project name" is the name of your project (you can't use special characters, + but you can use spaces, hyphens, underscores or even emojis). 1. The "Project description" is optional and will be shown in your project's dashboard so others can briefly understand what your project is about. 1. Select a [visibility level](../public_access/public_access.md). -- cgit v1.2.1 From 5c966f70fb218d6f4de0f888733604293f36c33e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rebeca=20M=C3=A9ndez?= Date: Mon, 29 Aug 2016 17:23:40 +0200 Subject: Issue #4270: Recursive option for files through API --- doc/api/repositories.md | 54 +++++++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 20 deletions(-) (limited to 'doc') diff --git a/doc/api/repositories.md b/doc/api/repositories.md index b6cca5d4e2a..bcf8b955044 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -13,44 +13,58 @@ Parameters: - `id` (required) - The ID of a project - `path` (optional) - The path inside repository. Used to get contend of subdirectories - `ref_name` (optional) - The name of a repository branch or tag or if not given the default branch +- `recursive` (optional) - Boolean value used to get a recursive tree (false by default) ```json [ { - "name": "assets", + "id": "a1e8f8d745cc87e3a9248358d9352bb7f9a0aeba", + "name": "html", "type": "tree", - "mode": "040000", - "id": "6229c43a7e16fcc7e95f923f8ddadb8281d9c6c6" + "path": "files/html", + "mode": "040000" }, { - "name": "contexts", + "id": "4535904260b1082e14f867f7a24fd8c21495bde3", + "name": "images", "type": "tree", - "mode": "040000", - "id": "faf1cdf33feadc7973118ca42d35f1e62977e91f" + "path": "files/images", + "mode": "040000" }, { - "name": "controllers", + "id": "31405c5ddef582c5a9b7a85230413ff90e2fe720", + "name": "js", "type": "tree", - "mode": "040000", - "id": "95633e8d258bf3dfba3a5268fb8440d263218d74" + "path": "files/js", + "mode": "040000" }, { - "name": "Rakefile", - "type": "blob", - "mode": "100644", - "id": "35b2f05cbb4566b71b34554cf184a9d0bd9d46d6" + "id": "cc71111cfad871212dc99572599a568bfe1e7e00", + "name": "lfs", + "type": "tree", + "path": "files/lfs", + "mode": "040000" }, { - "name": "VERSION", - "type": "blob", - "mode": "100644", - "id": "803e4a4f3727286c3093c63870c2b6524d30ec4f" + "id": "fd581c619bf59cfdfa9c8282377bb09c2f897520", + "name": "markdown", + "type": "tree", + "path": "files/markdown", + "mode": "040000" + }, + { + "id": "23ea4d11a4bdd960ee5320c5cb65b5b3fdbc60db", + "name": "ruby", + "type": "tree", + "path": "files/ruby", + "mode": "040000" }, { - "name": "config.ru", + "id": "7d70e02340bac451f281cecf0a980907974bd8be", + "name": "whitespace", "type": "blob", - "mode": "100644", - "id": "dfd2d862237323aa599be31b473d70a8a817943b" + "path": "files/whitespace", + "mode": "100644" } ] ``` -- cgit v1.2.1 From 5b6c633ed71d844ce26b12f5f5a3e8a0ed85a776 Mon Sep 17 00:00:00 2001 From: Ben Bodenmiller Date: Sat, 12 Nov 2016 19:18:28 +0000 Subject: use single quote for consistency --- doc/administration/high_availability/nfs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 537f4f3501d..5602d70f1ef 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -76,7 +76,7 @@ configuration to move each data location to a subdirectory: user['home'] = '/gitlab-data/home' git_data_dir '/gitlab-data/git-data' gitlab_rails['shared_path'] = '/gitlab-data/shared' -gitlab_rails['uploads_directory'] = "/gitlab-data/uploads" +gitlab_rails['uploads_directory'] = '/gitlab-data/uploads' gitlab_ci['builds_directory'] = '/gitlab-data/builds' ``` -- cgit v1.2.1 From 8b70a89c57d41d035e4b8ba06cc33859383f095a Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Sat, 12 Nov 2016 20:48:41 +0100 Subject: Fix link to index.md in development README.md [ci skip] --- doc/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/development/README.md b/doc/development/README.md index 87306b1501d..f88456a7a7a 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -14,7 +14,7 @@ contributing to documentation. - [SQL Migration Style Guide](migration_style_guide.md) for creating safe SQL migrations - [Testing standards and style guidelines](testing.md) -- [UX guide](ux_guide/README.md) for building GitLab with existing CSS styles and elements +- [UX guide](ux_guide/index.md) for building GitLab with existing CSS styles and elements - [Frontend guidelines](frontend.md) - [SQL guidelines](sql.md) for working with SQL queries - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers -- cgit v1.2.1 From 0279ad5c3b7dad73b55f94580079f68dce313f32 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Mon, 14 Nov 2016 12:21:17 +0200 Subject: Link to correct version of SanitizationFilter API docs for whitelist [ci skip] --- doc/user/markdown.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 7a7a0b864bd..dbc7e0f14e3 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -590,7 +590,7 @@ Quote break. You can also use raw HTML in your Markdown, and it'll mostly work pretty well. -See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubydoc.info/gems/html-pipeline/HTML/Pipeline/SanitizationFilter#WHITELIST-constant) class for the list of allowed HTML tags and attributes. In addition to the default `SanitizationFilter` whitelist, GitLab allows `span` elements. +See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubydoc.info/gems/html-pipeline/1.11.0/HTML/Pipeline/SanitizationFilter#WHITELIST-constant) class for the list of allowed HTML tags and attributes. In addition to the default `SanitizationFilter` whitelist, GitLab allows `span` elements. ```no-highlight
-- cgit v1.2.1 From b105d9da75d9e5131f2c118bc87f67df108d1214 Mon Sep 17 00:00:00 2001 From: Ben Bodenmiller Date: Mon, 14 Nov 2016 23:44:37 -0800 Subject: clarify best way to install custom CA --- doc/integration/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/integration/README.md b/doc/integration/README.md index c2fd299db07..e3f6d48904d 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -48,7 +48,11 @@ at SuperUser also has relevant information. **Omnibus Trusted Chain** -It is enough to concatenate the certificate to the main trusted certificate: +[Install the self signed certificate or custom certificate authorities](http://docs.gitlab.com/omnibus/common_installation_problems/README.html#using-self-signed-certificate-or-custom-certificate-authorities) +in to GitLab Omnibus. + +It is enough to concatenate the certificate to the main trusted certificate +however it may be overwritten during upgrades: ```bash cat jira.pem >> /opt/gitlab/embedded/ssl/certs/cacert.pem -- cgit v1.2.1 From 038ea5e79c561cfd87254abf658b697f71acd600 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bernhard=20H=C3=A4ussner?= Date: Tue, 15 Nov 2016 10:40:36 +0000 Subject: Fix invalid JSON in Builds API doc The JSON of the example responses were missing a few `,` and therefor invalid. This adds them to make those JSONs valid. --- doc/api/builds.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/api/builds.md b/doc/api/builds.md index 0476cac0eda..bca2f9e44ef 100644 --- a/doc/api/builds.md +++ b/doc/api/builds.md @@ -45,7 +45,7 @@ Example of response "ref": "master", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" - } + }, "ref": "master", "runner": null, "stage": "test", @@ -89,7 +89,7 @@ Example of response "ref": "master", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" - } + }, "ref": "master", "runner": null, "stage": "test", @@ -163,7 +163,7 @@ Example of response "ref": "master", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" - } + }, "ref": "master", "runner": null, "stage": "test", @@ -193,7 +193,7 @@ Example of response "ref": "master", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" - } + }, "ref": "master", "runner": null, "stage": "test", @@ -260,7 +260,7 @@ Example of response "ref": "master", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" - } + }, "ref": "master", "runner": null, "stage": "test", -- cgit v1.2.1 From 39a9af1594bf2365ba0da6457efef238d7e42e24 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 15 Nov 2016 14:07:38 +0100 Subject: Swap HA setups --- doc/administration/high_availability/README.md | 33 +++++++++------- doc/administration/high_availability/redis.md | 54 +++++++++++++------------- 2 files changed, 45 insertions(+), 42 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index d74a786ac24..d5a5aef7ec0 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -7,19 +7,10 @@ highly available. ## Architecture -### Active/Passive - -For pure high-availability/failover with no scaling you can use an -active/passive configuration. This utilizes DRBD (Distributed Replicated -Block Device) to keep all data in sync. DRBD requires a low latency link to -remain in sync. It is not advisable to attempt to run DRBD between data centers -or in different cloud availability zones. +There are two kinds of setups: -Components/Servers Required: - -- 2 servers/virtual machines (one active/one passive) - -![Active/Passive HA Diagram](../img/high_availability/active-passive-diagram.png) +- active/active +- active/passive ### Active/Active @@ -28,12 +19,24 @@ user requests simultaneously. The database, Redis, and GitLab application are all deployed on separate servers. The configuration is **only** highly-available if the database, Redis and storage are also configured as such. -![Active/Active HA Diagram](../img/high_availability/active-active-diagram.png) - -**Steps to configure active/active:** +Follow the steps below to configure an active/active setup: 1. [Configure the database](database.md) 1. [Configure Redis](redis.md) 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) + +![Active/Active HA Diagram](../img/high_availability/active-active-diagram.png) + +### Active/Passive + +For pure high-availability/failover with no scaling you can use an +active/passive configuration. This utilizes DRBD (Distributed Replicated +Block Device) to keep all data in sync. DRBD requires a low latency link to +remain in sync. It is not advisable to attempt to run DRBD between data centers +or in different cloud availability zones. + +Components/Servers Required: 2 servers/virtual machines (one active/one passive) + +![Active/Passive HA Diagram](../img/high_availability/active-passive-diagram.png) diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 3eaa0ffdcec..bb46de65e3c 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -16,10 +16,10 @@ Omnibus GitLab packages. - You are highly encouraged to read the [Redis Sentinel][sentinel] documentation before configuring Redis HA with GitLab to fully understand the topology and architecture. -- This is the documentation for the Omnibus packages. For installations from - source, follow the [Redis HA source install](redis_source.md) guide. -- Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the - Omnibus Community Edition and installations from source, follow the +- This is the documentation for the Omnibus GitLab packages. For installations + from source, follow the [Redis HA source install](redis_source.md) guide. +- Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only. + For the Omnibus Community Edition and installations from source, follow the [Redis HA source install](redis_source.md) guide. @@ -50,7 +50,6 @@ Omnibus GitLab packages. - [Troubleshooting Redis replication](#troubleshooting-redis-replication) - [Troubleshooting Sentinel](#troubleshooting-sentinel) - [Changelog](#changelog) - - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) @@ -76,15 +75,16 @@ making sure you have redundant connectivity between Redis / Sentinel and GitLab instances, otherwise the networks will become a single point of failure. -Read carefully how to configure the components below. +Make sure that you read this document once as a whole before configuring the +components below. ### High Availability with Sentinel > -- Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that - will monitor a group of Redis servers to provide failover support. -- With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and - improved the way you use and configure it. +- Starting with GitLab `8.11`, you can configure a list of Redis Sentinel + servers that will monitor a group of Redis servers to provide failover support. +- Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package + comes with Redis sentinel daemon support. High Availability with Redis requires a few things: @@ -136,20 +136,21 @@ the Omnibus GitLab package in `5` **independent** machines, both with Based on your infrastructure setup and how you have installed GitLab, there are multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or -Redis Sentinel bundled with them to save you the hassle to install them yourself. +Redis Sentinel bundled with them so you only need to focus on configuration. Pick the one that suits your needs. - [Installations from source][source]: You need to install Redis and Sentinel - yourself. Use the [Redis HA source install](redis_source.md) guide. -- [Omnibus package Community Edition (CE)][ce]: Redis is bundled, so you can use the - package with only the Redis service enabled (works for both master and slave - setups). -- [Omnibus package Enterprise Edition (EE)][ee]: Both Redis and Sentinel are bundled, - so you can use the EE package to setup the whole Redis HA infrastructure - (master, slave and Sentinel). - -Note that if you have installed GitLab using the Omnibus packages (both CE and EE), -you can also use an [external Redis server](#using-a-non-omnibus-external-redis-server). + yourself. Use the [Redis HA installation from source](redis_source.md) guide. +- [Omnibus package Community Edition (CE) package][ce]: Redis is bundled, so you + can use the package with only the Redis service enabled (works for both master + and slave setups). To install and configure Sentinel, you can use the + [Redis HA installation from source](redis_source.md) guide. +- [Omnibus package Enterprise Edition (EE) package][ee]: Both Redis and Sentinel + are bundled, so you can use the EE package to setup the whole Redis HA + infrastructure (master, slave and Sentinel). + +Note that if you have installed GitLab using the Omnibus GitLab packages (both +CE and EE), you can also use an [external Redis server](#using-a-non-omnibus-external-redis-server). ### Using a non-Omnibus external Redis server @@ -198,7 +199,7 @@ each other over the network. Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel detects that a Redis node is not responding, it will announce that to the -other sentinels. You have to reach the **quorum**, the minimum amount of +other sentinels. They have to reach the **quorum**, the minimum amount of sentinels that agrees that a node is down, to be able to start a failover. Whenever the **quorum** is met, you need the **majority** of all known @@ -267,9 +268,8 @@ This is the section where we install and setup the new Redis instances. already have it installed and running, read how to [switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha). - Redis nodes (both master and slaves) will need the same password defined in - `redis['password']` and `redis['master_password']`. At any time during a - failover the Sentinels can reconfigure a node and change its status - from master to slave and vice versa. + `redis['password']`. At any time during a failover the Sentinels can + reconfigure a node and change its status from master to slave and vice versa. A summary of what are we going to do: @@ -354,7 +354,7 @@ a failover, as the nodes will be managed by the Sentinels, and even after a `gitlab-ctl reconfigure`, they will get their configuration restored by the same Sentinels. -### Configuring the Sentinel instances +### Configuring the Redis Sentinel instances >**Note:** - Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the @@ -425,7 +425,7 @@ redis['enable'] = false If you fail to replicate first, you may loose data (unprocessed background jobs). -## Minimal example configuration with 1 master, 2 slaves and 3 sentinels +## Example of a minimal configuration with 1 master, 2 slaves and 3 sentinels In this example we consider that all servers have an internal network interface with IPs in the `10.0.0.x` range, and that they can connect -- cgit v1.2.1 From ff8194e0ec16092419862011d7cc048baa149c42 Mon Sep 17 00:00:00 2001 From: Dmitry Poray Date: Mon, 7 Nov 2016 20:11:54 +0300 Subject: Add ref parameter for triggerring builds with gitlab webhook from other project. --- doc/ci/triggers/README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) (limited to 'doc') diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 84048f1d25f..cf7c55f75f2 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -58,6 +58,22 @@ below. See the [Examples](#examples) section for more details on how to actually trigger a rebuild. +## Trigger a build from webhook + +> Introduced in GitLab 8.14. + +To trigger a build from webhook of another project you need to add the following +webhook url for Push and Tag push events: + +``` +https://gitlab.example.com/api/v3/projects/:id/ref/:ref/trigger/builds?token=TOKEN +``` + +> **Note**: +- `ref` should be passed as part of url in order to take precedence over `ref` + from webhook body that designates the branchref that fired the trigger in the source repository. +- `ref` should be url encoded if contains slashes. + ## Pass build variables to a trigger You can pass any number of arbitrary variables in the trigger API call and they @@ -169,6 +185,14 @@ curl --request POST \ https://gitlab.example.com/api/v3/projects/9/trigger/builds ``` +### Using webhook to trigger builds + +You can add the following webhook to another project in order to trigger a build: + +``` +https://gitlab.example.com/api/v3/projects/9/ref/master/trigger/builds?token=TOKEN&variables[UPLOAD_TO_S3]=true +``` + ### Using cron to trigger nightly builds Whether you craft a script or just run cURL directly, you can trigger builds -- cgit v1.2.1 From 6a5891185c6edde0a300adee30e936d44b6180e0 Mon Sep 17 00:00:00 2001 From: Drew Blessing Date: Tue, 15 Nov 2016 15:03:44 -0600 Subject: Clarify LDAP troubleshooting ldap_search example [ci skip] A customer noted an error/lack of clarity in the LDAP documentation with the `ldap_search` example. Previously, if taken literally, the customer may have expected the `$` variables to be automatically replaced or if they paste the exact `user_filter` contents the parentheses would have been incorrect. Let's just simply the filter and use exactly what's in the configuration. --- doc/administration/auth/ldap.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) (limited to 'doc') diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index fd23047f027..d3f216fb3bf 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -257,6 +257,24 @@ the LDAP server's SSL certificate is performed. ## Troubleshooting +### Debug LDAP user filter with ldapsearch + +This example uses ldapsearch and assumes you are using ActiveDirectory. The +following query returns the login names of the users that will be allowed to +log in to GitLab if you configure your own user_filter. + +``` +ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName +``` + +- Variables beginning with a `$` refer to a variable from the LDAP section of + your configuration file. +- Replace ldaps:// with ldap:// if you are using the plain authentication method. + Port `389` is the default `ldap://` port and `636` is the default `ldaps://` + port. +- We are assuming the password for the bind_dn user is in bind_dn_password.txt. + + ### Invalid credentials when logging in - Make sure the user you are binding with has enough permissions to read the user's -- cgit v1.2.1 From 9b70c09d07d180229501799f2d440191853aba57 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 15 Nov 2016 16:08:53 +0100 Subject: Refactor Redis HA docs [ci skip] --- doc/administration/high_availability/redis.md | 547 +++++++++++++++----------- 1 file changed, 314 insertions(+), 233 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index bb46de65e3c..6de92ae3741 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -17,10 +17,11 @@ Omnibus GitLab packages. before configuring Redis HA with GitLab to fully understand the topology and architecture. - This is the documentation for the Omnibus GitLab packages. For installations - from source, follow the [Redis HA source install](redis_source.md) guide. + from source, follow the [Redis HA source installation](redis_source.md) guide. - Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only. - For the Omnibus Community Edition and installations from source, follow the - [Redis HA source install](redis_source.md) guide. + For configuring Sentinel with the Omnibus GitLab Community Edition and + installations from source, follow the + [Redis HA source installation](redis_source.md) guide. @@ -31,16 +32,15 @@ Omnibus GitLab packages. - [High Availability with Sentinel](#high-availability-with-sentinel) - [Recommended setup](#recommended-setup) - [Available configuration setups](#available-configuration-setups) - - [Using a non-Omnibus external Redis server](#using-a-non-omnibus-external-redis-server) - [Redis setup overview](#redis-setup-overview) - [Sentinel setup overview](#sentinel-setup-overview) -- [Redis HA configuration](#redis-ha-configuration) - - [Configuring the Master Redis instance](#configuring-the-master-redis-instance) - - [Configuring the Slave Redis instances](#configuring-the-slave-redis-instances) - - [Configuring the Sentinel instances](#configuring-the-sentinel-instances) - - [Configuring the GitLab application](#configuring-the-gitlab-application) +- [Configuring Redis HA](#configuring-redis-ha) + - [Step 1. Configuring the Master Redis instance](#step-1-configuring-the-master-redis-instance) + - [Step 2. Configuring the Slave Redis instances](#step-2-configuring-the-slave-redis-instances) + - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) + - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) - [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha) -- [Minimal example configuration with 1 master, 2 slaves and 3 sentinels](#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels) +- [Example of a minimal configuration with 1 master, 2 slaves and 3 sentinels](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - [Configuration for Redis master](#configuration-for-redis-master) - [Configuration for Redis slaves](#configuration-for-redis-slaves) - [Configuration for Sentinels](#configuration-for-sentinels) @@ -50,6 +50,7 @@ Omnibus GitLab packages. - [Troubleshooting Redis replication](#troubleshooting-redis-replication) - [Troubleshooting Sentinel](#troubleshooting-sentinel) - [Changelog](#changelog) + - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) @@ -59,8 +60,6 @@ Before diving into the details of setting up Redis and Redis Sentinel for HA, make sure you read this Overview section to better understand how the components are tied together. -### Prerequisites - You need at least `3` independent machines: physical, or VMs running into distinct physical machines. It is essential that all master and slaves Redis instances run in different machines. If you fail to provision the machines in @@ -84,16 +83,16 @@ components below. - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel servers that will monitor a group of Redis servers to provide failover support. - Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package - comes with Redis sentinel daemon support. + comes with Redis Sentinel daemon support. High Availability with Redis requires a few things: - Multiple Redis instances -- Run Redis in a **Master** x **Slave** topology -- Multiple Sentinel instances + - Run Redis in a **Master** x **Slave** topology + - Multiple Sentinel instances - Application support and visibility to all Sentinel and Redis instances -Redis Sentinel can handle the most important tasks in a HA environment and that's +Redis Sentinel can handle the most important tasks in an HA environment and that's to help keep servers online with minimal to no downtime. Redis Sentinel: - Monitors **Master** and **Slaves** instances to see if they are available @@ -120,7 +119,8 @@ For a minimal setup, you will install the Omnibus GitLab package in `3` - Redis Slave + Sentinel If you are not sure or don't understand why and where the amount of nodes come -from, read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup). +from, read [Redis setup overview](#redis-setup-overview) and +[Sentinel setup overview](#sentinel-setup-overview). For a recommended setup that can resist more failures, you will install the Omnibus GitLab package in `5` **independent** machines, both with @@ -149,24 +149,9 @@ Pick the one that suits your needs. are bundled, so you can use the EE package to setup the whole Redis HA infrastructure (master, slave and Sentinel). -Note that if you have installed GitLab using the Omnibus GitLab packages (both -CE and EE), you can also use an [external Redis server](#using-a-non-omnibus-external-redis-server). - -### Using a non-Omnibus external Redis server - -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for Redis. For example, AWS offers a managed ElastiCache service -that runs Redis. - -Managed services can provide High Availability using their own proprietary -technology and provide a transparent proxy (which means that GitLab doesn't -need any additional change) or they will use Sentinel and manage it for you. - -If your provider uses Sentinel, see [GitLab Setup](#gitlab-setup) -to understand where you need to provide the list of servers and credentials. - -If you want to setup Redis by yourself, without using Omnibus, you can -read the documentation on [configuring Redis HA for source installs](redis_source.md). +Note that even if you have installed GitLab using the Omnibus GitLab packages +(both CE and EE), you can still use an +[external Redis server](#using-a-non-omnibus-external-redis-server). ### Redis setup overview @@ -197,13 +182,13 @@ each other over the network. ### Sentinel setup overview -Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel +Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel detects that a Redis node is not responding, it will announce that to the -other sentinels. They have to reach the **quorum**, the minimum amount of -sentinels that agrees that a node is down, to be able to start a failover. +other Sentinels. They have to reach the **quorum**, that is the minimum amount +of Sentinels that agrees a node is down, in order to be able to start a failover. -Whenever the **quorum** is met, you need the **majority** of all known -Sentinel nodes to be available and reachable, to elect the Sentinel **leader** +Whenever the **quorum** is met, the **majority** of all known Sentinel nodes +need to be available and reachable, so that they can elect the Sentinel **leader** who will take all the decisions to restore the service availability by: - Promoting a new **Master** @@ -212,7 +197,8 @@ who will take all the decisions to restore the service availability by: - Reconfigure the old **Master** and demote to **Slave** when it comes back online You must have at least `3` Redis Sentinel servers, and they need to -be each in a independent machine (that are believed to fail independently). +be each in a independent machine (that are believed to fail independently), +ideally in different geographical areas. You can configure them in the same machines where you've configured the other Redis servers, but understand that if a whole node goes down, you loose both @@ -230,7 +216,7 @@ Here are some examples: - With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin. - With `7` sentinels, a maximum of `3` nodes can go down. -The **Leader** election can sometimes fail the voting round when **consensus**, +The **Leader** election can sometimes fail the voting round when **consensus** is not achieved (see the odd number of nodes requirement above). In that case, a new attempt will be made after the amount of time defined in `sentinel['failover_timeout']` (in milliseconds). @@ -238,7 +224,7 @@ a new attempt will be made after the amount of time defined in >**Note:** We will see where `sentinel['failover_timeout']` is defined later. -The `failover_timeout` variable has a lot of different use cases, according to +The `failover_timeout` variable has a lot of different use cases. According to the official documentation: - The time needed to re-start a failover after a previous failover was @@ -259,7 +245,7 @@ the official documentation: the slaves will be reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. -## Redis HA configuration +## Configuring Redis HA This is the section where we install and setup the new Redis instances. @@ -271,95 +257,138 @@ This is the section where we install and setup the new Redis instances. `redis['password']`. At any time during a failover the Sentinels can reconfigure a node and change its status from master to slave and vice versa. -A summary of what are we going to do: +### Prerequisites -1. Provision the required number of instances specified previously: - - You can opt to install Redis and Sentinel in the same machine or each in - independent ones. - - Don't install Redis and Sentinel in the same machines your GitLab application - is running on. - - All machines must be able to talk to each other and accept incoming - connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you - change the default ports). - - GitLab machines must be able to access these machines and with the same - permissions. - - Protect them from access from external networks (Internet), - to harden the security. +The prerequisites for a HA Redis setup are the following: -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads) in each node. - - Do not complete other steps on the download page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. +1. Provision the minimum required number of instances as specified in the + [recommended setup](#recommended-setup) section. +1. **Do NOT** install Redis or Redis Sentinel in the same machines your + GitLab application is running on. You can however opt in to install Redis + and Sentinel in the same machine (each in independent ones is recommended + though). +1. All Redis nodes must be able to talk to each other and accept incoming + connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you + change the default ones). +1. The server that hosts the GitLab application must be able to access the + Redis nodes. +1. Protect the nodes from access from external networks (Internet), using + firewall. -1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. +### Step 1. Configuring the master Redis instance -1. Edit `/etc/gitlab/gitlab.rb` and make the changes based on the - [Example Configurations](#example-configurations). +1. SSH into the **master** Redis server and login as root: -### Configuring the Master Redis instance + ``` + sudo -i + ``` -You will need to configure the following in `/etc/gitlab/gitlab.rb`: +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. -1. Define `redis_master_role['enable']` to `true`, to disable other services - in the machine (you can still enable Sentinel) +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: -1. Define a `redis['bind']` address pointing to a local IP that your other machines - can reach you. - - If you really need to bind to an external accessible IP, make - sure you add extra firewall rules to prevent unauthorized access. - - You can also set bind to `0.0.0.0` which listen in all interfaces. + ```ruby + # Enable the master role and disable all other services in the machine + # (you can still enable Sentinel). + redis_master_role['enable'] = true + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.1' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'redis-password-goes-here' + ``` + +1. To prevent database migrations from running on upgrade, run: + + ``` + touch /etc/gitlab/skip-auto-migrations + ``` -1. Define a `redis['port']` so redis can listen for TCP requests which will - allow other machines to connect to it. + Only the primary GitLab application server should handle migrations. -1. Set up a password authentication with `redis['password']` and - `redis['master_password']` (use the same password in all nodes). +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -1. [Reconfigure Omnibus GitLab][reocnfigure] for the changes to take effect. +### Step 2. Configuring the slave Redis instances -### Configuring the Slave Redis instances +1. SSH into the **slave** Redis server and login as root: -You will need to configure the following in `/etc/gitlab/gitlab.rb`: + ``` + sudo -i + ``` -1. Define `redis_slaves_role['enable']` to `true`, to disable other services - in the machine (you can still enable Sentinel) - - This will also set automatically `redis['master'] = false`. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. -1. Define a `redis['bind']` address pointing to a local IP that your other machines - can reach you. - - If you really need to bind to an external accessible IP, make - sure you add extra firewall rules to prevent unauthorized access. - - You can also set bind to `0.0.0.0` which listen in all interfaces. +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Enable the slave role and disable all other services in the machine + # (you can still enable Sentinel). This will also set automatically + # `redis['master'] = false`. + redis_slave_role['enable'] = true -1. Define a `redis['port']` so redis can listen for TCP requests which will - allow other machines to connect to it. + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.2' -1. Set up a password authentication with `redis['password']` and - `redis['master_password']` (use the same password in all nodes). -1. Define `redis['master_ip']` with the IP of the **Master** Redis. + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 -1. Define `redis['master_port']` with the port of the **Master** Redis (default to `6379`). + # The same password for Redeis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' -Initial **Slave** nodes require `redis['master']` defined to `false` and -`redis['master_ip']` pointing to the initial **Master**. If you use the -simplified configuration by enabling `redis_slave_role['enable']`, you -just need to fill in the `redis['master_ip']`. + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' -This values don't have to be changed again in `/etc/gitlab/gitlab.rb` after + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + ``` + +1. To prevent database migrations from running on upgrade, run: + + ``` + touch /etc/gitlab/skip-auto-migrations + ``` + + Only the primary GitLab application server should handle migrations. + +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Go through the steps again for all the other slave nodes. + +--- + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the Sentinels, and even after a `gitlab-ctl reconfigure`, they will get their configuration restored by the same Sentinels. -### Configuring the Redis Sentinel instances +### Step 3. Configuring the Redis Sentinel instances >**Note:** -- Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the - Omnibus Community Edition and installations from source, follow the - [Redis HA source install](redis_source.md) guide. +Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. The +following section assumes you are using Omnibus GitLab Enterprise Edition. +For the Omnibus Community Edition and installations from source, follow the +[Redis HA source install](redis_source.md) guide. Now that the Redis servers are all set up, let's configure the Sentinel servers. @@ -369,15 +398,112 @@ correctly, please read the [Troubleshooting Replication](#troubleshooting-replic and fix it before proceeding with Sentinel setup. You must have at least `3` Redis Sentinel servers, and they need to -be each in a independent machine. You can configure them in the same +be each in an independent machine. You can configure them in the same machines where you've configured the other Redis servers. -With GitLab Enterprise Edition, you can use the Omnibus package to setup multiple -machines with the Sentinel daemon. +With GitLab Enterprise Edition, you can use the Omnibus package to setup +multiple machines with the Sentinel daemon. + +--- + +1. SSH into the server that will host Redis Sentinel and login as root: + + ``` + sudo -i + ``` + +1. **You can omit this step if the Sentinels will be hosted in the same node as + the other Redis instances.** + + [Download/install](https://about.gitlab.com/downloads-ee) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. -See [example configuration](#configuration-for-sentinel-ee-only) below. +1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the + Sentinels in the same node as the other Redis instances, some values might + be duplicate below): -### Configuring the GitLab application + + ```ruby + redis_sentinel_role['enable'] = true + + # Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + # The same password for Redis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' + + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel + sentinel['bind'] = '10.0.0.1' + + # Port that Sentinel listens on, uncomment to change to non default. Defaults + # to `26379`. + # sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to master failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the master. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the master being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + # sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## - The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + # sentinel['failover_timeout'] = 60000 + ``` + +1. To prevent database migrations from running on upgrade, run: + + ``` + touch /etc/gitlab/skip-auto-migrations + ``` + + Only the primary GitLab application server should handle migrations. + +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes. + +### Step 4. Configuring the GitLab application The final part is to inform the main GitLab application server of the Redis Sentinels servers and authentication credentials. @@ -395,14 +521,23 @@ which ideally should not have Redis or Sentinels on it for a HA setup. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: - - `redis['master_name']` - this is the `master-group-name` from sentinel (default: `gitlab-redis`) - - `redis['master_password']` - the same password you've defined before for Redis and Sentinels - - `gitlab_rails['redis_sentinels']` - a list of sentinels with `host` and `port` + ``` + # Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + # The same password for Redis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' + + # A list of sentinels with `host` and `port` + gitlab_rails['redis_sentinels'] = [ + {'host' => '10.0.0.1', 'port' => 26379}, + {'host' => '10.0.0.2', 'port' => 26379}, + {'host' => '10.0.0.3', 'port' => 26379} + ] + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -See [example configuration](#configuration-for-gitlab) below. - ## Switching from an existing single-machine installation to Redis HA If you already have a single-machine GitLab install running, you will need to @@ -436,14 +571,15 @@ unauthorized access from other machines and block traffic from the outside (Internet). We will use the same `3` nodes with **Redis** + **Sentinel** topology -discussed in [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup) -documentation. +discussed in [Redis setup overview](#redis-setup-overview) and +[Sentinel setup overview](#sentinel-setup-overview) documentation. Here is a list and description of each **machine** and the assigned **IP**: * `10.0.0.1`: Redis Master + Sentinel 1 * `10.0.0.2`: Redis Slave 1 + Sentinel 2 -* `10.0.0.2`: Redis Slave 2 + Sentinel 3 +* `10.0.0.3`: Redis Slave 2 + Sentinel 3 +* `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** @@ -454,7 +590,7 @@ The same thing will happen with `sentinel.conf` that will be overridden after th initial execution, after any new sentinel node starts watching the **Master**, or a failover promotes a different **Master** node. -### Configuration for Redis master +### Example configuration for Redis master **Example configation for Redis Master:** @@ -466,14 +602,13 @@ redis_master_role['enable'] = true redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' -redis['master_password'] = 'redis-password-goes-here' ``` [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -### Configuration for Redis slaves +### Example configuration for Redis slaves -**Example configation for Slave 1:** +**Example configuration for Slave 1:** In `/etc/gitlab/gitlab.rb`: @@ -489,9 +624,9 @@ redis['master_ip'] = '10.0.0.1' # IP of master Redis server #redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default ``` -Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -**Example configation for Slave 2:** +**Example configuration for Slave 2:** In `/etc/gitlab/gitlab.rb`: @@ -509,7 +644,7 @@ redis['master_ip'] = '10.0.0.1' # IP of master Redis server [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -### Configuration for Sentinels +### Example configuration for Sentinels >**Note:** Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the @@ -519,7 +654,7 @@ Omnibus Community Edition and installations from source, follow the Please note that some of the variables are already configured previously as they are required for Redis replication. -**Example configation for Sentinel 1:** +**Example configuration for Sentinel 1:** In `/etc/gitlab/gitlab.rb`: @@ -534,46 +669,17 @@ redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance ## Configure Sentinel sentinel['bind'] = '10.0.0.1' # sentinel['port'] = 26379 # uncomment to change default port - -## Quorum must reflect the amount of voting sentinels it take to start a failover. -## Value must NOT be greater then the ammount of sentinels. -## -## The quorum can be used to tune Sentinel in two ways: -## 1. If a the quorum is set to a value smaller than the majority of Sentinels -## we deploy, we are basically making Sentinel more sensible to master failures, -## triggering a failover as soon as even just a minority of Sentinels is no longer -## able to talk with the master. -## 1. If a quorum is set to a value greater than the majority of Sentinels, we are -## making Sentinel able to failover only when there are a very large number (larger -## than majority) of well connected Sentinels which agree about the master being down.s sentinel['quorum'] = 2 ## Consider unresponsive server down after x amount of ms. # sentinel['down_after_milliseconds'] = 10000 -## Specifies the failover timeout in milliseconds. It is used in many ways: -## -## - The time needed to re-start a failover after a previous failover was -## already tried against the same master by a given Sentinel, is two -## times the failover timeout. -## -## - The time needed for a slave replicating to a wrong master according -## to a Sentinel current configuration, to be forced to replicate -## with the right master, is exactly the failover timeout (counting since -## the moment a Sentinel detected the misconfiguration). -## -## - The time needed to cancel a failover that is already in progress but -## did not produced any configuration change (SLAVEOF NO ONE yet not -## acknowledged by the promoted slave). -## -## - The maximum time a failover in progress waits for all the slaves to be -## reconfigured as slaves of the new master. However even after this time -## the slaves will be reconfigured by the Sentinels anyway, but not with -## the exact parallel-syncs progression as specified. # sentinel['failover_timeout'] = 60000 ``` -**Example configation for Sentinel 2:** +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +**Example configuration for Sentinel 2:** In `/etc/gitlab/gitlab.rb`: @@ -589,45 +695,17 @@ redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance sentinel['bind'] = '10.0.0.2' # sentinel['port'] = 26379 # uncomment to change default port -## Quorum must reflect the amount of voting sentinels it take to start a failover. -## Value must NOT be greater then the ammount of sentinels. -## -## The quorum can be used to tune Sentinel in two ways: -## 1. If a the quorum is set to a value smaller than the majority of Sentinels -## we deploy, we are basically making Sentinel more sensible to master failures, -## triggering a failover as soon as even just a minority of Sentinels is no longer -## able to talk with the master. -## 1. If a quorum is set to a value greater than the majority of Sentinels, we are -## making Sentinel able to failover only when there are a very large number (larger -## than majority) of well connected Sentinels which agree about the master being down.s sentinel['quorum'] = 2 ## Consider unresponsive server down after x amount of ms. # sentinel['down_after_milliseconds'] = 10000 -## Specifies the failover timeout in milliseconds. It is used in many ways: -## -## - The time needed to re-start a failover after a previous failover was -## already tried against the same master by a given Sentinel, is two -## times the failover timeout. -## -## - The time needed for a slave replicating to a wrong master according -## to a Sentinel current configuration, to be forced to replicate -## with the right master, is exactly the failover timeout (counting since -## the moment a Sentinel detected the misconfiguration). -## -## - The time needed to cancel a failover that is already in progress but -## did not produced any configuration change (SLAVEOF NO ONE yet not -## acknowledged by the promoted slave). -## -## - The maximum time a failover in progress waits for all the slaves to be -## reconfigured as slaves of the new master. However even after this time -## the slaves will be reconfigured by the Sentinels anyway, but not with -## the exact parallel-syncs progression as specified. # sentinel['failover_timeout'] = 60000 ``` -**Example configation for Sentinel 3:** +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +**Example configuration for Sentinel 3:** In `/etc/gitlab/gitlab.rb`: @@ -643,44 +721,37 @@ redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance sentinel['bind'] = '10.0.0.3' # sentinel['port'] = 26379 # uncomment to change default port -## Quorum must reflect the amount of voting sentinels it take to start a failover. -## Value must NOT be greater then the ammount of sentinels. -## -## The quorum can be used to tune Sentinel in two ways: -## 1. If a the quorum is set to a value smaller than the majority of Sentinels -## we deploy, we are basically making Sentinel more sensible to master failures, -## triggering a failover as soon as even just a minority of Sentinels is no longer -## able to talk with the master. -## 1. If a quorum is set to a value greater than the majority of Sentinels, we are -## making Sentinel able to failover only when there are a very large number (larger -## than majority) of well connected Sentinels which agree about the master being down.s sentinel['quorum'] = 2 ## Consider unresponsive server down after x amount of ms. # sentinel['down_after_milliseconds'] = 10000 -## Specifies the failover timeout in milliseconds. It is used in many ways: -## -## - The time needed to re-start a failover after a previous failover was -## already tried against the same master by a given Sentinel, is two -## times the failover timeout. -## -## - The time needed for a slave replicating to a wrong master according -## to a Sentinel current configuration, to be forced to replicate -## with the right master, is exactly the failover timeout (counting since -## the moment a Sentinel detected the misconfiguration). -## -## - The time needed to cancel a failover that is already in progress but -## did not produced any configuration change (SLAVEOF NO ONE yet not -## acknowledged by the promoted slave). -## -## - The maximum time a failover in progress waits for all the slaves to be -## reconfigured as slaves of the new master. However even after this time -## the slaves will be reconfigured by the Sentinels anyway, but not with -## the exact parallel-syncs progression as specified. # sentinel['failover_timeout'] = 60000 ``` +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + +### Example configuration for the GitLab application + +In `/etc/gitlab/gitlab.rb`: + +``` +# Must be the same in every sentinel node +redis['master_name'] = 'gitlab-redis' + +# The same password for Redis authentication you set up for the master node. +redis['password'] = 'redis-password-goes-here' + +# A list of sentinels with `host` and `port` +gitlab_rails['redis_sentinels'] = [ + {'host' => '10.0.0.1', 'port' => 26379}, + {'host' => '10.0.0.2', 'port' => 26379}, + {'host' => '10.0.0.3', 'port' => 26379} +] +``` + +[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. + ## Advanced configuration Omnibus GitLab configures some things behind the curtains to make the sysadmins' @@ -688,8 +759,8 @@ lives easier. If you want to know what happens underneath keep reading. ### Control running services -In the previous example above we've used `redis_sentinel_role` and -`redis_master_role` which simplify the amount of configuration changes. +In the previous example, we've used `redis_sentinel_role` and +`redis_master_role` which simplifies the amount of configuration changes. If you want more control, here is what each one sets for you automatically when enabled: @@ -698,10 +769,10 @@ when enabled: ## Redis Sentinel Role redis_sentinel_role['enable'] = true -# When Sentinel Role is enabled, the following services are enabled/disabled: +# When Sentinel Role is enabled, the following services are also enabled sentinel['enable'] = true -# This others are disabled: +# The following services are disabled redis['enable'] = false bootstrap['enable'] = false nginx['enable'] = false @@ -709,17 +780,17 @@ postgresql['enable'] = false gitlab_rails['enable'] = false mailroom['enable'] = false -## Redis master/slave Role: +------- + +## Redis master/slave Role redis_master_role['enable'] = true # enable only one of them redis_slave_role['enable'] = true # enable only one of them -# When Redis Master or Slave role are enabled, the following services are enabled/disabled: -# (Note that if redis and sentinel roles are combined both services will be enabled) +# When Redis Master or Slave role are enabled, the following services are +# enabled/disabled. Note that if Redis and Sentinel roles are combined, both +# services will be enabled. -# When Sentinel Role is enabled, the following services are enabled/disabled: -redis['enable'] = true - -# This others are disabled: +# The following services are disabled sentinel['enable'] = false bootstrap['enable'] = false nginx['enable'] = false @@ -727,7 +798,7 @@ postgresql['enable'] = false gitlab_rails['enable'] = false mailroom['enable'] = false -# Redis Slave role also change this setting from default 'true' to 'false': +# For Redis Slave role, also change this setting from default 'true' to 'false': redis['master'] = false ``` @@ -856,6 +927,16 @@ To make sure your configuration is correct: Changes to Redis HA over time. +**8.14** + +- Redis Sentinel support is production-ready and bundled in the Omnibus GitLab + Enterprise Edition package +- Documentation restructure for better readability + +**8.11** + +- Experimental Redis Sentinel support was added + ### Experimental Redis Sentinel support > @@ -880,7 +961,7 @@ Read more on high-availability configuration: [gh-534]: https://github.com/redis/redis-rb/issues/534 [redis]: http://redis.io/ [sentinel]: http://redis.io/topics/sentinel -[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/82b7345b150f072c8673c79738ce893f92d0d652/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L134-159 +[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb [source]: ../../install/installation.md [ce]: https://about.gitlab.com/downloads [ee]: https://about.gitlab.com/downloads-ee -- cgit v1.2.1 From f92ea120bae61c071f8588befd2c82ec5ce0e302 Mon Sep 17 00:00:00 2001 From: Victor Wu Date: Mon, 14 Nov 2016 18:52:09 +0000 Subject: [ci skip] Link to the copy (messaging) page. Add new file for copy (messaging) guidelines Update copy guidelines Fix typo Add forms to guidelines Simplify Copy heading Refine copy page. Add images and table Fix toc link on Copy page --- doc/development/ux_guide/copy.md | 78 +++++++++++++++++++++ .../ux_guide/img/copy-form-addissuebutton.png | Bin 0 -> 26541 bytes .../ux_guide/img/copy-form-addissueform.png | Bin 0 -> 38242 bytes .../ux_guide/img/copy-form-editissuebutton.png | Bin 0 -> 16466 bytes .../ux_guide/img/copy-form-editissueform.png | Bin 0 -> 40660 bytes doc/development/ux_guide/index.md | 5 ++ 6 files changed, 83 insertions(+) create mode 100644 doc/development/ux_guide/copy.md create mode 100644 doc/development/ux_guide/img/copy-form-addissuebutton.png create mode 100644 doc/development/ux_guide/img/copy-form-addissueform.png create mode 100644 doc/development/ux_guide/img/copy-form-editissuebutton.png create mode 100644 doc/development/ux_guide/img/copy-form-editissueform.png (limited to 'doc') diff --git a/doc/development/ux_guide/copy.md b/doc/development/ux_guide/copy.md new file mode 100644 index 00000000000..03392a003ee --- /dev/null +++ b/doc/development/ux_guide/copy.md @@ -0,0 +1,78 @@ +# Copy + +The copy and messaging is a core part of the experience of GitLab and the conversation with our users. Follow the below conventions throughout GitLab. + +>**Note:** +We are currently inconsistent with this guidance. Images below are created to illustrate the point. As this guidance is refined, we will ensure that our experiences align. + +## Contents +* [Brevity](#brevity) +* [Forms](#forms) +* [Terminology](#terminology) + +--- + +## Brevity +Users will skim content, rather than read text carefully. +When familiar with a web app, users rely on muscle memory, and may read even less when moving quickly. +A good experience should quickly orient a user, regardless of their experience, to the purpose of the current screen. This should happen without the user having to consciously read long strings of text. +In general, text is burdensome and adds cognitive load. This is especially pronounced in a powerful productivity tool such as GitLab. +We should _not_ rely on words as a crutch to explain the purpose of a screen. +The current navigation and composition of the elements on the screen should get the user 95% there, with the remaining 5% being specific elements such as text. +This means that, as a rule, copy should be very short. A long message or label is a red flag hinting at design that needs improvement. + +>**Example:** +Use `Add` instead of `Add issue` as a button label. +Preferrably use context and placement of controls to make it obvious what clicking on them will do. + +--- + +## Forms + +### Adding items + +When viewing a list of issues, there is a button that is labeled `Add`. Given the context in the example, it is clearly referring to issues. If the context were not clear enough, the label could be `Add issue`. Clicking the button will bring you to the `Add issue` form. Other add flows should be similar. + +![Add issue button](img/copy-form-addissuebutton.png) + +The form should be titled `Add issue`. The submit button should be labeled `Save` or `Submit`. Do not use `Add`, `Create`, `New`, or `Save Changes`. The cancel button should be labeled `Cancel`. Do not use `Back`. + +![Add issue form](img/copy-form-addissueform.png) + +### Editing items + +When in context of an issue, the affordance to edit it is labeled `Edit`. If the context is not clear enough, `Edit issue` could be considered. Other edit flows should be similar. + +![Edit issue button](img/copy-form-editissuebutton.png) + +The form should be titled `Edit Issue`. The submit button should be labeled `Save`. Do not use `Edit`, `Update`, `New`, or `Save Changes`. The cancel button should be labeled `Cancel`. Do not use `Back`. + +![Edit issue form](img/copy-form-editissueform.png) + +--- + +## Terminology + +### Issues + +#### Adjectives (states) + +| Term | Use | +| ---- | --- | +| Open | Issue is active | +| Closed | Issue is no longer active | + +>**Example:** +Use `5 open issues` and do not use `5 pending issues`. +Only use the adjectives in the table above. + +#### Verbs (actions) + +| Term | Use | +| ---- | --- | +| Add | For adding an issue. Do not use `create` or `new` | +| View | View an issue | +| Edit | Edit an issue. Do not use `update` | +| Close | Closing an issue | +| Re-open | Re-open an issue. There should never be a need to use `open` as a verb | +| Delete | Deleting an issue. Do not use `remove` | \ No newline at end of file diff --git a/doc/development/ux_guide/img/copy-form-addissuebutton.png b/doc/development/ux_guide/img/copy-form-addissuebutton.png new file mode 100644 index 00000000000..18839d447e8 Binary files /dev/null and b/doc/development/ux_guide/img/copy-form-addissuebutton.png differ diff --git a/doc/development/ux_guide/img/copy-form-addissueform.png b/doc/development/ux_guide/img/copy-form-addissueform.png new file mode 100644 index 00000000000..e6838c06eca Binary files /dev/null and b/doc/development/ux_guide/img/copy-form-addissueform.png differ diff --git a/doc/development/ux_guide/img/copy-form-editissuebutton.png b/doc/development/ux_guide/img/copy-form-editissuebutton.png new file mode 100644 index 00000000000..2435820e14f Binary files /dev/null and b/doc/development/ux_guide/img/copy-form-editissuebutton.png differ diff --git a/doc/development/ux_guide/img/copy-form-editissueform.png b/doc/development/ux_guide/img/copy-form-editissueform.png new file mode 100644 index 00000000000..5ddeda33e68 Binary files /dev/null and b/doc/development/ux_guide/img/copy-form-editissueform.png differ diff --git a/doc/development/ux_guide/index.md b/doc/development/ux_guide/index.md index 1a61be4ed51..8aed11ebac3 100644 --- a/doc/development/ux_guide/index.md +++ b/doc/development/ux_guide/index.md @@ -26,6 +26,11 @@ The GitLab experience is broken apart into several surfaces. Each of these surfa --- +### [Copy](copy.md) +Conventions on text and messaging within labels, buttons, and other components. + +--- + ### [Features](features.md) The previous building blocks are combined into complete features in the GitLab UX. Examples include our navigation, filters, search results, and empty states. -- cgit v1.2.1 From a75c5f195124c12d3776d8c553bc71721cee121d Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 10:20:59 +0100 Subject: Add clear instructions for the different Redis HA setups --- doc/administration/high_availability/redis.md | 75 ++++++++++++++------------- 1 file changed, 40 insertions(+), 35 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 6de92ae3741..7ea49a2c22a 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -20,8 +20,9 @@ Omnibus GitLab packages. from source, follow the [Redis HA source installation](redis_source.md) guide. - Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only. For configuring Sentinel with the Omnibus GitLab Community Edition and - installations from source, follow the - [Redis HA source installation](redis_source.md) guide. + installations from source, read the + [Available configuration setups](#available-configuration-setups) section + below. @@ -40,7 +41,7 @@ Omnibus GitLab packages. - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) - [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha) -- [Example of a minimal configuration with 1 master, 2 slaves and 3 sentinels](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) +- [Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - [Configuration for Redis master](#configuration-for-redis-master) - [Configuration for Redis slaves](#configuration-for-redis-slaves) - [Configuration for Sentinels](#configuration-for-sentinels) @@ -79,17 +80,17 @@ components below. ### High Availability with Sentinel -> +>**Notes:** - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel servers that will monitor a group of Redis servers to provide failover support. - Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package - comes with Redis Sentinel daemon support. + comes with Redis Sentinel daemon built-in. High Availability with Redis requires a few things: - Multiple Redis instances - - Run Redis in a **Master** x **Slave** topology - - Multiple Sentinel instances +- Run Redis in a **Master** x **Slave** topology +- Multiple Sentinel instances - Application support and visibility to all Sentinel and Redis instances Redis Sentinel can handle the most important tasks in an HA environment and that's @@ -132,27 +133,6 @@ the Omnibus GitLab package in `5` **independent** machines, both with - Redis Slave + Sentinel - Redis Slave + Sentinel -### Available configuration setups - -Based on your infrastructure setup and how you have installed GitLab, there are -multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or -Redis Sentinel bundled with them so you only need to focus on configuration. -Pick the one that suits your needs. - -- [Installations from source][source]: You need to install Redis and Sentinel - yourself. Use the [Redis HA installation from source](redis_source.md) guide. -- [Omnibus package Community Edition (CE) package][ce]: Redis is bundled, so you - can use the package with only the Redis service enabled (works for both master - and slave setups). To install and configure Sentinel, you can use the - [Redis HA installation from source](redis_source.md) guide. -- [Omnibus package Enterprise Edition (EE) package][ee]: Both Redis and Sentinel - are bundled, so you can use the EE package to setup the whole Redis HA - infrastructure (master, slave and Sentinel). - -Note that even if you have installed GitLab using the Omnibus GitLab packages -(both CE and EE), you can still use an -[external Redis server](#using-a-non-omnibus-external-redis-server). - ### Redis setup overview You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they @@ -168,7 +148,7 @@ simultaneously down. Please note that there are different requirements for Sentinel nodes. If you host them in the same Redis machines, you may need to take that restrictions into consideration when calculating the amount of -nodes to be provisioned. See [Sentinel setup](#sentinel-setup) +nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview) documentation for more information. All Redis nodes should be configured the same way and with similar server specs, as @@ -245,6 +225,31 @@ the official documentation: the slaves will be reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. +### Available configuration setups + +Based on your infrastructure setup and how you have installed GitLab, there are +multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or +Redis Sentinel bundled with them so you only need to focus on configuration. +Pick the one that suits your needs. + +- [Installations from source][source]: You need to install Redis and Sentinel + yourself. Use the [Redis HA installation from source](redis_source.md) + documentation. +- [Omnibus GitLab **Community Edition** (CE) package][ce]: Redis is bundled, so you + can use the package with only the Redis service enabled as described in steps + 1 and 2 of this document (works for both master and slave setups). To install + and configure Sentinel, jump directly to the Sentinel section in the + [Redis HA installation from source](redis_source.md#step-3-configuring-the-redis-sentinel-instances) documentation. +- [Omnibus GitLab **Enterprise Edition** (EE) package][ee]: Both Redis and Sentinel + are bundled in the package, so you can use the EE package to setup the whole + Redis HA infrastructure (master, slave and Sentinel) which is described in + this document. +- If you have installed GitLab using the Omnibus GitLab packages (CE or EE), + but you want to use your own external Redis server, follow steps 1-3 in the + [Redis HA installation from source](redis_source.md) documentation, then go + straight to step 4 in this guide to + [set up the GitLab application](#step-4-configuring-the-gitlab-application). + ## Configuring Redis HA This is the section where we install and setup the new Redis instances. @@ -284,7 +289,7 @@ The prerequisites for a HA Redis setup are the following: ``` 1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab - package using **steps 1 and 2** from the GitLab downloads page. + package you want using **steps 1 and 2** from the GitLab downloads page. - Make sure you select the correct Omnibus package, with the same version and type (Community, Enterprise editions) of your current install. - Do not complete any other steps on the download page. @@ -313,7 +318,7 @@ The prerequisites for a HA Redis setup are the following: 1. To prevent database migrations from running on upgrade, run: ``` - touch /etc/gitlab/skip-auto-migrations + sudo touch /etc/gitlab/skip-auto-migrations ``` Only the primary GitLab application server should handle migrations. @@ -329,7 +334,7 @@ The prerequisites for a HA Redis setup are the following: ``` 1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab - package using **steps 1 and 2** from the GitLab downloads page. + package you want using **steps 1 and 2** from the GitLab downloads page. - Make sure you select the correct Omnibus package, with the same version and type (Community, Enterprise editions) of your current install. - Do not complete any other steps on the download page. @@ -367,7 +372,7 @@ The prerequisites for a HA Redis setup are the following: 1. To prevent database migrations from running on upgrade, run: ``` - touch /etc/gitlab/skip-auto-migrations + sudo touch /etc/gitlab/skip-auto-migrations ``` Only the primary GitLab application server should handle migrations. @@ -495,7 +500,7 @@ multiple machines with the Sentinel daemon. 1. To prevent database migrations from running on upgrade, run: ``` - touch /etc/gitlab/skip-auto-migrations + sudo touch /etc/gitlab/skip-auto-migrations ``` Only the primary GitLab application server should handle migrations. @@ -560,7 +565,7 @@ redis['enable'] = false If you fail to replicate first, you may loose data (unprocessed background jobs). -## Example of a minimal configuration with 1 master, 2 slaves and 3 sentinels +## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels In this example we consider that all servers have an internal network interface with IPs in the `10.0.0.x` range, and that they can connect -- cgit v1.2.1 From 178d5ae9719b0edf6a4873502422e3302d264528 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 10:21:44 +0100 Subject: Rearrange Redis HA source installations sections --- .../high_availability/redis_source.md | 57 +++++++++++++--------- 1 file changed, 33 insertions(+), 24 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 20630b070c0..a420d44f451 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -1,26 +1,34 @@ -# Configuring Redis for GitLab HA (Source Install) +# Configuring non-Omnibus Redis for GitLab HA -We highly recommend that you use Omnibus GitLab packages, as we can optimize -required packages specifically for GitLab, and we will take care of upgrading +This is the documentation for configuring a Highly Available Redis setup when +you have installed Redis all by yourself and not using the bundled one that +comes with the Omnibus packages. + +We cannot stress enough the importance of reading the +[Overview section](redis.md#overview) of the Omnibus Redis HA as it provides +some invaluable information to the configuration of Redis. Please proceed to +read it before going forward with this guide. + +We also highly recommend that you use the Omnibus GitLab packages, as we +optimize them specifically for GitLab, and we will take care of upgrading Redis to the latest supported version. -If you are building packages for a specific distro, or trying to build some -internal automation, you can check this documentation to learn about the -minimal setup, required changes, etc. +If you're not sure whether this guide is for you, please refer to +[Available configuration setups](redis.md#available-configuration-setups) in +the Omnibus Redis HA documentation. -If you want to see the documentation for Omnibus GitLab Install, please -[read it here](redis.md). +--- **Table of Contents** -- [Configure your own Redis server](#configure-your-own-redis-server) - - [Configuring Master Redis instance](#configuring-master-redis-instance) - - [Configuring Slave Redis instances](#configuring-slave-redis-instances) - - [Configuring Redis Sentinel instances](#configuring-redis-sentinel-instances) -- [GitLab setup](#gitlab-setup) -- [Example configurations](#example-configurations) +- [Configuring your own Redis server](#configuring-your-own-redis-server) + - [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance) + - [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances) + - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) + - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) +- [Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - [Configuring Redis Master](#configuring-redis-master) - [Configuring Redis Slaves](#configuring-redis-slaves) - [Configuring Redis Sentinel](#configuring-redis-sentinel) @@ -28,7 +36,7 @@ If you want to see the documentation for Omnibus GitLab Install, please -## Configure your own Redis server +## Configuring your own Redis server Redis server must be configured to use TCP connection instead of socket, and since Redis `3.2`, you must define a password to receive external @@ -41,8 +49,7 @@ To configure Redis to use TCP connection you need to define both `bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the IP of the desired interface (for ex. one from an internal network). - -### Configuring Master Redis instance +### Step 1. Configuring the master Redis instance You need to make the following changes in `redis.conf`: @@ -61,7 +68,7 @@ You need to make the following changes in `redis.conf`: See [example configuration](#configuring-redis-master) below. -### Configuring Slave Redis instances +### Step 2. Configuring the slave Redis instances 1. Follow same instructions for Redis Master @@ -71,7 +78,7 @@ See [example configuration](#configuring-redis-master) below. See [example configuration](#configuring-redis-slaves) below. -### Configuring Redis Sentinel instances +### Step 3. Configuring the Redis Sentinel instances Sentinel is a special type of Redis server. It inherits most of the basic configuration options you can define in `redis.conf`, with specific ones @@ -122,7 +129,7 @@ And the sentinel specific ones: See [example configuration](#configuring-redis-sentinel) below. -## GitLab setup +### Step 4. Configuring the GitLab application You can enable or disable Sentinel support at any time in new or existing installations. From the GitLab application perspective, all it requires is @@ -141,7 +148,7 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se 1. Restart GitLab for the changes to take effect. -## Example configurations +## Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels In this example we consider that all servers have an internal network interface with IPs in the `10.0.0.x` range, and that they can connect @@ -247,12 +254,13 @@ sentinel failover_timeout 30000 ## Troubleshooting -We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained in the documentation for Omnibus -Install. Here we will list only the things that are specific to a **Source** install. +We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained +in the documentation for Omnibus GitLab installations. Here we will list only +the things that are specific to a source installation. If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related -to [this issue][gh-531]. +to [this upstream issue][gh-531]. It's a bit non-intuitive the way you have to config `resque.yml` and `sentinel.conf`, otherwise `redis-rb` will not work properly. @@ -287,3 +295,4 @@ production: When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) [gh-531]: https://github.com/redis/redis-rb/issues/531 +[downloads]: https://about.gitlab.com/downloads -- cgit v1.2.1 From dc54239683fbe77b13216deb6a0728563a0a89d0 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 11:42:12 +0100 Subject: Merge examples of Redis master/slave + Sentinels --- doc/administration/high_availability/redis.md | 144 +++++++------------------- 1 file changed, 37 insertions(+), 107 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 7ea49a2c22a..02ca4999cd4 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -29,22 +29,23 @@ Omnibus GitLab packages. **Table of Contents** - [Overview](#overview) - - [Prerequisites](#prerequisites) - [High Availability with Sentinel](#high-availability-with-sentinel) - [Recommended setup](#recommended-setup) - - [Available configuration setups](#available-configuration-setups) - [Redis setup overview](#redis-setup-overview) - [Sentinel setup overview](#sentinel-setup-overview) + - [Available configuration setups](#available-configuration-setups) - [Configuring Redis HA](#configuring-redis-ha) - - [Step 1. Configuring the Master Redis instance](#step-1-configuring-the-master-redis-instance) - - [Step 2. Configuring the Slave Redis instances](#step-2-configuring-the-slave-redis-instances) + - [Prerequisites](#prerequisites) + - [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance) + - [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances) - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) - [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha) - [Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - - [Configuration for Redis master](#configuration-for-redis-master) - - [Configuration for Redis slaves](#configuration-for-redis-slaves) - - [Configuration for Sentinels](#configuration-for-sentinels) + - [Example configuration for Redis master and Sentinel 1](#example-configuration-for-redis-master-and-sentinel-1) + - [Example configuration for Redis slave 1 and Sentinel 2](#example-configuration-for-redis-slave-1-and-sentinel-2) + - [Example configuration for Redis slave 2 and Sentinel 3](#example-configuration-for-redis-slave-2-and-sentinel-3) + - [Example configuration for the GitLab application](#example-configuration-for-the-gitlab-application) - [Advanced configuration](#advanced-configuration) - [Control running services](#control-running-services) - [Troubleshooting](#troubleshooting) @@ -277,7 +278,7 @@ The prerequisites for a HA Redis setup are the following: change the default ones). 1. The server that hosts the GitLab application must be able to access the Redis nodes. -1. Protect the nodes from access from external networks (Internet), using +1. Protect the nodes from access from external networks ([Internet][it]), using firewall. ### Step 1. Configuring the master Redis instance @@ -527,13 +528,13 @@ which ideally should not have Redis or Sentinels on it for a HA setup. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: ``` - # Must be the same in every sentinel node + ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' - # The same password for Redis authentication you set up for the master node. + ## The same password for Redis authentication you set up for the master node. redis['password'] = 'redis-password-goes-here' - # A list of sentinels with `host` and `port` + ## A list of sentinels with `host` and `port` gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.1', 'port' => 26379}, {'host' => '10.0.0.2', 'port' => 26379}, @@ -567,6 +568,11 @@ If you fail to replicate first, you may loose data (unprocessed background jobs) ## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels +>**Note:** +Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For +different setups, read the +[available configuration setups](#available-configuration-setups) section. + In this example we consider that all servers have an internal network interface with IPs in the `10.0.0.x` range, and that they can connect to each other using these IPs. @@ -595,142 +601,70 @@ The same thing will happen with `sentinel.conf` that will be overridden after th initial execution, after any new sentinel node starts watching the **Master**, or a failover promotes a different **Master** node. -### Example configuration for Redis master - -**Example configation for Redis Master:** +### Example configuration for Redis master and Sentinel 1 In `/etc/gitlab/gitlab.rb`: ```ruby redis_master_role['enable'] = true - +redis_sentinel_role['enable'] = true redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' -``` - -[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. - -### Example configuration for Redis slaves - -**Example configuration for Slave 1:** - -In `/etc/gitlab/gitlab.rb`: - -```ruby -redis_slave_role['enable'] = true - -redis['bind'] = '10.0.0.2' -redis['port'] = 6379 -redis['password'] = 'redis-password-goes-here' -redis['master_password'] = 'redis-password-goes-here' - -redis['master_ip'] = '10.0.0.1' # IP of master Redis server -#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default -``` - -[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. - -**Example configuration for Slave 2:** - -In `/etc/gitlab/gitlab.rb`: - -```ruby -redis_slave_role['enable'] = true - -redis['bind'] = '10.0.0.3' -redis['port'] = 6379 -redis['password'] = 'redis-password-goes-here' -redis['master_password'] = 'redis-password-goes-here' - -redis['master_ip'] = '10.0.0.1' # IP of master Redis server -#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default -``` - -[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. - -### Example configuration for Sentinels - ->**Note:** -Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the -Omnibus Community Edition and installations from source, follow the -[Redis HA source install](redis_source.md) guide. - -Please note that some of the variables are already configured previously -as they are required for Redis replication. - -**Example configuration for Sentinel 1:** - -In `/etc/gitlab/gitlab.rb`: - -```ruby -redis_sentinel_role['enable'] = true - redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance #redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default - -## Configure Sentinel sentinel['bind'] = '10.0.0.1' # sentinel['port'] = 26379 # uncomment to change default port sentinel['quorum'] = 2 - -## Consider unresponsive server down after x amount of ms. # sentinel['down_after_milliseconds'] = 10000 - # sentinel['failover_timeout'] = 60000 ``` [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -**Example configuration for Sentinel 2:** +### Example configuration for Redis slave 1 and Sentinel 2 In `/etc/gitlab/gitlab.rb`: ```ruby +redis_slave_role['enable'] = true redis_sentinel_role['enable'] = true - +redis['bind'] = '10.0.0.2' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node -redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance -redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance -#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default - -## Configure Sentinel sentinel['bind'] = '10.0.0.2' # sentinel['port'] = 26379 # uncomment to change default port - sentinel['quorum'] = 2 - -## Consider unresponsive server down after x amount of ms. # sentinel['down_after_milliseconds'] = 10000 - # sentinel['failover_timeout'] = 60000 ``` [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -**Example configuration for Sentinel 3:** +### Example configuration for Redis slave 2 and Sentinel 3 In `/etc/gitlab/gitlab.rb`: ```ruby +redis_slave_role['enable'] = true redis_sentinel_role['enable'] = true - +redis['bind'] = '10.0.0.3' +redis['port'] = 6379 +redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' +redis['master_ip'] = '10.0.0.1' # IP of master Redis server +#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node -redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance -redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance -#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default - -## Configure Sentinel sentinel['bind'] = '10.0.0.3' # sentinel['port'] = 26379 # uncomment to change default port - sentinel['quorum'] = 2 - -## Consider unresponsive server down after x amount of ms. # sentinel['down_after_milliseconds'] = 10000 - # sentinel['failover_timeout'] = 60000 ``` @@ -740,14 +674,9 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: -``` -# Must be the same in every sentinel node +```ruby redis['master_name'] = 'gitlab-redis' - -# The same password for Redis authentication you set up for the master node. redis['password'] = 'redis-password-goes-here' - -# A list of sentinels with `host` and `port` gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.1', 'port' => 26379}, {'host' => '10.0.0.2', 'port' => 26379}, @@ -970,3 +899,4 @@ Read more on high-availability configuration: [source]: ../../install/installation.md [ce]: https://about.gitlab.com/downloads [ee]: https://about.gitlab.com/downloads-ee +[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png -- cgit v1.2.1 From b6447f3027b7439e33bb8864992f5b4e60ca5cc5 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 11:51:28 +0100 Subject: Add steps to Redis HA source installation --- .../high_availability/redis_source.md | 196 +++++++++++++-------- 1 file changed, 123 insertions(+), 73 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index a420d44f451..1660e26c784 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -38,45 +38,79 @@ the Omnibus Redis HA documentation. ## Configuring your own Redis server -Redis server must be configured to use TCP connection instead of socket, -and since Redis `3.2`, you must define a password to receive external -connections (`requirepass`). +This is the section where we install and setup the new Redis instances. -You will also need to define equal password for slave password definition -(`masterauth`), in the same instance, if you are using Redis with Sentinel. +### Prerequisites -To configure Redis to use TCP connection you need to define both -`bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the -IP of the desired interface (for ex. one from an internal network). +- All Redis servers in this guide must be configured to use a TCP connection + instead of a socket. To configure Redis to use TCP connections you need to + define both `bind` and `port` in the Redis config file. You can bind to all + interfaces (`0.0.0.0`) or specify the IP of the desired interface + (e.g., one from an internal network). +- Since Redis 3.2, you must define a password to receive external connections + (`requirepass`). +- If you are using Redis with Sentinel, you will also need to define the same + password for the slave password definition (`masterauth`) in the same instance. + +In addition, read the prerequisites as described in the +[Omnibus Redis HA document](redis.md#prerequisites) since they provide some +valuable information for the general setup. ### Step 1. Configuring the master Redis instance -You need to make the following changes in `redis.conf`: +Assuming that the Redis master instance IP is `10.0.0.1`: -1. Define a `bind` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external accessible IP, make - sure you add extra firewall rules to prevent unauthorized access: +1. [Install Redis](../../install/installation.md#6-redis) +1. Edit `/etc/redis/redis.conf`: -1. Define a `port` to force redis to listen on TCP so other machines can - connect to it (default port is `6379`). + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 -1. Set up password authentication (use the same password in all nodes). - The password should be defined equal for both `requirepass` and `masterauth` - when setting up Redis to use with Sentinel. + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 -1. Restart the Redis services for the changes to take effect. + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` -See [example configuration](#configuring-redis-master) below. +1. Restart the Redis service for the changes to take effect. ### Step 2. Configuring the slave Redis instances -1. Follow same instructions for Redis Master +Assuming that the Redis slave instance IP is `10.0.0.2`: + +1. [Install Redis](../../install/installation.md#6-redis) +1. Edit `/etc/redis/redis.conf`: + + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.2 + + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 -1. Define `slaveof` pointing to the Redis master instance with **IP** and **port**. + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here -1. Restart the Redis services for the changes to take effect. + ## Define `slaveof` pointing to the Redis master instance with IP and port. + slaveof 10.0.0.1 6379 + ``` -See [example configuration](#configuring-redis-slaves) below. +1. Restart the Redis service for the changes to take effect. +1. Go through the steps again for all the other slave nodes. ### Step 3. Configuring the Redis Sentinel instances @@ -84,50 +118,64 @@ Sentinel is a special type of Redis server. It inherits most of the basic configuration options you can define in `redis.conf`, with specific ones starting with `sentinel` prefix. -You will need to define the initial configs to enable connectivity: - -1. Define a `bind` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external accessible IP, make - sure you add extra firewall rules to prevent unauthorized access: - -1. Define a `port` to force sentinel to listen on TCP so other machines can - connect to it (default port is `26379`). - -And the sentinel specific ones: - -1. Define with `sentinel auth-pass` the same shared password you have - defined for both Redis **Master** and **Slaves** instances. - -1. Define with `sentinel monitor` the **IP** and **port** of the Redis - **Master** node, and the **quorum** required to start a failover. - If you need more information to understand about quorum, please - read the detailed explanation in the [HA documentation for Omnibus Installs](redis.md). - -1. Define with `sentinel down-after-milliseconds` the amount in `ms` of time - that an unresponsive server will be considered down. - -1. Define a value for `sentinel failover_timeout` in `ms`. This has multiple - meanings: - - * The time needed to re-start a failover after a previous failover was - already tried against the same master by a given Sentinel, is two - times the failover timeout. - - * The time needed for a slave replicating to a wrong master according - to a Sentinel current configuration, to be forced to replicate - with the right master, is exactly the failover timeout (counting since - the moment a Sentinel detected the misconfiguration). - - * The time needed to cancel a failover that is already in progress but - did not produced any configuration change (SLAVEOF NO ONE yet not - acknowledged by the promoted slave). - - * The maximum time a failover in progress waits for all the slaves to be - reconfigured as slaves of the new master. However even after this time - the slaves will be reconfigured by the Sentinels anyway, but not with - the exact parallel-syncs progression as specified. - -See [example configuration](#configuring-redis-sentinel) below. +Assuming that the Redis Sentinel is installed on the same instance as Redis +master with IP `10.0.0.1` (some settings might overlap with the master): + +1. [Install Redis](../../install/installation.md#6-redis) +1. Edit `/etc/redis/redis.conf`: + + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 + + ## Define a `port` to force Sentinel to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 26379 + + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + + ## Define with `sentinel auth-pass` the same shared password you have + ## defined for both Redis master and slaves instances. + sentinel auth-pass gitlab-redis redis-password-goes-here + + ## Define with `sentinel monitor` the IP and port of the Redis + ## master node, and the quorum required to start a failover. + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + + ## Define with `sentinel down-after-milliseconds` the time in `ms` + ## that an unresponsive server will be considered down. + sentinel down-after-milliseconds gitlab-redis 10000 + + ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple + ## meanings: + ## + ## * The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## * The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## * The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## * The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + sentinel failover_timeout 30000 + ``` +1. Restart the Redis service for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes. ### Step 4. Configuring the GitLab application @@ -136,17 +184,17 @@ installations. From the GitLab application perspective, all it requires is the correct credentials for the Sentinel nodes. While it doesn't require a list of all Sentinel nodes, in case of a failure, -it needs to access at one of listed ones. +it needs to access at least one of listed ones. ->**Note:** The following steps should be performed in the [GitLab application server](gitlab.md) -which ideally should not have Redis or Sentinels in the same machine for a HA setup. +which ideally should not have Redis or Sentinels in the same machine for a HA +setup: 1. Edit `/home/git/gitlab/config/resque.yml` following the example in `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels lines, pointing to the correct server credentials. -1. Restart GitLab for the changes to take effect. +1. [Restart GitLab][restart] for the changes to take effect. ## Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels @@ -156,7 +204,7 @@ to each other using these IPs. In a real world usage, you would also setup firewall rules to prevent unauthorized access from other machines, and block traffic from the -outside (Internet). +outside ([Internet][it]). We will use the same `3` nodes with **Redis** + **Sentinel** topology discussed in the [Configuring Redis for GitLab HA](redis.md) documentation. @@ -165,7 +213,7 @@ Here is a list and description of each **machine** and the assigned **IP**: * `10.0.0.1`: Redis Master + Sentinel 1 * `10.0.0.2`: Redis Slave 1 + Sentinel 2 -* `10.0.0.2`: Redis Slave 2 + Sentinel 3 +* `10.0.0.3`: Redis Slave 2 + Sentinel 3 Please note that after the initial configuration, if a failover is initiated by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** @@ -296,3 +344,5 @@ When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics [gh-531]: https://github.com/redis/redis-rb/issues/531 [downloads]: https://about.gitlab.com/downloads +[restart]: ../restart_gitlab.md#installations-from-source +[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png -- cgit v1.2.1 From 850405b42ef408785c6133ab7ba9c7f7303f18b6 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 12:04:33 +0100 Subject: Merge examples of Redis master/slave + Sentinels for source docs [ci skip] --- .../high_availability/redis_source.md | 173 +++++++++++++-------- 1 file changed, 106 insertions(+), 67 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 1660e26c784..e7a8f47355d 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -24,14 +24,16 @@ the Omnibus Redis HA documentation. **Table of Contents** - [Configuring your own Redis server](#configuring-your-own-redis-server) + - [Prerequisites](#prerequisites) - [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance) - [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances) - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) - [Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - - [Configuring Redis Master](#configuring-redis-master) - - [Configuring Redis Slaves](#configuring-redis-slaves) - - [Configuring Redis Sentinel](#configuring-redis-sentinel) + - [Example configuration for Redis master and Sentinel 1](#example-configuration-for-redis-master-and-sentinel-1) + - [Example configuration for Redis slave 1 and Sentinel 2](#example-configuration-for-redis-slave-1-and-sentinel-2) + - [Example configuration for Redis slave 2 and Sentinel 3](#example-configuration-for-redis-slave-2-and-sentinel-3) + - [Example configuration of the GitLab application](#example-configuration-of-the-gitlab-application) - [Troubleshooting](#troubleshooting) @@ -121,8 +123,8 @@ starting with `sentinel` prefix. Assuming that the Redis Sentinel is installed on the same instance as Redis master with IP `10.0.0.1` (some settings might overlap with the master): -1. [Install Redis](../../install/installation.md#6-redis) -1. Edit `/etc/redis/redis.conf`: +1. [Install Redis Sentinel](http://redis.io/topics/sentinel) +1. Edit `/etc/redis/sentinel.conf`: ```conf ## Define a `bind` address pointing to a local IP that your other machines @@ -191,8 +193,24 @@ which ideally should not have Redis or Sentinels in the same machine for a HA setup: 1. Edit `/home/git/gitlab/config/resque.yml` following the example in - `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels - lines, pointing to the correct server credentials. + [resque.yml.example][resque], and uncomment the Sentinel lines, pointing to + the correct server credentials: + + ```yaml + # resque.yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` 1. [Restart GitLab][restart] for the changes to take effect. @@ -206,14 +224,16 @@ In a real world usage, you would also setup firewall rules to prevent unauthorized access from other machines, and block traffic from the outside ([Internet][it]). -We will use the same `3` nodes with **Redis** + **Sentinel** topology -discussed in the [Configuring Redis for GitLab HA](redis.md) documentation. +For this example, **Sentinel 1** will be configured in the same machine as the +**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the +**Slave 1** and **Slave 2** respectively. Here is a list and description of each **machine** and the assigned **IP**: * `10.0.0.1`: Redis Master + Sentinel 1 * `10.0.0.2`: Redis Slave 1 + Sentinel 2 * `10.0.0.3`: Redis Slave 2 + Sentinel 3 +* `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** @@ -224,81 +244,100 @@ The same thing will happen with `sentinel.conf` that will be overridden after th initial execution, after any new sentinel node starts watching the **Master**, or a failover promotes a different **Master** node. -### Configuring Redis Master +### Example configuration for Redis master and Sentinel 1 -**Example configation for Redis Master - `redis.conf`:** +1. In `/etc/redis/redis.conf`: -```conf -bind 10.0.0.1 -port 6379 -requirepass redis-password-goes-here -masterauth redis-password-goes-here -``` + ```conf + bind 10.0.0.1 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` + +1. In `/etc/redis/sentinel.conf`: -### Configuring Redis Slaves + ```conf + bind 10.0.0.1 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` -**Example configation for Slave 1 - `redis.conf`:** +1. Restart the Redis service for the changes to take effect. -```conf -bind 10.0.0.2 -port 6379 -requirepass redis-password-goes-here -masterauth redis-password-goes-here +### Example configuration for Redis slave 1 and Sentinel 2 -# IP and port of the master Redis server -slaveof 10.0.0.1 6379 -``` +1. In `/etc/redis/redis.conf`: -**Example configation for Slave 2 - `redis.conf`:** + ```conf + bind 10.0.0.2 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` -```conf -bind 10.0.0.3 -port 6379 -requirepass redis-password-goes-here -masterauth redis-password-goes-here +1. In `/etc/redis/sentinel.conf`: -# IP and port of the master Redis server -slaveof 10.0.0.1 6379 -``` + ```conf + bind 10.0.0.2 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` -### Configuring Redis Sentinel +1. Restart the Redis service for the changes to take effect. -For this example, **Sentinel 1** will be configured in the same machine as the -**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the -**Slave 1** and **Slave 2** respectively. +### Example configuration for Redis slave 2 and Sentinel 3 -**Example configation for Sentinel 1 - `sentinel.conf`:** +1. In `/etc/redis/redis.conf`: -```conf -bind 10.0.0.1 -port 26379 -sentinel auth-pass gitlab-redis redis-password-goes-here -sentinel monitor gitlab-redis 10.0.0.1 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel failover_timeout 30000 -``` + ```conf + bind 10.0.0.3 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` -**Example configation for Sentinel 2 - `sentinel.conf`:** +1. In `/etc/redis/sentinel.conf`: -```conf -bind 10.0.0.2 -port 26379 -sentinel auth-pass gitlab-redis redis-password-goes-here -sentinel monitor gitlab-redis 10.0.0.1 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel failover_timeout 30000 -``` + ```conf + bind 10.0.0.3 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` -**Example configation for Sentinel 3 - `sentinel.conf`:** +1. Restart the Redis service for the changes to take effect. -```conf -bind 10.0.0.3 -port 26379 -sentinel auth-pass gitlab-redis redis-password-goes-here -sentinel monitor gitlab-redis 10.0.0.1 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel failover_timeout 30000 -``` +### Example configuration of the GitLab application + +1. Edit `/home/git/gitlab/config/resque.yml`: + + ```yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` + +1. [Restart GitLab][restart] for the changes to take effect. ## Troubleshooting -- cgit v1.2.1 From 04f2dd65fb355eb3bf20e0d3deef9123132aacad Mon Sep 17 00:00:00 2001 From: Marin Jankovski Date: Fri, 11 Nov 2016 15:49:51 +0100 Subject: Add idle_timeout to reply by email doc. --- doc/administration/reply_by_email.md | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'doc') diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 5a9a1582877..b42892eef68 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -105,6 +105,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". gitlab_rails['incoming_email_mailbox_name'] = "inbox" + # The IDLE command timeout. + gitlab_rails['incoming_email_idle_timeout'] = 60 ``` ```ruby @@ -133,6 +135,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". gitlab_rails['incoming_email_mailbox_name'] = "inbox" + # The IDLE command timeout. + gitlab_rails['incoming_email_idle_timeout'] = 60 ``` 1. Reconfigure GitLab and restart mailroom for the changes to take effect: @@ -192,6 +196,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" + # The IDLE command timeout. + email_idle_timeout: 60 ``` ```yaml @@ -221,6 +227,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" + # The IDLE command timeout. + email_idle_timeout: 60 ``` 1. Enable `mail_room` in the init script at `/etc/default/gitlab`: @@ -277,6 +285,8 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" + # The IDLE command timeout. + email_idle_timeout: 60 ``` As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. -- cgit v1.2.1 From baf09d2a7a78f9260f3dff601ed7a10d18d3bec4 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Thu, 22 Sep 2016 06:05:28 -0300 Subject: Improved redis sentinel documentation for CE --- doc/administration/high_availability/redis.md | 62 +++++++++++---------------- 1 file changed, 26 insertions(+), 36 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 8656b42f274..a1426575abd 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -143,6 +143,7 @@ the master, and `masterauth` in slaves. redis['port'] = 6379 ## Slave redis instance + redis['master'] = false redis['master_ip'] = '10.10.10.10' # IP of master Redis server redis['master_port'] = 6379 # Port of master Redis server redis['master_password'] = "" @@ -157,31 +158,19 @@ servers. ### Sentinel setup -We don't provide yet an automated way to setup and run the Sentinel daemon -from Omnibus installation method. You must follow the instructions below and -run it by yourself. +We provide an automated way to setup and run the Sentinel daemon +with GitLab EE. -The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531). -While you can give any name for the `master-group-name` part of the -configuration, as in this example: - -```conf -sentinel monitor -``` - -,for it to work in Ruby, you have to use the "hostname" of the master Redis -server, otherwise you will get an error message like: -`Redis::CannotConnectError: No sentinels available.`. Read -[Sentinel troubleshooting](#sentinel-troubleshooting) for more information. +See the instructions below how to setup it by yourself. Here is an example configuration file (`sentinel.conf`) for a Sentinel node: ```conf port 26379 -sentinel monitor master-redis.example.com 10.10.10.10 6379 1 -sentinel down-after-milliseconds master-redis.example.com 10000 -sentinel config-epoch master-redis.example.com 0 -sentinel leader-epoch master-redis.example.com 0 +sentinel monitor gitlab-redis 10.0.0.1 6379 1 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 ``` --- @@ -213,10 +202,11 @@ The following steps should be performed in the [GitLab application server](gitla 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: ```ruby - gitlab-rails['redis_host'] = "master-redis.example.com" - gitlab-rails['redis_port'] = 6379 - gitlab-rails['redis_password'] = '' - gitlab-rails['redis_sentinels'] = [ + redis['master_name'] = "gitlab-redis" + redis['master_ip'] = "10.0.0.1" + redis['master_port'] = 6379 + redis['master_password'] = '' + gitlab_rails['redis_sentinels'] = [ {'host' => '10.10.10.1', 'port' => 26379}, {'host' => '10.10.10.2', 'port' => 26379}, {'host' => '10.10.10.3', 'port' => 26379} @@ -229,33 +219,33 @@ The following steps should be performed in the [GitLab application server](gitla If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related -to [this issue][gh-531] ([pull request][gh-534] that should make things better). +to [this issue][gh-531]. -It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`, -otherwise `redis-rb` will not work properly. +It's a bit non-intuitive the way you have to config `resque.yml` and +`sentinel.conf`, otherwise `redis-rb` will not work properly. -The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`) -**must** match the one configured in GitLab (`resque.yml` for source installations -or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex: +The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) +**must** be used as the hostname in GitLab (`resque.yml` for source installations +or `gitlab-rails['redis_*']` in Omnibus): ```conf # sentinel.conf: -sentinel monitor my-primary-redis 10.10.10.10 6379 1 -sentinel down-after-milliseconds my-primary-redis 10000 -sentinel config-epoch my-primary-redis 0 -sentinel leader-epoch my-primary-redis 0 +sentinel monitor gitlab-redis 10.10.10.10 6379 1 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 ``` ```yaml # resque.yaml production: - url: redis://my-primary-redis:6378 + url: redis://:myredispassword@gitlab-redis/ sentinels: - - host: slave1 + host: slave1.example.com # or use ip port: 26380 # point to sentinel, not to redis port - - host: slave2 + host: slave2.exampl.com # or use ip port: 26381 # point to sentinel, not to redis port ``` -- cgit v1.2.1 From eb73cd08086c445210532c746ce305938bc1b53c Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Fri, 7 Oct 2016 01:16:32 +0200 Subject: Small fixes on Sentinel documentation for CE --- doc/administration/high_availability/redis.md | 2 -- 1 file changed, 2 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index a1426575abd..97842f50b81 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -203,8 +203,6 @@ The following steps should be performed in the [GitLab application server](gitla ```ruby redis['master_name'] = "gitlab-redis" - redis['master_ip'] = "10.0.0.1" - redis['master_port'] = 6379 redis['master_password'] = '' gitlab_rails['redis_sentinels'] = [ {'host' => '10.10.10.1', 'port' => 26379}, -- cgit v1.2.1 From e26d8e0272d822a5086ce073a67039cf0107c9c8 Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Fri, 7 Oct 2016 01:47:22 +0200 Subject: Updated password examples and improved omnibus troubleshooting --- doc/administration/high_availability/redis.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 97842f50b81..acdfc2773a9 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -45,7 +45,7 @@ Redis. redis['bind'] = '0.0.0.0' # If you wish to use Redis authentication (recommended) - redis['password'] = 'Redis Password' + redis['password'] = 'redis-password-goes-here' ``` 1. Run `sudo touch /etc/gitlab/skip-auto-migrations` to prevent database migrations @@ -132,7 +132,7 @@ the master, and `masterauth` in slaves. redis['port'] = 6379 ## Master redis instance - redis['password'] = '' + redis['password'] = 'redis-password-goes-here' ``` 1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines): @@ -146,7 +146,7 @@ the master, and `masterauth` in slaves. redis['master'] = false redis['master_ip'] = '10.10.10.10' # IP of master Redis server redis['master_port'] = 6379 # Port of master Redis server - redis['master_password'] = "" + redis['master_password'] = "redis-password-goes-here" ``` 1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` @@ -203,7 +203,7 @@ The following steps should be performed in the [GitLab application server](gitla ```ruby redis['master_name'] = "gitlab-redis" - redis['master_password'] = '' + redis['master_password'] = 'redis-password-goes-here' gitlab_rails['redis_sentinels'] = [ {'host' => '10.10.10.1', 'port' => 26379}, {'host' => '10.10.10.2', 'port' => 26379}, @@ -215,6 +215,21 @@ The following steps should be performed in the [GitLab application server](gitla ### Sentinel troubleshooting +#### Omnibus install + +If you get an error like: `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this issue][gh-531]. + +You must make sure you are defining the same value in `redis['master_name']` +and `redis['master_pasword']` as you defined for your sentinel node. + +The way the redis connector `redis-rb` works with sentinel is a bit +non-intuitive. We try to hide the complexity in omnibus, but it still requires +a few extra configs. + +#### Source install + If you get an error like: `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related to [this issue][gh-531]. -- cgit v1.2.1 From 95f6cf339a8c0f5d63178a92eaef2d02bcfb297f Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Sat, 15 Oct 2016 05:40:15 +0200 Subject: Improved documentation on HA sentinel part and Redis replication troubleshooting. --- doc/administration/high_availability/redis.md | 316 +++++++++++++++++++++----- 1 file changed, 256 insertions(+), 60 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index acdfc2773a9..e416514bedf 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -8,6 +8,27 @@ that comes bundled with GitLab Omnibus packages. information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service. + + +**Table of Contents** + +- [Configure your own Redis server](#configure-your-own-redis-server) +- [Configure Redis using Omnibus](#configure-redis-using-omnibus) +- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) + - [Redis setup](#redis-setup) + - [Source install](#source-install) + - [Omnibus Install](#omnibus-install) + - [Troubleshooting Replication](#troubleshooting-replication) + - [Sentinel](#sentinel) + - [Sentinel setup (Community Edition)](#sentinel-setup-community-edition) + - [Sentinel setup (EE Only)](#sentinel-setup-ee-only) + - [GitLab setup](#gitlab-setup) + - [Sentinel troubleshooting](#sentinel-troubleshooting) + - [Omnibus install](#omnibus-install) + - [Source install](#source-install-1) + + + ## Configure your own Redis server If you're hosting GitLab on a cloud provider, you can optionally use a @@ -37,6 +58,7 @@ Redis. unicorn['enable'] = false sidekiq['enable'] = false postgresql['enable'] = false + gitlab_rails['enable'] = false gitlab_workhorse['enable'] = false mailroom['enable'] = false @@ -59,120 +81,294 @@ Redis. ## Experimental Redis Sentinel support -> [Introduced][ce-1877] in GitLab 8.11. +> [Introduced][ce-1877] in GitLab 8.11, improved in 8.13. Since GitLab 8.11, you can configure a list of Redis Sentinel servers that will monitor a group of Redis servers to provide you with a standard failover support. -There is currently one exception to the Sentinel support: `mail_room`, the -component that processes incoming emails. It doesn't support Sentinel yet, but -we hope to integrate a future release that does support it. - To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as failing to configure it correctly can lead to data loss. +Redis Sentinel can handle the most important tasks in a HA environment to help +keep servers online with minimal to no downtime: + +- Monitors master and slave instances to see if they are available +- Promote a slave to master when the master fails. +- Demote a master to slave when failed master comes back online (to prevent + data-partitioning). +- Can be queried by clients to always connect to the correct master server. + +There is currently one exception to the Sentinel support: `mail_room`, the +component that processes incoming emails. It doesn't support Sentinel yet, but +we hope to integrate a future release that does support it soon. + The configuration consists of three parts: -- Redis setup -- Sentinel setup -- GitLab setup +- Setup Redis Master and Slave nodes +- Setup Sentinel nodes +- Setup GitLab + +> **IMPORTANT**: You need at least 3 independent machines: physical, or VMs +running into distinct physical machines. If you fail to provision the +machines in that specific way, any issue with the shared environment can +bring your entire setup down. Read carefully how to configure those components below. ### Redis setup -You must have at least 2 Redis servers: 1 Master, 1 or more Slaves. +You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to +be each in a independent machine (see explanation above). + They should be configured the same way and with similar server specs, as -in a failover situation, any Slave can be elected as the new Master by +in a failover situation, any `Slave` can be elected as the new `Master` by the Sentinel servers. -In a minimal setup, the only required change for the slaves in `redis.conf` -is the addition of a `slaveof` line pointing to the initial master. -You can increase the security by defining a `requirepass` configuration in -the master, and `masterauth` in slaves. +With Sentinel, you must define a password to protect the access as both +Sentinel instances and other redis instances should be able to talk to +each other over the network. ---- +You'll need to define both `requirepass` and `masterauth` in all +nodes because they can be re-configured at any time by the Sentinels +during a failover, and change it's status as `Master` or `Slave`. -**Configuring your own Redis server** +Initial `Slave` nodes will have in `redis.conf` an additional `slaveof` line +pointing to the initial `Master`. -1. Add to the slaves' `redis.conf`: +#### Source install - ```conf - # IP and port of the master Redis server - slaveof 10.10.10.10 6379 - ``` +**Master Redis instance** -1. Optionally, set up password authentication for increased security. - Add the following to master's `redis.conf`: +You need to make the following changes in `redis.conf`: - ```conf - # Optional password authentication for increased security - requirepass "" - ``` +1. Define a `bind` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access: -1. Then add this line to all the slave servers' `redis.conf`: + ```conf + # By default, if no "bind" configuration directive is specified, Redis listens + # for connections from all the network interfaces available on the server. + # It is possible to listen to just one or multiple selected interfaces using + # the "bind" configuration directive, followed by one or more IP addresses. + # + # Examples: + # + # bind 192.168.1.100 10.0.0.1 + # bind 127.0.0.1 ::1 + bind 0.0.0.0 # This will bind to all interfaces + ``` + +1. Define a `port` to force redis to listin on TCP so other machines can + connect to it: + + ```conf + # Accept connections on the specified port, default is 6379 (IANA #815344). + # If port 0 is specified Redis will not listen on a TCP socket. + port 6379 + ``` + +1. Set up password authentication (use the same password in all nodes) ```conf - masterauth "" + requirepass "redis-password-goes-here" + masterauth "redis-password-goes-here" ``` 1. Restart the Redis services for the changes to take effect. ---- +**Slave Redis instance** -**Using Redis via Omnibus** +1. Follow same instructions from master with the extra change in `redis.conf`: -1. Edit `/etc/gitlab/gitlab.rb` of a master Redis machine (usualy a single machine): + ```conf + # IP and port of the master Redis server + slaveof 10.10.10.10 6379 + ``` - ```ruby - ## Redis TCP support (will disable UNIX socket transport) - redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one - redis['port'] = 6379 +1. Restart the Redis services for the changes to take effect. - ## Master redis instance - redis['password'] = 'redis-password-goes-here' - ``` +#### Omnibus Install -1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines): +You need to install the omnibus package in 3 different and independent machines. +We will elect one as the initial `Master` and the other 2 as `Slaves`. - ```ruby - ## Redis TCP support (will disable UNIX socket transport) - redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one - redis['port'] = 6379 +If you are migrating from a single machine install, you may want to setup the +machines as Slaves, pointing to the original machine as `Master`, to migrate +the data first, and than switch to this setup. - ## Slave redis instance - redis['master'] = false - redis['master_ip'] = '10.10.10.10' # IP of master Redis server - redis['master_port'] = 6379 # Port of master Redis server - redis['master_password'] = "redis-password-goes-here" - ``` +To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: + +```ruby +redis['enable'] = false +``` + +**Master Redis instances** + +You need to make the following changes in `/etc/gitlab/gitlab.rb`: + +1. Define a `redis['bind']` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access. +1. Define a `redis['port']` to force redis to listin on TCP so other machines can + connect to it. +1. Set up password authentication with `redis['master_password']` (use the same + password in all nodes). -1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure` +```ruby +## Redis TCP support (will disable UNIX socket transport) +redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis['port'] = 6379 +redis['requirepass'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' +``` + +Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` + +**Slave Redis instances** + +You need to make the same changes listed for the `Master` instance, +with an additional `Slave` section as in the example below: + +```ruby +## Redis TCP support (will disable UNIX socket transport) +redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +redis['port'] = 6379 +redis['requirepass'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' + +## Slave redis instance +redis['master'] = false +redis['master_ip'] = '10.10.10.10' # IP of master Redis server +redis['master_port'] = 6379 # Port of master Redis server +``` + +Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` + +#### Troubleshooting Replication + +You can check if everything is correct by connecting to each server using +`redis-cli` application, and sending the `INFO` command. + +If authentication was correctly defined, it should fail with: +`NOAUTH Authentication required` error. Try to authenticate with the +previous defined password with `AUTH redis-password-goes-here` and +try the `INFO` command again. + +Look for the `# Replication` section where you should see some important +information like the `role` of the server. + +When connected to a `master` redis, you will see the number of connected +`slaves`, and a list of each with connection details. + +When it's a `slave`, you will see details of the master connection and if +its `up` or `down`. --- Now that the Redis servers are all set up, let's configure the Sentinel servers. -### Sentinel setup +If you are not sure if your Redis servers are working and replicating +correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) +and fix it before proceeding with Sentinel setup. + +### Sentinel + +You must have at least `3` Redis Sentinel servers, and they need to +be each in a independent machine. You can install them in the same +machines you installed the other `3` Redis servers. + +This number is required for the consensus algorithm to be effective +in the case of a failure. You should always have and `odd` number +of Sentinel nodes provisioned. -We provide an automated way to setup and run the Sentinel daemon -with GitLab EE. +Here is a simple explanation on how Sentinel handles a failover: -See the instructions below how to setup it by yourself. +When a number of Sentinels (`quorum` value) agree the fact the `master` is +not reachable, the **majority** of the sentinels must elect a temporary +Sentinel `leader`, that will be responsible to start the failover proceedings. -Here is an example configuration file (`sentinel.conf`) for a Sentinel node: +As an example, for a cluster of `3` Sentinels, at least `2` must agree on a +`leader`. If you have total of `5` at least `3` must agree on the leader. + +The `quorum` is only used to detect failure, not to elect the `leader`. + +Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments) +also lists different network topologies and warns againts situations like +network partition and how it can affect the state of the HA solution. Make +sure you read it carefully and understand the implications in your current +setup. + +To make Sentinel setup easier, ee provide an [automated way to setup and run](#sentinel-setup-ee-only) +the Sentinel daemon with GitLab EE. + +#### Sentinel setup (Community Edition) + +For GitLab CE, you need to install, configure, execute and monitor Sentinel +by yourself. + +Here is an example configuration file (`sentinel.conf`) for a minimal Sentinel +node: ```conf -port 26379 -sentinel monitor gitlab-redis 10.0.0.1 6379 1 +bind 0.0.0.0 # bind to all interfaces or change to a specific IP +port 26379 # default sentinel port +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 sentinel down-after-milliseconds gitlab-redis 10000 sentinel config-epoch gitlab-redis 0 sentinel leader-epoch gitlab-redis 0 ``` +#### Sentinel setup (EE Only) + +To setup sentinel, you must edit `/etc/gitlab/gitlab.rb` file. +This is a minimal configuration required to run the daemon: + +```ruby +redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node +redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance +redis['master_port'] = 6379 # port of the initial master redis instance +redis['master_password'] = 'your-secure-password-here' # the same value defined in redis['password'] in the master instance + +sentinel['enable'] = true +# sentinel['port'] = 26379 + +## Quorum must reflect the amount of voting sentinels it take to start a failover. +sentinel['quorum'] = 2 + +## Consider unresponsive server down after x amount of ms. +# sentinel['down_after_milliseconds'] = 10000 + +# sentinel['failover_timeout'] = 60000 +``` + +When you install Sentinel in a separate machine, you need to control which +other services will be running in it. Take a look at the following variables +and enable or disable whenever it fits your strategy: + +```ruby +# Enabled Redis and Sentinel services +redis['enable'] = true +sentinel['enable'] = true + +# Disabled all other services +redis['enable'] = false +bootstrap['enable'] = false +nginx['enable'] = false +unicorn['enable'] = false +sidekiq['enable'] = false +postgresql['enable'] = false +gitlab_workhorse['enable'] = false +gitlab_rails['enable'] = false +mailroom['enable'] = false +``` + +Remember that enabling a new service may also require additional configuration +params (like `redis` for example). + --- The final part is to inform the main GitLab application server of the Redis @@ -243,7 +439,7 @@ or `gitlab-rails['redis_*']` in Omnibus): ```conf # sentinel.conf: -sentinel monitor gitlab-redis 10.10.10.10 6379 1 +sentinel monitor gitlab-redis 10.10.10.10 6379 2 sentinel down-after-milliseconds gitlab-redis 10000 sentinel config-epoch gitlab-redis 0 sentinel leader-epoch gitlab-redis 0 @@ -276,7 +472,7 @@ To make sure your configuration is correct: sudo gitlab-rails console # For source installations - sudo -u git rails console RAILS_ENV=production + sudo -u git rails console production ``` 1. Run in the console: -- cgit v1.2.1 From 0f951737af830163260a9535990a18fba130b443 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 13:56:31 +0100 Subject: Remove experimental heading and move note at the top [ci skip] --- doc/administration/high_availability/redis.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index a87c1dde55d..f9bc4f59345 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,5 +1,11 @@ # Configuring Redis for GitLab HA +> +Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. +Starting with 8.14, Redis Sentinel is no longer experimental. +If you've used it with versions `< 8.14` before, please check the updated +documentation here. + High Availability with [Redis] is possible using a **Master** x **Slave** topology with a [Redis Sentinel][sentinel] service to watch and automatically start the failover procedure. @@ -52,7 +58,6 @@ Omnibus GitLab packages. - [Troubleshooting Redis replication](#troubleshooting-redis-replication) - [Troubleshooting Sentinel](#troubleshooting-sentinel) - [Changelog](#changelog) - - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) - [Further reading](#further-reading) @@ -871,14 +876,6 @@ Changes to Redis HA over time. - Experimental Redis Sentinel support was added -### Experimental Redis Sentinel support - -> -Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. -Starting with 8.14, Redis Sentinel is no longer experimental. -If you used with versions `< 8.14` before, please check the updated -documentation here. - ## Further reading Read more on High Availability: -- cgit v1.2.1 From d5dc44f0588690fb79ddc2aa3ee150012d5e4873 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 13:59:52 +0100 Subject: Minor edits in Redis HA source install [ci skip] --- doc/administration/high_availability/redis_source.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index e7a8f47355d..8558ba82d63 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -345,12 +345,12 @@ We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained in the documentation for Omnibus GitLab installations. Here we will list only the things that are specific to a source installation. -If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`, +If you get an error in GitLab like `Redis::CannotConnectError: No sentinels available.`, there may be something wrong with your configuration files or it can be related to [this upstream issue][gh-531]. -It's a bit non-intuitive the way you have to config `resque.yml` and -`sentinel.conf`, otherwise `redis-rb` will not work properly. +You must make sure that `resque.yml` and `sentinel.conf` are configured correctly, +otherwise `redis-rb` will not work properly. The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) **must** be used as the hostname in GitLab (`resque.yml`): @@ -379,7 +379,7 @@ production: port: 26379 # point to sentinel, not to redis port ``` -When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) +When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel). [gh-531]: https://github.com/redis/redis-rb/issues/531 [downloads]: https://about.gitlab.com/downloads -- cgit v1.2.1 From 465c61ee6889e03016712cbda7fd48e0e973cb76 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 14:17:58 +0100 Subject: Remove login as root step from Redis HA docs [ci skip] --- doc/administration/high_availability/redis.md | 31 ++++++++------------------- 1 file changed, 9 insertions(+), 22 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index f9bc4f59345..1fa3560e932 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -107,10 +107,12 @@ to help keep servers online with minimal to no downtime. Redis Sentinel: - Promotes a **Slave** to **Master** when the **Master** fails - Demotes a **Master** to **Slave** when the failed **Master** comes back online (to prevent data-partitioning) -- Can be queried by clients to always connect to the current **Master** server +- Can be queried by the application to always connect to the current **Master** + server -When a **Master** fails to respond, it's the client's responsibility to handle -timeout and reconnect (querying a **Sentinel** for a new **Master**). +When a **Master** fails to respond, it's the application's responsibility +(in our case GitLab) to handle timeout and reconnect (querying a **Sentinel** +for a new **Master**). To get a better understanding on how to correctly setup Sentinel, please read the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as @@ -289,12 +291,7 @@ The prerequisites for a HA Redis setup are the following: ### Step 1. Configuring the master Redis instance -1. SSH into the **master** Redis server and login as root: - - ``` - sudo -i - ``` - +1. SSH into the **master** Redis server. 1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - Make sure you select the correct Omnibus package, with the same version @@ -334,12 +331,7 @@ The prerequisites for a HA Redis setup are the following: ### Step 2. Configuring the slave Redis instances -1. SSH into the **slave** Redis server and login as root: - - ``` - sudo -i - ``` - +1. SSH into the **slave** Redis server. 1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - Make sure you select the correct Omnibus package, with the same version @@ -417,12 +409,7 @@ multiple machines with the Sentinel daemon. --- -1. SSH into the server that will host Redis Sentinel and login as root: - - ``` - sudo -i - ``` - +1. SSH into the server that will host Redis Sentinel. 1. **You can omit this step if the Sentinels will be hosted in the same node as the other Redis instances.** @@ -437,7 +424,6 @@ multiple machines with the Sentinel daemon. Sentinels in the same node as the other Redis instances, some values might be duplicate below): - ```ruby redis_sentinel_role['enable'] = true @@ -530,6 +516,7 @@ it needs to access at least one of the listed. The following steps should be performed in the [GitLab application server](gitlab.md) which ideally should not have Redis or Sentinels on it for a HA setup. +1. SSH into the server where the GitLab application is installed. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: ``` -- cgit v1.2.1 From 6715091bbc035142237f5121623bb0b84c3def51 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 19 Oct 2016 11:54:38 +0200 Subject: WIP refactor environments --- doc/ci/environments.md | 159 +++++++++++++++++++++++++++++++-------- doc/ci/img/deployments_view.png | Bin 0 -> 179909 bytes doc/ci/img/environments_view.png | Bin 0 -> 57534 bytes doc/ci/yaml/README.md | 62 ++++++++++++++- doc/user/permissions.md | 4 +- 5 files changed, 190 insertions(+), 35 deletions(-) create mode 100644 doc/ci/img/deployments_view.png create mode 100644 doc/ci/img/environments_view.png (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index e070302fb82..9d65621fbbb 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -3,69 +3,166 @@ >**Note:** Introduced in GitLab 8.9. -## Environments +During the development of a software there can be many stages until it's ready +for public consumption. You sure want to first see your code in a testing or +staging environment before you release it to the public. That way you can +prevent bugs not only in your software, but in the deployment process as well. + +With environments you can control the Continuous Deployment of your software all +within GitLab. All you need to do is define them in your project's +[`.gitlab-ci.yml`][yaml]. + +In the following sections, we'll see how that works. + +## An environments example + +Let's assume that you have + +1. Define the environments in `.gitlab-ci.yml` +1. Push the repository to GitLab +1. Runner picks up the job +1. The job finishes successfully +1. The environments get created if they don't already exist +1. A deployment is recorded remembering the environment name and the Git SHA of + the last commit of the pipeline + +Further runs of the CI will + +Actions + +View environments +View deployments + Rollback deployments + Run deployments +View link to environment URL +View last commit message of deployment +View person who performed the deployment +View commit SHA that triggered the deployment +View branch the deployment was based on +View time ago the deployment was performed + +Environments are like tags for your CI jobs, describing where code gets deployed. + +You can think of names such as testing, staging or production. -Environments are places where code gets deployed, such as staging or production. CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. + Defining environments in a project's `.gitlab-ci.yml` lets developers track [deployments] to these environments. -## Deployments +The environments page can only be viewed by Reporters and above. For more +information on the permissions, see the [permissions documentation][permissions]. -Deployments are created when [jobs] deploy versions of code to [environments]. +### Defining environments -### Checkout deployments locally +While you can create and delete environments manually in the web interface, we +recommend that you define your environments in `.gitlab-ci.yml` first. They will +be automatically created for you after the first deploy. -Since 8.13, a reference in the git repository is saved for each deployment. So -knowing what the state is of your current environments is only a `git fetch` -away. +The `environment` keyword is just a hint for GitLab that this job actually +deploys to this environment. Each time the job succeeds, a deployment is +recorded, remembering the Git SHA and environment name. -In your git config, append the `[remote ""]` block with an extra -fetch line: +Add something like this to your `.gitlab-ci.yml`: ``` -fetch = +refs/environments/*:refs/remotes/origin/environments/* +production: + stage: deploy + script: make deploy-to-prod + environment: + name: production ``` -## Defining environments +See the [yaml definition](yaml/README.md#environment) of environments. -You can create and delete environments manually in the web interface, but we -recommend that you define your environments in `.gitlab-ci.yml` first, which -will automatically create environments for you after the first deploy. +### View the environment status -The `environment` is just a hint for GitLab that this job actually deploys to -this environment. Each time the job succeeds, a deployment is recorded, -remembering the git SHA and environment. +GitLab keeps track of your deployments, so you always know what is currently +being deployed on your servers. You can find the environment list under +**Pipelines > Environments** for your project. You'll see the git SHA and date +of the last deployment to each environment defined. + +![Environments](img/environments_view.png) + +>**Note:** +Only deploys that happen after your `.gitlab-ci.yml` is properly configured will +show up in the "Environment" and "Last deployment" lists. + +## Manually deploying to environments + + +## Dynamic environments + +As the name suggests, it is possible to create environments on the fly by just +declaring their names dynamically in `.gitlab-ci.yml`. + +GitLab Runner exposes various [environment variables][variables] when a job runs, +and as such you can use them -Add something like this to your `.gitlab-ci.yml`: ``` -production: +review: stage: deploy - script: dpl... - environment: production + script: + - rsync -av --delete public /srv/nginx/pages/$CI_BUILD_REF_NAME + environment: + name: review/$CI_BUILD_REF_NAME + url: https://$CI_BUILD_REF_NAME.example.com ``` -See full [documentation](yaml/README.md#environment). +### Closing an environment -## Seeing environment status +``` +review: + stage: deploy + script: + - rsync -av --delete public /srv/nginx/pages/$CI_BUILD_REF_NAME + environment: + name: review/$CI_BUILD_REF_NAME + url: http://$CI_BUILD_REF_NAME.$APPS_DOMAIN + on_stop: stop_review + +stop_review: + script: rm -rf /srv/nginx/pages/$CI_BUILD_REF_NAME + when: manual + environment: + name: review/$CI_BUILD_REF_NAME + action: stop +``` -You can find the environment list under **Pipelines > Environments** for your -project. You'll see the git SHA and date of the last deployment to each -environment defined. +## The relationship between deployments and environments ->**Note:** -Only deploys that happen after your `.gitlab-ci.yml` is properly configured will -show up in the environments and deployments lists. +Deployments are created when [jobs] deploy versions of code to [environments], +so every environment can have one or more deployments. GitLab keeps track of +your deployments, so you always know what is currently being deployed on your +servers. -## Seeing deployment history +### View the deployment history Clicking on an environment will show the history of deployments. +![Deployments](img/deployments_view.png) + >**Note:** Only deploys that happen after your `.gitlab-ci.yml` is properly configured will show up in the environments and deployments lists. +### Checkout deployments locally + +Since 8.13, a reference in the git repository is saved for each deployment. So +knowing what the state is of your current environments is only a `git fetch` +away. + +In your git config, append the `[remote ""]` block with an extra +fetch line: + +``` +fetch = +refs/environments/*:refs/remotes/origin/environments/* +``` + [Pipelines]: pipelines.md [jobs]: yaml/README.md#jobs +[yaml]: yaml/README.md [environments]: #environments [deployments]: #deployments +[permissions]: ../user/permissions.md +[variables]: variables/README.md diff --git a/doc/ci/img/deployments_view.png b/doc/ci/img/deployments_view.png new file mode 100644 index 00000000000..250e051bf42 Binary files /dev/null and b/doc/ci/img/deployments_view.png differ diff --git a/doc/ci/img/environments_view.png b/doc/ci/img/environments_view.png new file mode 100644 index 00000000000..131a9718cc4 Binary files /dev/null and b/doc/ci/img/environments_view.png differ diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 5c0e1c44e3f..5c7d300e3d9 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -573,7 +573,8 @@ In its simplest form, the `environment` keyword can be defined like: deploy to production: stage: deploy script: git push production HEAD:master - environment: production + environment: + name: production ``` In the above example, the `deploy to production` job will be marked as doing a @@ -673,6 +674,61 @@ The `stop_review_app` job is **required** to have the following keywords defined - `environment:name` - `environment:action` +#### environment:name + +#### environment:url + +Optional. + +#### environment:on_stop + +> [Introduced][ce-6669] in GitLab 8.13. + +Closing environments can be achieved with the `on_stop` keyword defined under +`environment`. It declares a different job that has to be run in order to close +the environment. + +This job is required to have the following keywords defined: + +- `when` - [reference](#when) +- `environment:name` +- `environment:action` - reference below + +See below for an example. + +#### environment:action + +> [Introduced][ce-6669] in GitLab 8.13. + +The `action` keyword is to be used in conjunction with `on_stop` and is defined +in the job that depends on the one that was called from. + +Take for instance: + +```yaml +review: + stage: deploy + script: make deploy-app + environment: + name: review + on_stop: stop_review + +stop_review: + stage: deploy + script: make delete-app + when: manual + environment: + name: review + action: stop +``` + +In the above example we set up the `review` job to deploy to the `review` +environment, and we also defined a new `stop_review` job under `on_stop`. +Once the `review` job is successfully finished, it will trigger the `stop_review` +job based on what is defined under `when`. In this case we set it up to `manual` +so it will need a [manual action](#manual-actions) via GitLab's web interface +in order to run. + #### dynamic environments > [Introduced][ce-6323] in GitLab 8.12 and GitLab Runner 1.6. @@ -681,7 +737,9 @@ The `stop_review_app` job is **required** to have the following keywords defined These parameters can use any of the defined [CI variables](#variables) (including predefined, secure variables and `.gitlab-ci.yml` variables). -For example: +--- + +**Example configurations** ``` deploy as review app: diff --git a/doc/user/permissions.md b/doc/user/permissions.md index d6216a8dd50..a33183fa01c 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -32,6 +32,8 @@ The following table depicts the various user permission levels in a project. | See a commit status | | ✓ | ✓ | ✓ | ✓ | | See a container registry | | ✓ | ✓ | ✓ | ✓ | | See environments | | ✓ | ✓ | ✓ | ✓ | +| Create new environments | | | ✓ | ✓ | ✓ | +| Delete environments | | | | ✓ | ✓ | | See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | Manage/Accept merge requests | | | ✓ | ✓ | ✓ | | Create new merge request | | | ✓ | ✓ | ✓ | @@ -45,7 +47,6 @@ The following table depicts the various user permission levels in a project. | Create or update commit status | | | ✓ | ✓ | ✓ | | Update a container registry | | | ✓ | ✓ | ✓ | | Remove a container registry image | | | ✓ | ✓ | ✓ | -| Create new environments | | | ✓ | ✓ | ✓ | | Create new milestones | | | | ✓ | ✓ | | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | @@ -58,7 +59,6 @@ The following table depicts the various user permission levels in a project. | Manage runners | | | | ✓ | ✓ | | Manage build triggers | | | | ✓ | ✓ | | Manage variables | | | | ✓ | ✓ | -| Delete environments | | | | ✓ | ✓ | | Switch visibility level | | | | | ✓ | | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | -- cgit v1.2.1 From 8666fd7613e0e4c3eaaf3e730f0cb0ac75aff245 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 4 Nov 2016 11:20:44 +0100 Subject: Begin writing Overview section --- doc/ci/environments.md | 57 +++++++++++++++++++++++----------------------- doc/ci/yaml/README.md | 62 ++------------------------------------------------ 2 files changed, 30 insertions(+), 89 deletions(-) (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 9d65621fbbb..60541c44016 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -3,20 +3,38 @@ >**Note:** Introduced in GitLab 8.9. -During the development of a software there can be many stages until it's ready -for public consumption. You sure want to first see your code in a testing or -staging environment before you release it to the public. That way you can -prevent bugs not only in your software, but in the deployment process as well. - -With environments you can control the Continuous Deployment of your software all +During the development of a software, there can be many stages until it's ready +for public consumption. You sure want to first test your code and then deploy it +in a testing or staging environment before you release it to the public. That +way you can prevent bugs not only in your software, but in the deployment +process as well. + +In case you use GitLab CI to not only test or build your project, but also +deploy it in your infrastructure, GitLab provides a way to track your deployments +so you always know what is currently being deployed on your servers. With +environments you can control the Continuous Deployment of your software all within GitLab. All you need to do is define them in your project's -[`.gitlab-ci.yml`][yaml]. +[`.gitlab-ci.yml`][yaml]. GitLab provides a full history of your deployments per +every environment. + +## Overview + +Deployments are created when [jobs] deploy versions of code to environments, +so every environment can have one or more deployments. GitLab keeps track of +your deployments, so you always know what is currently being deployed on your +servers. + +Environments are like tags for your CI jobs, describing where code gets deployed. +CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. +You can think of names such as testing, staging or production. -In the following sections, we'll see how that works. +Defining environments in a project's `.gitlab-ci.yml` lets developers track +[deployments] to these environments. -## An environments example +The environments page can only be viewed by Reporters and above. For more +information on the permissions, see the [permissions documentation][permissions]. -Let's assume that you have +Let's assume that you have: 1. Define the environments in `.gitlab-ci.yml` 1. Push the repository to GitLab @@ -41,18 +59,6 @@ View commit SHA that triggered the deployment View branch the deployment was based on View time ago the deployment was performed -Environments are like tags for your CI jobs, describing where code gets deployed. - -You can think of names such as testing, staging or production. - -CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. - -Defining environments in a project's `.gitlab-ci.yml` lets developers track -[deployments] to these environments. - -The environments page can only be viewed by Reporters and above. For more -information on the permissions, see the [permissions documentation][permissions]. - ### Defining environments While you can create and delete environments manually in the web interface, we @@ -129,13 +135,6 @@ stop_review: action: stop ``` -## The relationship between deployments and environments - -Deployments are created when [jobs] deploy versions of code to [environments], -so every environment can have one or more deployments. GitLab keeps track of -your deployments, so you always know what is currently being deployed on your -servers. - ### View the deployment history Clicking on an environment will show the history of deployments. diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 5c7d300e3d9..5c0e1c44e3f 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -573,8 +573,7 @@ In its simplest form, the `environment` keyword can be defined like: deploy to production: stage: deploy script: git push production HEAD:master - environment: - name: production + environment: production ``` In the above example, the `deploy to production` job will be marked as doing a @@ -674,61 +673,6 @@ The `stop_review_app` job is **required** to have the following keywords defined - `environment:name` - `environment:action` -#### environment:name - -#### environment:url - -Optional. - -#### environment:on_stop - -> [Introduced][ce-6669] in GitLab 8.13. - -Closing environments can be achieved with the `on_stop` keyword defined under -`environment`. It declares a different job that has to be run in order to close -the environment. - -This job is required to have the following keywords defined: - -- `when` - [reference](#when) -- `environment:name` -- `environment:action` - reference below - -See below for an example. - -#### environment:action - -> [Introduced][ce-6669] in GitLab 8.13. - -The `action` keyword is to be used in conjunction with `on_stop` and is defined -in the job that depends on the one that was called from. - -Take for instance: - -```yaml -review: - stage: deploy - script: make deploy-app - environment: - name: review - on_stop: stop_review - -stop_review: - stage: deploy - script: make delete-app - when: manual - environment: - name: review - action: stop -``` - -In the above example we set up the `review` job to deploy to the `review` -environment, and we also defined a new `stop_review` job under `on_stop`. -Once the `review` job is successfully finished, it will trigger the `stop_review` -job based on what is defined under `when`. In this case we set it up to `manual` -so it will need a [manual action](#manual-actions) via GitLab's web interface -in order to run. - #### dynamic environments > [Introduced][ce-6323] in GitLab 8.12 and GitLab Runner 1.6. @@ -737,9 +681,7 @@ in order to run. These parameters can use any of the defined [CI variables](#variables) (including predefined, secure variables and `.gitlab-ci.yml` variables). ---- - -**Example configurations** +For example: ``` deploy as review app: -- cgit v1.2.1 From dd3c2714689421ad782bbbb3f9c09e81b8b20e71 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Mon, 7 Nov 2016 20:57:18 +0100 Subject: Move paragraph to overview --- doc/ci/environments.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 60541c44016..0c1557eeaef 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -11,20 +11,21 @@ process as well. In case you use GitLab CI to not only test or build your project, but also deploy it in your infrastructure, GitLab provides a way to track your deployments -so you always know what is currently being deployed on your servers. With -environments you can control the Continuous Deployment of your software all -within GitLab. All you need to do is define them in your project's -[`.gitlab-ci.yml`][yaml]. GitLab provides a full history of your deployments per -every environment. +so you always know what is currently being deployed on your servers. ## Overview +With environments, you can control the Continuous Deployment of your software all +within GitLab. All you need to do is define them in your project's +[`.gitlab-ci.yml`][yaml] as we will explore below. GitLab provides a full +history of your deployments per every environment. + +Environments are like tags for your CI jobs, describing where code gets deployed. Deployments are created when [jobs] deploy versions of code to environments, so every environment can have one or more deployments. GitLab keeps track of your deployments, so you always know what is currently being deployed on your servers. -Environments are like tags for your CI jobs, describing where code gets deployed. CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. You can think of names such as testing, staging or production. -- cgit v1.2.1 From c61a9b0f1a12972dd3c2ae4e968c11c72ced6756 Mon Sep 17 00:00:00 2001 From: Marin Jankovski Date: Wed, 16 Nov 2016 16:24:46 +0100 Subject: Correct the idle_timeout in the docs for installation from source. --- doc/administration/reply_by_email.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index b42892eef68..14cd7a03826 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -197,7 +197,7 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" # The IDLE command timeout. - email_idle_timeout: 60 + idle_timeout: 60 ``` ```yaml @@ -228,7 +228,7 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" # The IDLE command timeout. - email_idle_timeout: 60 + idle_timeout: 60 ``` 1. Enable `mail_room` in the init script at `/etc/default/gitlab`: @@ -286,7 +286,7 @@ To set up a basic Postfix mail server with IMAP access on Ubuntu, follow # The mailbox where incoming mail will end up. Usually "inbox". mailbox: "inbox" # The IDLE command timeout. - email_idle_timeout: 60 + idle_timeout: 60 ``` As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. -- cgit v1.2.1 From 21f1bee2c6ebb4ab307fab19e6560245a9fdb0d7 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 17:13:08 +0100 Subject: Add section on defining environments [ci skip] --- doc/ci/environments.md | 166 ++++++++++++++++++-------- doc/ci/img/environments_available_staging.png | Bin 0 -> 27398 bytes 2 files changed, 114 insertions(+), 52 deletions(-) create mode 100644 doc/ci/img/environments_available_staging.png (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 0c1557eeaef..b3d4ec40830 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -3,20 +3,21 @@ >**Note:** Introduced in GitLab 8.9. -During the development of a software, there can be many stages until it's ready +During the development of software, there can be many stages until it's ready for public consumption. You sure want to first test your code and then deploy it in a testing or staging environment before you release it to the public. That way you can prevent bugs not only in your software, but in the deployment process as well. -In case you use GitLab CI to not only test or build your project, but also -deploy it in your infrastructure, GitLab provides a way to track your deployments -so you always know what is currently being deployed on your servers. +GitLab CI is capable of not only testing or building your projects, but also +deploying them in your infrastructure, with the added benefit of giving you a +way to track your deployments. In other words, you can always know what is +currently being deployed or has been deployed on your servers. ## Overview -With environments, you can control the Continuous Deployment of your software all -within GitLab. All you need to do is define them in your project's +With environments, you can control the Continuous Deployment of your software +all within GitLab. All you need to do is define them in your project's [`.gitlab-ci.yml`][yaml] as we will explore below. GitLab provides a full history of your deployments per every environment. @@ -26,63 +27,97 @@ so every environment can have one or more deployments. GitLab keeps track of your deployments, so you always know what is currently being deployed on your servers. -CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. -You can think of names such as testing, staging or production. +To better understand how environments and deployments work, let's consider an +example. We assume that you have already created a project in GitLab and set up +a Runner. The example will cover the following: -Defining environments in a project's `.gitlab-ci.yml` lets developers track -[deployments] to these environments. +- We are developing an application +- We want to run tests and build our app on all branches +- Our default branch is `master` +- We deploy the app only when a pipeline on `master` branch is run -The environments page can only be viewed by Reporters and above. For more -information on the permissions, see the [permissions documentation][permissions]. +Let's see how it all ties together. -Let's assume that you have: +## Defining environments -1. Define the environments in `.gitlab-ci.yml` -1. Push the repository to GitLab -1. Runner picks up the job -1. The job finishes successfully -1. The environments get created if they don't already exist -1. A deployment is recorded remembering the environment name and the Git SHA of - the last commit of the pipeline +Let's consider the following `.gitlab-ci.yml` example: -Further runs of the CI will +```yaml +stages: + - test + - build + - deploy -Actions +test: + stage: test + script: echo "Running tests" -View environments -View deployments - Rollback deployments - Run deployments -View link to environment URL -View last commit message of deployment -View person who performed the deployment -View commit SHA that triggered the deployment -View branch the deployment was based on -View time ago the deployment was performed +build: + stage: build + script: echo "Building the app" -### Defining environments +deploy_staging: + stage: deploy + script: + - echo "Deploy to staging server" + environment: + name: staging + url: https://staging.example.com + only: + - master +``` -While you can create and delete environments manually in the web interface, we -recommend that you define your environments in `.gitlab-ci.yml` first. They will -be automatically created for you after the first deploy. +We have defined 3 [stages](yaml/README.md#stages): -The `environment` keyword is just a hint for GitLab that this job actually -deploys to this environment. Each time the job succeeds, a deployment is -recorded, remembering the Git SHA and environment name. +- test +- build +- deploy -Add something like this to your `.gitlab-ci.yml`: +The jobs assigned to these stages will run in this order. If a job fails, then +the builds that are assigned to the next stage won't run, rendering the pipeline +as failed. In our case, the `test` job will run first, then the `build` and +lastly the `deploy_staging`. With this, we ensure that first the tests pass, +then our app is able to be built successfully, and lastly we deploy to the +staging server. -``` -production: - stage: deploy - script: make deploy-to-prod - environment: - name: production -``` +With the above `.gitlab-ci.yml` we have achieved that: -See the [yaml definition](yaml/README.md#environment) of environments. +- All branches will run the `test` and `build` jobs. +- The `deploy_staging` job will [only](yaml/README.md#only) run on the `master` + branch which means all merge requests +- When a merge request is merged, all jobs will run and the `deploy_staging` + in particular will deploy our code to a staging server while the deployment + will be recorded in an environment named `staging`. -### View the environment status +The `environment` keyword is just a hint for GitLab that this job actually +deploys to this environment. Each time the job succeeds, a deployment is +recorded, remembering the Git SHA and environment name. Here's how the +Environments page looks so far. + +![Staging environment view](img/environments_available_staging.png) + +TODO: describe what the above page means + +>**Notes:** +- While you can create environments manually in the web interface, we recommend + that you define your environments in `.gitlab-ci.yml` first. They will + be automatically created for you after the first deploy. +- The environments page can only be viewed by Reporters and above. For more + information on the permissions, see the [permissions documentation][permissions]. + +As we've pointed in the Overview section, environments are like tags for your +CI jobs, describing where code gets deployed. Here's what happened behind the +scenes: + +1. The jobs and environments were defined in `.gitlab-ci.yml` +1. Changes were pushed to the repository in GitLab +1. The Runner(s) picked up the jobs +1. The jobs finished successfully +1. The environments got created if they didn't already exist +1. A deployment was recorded remembering the environment name and the Git SHA of + the last commit of the pipeline + +## View the environment status GitLab keeps track of your deployments, so you always know what is currently being deployed on your servers. You can find the environment list under @@ -97,6 +132,9 @@ show up in the "Environment" and "Last deployment" lists. ## Manually deploying to environments +CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. +You can think of names such as testing, staging or production. + ## Dynamic environments @@ -116,7 +154,7 @@ review: url: https://$CI_BUILD_REF_NAME.example.com ``` -### Closing an environment +## Closing an environment ``` review: @@ -136,7 +174,7 @@ stop_review: action: stop ``` -### View the deployment history +## View the deployment history Clicking on an environment will show the history of deployments. @@ -146,7 +184,7 @@ Clicking on an environment will show the history of deployments. Only deploys that happen after your `.gitlab-ci.yml` is properly configured will show up in the environments and deployments lists. -### Checkout deployments locally +## Checkout deployments locally Since 8.13, a reference in the git repository is saved for each deployment. So knowing what the state is of your current environments is only a `git fetch` @@ -159,6 +197,30 @@ fetch line: fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` +## Further reading + +Below are some links you may find interesting: + +- [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) +- [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) +- [Review Apps](review_apps.md) Expand dynamic environments to deploy your code for every branch + + +## WIP + +Actions + +View environments +View deployments + Rollback deployments + Run deployments +View link to environment URL +View last commit message of deployment +View person who performed the deployment +View commit SHA that triggered the deployment +View branch the deployment was based on +View time ago the deployment was performed + [Pipelines]: pipelines.md [jobs]: yaml/README.md#jobs [yaml]: yaml/README.md diff --git a/doc/ci/img/environments_available_staging.png b/doc/ci/img/environments_available_staging.png new file mode 100644 index 00000000000..784c4fd944c Binary files /dev/null and b/doc/ci/img/environments_available_staging.png differ -- cgit v1.2.1 From 5cac52d8e05492d78869974c4ee9a928da7c6983 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 16 Nov 2016 17:20:07 +0100 Subject: Change deployments view image [ci skip] --- doc/ci/img/deployments_view.png | Bin 179909 -> 57598 bytes 1 file changed, 0 insertions(+), 0 deletions(-) (limited to 'doc') diff --git a/doc/ci/img/deployments_view.png b/doc/ci/img/deployments_view.png index 250e051bf42..ca6097cbea4 100644 Binary files a/doc/ci/img/deployments_view.png and b/doc/ci/img/deployments_view.png differ -- cgit v1.2.1 From 600282799606ed260a1149d8312494a2f6a83290 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Coutable?= Date: Thu, 10 Nov 2016 15:24:23 +0100 Subject: Add a gotcha about FactoryGirl sequences MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/24341 Signed-off-by: Rémy Coutable --- doc/development/gotchas.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++ doc/development/testing.md | 1 + 2 files changed, 90 insertions(+) (limited to 'doc') diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index b25ce79e89f..7bfc9cb361f 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -32,6 +32,95 @@ spec/models/user_spec.rb|6 error| Failure/Error: u = described_class.new NoMeth Except for the top-level `describe` block, always provide a String argument to `describe`. +## Don't assert against the absolute value of a sequence-generated attribute + +Consider the following factory: + +```ruby +FactoryGirl.define do + factory :label do + sequence(:title) { |n| "label#{n}" } + end +end +``` + +Consider the following API spec: + +```ruby +require 'rails_helper' + +describe API::Labels do + it 'creates a first label' do + create(:label) + + get api("/projects/#{project.id}/labels", user) + + expect(response).to have_http_status(200) + expect(json_response.first['name']).to eq('label1') + end + + it 'creates a second label' do + create(:label) + + get api("/projects/#{project.id}/labels", user) + + expect(response).to have_http_status(200) + expect(json_response.first['name']).to eq('label1') + end +end +``` + +When run, this spec doesn't do what we might expect: + +```sh +1) API::API reproduce sequence issue creates a second label + Failure/Error: expect(json_response.first['name']).to eq('label1') + + expected: "label1" + got: "label2" + + (compared using ==) +``` + +That's because FactoryGirl sequences are not reseted for each example. + +Please remember that sequence-generated values exist only to avoid having to +explicitly set attributes that have a uniqueness constraint when using a factory. + +### Solution + +If you assert against a sequence-generated attribute's value, you should set it +explicitly. Also, the value you set shouldn't match the sequence pattern. + +For instance, using our `:label` factory, writing `create(:label, title: 'foo')` +is ok, but `create(:label, title: 'label1')` is not. + +Following is the fixed API spec: + +```ruby +require 'rails_helper' + +describe API::Labels do + it 'creates a first label' do + create(:label, title: 'foo') + + get api("/projects/#{project.id}/labels", user) + + expect(response).to have_http_status(200) + expect(json_response.first['name']).to eq('foo') + end + + it 'creates a second label' do + create(:label, title: 'bar') + + get api("/projects/#{project.id}/labels", user) + + expect(response).to have_http_status(200) + expect(json_response.first['name']).to eq('bar') + end +end +``` + ## Don't `rescue Exception` See ["Why is it bad style to `rescue Exception => e` in Ruby?"][Exception]. diff --git a/doc/development/testing.md b/doc/development/testing.md index b0b26ccf57a..4dc535fb359 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -64,6 +64,7 @@ the command line via `bundle exec teaspoon`, or via a web browser at methods. - Use `context` to test branching logic. - Don't `describe` symbols (see [Gotchas](gotchas.md#dont-describe-symbols)). +- Don't assert against the absolute value of a sequence-generated attribute (see [Gotchas](gotchas.md#dont-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Don't supply the `:each` argument to hooks since it's the default. - Prefer `not_to` to `to_not` (_this is enforced by Rubocop_). - Try to match the ordering of tests to the ordering within the class. -- cgit v1.2.1 From 3a6e38db5316a73809c3303de2aa8a0ece866fcc Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 00:04:28 +0100 Subject: Finish most of environments [ci skip] --- doc/ci/environments.md | 184 +++++++++++++++------ doc/ci/img/environments_manual_action_builds.png | Bin 0 -> 27170 bytes .../img/environments_manual_action_deployments.png | Bin 0 -> 34504 bytes .../environments_manual_action_environments.png | Bin 0 -> 40297 bytes .../img/environments_manual_action_pipelines.png | Bin 0 -> 42212 bytes .../environments_manual_action_single_pipeline.png | Bin 0 -> 42233 bytes 6 files changed, 132 insertions(+), 52 deletions(-) create mode 100644 doc/ci/img/environments_manual_action_builds.png create mode 100644 doc/ci/img/environments_manual_action_deployments.png create mode 100644 doc/ci/img/environments_manual_action_environments.png create mode 100644 doc/ci/img/environments_manual_action_pipelines.png create mode 100644 doc/ci/img/environments_manual_action_single_pipeline.png (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index b3d4ec40830..a96e975bb92 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -59,10 +59,10 @@ build: deploy_staging: stage: deploy script: - - echo "Deploy to staging server" + - echo "Deploy to staging server" environment: - name: staging - url: https://staging.example.com + name: staging + url: https://staging.example.com only: - master ``` @@ -80,23 +80,45 @@ lastly the `deploy_staging`. With this, we ensure that first the tests pass, then our app is able to be built successfully, and lastly we deploy to the staging server. -With the above `.gitlab-ci.yml` we have achieved that: +The `environment` keyword is just a hint for GitLab that this job actually +deploys to this environment's `name`. It can also have a `url` which, as we +will later see, is exposed in various places within GitLab. Each time a job that +has an environment specified and succeeds, a deployment is recorded, remembering +the Git SHA and environment name. + +To sum up, with the above `.gitlab-ci.yml` we have achieved that: - All branches will run the `test` and `build` jobs. -- The `deploy_staging` job will [only](yaml/README.md#only) run on the `master` - branch which means all merge requests +- The `deploy_staging` job will run [only](yaml/README.md#only) on the `master` + branch which means all merge requests that are created from branches don't + get to deploy to the staging server - When a merge request is merged, all jobs will run and the `deploy_staging` in particular will deploy our code to a staging server while the deployment will be recorded in an environment named `staging`. -The `environment` keyword is just a hint for GitLab that this job actually -deploys to this environment. Each time the job succeeds, a deployment is -recorded, remembering the Git SHA and environment name. Here's how the -Environments page looks so far. +Let's now see how that information is exposed within GitLab. + +## Viewing the current status of an environment + +The environment list under your project's **Pipelines ➔ Environments**, is +where you can find information of the last deployment status of an environment. + +Here's how the Environments page looks so far. ![Staging environment view](img/environments_available_staging.png) -TODO: describe what the above page means +There's a bunch of information there, specifically you can see: + +- The environment's name with a link to its deployments +- The last deployment ID number and who performed it +- The build ID of the last deployment with its respective job name +- The commit information of the last deployment such as who committed, to what + branch and the Git SHA of the commit +- The exact time the last deployment was performed +- A button that takes you to the URL that you have defined under the + `environment` keyword in `.gitlab-ci.yml` +- A button that re-deploys the latest deployment, meaning it runs the job + defined by the environment name for that specific commit >**Notes:** - While you can create environments manually in the web interface, we recommend @@ -104,37 +126,105 @@ TODO: describe what the above page means be automatically created for you after the first deploy. - The environments page can only be viewed by Reporters and above. For more information on the permissions, see the [permissions documentation][permissions]. +- Only deploys that happen after your `.gitlab-ci.yml` is properly configured + will show up in the "Environment" and "Last deployment" lists. -As we've pointed in the Overview section, environments are like tags for your -CI jobs, describing where code gets deployed. Here's what happened behind the -scenes: +The information shown in the Environments page is limited to the latest +deployments, but as you may have guessed an environment can have multiple +deployments. -1. The jobs and environments were defined in `.gitlab-ci.yml` -1. Changes were pushed to the repository in GitLab -1. The Runner(s) picked up the jobs -1. The jobs finished successfully -1. The environments got created if they didn't already exist -1. A deployment was recorded remembering the environment name and the Git SHA of - the last commit of the pipeline - -## View the environment status +## Viewing the deployment history of an environment GitLab keeps track of your deployments, so you always know what is currently -being deployed on your servers. You can find the environment list under -**Pipelines > Environments** for your project. You'll see the git SHA and date -of the last deployment to each environment defined. +being deployed on your servers. That way you can have the full history of your +deployments per every environment right in your browser. Clicking on an +environment will show the history of its deployments. Assuming you have deployed +multiple times already, here's how a specific environment's page looks like. + +![Deployments](img/deployments_view.png) -![Environments](img/environments_view.png) +We can see the same information as when in the Environments page, but this time +all deployments are shown. As you may have noticed, apart from the **Re-deploy** +button there are now **Rollback** buttons for each deployment. Let's see how +that works. + +## Rolling back changes + +You can't control everything, so sometimes things go wrong. When that unfortunate +time comes GitLab has you covered. Simply by clicking the **Rollback** button +that can be found in the deployments page +(**Pipelines ➔ Environments ➔ `environment name`**) you can relaunch the +job with the commit associated with it. >**Note:** -Only deploys that happen after your `.gitlab-ci.yml` is properly configured will -show up in the "Environment" and "Last deployment" lists. +Bare in mind that your mileage will vary and it's entirely up to how you define +the deployment process in the job's `script` whether the rollback succeeds or not. +GitLab CI is just following orders. + +Thankfully that was the staging server that we had to rollback, and since we +learn from our mistakes, we decided to not make the same again when we deploy +to the production server. Enter manual actions for deployments. ## Manually deploying to environments -CI/CD [Pipelines] usually have one or more [jobs] that deploy to an environment. -You can think of names such as testing, staging or production. +Turning a job from running automatically to a manual action is as simple as +adding `when: manual` to it. To expand on our previous example, let's add +another job that this time deploys our app to a production server and is +tracked by a `production` environment. The `.gitlab-ci.yml` looks like this +so far: +```yaml +stages: + - test + - build + - deploy + +test: + stage: test + script: echo "Running tests" + +build: + stage: build + script: echo "Building the app" + +deploy_staging: + stage: deploy + script: + - echo "Deploy to staging server" + environment: + name: staging + url: https://staging.example.com + only: + - master + +deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + environment: + name: production + url: https://example.com + when: manual + only: + - master +``` + +The `when: manual` action exposes a play button in GitLab's UI and the +`deploy_prod` job will only be triggered if and when we click that play button. +You can find it in the pipeline, build, environment, and deployment views. + +| Pipelines | Single pipeline | Environments | Deployments | Builds | +| --------- | ----------------| ------------ | ----------- | -------| +| ![Pipelines manual action](img/environments_manual_action_pipelines.png) | ![Pipelines manual action](img/environments_manual_action_single_pipeline.png) | ![Environments manual action](img/environments_manual_action_environments.png) | ![Deployments manual action](img/environments_manual_action_deployments.png) | ![Builds manual action](img/environments_manual_action_builds.png) | + +Clicking on the play button in either of these places will trigger the +`deploy_prod` job, and the deployment will be recorded under a new +environment named `production`. + +While this is fine for deploying to some stable environments like staging or +production, what happens for branches? So far we haven't defined anything +regarding deployments for branches other than `master`. Dynamic environments +will help us achieve that. ## Dynamic environments @@ -174,16 +264,6 @@ stop_review: action: stop ``` -## View the deployment history - -Clicking on an environment will show the history of deployments. - -![Deployments](img/deployments_view.png) - ->**Note:** -Only deploys that happen after your `.gitlab-ci.yml` is properly configured will -show up in the environments and deployments lists. - ## Checkout deployments locally Since 8.13, a reference in the git repository is saved for each deployment. So @@ -206,20 +286,20 @@ Below are some links you may find interesting: - [Review Apps](review_apps.md) Expand dynamic environments to deploy your code for every branch -## WIP +## TODO Actions -View environments -View deployments - Rollback deployments - Run deployments -View link to environment URL -View last commit message of deployment -View person who performed the deployment -View commit SHA that triggered the deployment -View branch the deployment was based on -View time ago the deployment was performed +- View environments + +- View deployments + + - Rollback deployments + + - Run deployments + +- View link to environment URL +- View last commit message of deployment + +- View person who performed the deployment + +- View commit SHA that triggered the deployment + +- View branch the deployment was based on + +- View time ago the deployment was performed + [Pipelines]: pipelines.md [jobs]: yaml/README.md#jobs diff --git a/doc/ci/img/environments_manual_action_builds.png b/doc/ci/img/environments_manual_action_builds.png new file mode 100644 index 00000000000..d4bb7ccdbae Binary files /dev/null and b/doc/ci/img/environments_manual_action_builds.png differ diff --git a/doc/ci/img/environments_manual_action_deployments.png b/doc/ci/img/environments_manual_action_deployments.png new file mode 100644 index 00000000000..c2477381c80 Binary files /dev/null and b/doc/ci/img/environments_manual_action_deployments.png differ diff --git a/doc/ci/img/environments_manual_action_environments.png b/doc/ci/img/environments_manual_action_environments.png new file mode 100644 index 00000000000..56601c0db2d Binary files /dev/null and b/doc/ci/img/environments_manual_action_environments.png differ diff --git a/doc/ci/img/environments_manual_action_pipelines.png b/doc/ci/img/environments_manual_action_pipelines.png new file mode 100644 index 00000000000..eb6e87cd956 Binary files /dev/null and b/doc/ci/img/environments_manual_action_pipelines.png differ diff --git a/doc/ci/img/environments_manual_action_single_pipeline.png b/doc/ci/img/environments_manual_action_single_pipeline.png new file mode 100644 index 00000000000..9713ad212e2 Binary files /dev/null and b/doc/ci/img/environments_manual_action_single_pipeline.png differ -- cgit v1.2.1 From ff97de61df3ebbeea31e5fef833e8a477c13ab3c Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 08:54:51 +0100 Subject: Move name rules under environment:name in yaml readme --- doc/ci/yaml/README.md | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) (limited to 'doc') diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 5c0e1c44e3f..bc9c0897f5a 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -552,28 +552,14 @@ An example usage of manual actions is deployment to production. If `environment` is specified and no environment under that name exists, a new one will be created automatically. -The `environment` name can contain: - -- letters -- digits -- spaces -- `-` -- `_` -- `/` -- `$` -- `{` -- `}` - -Common names are `qa`, `staging`, and `production`, but you can use whatever -name works with your workflow. - In its simplest form, the `environment` keyword can be defined like: ``` deploy to production: stage: deploy script: git push production HEAD:master - environment: production + environment: + name: production ``` In the above example, the `deploy to production` job will be marked as doing a @@ -588,6 +574,21 @@ Before GitLab 8.11, the name of an environment could be defined as a string like `environment: production`. The recommended way now is to define it under the `name` keyword. +The `environment` name can contain: + +- letters +- digits +- spaces +- `-` +- `_` +- `/` +- `$` +- `{` +- `}` + +Common names are `qa`, `staging`, and `production`, but you can use whatever +name works with your workflow. + Instead of defining the name of the environment right after the `environment` keyword, it is also possible to define it as a separate value. For that, use the `name` keyword under `environment`: -- cgit v1.2.1 From 54fc574ebad57fddeaede8edeae04d53031d7712 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 10:36:58 +0100 Subject: Finish dynamic environments and URLs sections [ci skip] --- doc/ci/environments.md | 160 ++++++++++++++++++++++- doc/ci/img/environments_link_url.png | Bin 0 -> 33561 bytes doc/ci/img/environments_link_url_deployments.png | Bin 0 -> 19652 bytes doc/ci/img/environments_link_url_mr.png | Bin 0 -> 47347 bytes doc/ci/img/environments_mr_review_app.png | Bin 0 -> 39780 bytes 5 files changed, 156 insertions(+), 4 deletions(-) create mode 100644 doc/ci/img/environments_link_url.png create mode 100644 doc/ci/img/environments_link_url_deployments.png create mode 100644 doc/ci/img/environments_link_url_mr.png create mode 100644 doc/ci/img/environments_mr_review_app.png (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index a96e975bb92..428e8a0cf25 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -221,6 +221,11 @@ Clicking on the play button in either of these places will trigger the `deploy_prod` job, and the deployment will be recorded under a new environment named `production`. +>**Note:** +Remember that if your environment's name is `production` (all lowercase), then +it will get recorded in [Cycle Analytics](../user/project/cycle_analytics.md). +Double the benefit! + While this is fine for deploying to some stable environments like staging or production, what happens for branches? So far we haven't defined anything regarding deployments for branches other than `master`. Dynamic environments @@ -229,21 +234,162 @@ will help us achieve that. ## Dynamic environments As the name suggests, it is possible to create environments on the fly by just -declaring their names dynamically in `.gitlab-ci.yml`. +declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is +the base of [Review apps](review_apps.md). GitLab Runner exposes various [environment variables][variables] when a job runs, -and as such you can use them +and as such, you can use them as environment names. Let's add another job in +our example which will deploy to all branches except `master`: +```yaml +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_BUILD_REF_NAME + url: https://$CI_BUILD_REF_NAME.example.com + only: + - branches + except: + - master ``` -review: + +Let's break it down in pieces. The job's name is `deploy_review` and it runs +on the `deploy` stage. The `script` at this point is fictional, you'd have to +use your own based on your deployment. Then, we set the `environment` with the +`environment:name` being `review/$CI_BUILD_REF_NAME`. Now that's an interesting +one. Since the [environment name][env-name] can contain also slashes (`/`), we +can use this pattern to distinguish between dynamic environments and the regular +ones. + +So, the first part is `review`, followed by a `/` and then `$CI_BUILD_REF_NAME` +which takes the value of the branch name. We also use the same +`$CI_BUILD_REF_NAME` value in the `environment:url` so that the environment +can get a specific and distinct URL for each branch. Again, the way you set up +the webserver to serve these requests is based on your setup. + +Last but not least, we tell the job to run [`only`][only] on branches +[`except`][only] master. + +>**Note:** +You are not bound to use only slashes in the dynamic environments' names (`/`), +but as we will see later, this will enable the "grouping similar environments" +feature. + +The whole `.gitlab-ci.yml` looks like this so far: + +```yaml +stages: + - test + - build + - deploy + +test: + stage: test + script: echo "Running tests" + +build: + stage: build + script: echo "Building the app" + +deploy_review: stage: deploy script: - - rsync -av --delete public /srv/nginx/pages/$CI_BUILD_REF_NAME + - echo "Deploy a review app" environment: name: review/$CI_BUILD_REF_NAME url: https://$CI_BUILD_REF_NAME.example.com + only: + - branches + except: + - master + +deploy_staging: + stage: deploy + script: + - echo "Deploy to staging server" + environment: + name: staging + url: https://staging.example.com + only: + - master + +deploy_prod: + stage: deploy + script: + - echo "Deploy to production server" + environment: + name: production + url: https://example.com + when: manual + only: + - master ``` +A more realistic example would include copying files to a location where a +webserver (NGINX) could then read and serve. The example below will copy the +`public` directory to `/srv/nginx/$CI_BUILD_REF_NAME/public`: + +```yaml +review_app: + stage: deploy + script: + - rsync -av --delete public /srv/nginx/$CI_BUILD_REF_NAME + environment: + name: review/$CI_BUILD_REF_NAME + url: https://$CI_BUILD_REF_NAME.example.com +``` + +It is assumed that the user has already setup NGINX and GitLab Runner in the +server this job will run on. + +--- + +The development workflow would now be: + +- Developer creates a branch locally +- Developer makes changes, commits and pushes the branch to GitLab +- Developer creates a merge request + +Behind the scenes: + +- GitLab Runner picks up the changes and starts running the jobs +- The jobs run sequentially as defined in `stages` + - First, the tests pass + - Then, the build begins and successfully also passes + - Lastly, the app is deployed to an environment with a name specific to the + branch + +So now, every branch gets its own environment and is deployed to its own place +with the added benefit of having a [history of deployments](#viewing-the-deployment-history-of-an-environment) +and also being able to [rollback changes](#rolling-back-changes) if needed. +Let's briefly see where URL that's defined in the environments is exposed. + +## Making use of the environment URL + +The environment URL is exposed in a few places within GitLab. + +| In a merge request widget as a link | In the Environments view as a button | In the Deployments view as a button | +| -------------------- | ------------ | ----------- | +| ![Environment URL in merge request](img/environments_mr_review_app.png) | ![Environment URL in environments](img/environments_link_url.png) | ![Environment URL in deployments](img/environments_link_url_deployments.png) | + +If a merge request is eventually merged to the default branch (in our case +`master`) and that branch also deploys to an environment (in our case `staging` +and/or `production`) you can see this information in the merge request itself. + +![Environment URLs in merge request](img/environments_link_url_mr.png) + +--- + +We now have a full development cycle, where our app is tested, built, deployed +as a Review app, deployed to a staging server once the merge request is merged, +and finally manually deployed to the production server. What we just described +is a single workflow, but imagine tens of developers working on a project +at the same time. They each push to their branches, and dynamic environments are +created all the time. In that case, we probably need to do some clean up. Read +next how environments can be closed. + ## Closing an environment ``` @@ -264,6 +410,10 @@ stop_review: action: stop ``` +## Grouping similar environments + +folders in environments page + ## Checkout deployments locally Since 8.13, a reference in the git repository is saved for each deployment. So @@ -308,3 +458,5 @@ Actions [deployments]: #deployments [permissions]: ../user/permissions.md [variables]: variables/README.md +[env-name]: yaml/README.md#environment-name +[only]: yaml/README.md#only-and-except diff --git a/doc/ci/img/environments_link_url.png b/doc/ci/img/environments_link_url.png new file mode 100644 index 00000000000..224c21adfb5 Binary files /dev/null and b/doc/ci/img/environments_link_url.png differ diff --git a/doc/ci/img/environments_link_url_deployments.png b/doc/ci/img/environments_link_url_deployments.png new file mode 100644 index 00000000000..9419668a9bd Binary files /dev/null and b/doc/ci/img/environments_link_url_deployments.png differ diff --git a/doc/ci/img/environments_link_url_mr.png b/doc/ci/img/environments_link_url_mr.png new file mode 100644 index 00000000000..3276dfb6096 Binary files /dev/null and b/doc/ci/img/environments_link_url_mr.png differ diff --git a/doc/ci/img/environments_mr_review_app.png b/doc/ci/img/environments_mr_review_app.png new file mode 100644 index 00000000000..a2ae25d62fa Binary files /dev/null and b/doc/ci/img/environments_mr_review_app.png differ -- cgit v1.2.1 From aee501cf43811f819e95541d5a0c0c6d702bf8da Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 10:53:42 +0100 Subject: Remove ToC and fix headings in Markdown docs [ci skip] --- doc/user/markdown.md | 156 +++++++++++++++++---------------------------------- 1 file changed, 53 insertions(+), 103 deletions(-) (limited to 'doc') diff --git a/doc/user/markdown.md b/doc/user/markdown.md index dbc7e0f14e3..162d1bd7ed4 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -1,43 +1,5 @@ # Markdown -## Table of Contents - -**[GitLab Flavored Markdown](#gitlab-flavored-markdown-gfm)** - -* [Newlines](#newlines) -* [Multiple underscores in words](#multiple-underscores-in-words) -* [URL auto-linking](#url-auto-linking) -* [Multiline Blockquote](#multiline-blockquote) -* [Code and Syntax Highlighting](#code-and-syntax-highlighting) -* [Inline Diff](#inline-diff) -* [Emoji](#emoji) -* [Special GitLab references](#special-gitlab-references) -* [Task Lists](#task-lists) -* [Videos](#videos) - -**[Standard Markdown](#standard-markdown)** - -* [Headers](#headers) -* [Emphasis](#emphasis) -* [Lists](#lists) -* [Links](#links) -* [Images](#images) -* [Blockquotes](#blockquotes) -* [Inline HTML](#inline-html) -* [Horizontal Rule](#horizontal-rule) -* [Line Breaks](#line-breaks) -* [Tables](#tables) -* [Footnotes](#footnotes) - -**[Wiki-Specific Markdown](#wiki-specific-markdown)** - -* [Wiki - Direct page link](#wiki-direct-page-link) -* [Wiki - Direct file link](#wiki-direct-file-link) -* [Wiki - Hierarchical link](#wiki-hierarchical-link) -* [Wiki - Root link](#wiki-root-link) - -**[References](#references)** - ## GitLab Flavored Markdown (GFM) > **Note:** @@ -64,7 +26,7 @@ You can use GFM in the following areas: You can also use other rich text files in GitLab. You might have to install a dependency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. -## Newlines +### Newlines > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#newlines @@ -84,7 +46,7 @@ Violets are blue Sugar is sweet -## Multiple underscores in words +### Multiple underscores in words > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiple-underscores-in-words @@ -99,7 +61,7 @@ perform_complicated_task do_this_and_do_that_and_another_thing -## URL auto-linking +### URL auto-linking > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#url-auto-linking @@ -120,7 +82,7 @@ GFM will autolink almost any URL you copy and paste into your text: * irc://irc.freenode.net/gitlab * http://localhost:3000 -## Multiline Blockquote +### Multiline Blockquote > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#multiline-blockquote @@ -154,7 +116,7 @@ multiple lines, you can quote that without having to manually prepend `>` to every line! >>> -## Code and Syntax Highlighting +### Code and Syntax Highlighting > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#code-and-syntax-highlighting @@ -224,7 +186,7 @@ s = "There is no highlighting for this." But let's throw in a tag. ``` -## Inline Diff +### Inline Diff > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#inline-diff @@ -240,7 +202,7 @@ However the wrapping tags cannot be mixed as such: - {- deletions -] - [- deletions -} -## Emoji +### Emoji > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji @@ -265,7 +227,7 @@ If you are new to this, don't be :fearful:. You can easily join the emoji :famil Consult the [Emoji Cheat Sheet](http://emoji.codes) for a list of all supported emoji codes. :thumbsup: -## Special GitLab References +### Special GitLab References GFM recognizes special references. @@ -305,7 +267,7 @@ GFM also recognizes certain cross-project references: | `namespace/project@9ba12248...b19a04f5` | commit range comparison | | `namespace/project~"Some label"` | issues with given label | -## Task Lists +### Task Lists > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#task-lists @@ -328,7 +290,7 @@ You can add task lists to issues, merge requests and comments. To create a task Task lists can only be created in descriptions, not in titles. Task item state can be managed by editing the description's Markdown or by toggling the rendered check boxes. -## Videos +### Videos > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#videos @@ -345,9 +307,9 @@ Here's a sample video: ![Sample Video](img/markdown_video.mp4) -# Standard Markdown +## Standard Markdown -## Headers +### Headers ```no-highlight # H1 @@ -366,21 +328,6 @@ Alt-H2 ------ ``` -# H1 -## H2 -### H3 -#### H4 -##### H5 -###### H6 - -Alternatively, for H1 and H2, an underline-ish style: - -Alt-H1 -====== - -Alt-H2 ------- - ### Header IDs and links All Markdown-rendered headers automatically get IDs, except in comments. @@ -416,7 +363,7 @@ Would generate the following link IDs: Note that the Emoji processing happens before the header IDs are generated, so the Emoji is converted to an image which then gets removed from the ID. -## Emphasis +### Emphasis ```no-highlight Emphasis, aka italics, with *asterisks* or _underscores_. @@ -436,7 +383,7 @@ Combined emphasis with **asterisks and _underscores_**. Strikethrough uses two tildes. ~~Scratch this.~~ -## Lists +### Lists ```no-highlight 1. First ordered list item @@ -492,7 +439,7 @@ the second list item will be incorrectly labeled as `1`. Second paragraph of first item. 2. Another item -## Links +### Links There are two ways to create links, inline-style and reference-style. @@ -501,9 +448,9 @@ There are two ways to create links, inline-style and reference-style. [I'm a reference-style link][Arbitrary case-insensitive reference text] [I'm a relative reference to a repository file](LICENSE) - + [I am an absolute reference within the repository](/doc/user/markdown.md) - + [I link to the Milestones page](/../milestones) [You can use numbers for reference-style link definitions][1] @@ -523,9 +470,9 @@ There are two ways to create links, inline-style and reference-style. [I'm a relative reference to a repository file](LICENSE)[^1] [I am an absolute reference within the repository](/doc/user/markdown.md) - + [I link to the Milestones page](/../milestones) - + [You can use numbers for reference-style link definitions][1] Or leave it empty and use the [link text itself][] @@ -544,7 +491,8 @@ Relative links do not allow referencing project files in a wiki page or wiki pag will point the link to `wikis/style` when the link is inside of a wiki markdown file. -## Images + +### Images Here's our logo (hover to see the title text): @@ -568,7 +516,7 @@ Reference-style: [logo]: img/markdown_logo.png -## Blockquotes +### Blockquotes ```no-highlight > Blockquotes are very handy in email to emulate reply text. @@ -586,7 +534,7 @@ Quote break. > This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote. -## Inline HTML +### Inline HTML You can also use raw HTML in your Markdown, and it'll mostly work pretty well. @@ -610,7 +558,7 @@ See the documentation for HTML::Pipeline's [SanitizationFilter](http://www.rubyd
Does *not* work **very** well. Use HTML tags.
-## Horizontal Rule +### Horizontal Rule ``` Three or more... @@ -642,7 +590,7 @@ ___ Underscores -## Line Breaks +### Line Breaks My basic recommendation for learning how line breaks work is to experiment and discover -- hit <Enter> once (i.e., insert one newline), then hit it twice (i.e., insert two newlines), see what happens. You'll soon learn to get what you want. "Markdown Toggle" is your friend. @@ -672,7 +620,7 @@ This line is also a separate paragraph, and... This line is on its own line, because the previous line ends with two spaces. -## Tables +### Tables Tables aren't part of the core Markdown spec, but they are part of GFM and Markdown Here supports them. @@ -708,16 +656,15 @@ By including colons in the header row, you can align the text within that column | Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | | Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | -## Footnotes - -You can add footnotes to your text as follows.[^1] -[^1]: This is my awesome footnote. +### Footnotes ``` -You can add footnotes to your text as follows.[^1] -[^1]: This is my awesome footnote. +You can add footnotes to your text as follows.[^2] +[^2]: This is my awesome footnote. ``` +You can add footnotes to your text as follows.[^2] + ## Wiki-specific Markdown The following examples show how links inside wikis behave. @@ -752,30 +699,30 @@ A link can be constructed relative to the current wiki page using `./`, - If this snippet was placed on a page at `/documentation/main`, it would link to `/documentation/related`: - ```markdown - [Link to Related Page](./related) - ``` + ```markdown + [Link to Related Page](./related) + ``` - If this snippet was placed on a page at `/documentation/related/content`, it would link to `/documentation/main`: - ```markdown - [Link to Related Page](../main) - ``` + ```markdown + [Link to Related Page](../main) + ``` - If this snippet was placed on a page at `/documentation/main`, it would link to `/documentation/related.md`: - ```markdown - [Link to Related Page](./related.md) - ``` + ```markdown + [Link to Related Page](./related.md) + ``` - If this snippet was placed on a page at `/documentation/related/content`, it would link to `/documentation/main.md`: - ```markdown - [Link to Related Page](../main.md) - ``` + ```markdown + [Link to Related Page](../main.md) + ``` ### Wiki - Root link @@ -783,22 +730,25 @@ A link starting with a `/` is relative to the wiki root. - This snippet links to `/documentation`: - ```markdown - [Link to Related Page](/documentation) - ``` + ```markdown + [Link to Related Page](/documentation) + ``` - This snippet links to `/miscellaneous.md`: - ```markdown - [Link to Related Page](/miscellaneous.md) - ``` + ```markdown + [Link to Related Page](/miscellaneous.md) + ``` + ## References - This document leveraged heavily from the [Markdown-Cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet). - The [Markdown Syntax Guide](https://daringfireball.net/projects/markdown/syntax) at Daring Fireball is an excellent resource for a detailed explanation of standard markdown. - [Dillinger.io](http://dillinger.io) is a handy tool for testing standard markdown. +[^1]: This link will be broken if you see this document from the Help page or docs.gitlab.com +[^2]: This is my awesome footnote. + [markdown.md]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md [rouge]: http://rouge.jneen.net/ "Rouge website" [redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" -[^1]: This link will be broken if you see this document from the Help page or docs.gitlab.com -- cgit v1.2.1 From 556aaebc22b58d1e496512323173ea1ca28370d1 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 11:58:57 +0100 Subject: Finish "Stopping envs" and "Grouping similar envs" sections [ci skip] --- doc/ci/environments.md | 83 +++++++++++++++++++---------- doc/ci/img/environments_dynamic_groups.png | Bin 0 -> 134164 bytes 2 files changed, 56 insertions(+), 27 deletions(-) create mode 100644 doc/ci/img/environments_dynamic_groups.png (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 428e8a0cf25..861a639b3fc 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -273,9 +273,9 @@ Last but not least, we tell the job to run [`only`][only] on branches [`except`][only] master. >**Note:** -You are not bound to use only slashes in the dynamic environments' names (`/`), -but as we will see later, this will enable the "grouping similar environments" -feature. +You are not bound to use the same prefix or only slashes in the dynamic +environments' names (`/`), but as we will see later, this will enable the +[grouping similar environments](#grouping-similar-environments) feature. The whole `.gitlab-ci.yml` looks like this so far: @@ -388,31 +388,74 @@ and finally manually deployed to the production server. What we just described is a single workflow, but imagine tens of developers working on a project at the same time. They each push to their branches, and dynamic environments are created all the time. In that case, we probably need to do some clean up. Read -next how environments can be closed. +next how environments can be stopped. -## Closing an environment +## Stopping an environment -``` -review: +By stopping an environment, you are effectively terminating its recording of the +deployments that happen in it. + +>**Note:** +Starting with GitLab 8.14, dynamic environments will be stopped automatically +when their associated branch is removed. + +There is a special case where environments can be manually closed. That can +happen if you provide another job for that matter. The syntax is a little +tricky since a job calls another job to do the job. + +Consider the following example where the `deploy_review` calls the `stop_review` +to clean up and stop the environment: + +```yaml +deploy_review: stage: deploy script: - - rsync -av --delete public /srv/nginx/pages/$CI_BUILD_REF_NAME + - echo "Deploy a review app" environment: name: review/$CI_BUILD_REF_NAME - url: http://$CI_BUILD_REF_NAME.$APPS_DOMAIN + url: https://$CI_BUILD_REF_NAME.example.com on_stop: stop_review + only: + - branches + except: + - master stop_review: - script: rm -rf /srv/nginx/pages/$CI_BUILD_REF_NAME + script: + - echo "Remove review app" when: manual environment: name: review/$CI_BUILD_REF_NAME action: stop ``` +You can read more in the [`.gitlab-ci.yml` reference][onstop]. + ## Grouping similar environments -folders in environments page +> [Introduced][ce-7015] in GitLab 8.14. + +As we've seen in the [dynamic environments](#dynamic-environments), you can + +In short, environments that are named like `type/foo` are presented under a +group named `type`. + +In our minimal example, we name the environments `review/$CI_BUILD_REF_NAME` +where `$CI_BUILD_REF_NAME` is the branch name: + +```yaml +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_BUILD_REF_NAME +``` + +In that case, if you visit the Environments page, and provided the branches +exist, you should see something like: + +![Environment groups](img/environments_dynamic_groups.png) ## Checkout deployments locally @@ -435,22 +478,6 @@ Below are some links you may find interesting: - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - [Review Apps](review_apps.md) Expand dynamic environments to deploy your code for every branch - -## TODO - -Actions - -- View environments + -- View deployments + - - Rollback deployments + - - Run deployments + -- View link to environment URL -- View last commit message of deployment + -- View person who performed the deployment + -- View commit SHA that triggered the deployment + -- View branch the deployment was based on + -- View time ago the deployment was performed + - [Pipelines]: pipelines.md [jobs]: yaml/README.md#jobs [yaml]: yaml/README.md @@ -460,3 +487,5 @@ Actions [variables]: variables/README.md [env-name]: yaml/README.md#environment-name [only]: yaml/README.md#only-and-except +[onstop]: yaml/README.md#environment-on_stop +[ce-7015]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7015 diff --git a/doc/ci/img/environments_dynamic_groups.png b/doc/ci/img/environments_dynamic_groups.png new file mode 100644 index 00000000000..e89b66c502c Binary files /dev/null and b/doc/ci/img/environments_dynamic_groups.png differ -- cgit v1.2.1 From 9ea4a5bca589aa095dd3b633245b551b4c758d82 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 12:15:06 +0100 Subject: Add note about auto-stopping of environments --- doc/ci/environments.md | 17 +++++++++++++---- doc/ci/yaml/README.md | 7 ++++++- 2 files changed, 19 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 861a639b3fc..cc65f0ad8ad 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -395,11 +395,12 @@ next how environments can be stopped. By stopping an environment, you are effectively terminating its recording of the deployments that happen in it. ->**Note:** -Starting with GitLab 8.14, dynamic environments will be stopped automatically -when their associated branch is removed. +A branch is associated with an environment when the CI pipeline that is created +for this branch, was recently deployed to this environment. You can think of +the CI pipeline as the glue between the branch and the environment: +`branch ➔ CI pipeline ➔ environment`. -There is a special case where environments can be manually closed. That can +There is a special case where environments can be manually stopped. That can happen if you provide another job for that matter. The syntax is a little tricky since a job calls another job to do the job. @@ -429,6 +430,14 @@ stop_review: action: stop ``` +>**Note:** +Starting with GitLab 8.14, dynamic environments will be stopped automatically +when their associated branch is deleted. + +When you have an environment that has a stop action defined (typically when +the environment describes a review app), GitLab will automatically trigger a +stop action when the associated branch is deleted. + You can read more in the [`.gitlab-ci.yml` reference][onstop]. ## Grouping similar environments diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index bc9c0897f5a..1e096444903 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -627,7 +627,12 @@ deploy to production: #### environment:on_stop -> [Introduced][ce-6669] in GitLab 8.13. +> +**Notes:** +- [Introduced][ce-6669] in GitLab 8.13. +- Starting with GitLab 8.14, when you have an environment that has a stop action + defined, GitLab will automatically trigger a stop action when the associated + branch is deleted. Closing (stoping) environments can be achieved with the `on_stop` keyword defined under `environment`. It declares a different job that runs in order to close -- cgit v1.2.1 From e5aebccb640636cf0ea74b3569772b8a9156bb48 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 12:16:13 +0100 Subject: Add stop environment permissions and remove delete [ci skip] --- doc/user/permissions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/user/permissions.md b/doc/user/permissions.md index a33183fa01c..cea78864df2 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -33,7 +33,7 @@ The following table depicts the various user permission levels in a project. | See a container registry | | ✓ | ✓ | ✓ | ✓ | | See environments | | ✓ | ✓ | ✓ | ✓ | | Create new environments | | | ✓ | ✓ | ✓ | -| Delete environments | | | | ✓ | ✓ | +| Stop environments | | | ✓ | ✓ | ✓ | | See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | Manage/Accept merge requests | | | ✓ | ✓ | ✓ | | Create new merge request | | | ✓ | ✓ | ✓ | -- cgit v1.2.1 From 1ccc69d0dc25e1b59e68e1737c217f3003b3b372 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 19 Oct 2016 11:55:27 +0200 Subject: Add Review apps link to CI README --- doc/ci/README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/ci/README.md b/doc/ci/README.md index 6b90940c047..545cc72682d 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -1,6 +1,6 @@ -## GitLab CI Documentation +# GitLab CI Documentation -### CI User documentation +## CI User documentation - [Get started with GitLab CI](quick_start/README.md) - [CI examples for various languages](examples/README.md) @@ -20,4 +20,8 @@ - [API](../api/ci/README.md) - [CI services (linked docker containers)](services/README.md) - [CI/CD pipelines settings](../user/project/pipelines/settings.md) -- [**New CI build permissions model**](../user/project/new_ci_build_permissions_model.md) Read about what changed in GitLab 8.12 and how that affects your builds. There's a new way to access your Git submodules and LFS objects in builds. +- [Review Apps](review_apps/index.md) + +## Breaking changes + +- [New CI build permissions model](../user/project/new_ci_build_permissions_model.md) Read about what changed in GitLab 8.12 and how that affects your builds. There's a new way to access your Git submodules and LFS objects in builds. -- cgit v1.2.1 From eee65421feedaf5aa510e05794f4da0618401e96 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 19 Oct 2016 11:55:46 +0200 Subject: WIP review apps Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/23484 [ci skip] --- doc/ci/review_apps/index.md | 82 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 doc/ci/review_apps/index.md (limited to 'doc') diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md new file mode 100644 index 00000000000..ee279025a9d --- /dev/null +++ b/doc/ci/review_apps/index.md @@ -0,0 +1,82 @@ +# Review apps + +> [Introduced][ce-21971] in GitLab 8.12. Further additions were made in GitLab 8.13. + +Review apps are automatically-created [environments] that run your code for each +branch. That means [merge requests] can be reviewed in a live-running environment. + +They mostly make sense to be used with web applications. + +Review Apps can make a huge impact on your development flow. Reviewing anything +from performance to interface changes becomes much easier with a live environment. + +> +Inspired by [Heroku's Review Apps][heroku-apps] which itself was inspired by +[Fourchette]. + +## Dynamic environments + +You can now use predefined CI variables as a name for environments. In addition, +you can specify a URL for the environment configuration in your .gitlab-ci.yml +file. + +- Mapping branch with environment + +This issue describes dynamic environments implementation, mostly to be used with dynamically create applications. +These application will be mostly used for Review Apps. + +## Assumptions + +1. We will allow to create dynamic environments from `.gitlab-ci.yml`, by allowing to specify environment variables: `review_apps_${CI_BUILD_REF_NAME}`, +1. We will use multiple deployments of the same application per environment, +1. The URL will be assigned to environment on the creation, and updated later if necessary, +1. The URL will be specified in `.gitlab-ci.yml`, possibly introducing regexp for getting an URL from build log if required, +1. We need some form to distinguish between production/staging and review app environment, +1. We don't try to manage life cycle of deployments in the first iteration, possibly we will extend a Pipeline to add jobs that will be responsible either for cleaning up or removing old deployments and closing environments. + +## Configuration + +``` +review_apps: + environment: + name: review/$CI_BUILD_REF_NAME + url: http://$CI_BUILD_REF_NAME.review.gitlab.com/ +``` + +### Remarks + +1. We are limited to use only CI predefined variables, since this is not easy task to pass the URL from build, +2. I do prefer nested `url:` since this allows us in the future to extend that with more `environment:` configuration or constraints, like: `required_variables:` or `access_level` of user allowed to use that. +3. Following the problem from (1) we can extend `url:` with a `regexp` that will be used to fish a URL from build log. + +## Distinguish between production and review apps + +### Convention over configuration + +We would expect the environments to be of `type/name`: + +1. This would allow us to have a clear distinction between different environment types: `production/gitlab.com`, `staging/dev`, `review-apps/feature/branch`, +2. Since we use a folder structure we could group all environments by `type` and strip that from environment name, +3. We would be aware of some of these types and for example for `review-apps` show them differently in context of Merge Requests, ex. calculating `deployed ago` a little differently. +3. We could easily group all `types` across from group from all projects. + +The `type/name` also plays nice with `Variables` and `Runners`, because we can limit their usage: + +1. We could extend the resources with a field that would allow us to filter for what types it can be used, ex.: `production/*` or `review-apps/*` +2. We could limit runners to be used only by `review-apps/*`, + +## Destroying Review Apps + + +## Examples + +- Use with NGINX +- Use with Amazon S3 +- Use on Heroku with dpl +- Use with OpenShift/kubernetes + +[ce-21971]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21971 +[heroku-apps]: https://devcenter.heroku.com/articles/github-integration-review-apps +[environments]: ../environments.md +[merge requests]: ../../user/project/merge_requests.md +[fourchette]: https://github.com/rainforestapp/fourchette -- cgit v1.2.1 From 0881c3a82a6c4d198856b6a1e976b932b2b0f837 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 27 Oct 2016 14:18:41 +0200 Subject: Add an intro and an Overview section for Review Apps [ci skip] --- doc/ci/review_apps/index.md | 126 +++++++++++++++++++++++++++++--------------- 1 file changed, 83 insertions(+), 43 deletions(-) (limited to 'doc') diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index ee279025a9d..ce1fbaaab04 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,29 +1,87 @@ -# Review apps +# Getting started with Review Apps > [Introduced][ce-21971] in GitLab 8.12. Further additions were made in GitLab 8.13. +> +Inspired by [Heroku's Review Apps][heroku-apps] which itself was inspired by +[Fourchette]. -Review apps are automatically-created [environments] that run your code for each -branch. That means [merge requests] can be reviewed in a live-running environment. +The base of Review Apps is the [dynamic environments] which allow you to create +a new environment (dynamically) for each one of your branches. -They mostly make sense to be used with web applications. +A Review App can then be visible as a link when you visit the [merge request] +relevant to the branch. That way, you are able to see live all changes introduced +by the merge request changes. Reviewing anything, from performance to interface +changes, becomes much easier with a live environment and as such, Review Apps +can make a huge impact on your development flow. -Review Apps can make a huge impact on your development flow. Reviewing anything -from performance to interface changes becomes much easier with a live environment. +They mostly make sense to be used with web applications, but you can use them +any way you'd like. -> -Inspired by [Heroku's Review Apps][heroku-apps] which itself was inspired by -[Fourchette]. +## Overview + +Simply put, a Review App is a mapping of +a branch with an environment as there is a 1:1 relation between them. + +To get a better understanding of Review Apps, you must first learn how +environments and deployments work. The following docs will help you grasp that +knowledge: + +1. First, learn about [environments][] and their role in the development workflow. +1. Then make a small stop to learn about [CI variables][variables] and how they + can be used in your CI jobs. +1. Next, explore the [`environment` syntax][yaml-env] as defined in `.gitlab-ci.yml`. + This will be your primary reference when you are finally comfortable with + how environments work. +1. Additionally, find out about [manual actions][] and how you can use them to + deploy to critical environments like production with the push of a button. +1. And as a last step, follow the [NGINX example tutorial][app-nginx] which will + guide you step by step to set up the infrastructure and make use of + Review Apps with a simple HTML site and NGINX. + +## Configuration + +For configuration examples see the [dynamic environments] documentation. + +## Creating and destroying Review Apps + +The creation and destruction of a Review App is defined in `.gitlab-ci.yml`. +Check the [environments] documentation how to do so. + +## A simple workflow + +The process of adding Review Apps in your workflow would look like: + +1. Set up the infrastructure to host and deploy the Review Apps. +1. Install and configure a Runner that does the deployment. +1. Set up a job in `.gitlab-ci.yml` that uses the predefined + [predefined CI environment variable][variables] `${CI_BUILD_REF_NAME}` to + create dynamic environments and restrict it to run only on branches. +1. Optionally set a job that stops the Review Apps. + +From there on, you would follow the branched Git flow: + +1. Push a branch and let the Runner deploy the Review App based on the `script` + definition of the dynamic environment job. +1. Wait for the Runner to build and/or deploy your web app. +1. Click on the link that's present in the MR related to the branch and see the + changes live. + +### Limitations + +We are limited to use only [CI predefined variables][variables]. -## Dynamic environments +## Examples + +A list of examples used with Review Apps can be found below: -You can now use predefined CI variables as a name for environments. In addition, -you can specify a URL for the environment configuration in your .gitlab-ci.yml -file. +- [Use with NGINX][app-nginx] - Use NGINX and the shell executor of GitLab Runner + to deploy a simple HTML website. -- Mapping branch with environment +And below is a soon to be added examples list: -This issue describes dynamic environments implementation, mostly to be used with dynamically create applications. -These application will be mostly used for Review Apps. +- Use with Amazon S3 +- Use on Heroku with dpl +- Use with OpenShift/kubernetes ## Assumptions @@ -34,23 +92,10 @@ These application will be mostly used for Review Apps. 1. We need some form to distinguish between production/staging and review app environment, 1. We don't try to manage life cycle of deployments in the first iteration, possibly we will extend a Pipeline to add jobs that will be responsible either for cleaning up or removing old deployments and closing environments. -## Configuration - -``` -review_apps: - environment: - name: review/$CI_BUILD_REF_NAME - url: http://$CI_BUILD_REF_NAME.review.gitlab.com/ -``` - -### Remarks - -1. We are limited to use only CI predefined variables, since this is not easy task to pass the URL from build, -2. I do prefer nested `url:` since this allows us in the future to extend that with more `environment:` configuration or constraints, like: `required_variables:` or `access_level` of user allowed to use that. -3. Following the problem from (1) we can extend `url:` with a `regexp` that will be used to fish a URL from build log. - ## Distinguish between production and review apps +- Are dynamic environments distinguishable by the slash in `environment:url`? + ### Convention over configuration We would expect the environments to be of `type/name`: @@ -65,18 +110,13 @@ The `type/name` also plays nice with `Variables` and `Runners`, because we can l 1. We could extend the resources with a field that would allow us to filter for what types it can be used, ex.: `production/*` or `review-apps/*` 2. We could limit runners to be used only by `review-apps/*`, -## Destroying Review Apps - - -## Examples - -- Use with NGINX -- Use with Amazon S3 -- Use on Heroku with dpl -- Use with OpenShift/kubernetes - +[app-nginx]: nginx_guide.md [ce-21971]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21971 -[heroku-apps]: https://devcenter.heroku.com/articles/github-integration-review-apps +[dynamic environments]: ../environments.md#dynamic-environments [environments]: ../environments.md -[merge requests]: ../../user/project/merge_requests.md [fourchette]: https://github.com/rainforestapp/fourchette +[heroku-apps]: https://devcenter.heroku.com/articles/github-integration-review-apps +[manual actions]: ../environments.md#manual-actions +[merge request]: ../../user/project/merge_requests.md +[variables]: ../variables/README.md +[yaml-env]: ../yaml/README.md#environment -- cgit v1.2.1 From 17cf2adf0438f306f0d47e0178f0718a46471fb2 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 27 Oct 2016 14:19:43 +0200 Subject: Add note about current limitation in $CI_BUILD_REF_NAME [ci skip] --- doc/ci/yaml/README.md | 7 +++++++ 1 file changed, 7 insertions(+) (limited to 'doc') diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 1e096444903..b73cc41f95d 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -687,6 +687,13 @@ The `stop_review_app` job is **required** to have the following keywords defined These parameters can use any of the defined [CI variables](#variables) (including predefined, secure variables and `.gitlab-ci.yml` variables). +>**Note:** +Be aware than if the branch name contains special characters and you use the +`$CI_BUILD_REF_NAME` variable to dynamically create environments, there might +be complications during deployment. Follow the +[issue 22849](https://gitlab.com/gitlab-org/gitlab-ce/issues/22849) for more +information. + For example: ``` -- cgit v1.2.1 From 46a7e681e0dad51b7465f056d18e187310049694 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 27 Oct 2016 15:31:06 +0200 Subject: Get rid most of the irrelevant sections [ci skip] --- .../review_apps/img/review_apps_preview_in_mr.png | Bin 0 -> 28689 bytes doc/ci/review_apps/index.md | 59 ++++++++++----------- 2 files changed, 27 insertions(+), 32 deletions(-) create mode 100644 doc/ci/review_apps/img/review_apps_preview_in_mr.png (limited to 'doc') diff --git a/doc/ci/review_apps/img/review_apps_preview_in_mr.png b/doc/ci/review_apps/img/review_apps_preview_in_mr.png new file mode 100644 index 00000000000..15bcb90518c Binary files /dev/null and b/doc/ci/review_apps/img/review_apps_preview_in_mr.png differ diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index ce1fbaaab04..b4abb8c0e5c 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -19,8 +19,19 @@ any way you'd like. ## Overview -Simply put, a Review App is a mapping of -a branch with an environment as there is a 1:1 relation between them. +Simply put, a Review App is a mapping of a branch with an environment as there +is a 1:1 relation between them. + +Here's an example of what it looks like when viewing a merge request with a +dynamically set environment. + +![Review App in merge request](img/review_apps_preview_in_mr.png) + +In the image above you can see that the `add-new-line` branch was successfully +built and deployed under a dynamic environment and can be previewed with an +also dynamically URL. + +--- To get a better understanding of Review Apps, you must first learn how environments and deployments work. The following docs will help you grasp that @@ -34,9 +45,9 @@ knowledge: how environments work. 1. Additionally, find out about [manual actions][] and how you can use them to deploy to critical environments like production with the push of a button. -1. And as a last step, follow the [NGINX example tutorial][app-nginx] which will +1. And as a last step, follow the [example tutorials](#examples) which will guide you step by step to set up the infrastructure and make use of - Review Apps with a simple HTML site and NGINX. + Review Apps. ## Configuration @@ -66,7 +77,18 @@ From there on, you would follow the branched Git flow: 1. Click on the link that's present in the MR related to the branch and see the changes live. -### Limitations +## Distinguish between production and review apps + +1. We need some form to distinguish between production/staging and review app environment, + +- Are dynamic environments distinguishable by the slash in `environment:url`? + +We would expect the environments to be of `type/name`: + +1. This would allow us to have a clear distinction between different environment types: `production/gitlab.com`, `staging/dev`, `review-apps/feature/branch`, +3. We would be aware of some of these types and for example for `review-apps` show them differently in context of Merge Requests, ex. calculating `deployed ago` a little differently. + +## Limitations We are limited to use only [CI predefined variables][variables]. @@ -83,33 +105,6 @@ And below is a soon to be added examples list: - Use on Heroku with dpl - Use with OpenShift/kubernetes -## Assumptions - -1. We will allow to create dynamic environments from `.gitlab-ci.yml`, by allowing to specify environment variables: `review_apps_${CI_BUILD_REF_NAME}`, -1. We will use multiple deployments of the same application per environment, -1. The URL will be assigned to environment on the creation, and updated later if necessary, -1. The URL will be specified in `.gitlab-ci.yml`, possibly introducing regexp for getting an URL from build log if required, -1. We need some form to distinguish between production/staging and review app environment, -1. We don't try to manage life cycle of deployments in the first iteration, possibly we will extend a Pipeline to add jobs that will be responsible either for cleaning up or removing old deployments and closing environments. - -## Distinguish between production and review apps - -- Are dynamic environments distinguishable by the slash in `environment:url`? - -### Convention over configuration - -We would expect the environments to be of `type/name`: - -1. This would allow us to have a clear distinction between different environment types: `production/gitlab.com`, `staging/dev`, `review-apps/feature/branch`, -2. Since we use a folder structure we could group all environments by `type` and strip that from environment name, -3. We would be aware of some of these types and for example for `review-apps` show them differently in context of Merge Requests, ex. calculating `deployed ago` a little differently. -3. We could easily group all `types` across from group from all projects. - -The `type/name` also plays nice with `Variables` and `Runners`, because we can limit their usage: - -1. We could extend the resources with a field that would allow us to filter for what types it can be used, ex.: `production/*` or `review-apps/*` -2. We could limit runners to be used only by `review-apps/*`, - [app-nginx]: nginx_guide.md [ce-21971]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21971 [dynamic environments]: ../environments.md#dynamic-environments -- cgit v1.2.1 From a81546de3c512bb4b480582522dcc6121c0136b3 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 27 Oct 2016 16:03:56 +0200 Subject: Link to NGINX example project for the time being [ci skip] --- doc/ci/review_apps/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index b4abb8c0e5c..2fe41197206 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -105,7 +105,7 @@ And below is a soon to be added examples list: - Use on Heroku with dpl - Use with OpenShift/kubernetes -[app-nginx]: nginx_guide.md +[app-nginx]: https://gitlab.com/gitlab-examples/review-apps-nginx [ce-21971]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21971 [dynamic environments]: ../environments.md#dynamic-environments [environments]: ../environments.md -- cgit v1.2.1 From 35c45f7eac2392db136275a8352d3c6c68176a92 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 12:50:05 +0100 Subject: Add a prerequisites section, add some links --- doc/ci/review_apps/index.md | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 2fe41197206..aea6a8c2ea0 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -31,7 +31,16 @@ In the image above you can see that the `add-new-line` branch was successfully built and deployed under a dynamic environment and can be previewed with an also dynamically URL. ---- +The details of the Review Apps implementation depend widely on your real +technology stack and on your deployment process. The simplest case it to +deploy a simple static HTML website, but it will not be that straightforward +when your app is using a database for example. To make a branch be deployed +on a temporary instance and booting up this instance with all required software +and services automatically on the fly is not a trivial task. However, it is +doable, especially if you use Docker, or at least a configuration management +tool like Chef, Puppet, Ansible or Salt. + +## Prerequisites To get a better understanding of Review Apps, you must first learn how environments and deployments work. The following docs will help you grasp that @@ -51,11 +60,15 @@ knowledge: ## Configuration -For configuration examples see the [dynamic environments] documentation. +The configuration of Review apps depends on your technology stack and your +infrastructure. Read the [dynamic environments] documentation to understand +how to define and create them. ## Creating and destroying Review Apps -The creation and destruction of a Review App is defined in `.gitlab-ci.yml`. +The creation and destruction of a Review App is defined in `.gitlab-ci.yml` +at a job level under the `environment` keyword. + Check the [environments] documentation how to do so. ## A simple workflow @@ -63,11 +76,12 @@ Check the [environments] documentation how to do so. The process of adding Review Apps in your workflow would look like: 1. Set up the infrastructure to host and deploy the Review Apps. -1. Install and configure a Runner that does the deployment. +1. [Install][install-runner] and [configure][conf-runner] a Runner that does + the deployment. 1. Set up a job in `.gitlab-ci.yml` that uses the predefined [predefined CI environment variable][variables] `${CI_BUILD_REF_NAME}` to create dynamic environments and restrict it to run only on branches. -1. Optionally set a job that stops the Review Apps. +1. Optionally set a job that [manually stops][manual-env] the Review Apps. From there on, you would follow the branched Git flow: @@ -115,3 +129,6 @@ And below is a soon to be added examples list: [merge request]: ../../user/project/merge_requests.md [variables]: ../variables/README.md [yaml-env]: ../yaml/README.md#environment +[install-runner]: https://docs.gitlab.com/runner/install/ +[conf-runner]: https://docs.gitlab.com/runner/commands/ +[manual-env]: ../environments.md#stopping-an-environment -- cgit v1.2.1 From 11c5fcb43ce55c9b96428f440f19d459ae09166b Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 12:50:40 +0100 Subject: Fix URL to review apps docs --- doc/ci/environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index cc65f0ad8ad..1d663060241 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -485,7 +485,7 @@ Below are some links you may find interesting: - [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) -- [Review Apps](review_apps.md) Expand dynamic environments to deploy your code for every branch +- [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) [Pipelines]: pipelines.md [jobs]: yaml/README.md#jobs -- cgit v1.2.1 From ffadc93a124eb6ba03b8f21e259cd6292191ea48 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 12:50:57 +0100 Subject: Add link to environments docs --- doc/ci/yaml/README.md | 3 +++ 1 file changed, 3 insertions(+) (limited to 'doc') diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index b73cc41f95d..6fee750c709 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -541,6 +541,8 @@ same manual action multiple times. An example usage of manual actions is deployment to production. +Read more at the [environments documentation][env-manual]. + ### environment > Introduced in GitLab 8.9. @@ -1223,6 +1225,7 @@ capitalization, the commit will be created but the builds will be skipped. Visit the [examples README][examples] to see a list of examples using GitLab CI with various languages. +[env-manual]: ../environments.md#manually-deploying-to-environments [examples]: ../examples/README.md [ce-6323]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6323 [environment]: ../environments.md -- cgit v1.2.1 From 184c1489d88ac7d9a5d6c34c6bfb5b973f120870 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 12:57:08 +0100 Subject: Add Limitations sections to environments and review apps docs [ci skip] --- doc/ci/environments.md | 9 +++++++++ doc/ci/review_apps/index.md | 13 +------------ 2 files changed, 10 insertions(+), 12 deletions(-) (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 1d663060241..aed17325b3a 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -479,6 +479,15 @@ fetch line: fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` +## Limitations + +- You are limited to use only the [CI predefined variables][variables] in the + `environment: name`. Any variables defined inside `script` will not work. +- If the branch name contains special characters and you use the + `$CI_BUILD_REF_NAME` variable to dynamically create environments, there might + be complications during deployment. Follow the [issue 22849][ce-22849] for + more information. + ## Further reading Below are some links you may find interesting: diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index aea6a8c2ea0..42b5ec740d9 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -91,20 +91,9 @@ From there on, you would follow the branched Git flow: 1. Click on the link that's present in the MR related to the branch and see the changes live. -## Distinguish between production and review apps - -1. We need some form to distinguish between production/staging and review app environment, - -- Are dynamic environments distinguishable by the slash in `environment:url`? - -We would expect the environments to be of `type/name`: - -1. This would allow us to have a clear distinction between different environment types: `production/gitlab.com`, `staging/dev`, `review-apps/feature/branch`, -3. We would be aware of some of these types and for example for `review-apps` show them differently in context of Merge Requests, ex. calculating `deployed ago` a little differently. - ## Limitations -We are limited to use only [CI predefined variables][variables]. +Check the [environments limitations](../environments.md#limitations). ## Examples -- cgit v1.2.1 From 642e668c295e6758bf3498daab06414f498607a0 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 13:00:37 +0100 Subject: Add 8.14 to versions with further additions to review apps [ci skip] --- doc/ci/review_apps/index.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 42b5ec740d9..b41ae130bc2 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,9 +1,10 @@ # Getting started with Review Apps -> [Introduced][ce-21971] in GitLab 8.12. Further additions were made in GitLab 8.13. > -Inspired by [Heroku's Review Apps][heroku-apps] which itself was inspired by -[Fourchette]. +- [Introduced][ce-21971] in GitLab 8.12. Further additions were made in GitLab + 8.13 and 8.14. +- Inspired by [Heroku's Review Apps][heroku-apps] which itself was inspired by + [Fourchette]. The base of Review Apps is the [dynamic environments] which allow you to create a new environment (dynamically) for each one of your branches. -- cgit v1.2.1 From 5fa7169618e2be9956482e2feedaff651f080761 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 13:03:29 +0100 Subject: Remove ToC since it's now supported in the docs portal itself [ci skip] --- doc/administration/high_availability/redis.md | 32 ---------------------- .../high_availability/redis_source.md | 21 -------------- 2 files changed, 53 deletions(-) (limited to 'doc') diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 1fa3560e932..f532a106bc6 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -30,38 +30,6 @@ Omnibus GitLab packages. [Available configuration setups](#available-configuration-setups) section below. - - -**Table of Contents** - -- [Overview](#overview) - - [High Availability with Sentinel](#high-availability-with-sentinel) - - [Recommended setup](#recommended-setup) - - [Redis setup overview](#redis-setup-overview) - - [Sentinel setup overview](#sentinel-setup-overview) - - [Available configuration setups](#available-configuration-setups) -- [Configuring Redis HA](#configuring-redis-ha) - - [Prerequisites](#prerequisites) - - [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance) - - [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances) - - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) - - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) -- [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha) -- [Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - - [Example configuration for Redis master and Sentinel 1](#example-configuration-for-redis-master-and-sentinel-1) - - [Example configuration for Redis slave 1 and Sentinel 2](#example-configuration-for-redis-slave-1-and-sentinel-2) - - [Example configuration for Redis slave 2 and Sentinel 3](#example-configuration-for-redis-slave-2-and-sentinel-3) - - [Example configuration for the GitLab application](#example-configuration-for-the-gitlab-application) -- [Advanced configuration](#advanced-configuration) - - [Control running services](#control-running-services) -- [Troubleshooting](#troubleshooting) - - [Troubleshooting Redis replication](#troubleshooting-redis-replication) - - [Troubleshooting Sentinel](#troubleshooting-sentinel) -- [Changelog](#changelog) -- [Further reading](#further-reading) - - - ## Overview Before diving into the details of setting up Redis and Redis Sentinel for HA, diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 8558ba82d63..3629772b8af 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -17,27 +17,6 @@ If you're not sure whether this guide is for you, please refer to [Available configuration setups](redis.md#available-configuration-setups) in the Omnibus Redis HA documentation. ---- - - - -**Table of Contents** - -- [Configuring your own Redis server](#configuring-your-own-redis-server) - - [Prerequisites](#prerequisites) - - [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance) - - [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances) - - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances) - - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application) -- [Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-minimal-configuration-with-1-master-2-slaves-and-3-sentinels) - - [Example configuration for Redis master and Sentinel 1](#example-configuration-for-redis-master-and-sentinel-1) - - [Example configuration for Redis slave 1 and Sentinel 2](#example-configuration-for-redis-slave-1-and-sentinel-2) - - [Example configuration for Redis slave 2 and Sentinel 3](#example-configuration-for-redis-slave-2-and-sentinel-3) - - [Example configuration of the GitLab application](#example-configuration-of-the-gitlab-application) -- [Troubleshooting](#troubleshooting) - - - ## Configuring your own Redis server This is the section where we install and setup the new Redis instances. -- cgit v1.2.1 From 7a37e5c6ab6f96daae2612ef278899899359e271 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Thu, 17 Nov 2016 14:10:55 +0100 Subject: Mention Git strategy none [ci skip] --- doc/ci/environments.md | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'doc') diff --git a/doc/ci/environments.md b/doc/ci/environments.md index cc65f0ad8ad..91e6c042e0d 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -422,6 +422,8 @@ deploy_review: - master stop_review: + variables: + GIT_STRATEGY: none script: - echo "Remove review app" when: manual @@ -430,6 +432,10 @@ stop_review: action: stop ``` +Setting the [`GIT_STRATEGY`][git-strategy] to `none` is necessary on the +`stop_review` job so that the [GitLab Runner] won't try to checkout the code +after the branch is deleted. + >**Note:** Starting with GitLab 8.14, dynamic environments will be stopped automatically when their associated branch is deleted. @@ -445,6 +451,8 @@ You can read more in the [`.gitlab-ci.yml` reference][onstop]. > [Introduced][ce-7015] in GitLab 8.14. As we've seen in the [dynamic environments](#dynamic-environments), you can +prepend their name with a word, then followed by a `/` and finally the branch +name which is automatically defined by the `CI_BUILD_REF_NAME` variable. In short, environments that are named like `type/foo` are presented under a group named `type`. @@ -498,3 +506,5 @@ Below are some links you may find interesting: [only]: yaml/README.md#only-and-except [onstop]: yaml/README.md#environment-on_stop [ce-7015]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7015 +[gitlab runner]: https://docs.gitlab.com/runner/ +[git-strategy]: yaml/README.md#git-strategy -- cgit v1.2.1