From e32712e3a613106121a2afe8ea1fb1f9dad93783 Mon Sep 17 00:00:00 2001 From: Sean McGivern Date: Tue, 24 Oct 2017 09:10:42 +0000 Subject: Merge branch 'bvl-circuitbreaker-backoff' into 'master' Circuitbreaker backoff and retries Closes #37383 and #38231 See merge request gitlab-org/gitlab-ce!14933 --- doc/administration/img/circuitbreaker_config.png | Bin 213210 -> 335073 bytes doc/administration/repository_storage_paths.md | 14 ++++++++++++++ doc/api/settings.md | 2 ++ 3 files changed, 16 insertions(+) (limited to 'doc') diff --git a/doc/administration/img/circuitbreaker_config.png b/doc/administration/img/circuitbreaker_config.png index 9250d38297c..e811d173634 100644 Binary files a/doc/administration/img/circuitbreaker_config.png and b/doc/administration/img/circuitbreaker_config.png differ diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index efcabd69822..96f436fa7c3 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -109,6 +109,11 @@ This can be configured from the admin interface: ![circuitbreaker configuration](img/circuitbreaker_config.png) +**Number of access attempts**: The number of attempts GitLab will make to access a +storage when probing a shard. + +**Number of failures before backing off**: The number of failures after which +GitLab will start temporarily disabling access to a storage shard on a host. **Maximum git storage failures:** The number of failures of after which GitLab will completely prevent access to the storage. The number of failures can be reset in @@ -126,6 +131,15 @@ mount is reset. **Seconds to wait for a storage access attempt:** The time in seconds GitLab will try to access storage. After this time a timeout error will be raised. +To enable the circuitbreaker for repository storage you can flip the feature flag from a rails console: + +``` +Feature.enable('git_storage_circuit_breaker') +``` + +Alternatively it can be enabled by setting `true` in the `GIT_STORAGE_CIRCUIT_BREAKER` environment variable. +This approach would be used when enabling the circuit breaker on a single host. + When storage failures occur, this will be visible in the admin interface like this: ![failing storage](img/failing_storage.png) diff --git a/doc/api/settings.md b/doc/api/settings.md index 5ec96b8a6a2..5fce6ac7810 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -96,6 +96,8 @@ PUT /application/settings | `dsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded DSA key. Default is `0` (no restriction). `-1` disables DSA keys. | `ecdsa_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ECDSA key. Default is `0` (no restriction). `-1` disables ECDSA keys. | `ed25519_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ED25519 key. Default is `0` (no restriction). `-1` disables ED25519 keys. +| `circuitbreaker_access_retries | integer | no | The number of attempts GitLab will make to access a storage. | +| `circuitbreaker_backoff_threshold | integer | no | The number of failures after which GitLab will start temporarily disabling access to a storage shard on a host. | | `circuitbreaker_failure_count_threshold` | integer | no | The number of failures of after which GitLab will completely prevent access to the storage. | | `circuitbreaker_failure_reset_time` | integer | no | Time in seconds GitLab will keep storage failure information. When no failures occur during this time, the failure information is reset. | | `circuitbreaker_failure_wait_time` | integer | no | Time in seconds GitLab will block access to a failing storage to allow it to recover. | -- cgit v1.2.1