diff options
Diffstat (limited to 'doc/development/chaos_endpoints.md')
| -rw-r--r-- | doc/development/chaos_endpoints.md | 48 |
1 files changed, 24 insertions, 24 deletions
diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 63218af857d..9104c01c980 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -1,14 +1,14 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Generating chaos in a test GitLab instance As [Werner Vogels](https://twitter.com/Werner), the CTO at Amazon Web Services, famously put it, **Everything fails, all the time**. -As a developer, it's as important to consider the failure modes in which your software will operate as much as normal operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users and a full site outage that affects all users for an extended period. +As a developer, it's as important to consider the failure modes in which your software may operate as much as normal operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users, and a full site outage that affects all users for an extended period. To paraphrase [Tolstoy](https://en.wikipedia.org/wiki/Anna_Karenina_principle), _all happy servers are alike, but all failing servers are failing in their own way_. Luckily, there are ways we can attempt to simulate these failure modes, and the chaos endpoints are tools for assisting in this process. @@ -24,7 +24,7 @@ Currently, there are four endpoints for simulating the following conditions: For obvious reasons, these endpoints are not enabled by default on `production`. They are enabled by default on **development** environments. -DANGER: **Warning:** +WARNING: It is required that you secure access to the chaos endpoints using a secret token. You should not enable them in production unless you absolutely know what you're doing. @@ -40,17 +40,17 @@ Replace `secret` with your own secret token. ## Invoking chaos -Once you have enabled the chaos endpoints and restarted the application, you can start testing using the endpoints. +After you have enabled the chaos endpoints and restarted the application, you can start testing using the endpoints. -By default, when invoking a chaos endpoint, the web worker process which receives the request will handle it. This means, for example, that if the Kill -operation is invoked, the Puma or Unicorn worker process handling the request will be killed. To test these operations in Sidekiq, the `async` parameter on -each endpoint can be set to `true`. This will run the chaos process in a Sidekiq worker. +By default, when invoking a chaos endpoint, the web worker process which receives the request handles it. This means, for example, that if the Kill +operation is invoked, the Puma or Unicorn worker process handling the request is killed. To test these operations in Sidekiq, the `async` parameter on +each endpoint can be set to `true`. This runs the chaos process in a Sidekiq worker. ## Memory leaks To simulate a memory leak in your application, use the `/-/chaos/leakmem` endpoint. -The memory is not retained after the request finishes. After the request has completed, the Ruby garbage collector will attempt to recover the memory. +The memory is not retained after the request finishes. After the request has completed, the Ruby garbage collector attempts to recover the memory. ```plaintext GET /-/chaos/leakmem @@ -66,8 +66,8 @@ GET /-/chaos/leakmem?memory_mb=1024&duration_s=50&async=true | `async` | boolean | no | Set to true to leak memory in a Sidekiq background worker process | ```shell -curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10 --header 'X-Chaos-Secret: secret' -curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=secret +curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=secret" ``` ## CPU spin @@ -85,12 +85,12 @@ GET /-/chaos/cpu_spin?duration_s=50&async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------- | -| `duration_s` | integer | no | Duration, in seconds, that the core will be used. Defaults to 30s | +| `duration_s` | integer | no | Duration, in seconds, that the core is used. Defaults to 30s | | `async` | boolean | no | Set to true to consume CPU in a Sidekiq background worker process | ```shell -curl http://localhost:3000/-/chaos/cpu_spin?duration_s=60 --header 'X-Chaos-Secret: secret' -curl http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret +curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret" ``` ## DB spin @@ -110,19 +110,19 @@ GET /-/chaos/db_spin?duration_s=50&async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------------- | | `interval_s` | float | no | Interval, in seconds, for every DB request. Defaults to 1s | -| `duration_s` | integer | no | Duration, in seconds, that the core will be used. Defaults to 30s | +| `duration_s` | integer | no | Duration, in seconds, that the core is used. Defaults to 30s | | `async` | boolean | no | Set to true to perform the operation in a Sidekiq background worker process | ```shell -curl http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60 --header 'X-Chaos-Secret: secret' -curl http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60&token=secret +curl "http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60&token=secret" ``` ## Sleep -This endpoint is similar to the CPU Spin endpoint but simulates off-processor activity, such as network calls to backend services. It will sleep for a given duration_s. +This endpoint is similar to the CPU Spin endpoint but simulates off-processor activity, such as network calls to backend services. It sleeps for a given `duration_s`. -As with the CPU Spin endpoint, this may lead to your request timing out if duration_s exceeds the configured limit. +As with the CPU Spin endpoint, this may lead to your request timing out if `duration_s` exceeds the configured limit. ```plaintext GET /-/chaos/sleep @@ -132,17 +132,17 @@ GET /-/chaos/sleep?duration_s=50&async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ---------------------------------------------------------------------- | -| `duration_s` | integer | no | Duration, in seconds, that the request will sleep for. Defaults to 30s | +| `duration_s` | integer | no | Duration, in seconds, that the request sleeps for. Defaults to 30s | | `async` | boolean | no | Set to true to sleep in a Sidekiq background worker process | ```shell -curl http://localhost:3000/-/chaos/sleep?duration_s=60 --header 'X-Chaos-Secret: secret' -curl http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret +curl "http://localhost:3000/-/chaos/sleep?duration_s=60" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret" ``` ## Kill -This endpoint will simulate the unexpected death of a worker process using a `kill` signal. +This endpoint simulates the unexpected death of a worker process using a `kill` signal. Because this endpoint uses the `KILL` signal, the worker isn't given an opportunity to cleanup or shutdown. @@ -157,6 +157,6 @@ GET /-/chaos/kill?async=true | `async` | boolean | no | Set to true to kill a Sidekiq background worker process | ```shell -curl http://localhost:3000/-/chaos/kill --header 'X-Chaos-Secret: secret' -curl http://localhost:3000/-/chaos/kill?token=secret +curl "http://localhost:3000/-/chaos/kill" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/kill?token=secret" ``` |