diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-03-02 18:07:42 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-03-02 18:07:42 +0000 |
commit | 7b52c7cb634ef7047d30b0337fe477bcdcedf41d (patch) | |
tree | 374ca9e908204488422046f10e340d1500780362 /doc/development | |
parent | b375c6c05fbd03aea33a9ee9f82e678bdaa8c3cc (diff) | |
download | gitlab-ce-7b52c7cb634ef7047d30b0337fe477bcdcedf41d.tar.gz |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development')
-rw-r--r-- | doc/development/licensing.md | 4 | ||||
-rw-r--r-- | doc/development/sidekiq_style_guide.md | 65 | ||||
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 5 |
3 files changed, 38 insertions, 36 deletions
diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 2dc77b2eec8..a716db6b407 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -46,8 +46,8 @@ More detailed information on how the gem and its commands work is available in t Libraries with the following licenses are acceptable for use: -- [The MIT License](https://choosealicense.com/licenses/mit/) (the MIT Expat License specifically): The MIT License requires that the license itself is included with all copies of the source. It is a permissive (non-copyleft) license as defined by the Open Source Initiative. -- [LGPL](https://choosealicense.com/licenses/lgpl-3.0/) (version 2, version 3): GPL constraints regarding modification and redistribution under the same license are not required of projects using an LGPL library, only upon modification of the LGPL-licensed library itself. +- [MIT License](https://choosealicense.com/licenses/mit/) (the MIT Expat License specifically): The MIT License requires that the license itself is included with all copies of the source. It is a permissive (non-copyleft) license as defined by the Open Source Initiative. +- [GNU Lesser General Public License (GNU LGPL)](https://choosealicense.com/licenses/lgpl-3.0/) (version 2, version 3): GPL constraints regarding modification and redistribution under the same license are not required of projects using an LGPL library, only upon modification of the LGPL-licensed library itself. - [Apache 2.0 License](https://choosealicense.com/licenses/apache-2.0/): A permissive license that also provides an express grant of patent rights from contributors to users. - [Ruby 1.8 License][ruby-1.8]: Dual-licensed under either itself or the GPLv2, defer to the Ruby License itself. Acceptable because of point 3b: "You may distribute the software in object code or binary form, provided that you do at least ONE of the following: b) accompany the distribution with the machine-readable source of the software." - [Ruby 1.9 License][ruby-1.9]: Dual-licensed under either itself or the BSD 2-Clause License, defer to BSD 2-Clause. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e15daab0fdb..89a05f91a9e 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -121,7 +121,30 @@ NOTE: **Note:** Note that a cop will fail if the worker class is not marked as idempotent. Consider skipping the cop if you're not confident your job can safely run multiple times. -## Latency Sensitive Jobs +## Job urgency + +Jobs can have an `urgency` attribute set, which can be `:high`, +`:default`, or `:none`. These have the below targets: + +| **Urgency** | **Queue Scheduling Target** | **Execution Latency Requirement** | +|-------------|-----------------------------|------------------------------------| +| `:high` | 100 milliseconds | p50 of 1 second, p99 of 10 seconds | +| `:default` | 1 minute | Maximum run time of 1 hour | +| `:none` | None | Maximum run time of 1 hour | + +To set a job's urgency, use the `urgency` class method: + +```ruby +class HighUrgencyWorker + include ApplicationWorker + + urgency :high + + # ... +end +``` + +### Latency sensitive jobs If a large number of background jobs get scheduled at once, queueing of jobs may occur while jobs wait for a worker node to be become available. This is normal @@ -140,7 +163,7 @@ of these jobs include: When these jobs are delayed, the user may perceive the delay as a bug: for example, they may push a branch and then attempt to create a merge request for that branch, but be told in the UI that the branch does not exist. We deem these -jobs to be `latency_sensitive`. +jobs to be `urgency :high`. Extra effort is made to ensure that these jobs are started within a very short period of time after being scheduled. However, in order to ensure throughput, @@ -150,31 +173,11 @@ these jobs also have very strict execution duration requirements: 1. 99% of jobs should complete within 10 seconds. If a worker cannot meet these expectations, then it cannot be treated as a -`latency_sensitive` worker: consider redesigning the worker, or splitting the -work between two different workers, one with `latency_sensitive` code that -executes quickly, and the other with non-`latency_sensitive`, which has no +`urgency :high` worker: consider redesigning the worker, or splitting the +work between two different workers, one with `urgency :high` code that +executes quickly, and the other with `urgency :default`, which has no execution latency requirements (but also has lower scheduling targets). -This can be summed up in the following table: - -| **Latency Sensitivity** | **Queue Scheduling Target** | **Execution Latency Requirement** | -|-------------------------|-----------------------------|-------------------------------------| -| Not `latency_sensitive` | 1 minute | Maximum run time of 1 hour | -| `latency_sensitive` | 100 milliseconds | p50 of 1 second, p99 of 10 seconds | - -To mark a worker as being `latency_sensitive`, use the -`latency_sensitive_worker!` attribute, as shown in this example: - -```ruby -class LatencySensitiveWorker - include ApplicationWorker - - latency_sensitive_worker! - - # ... -end -``` - ## Jobs with External Dependencies Most background jobs in the GitLab application communicate with other GitLab @@ -194,7 +197,7 @@ the background processing cluster in several ways: therefore we cannot guarantee the execution latencies on these jobs. Since we cannot guarantee execution latency, we cannot ensure throughput and therefore, in high-traffic environments, we need to ensure that jobs with - external dependencies are separated from `latency_sensitive` jobs, to ensure + external dependencies are separated from high urgency jobs, to ensure throughput on those queues. 1. Errors in jobs with external dependencies have higher alerting thresholds as there is a likelihood that the cause of the error is external. @@ -212,7 +215,7 @@ class ExternalDependencyWorker end ``` -NOTE: **Note:** Note that a job cannot be both latency sensitive and have +NOTE: **Note:** Note that a job cannot be both high urgency and have external dependencies. ## CPU-bound and Memory-bound Workers @@ -246,14 +249,14 @@ bespoke low concurrency, high memory fleet. Note that memory-bound workers create heavy GC workloads, with pauses of 10-50ms. This will have an impact on the latency requirements for the -worker. For this reason, `memory` bound, `latency_sensitive` jobs are not +worker. For this reason, `memory` bound, `urgency :high` jobs are not permitted and will fail CI. In general, `memory` bound workers are discouraged, and alternative approaches to processing the work should be considered. -If a worker needs large amounts of both memory and CPU time, it should be marked as -memory-bound, due to the above restrction on latency-sensitive memory-bound -workers. +If a worker needs large amounts of both memory and CPU time, it should +be marked as memory-bound, due to the above restrction on high urgency +memory-bound workers. ## Declaring a Job as CPU-bound diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index f39151cfa59..ef8676ddf32 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -44,9 +44,8 @@ On our CI, we use [rspec-retry] to automatically retry a failing example a few times (see [`spec/spec_helper.rb`] for the precise retries count). We also use a home-made `RspecFlaky::Listener` listener which records flaky -examples in a JSON report file on `master` (`retrieve-tests-metadata` and `update-tests-metadata` jobs), and warns when a new flaky example -is detected in any other branch (`flaky-examples-check` job). In the future, the -`flaky-examples-check` job will not be allowed to fail. +examples in a JSON report file on `master` (`retrieve-tests-metadata` and +`update-tests-metadata` jobs). This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021>. |