summaryrefslogtreecommitdiff
path: root/doc/development
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2020-03-02 18:07:42 +0000
committerGitLab Bot <gitlab-bot@gitlab.com>2020-03-02 18:07:42 +0000
commit7b52c7cb634ef7047d30b0337fe477bcdcedf41d (patch)
tree374ca9e908204488422046f10e340d1500780362 /doc/development
parentb375c6c05fbd03aea33a9ee9f82e678bdaa8c3cc (diff)
downloadgitlab-ce-7b52c7cb634ef7047d30b0337fe477bcdcedf41d.tar.gz
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development')
-rw-r--r--doc/development/licensing.md4
-rw-r--r--doc/development/sidekiq_style_guide.md65
-rw-r--r--doc/development/testing_guide/flaky_tests.md5
3 files changed, 38 insertions, 36 deletions
diff --git a/doc/development/licensing.md b/doc/development/licensing.md
index 2dc77b2eec8..a716db6b407 100644
--- a/doc/development/licensing.md
+++ b/doc/development/licensing.md
@@ -46,8 +46,8 @@ More detailed information on how the gem and its commands work is available in t
Libraries with the following licenses are acceptable for use:
-- [The MIT License](https://choosealicense.com/licenses/mit/) (the MIT Expat License specifically): The MIT License requires that the license itself is included with all copies of the source. It is a permissive (non-copyleft) license as defined by the Open Source Initiative.
-- [LGPL](https://choosealicense.com/licenses/lgpl-3.0/) (version 2, version 3): GPL constraints regarding modification and redistribution under the same license are not required of projects using an LGPL library, only upon modification of the LGPL-licensed library itself.
+- [MIT License](https://choosealicense.com/licenses/mit/) (the MIT Expat License specifically): The MIT License requires that the license itself is included with all copies of the source. It is a permissive (non-copyleft) license as defined by the Open Source Initiative.
+- [GNU Lesser General Public License (GNU LGPL)](https://choosealicense.com/licenses/lgpl-3.0/) (version 2, version 3): GPL constraints regarding modification and redistribution under the same license are not required of projects using an LGPL library, only upon modification of the LGPL-licensed library itself.
- [Apache 2.0 License](https://choosealicense.com/licenses/apache-2.0/): A permissive license that also provides an express grant of patent rights from contributors to users.
- [Ruby 1.8 License][ruby-1.8]: Dual-licensed under either itself or the GPLv2, defer to the Ruby License itself. Acceptable because of point 3b: "You may distribute the software in object code or binary form, provided that you do at least ONE of the following: b) accompany the distribution with the machine-readable source of the software."
- [Ruby 1.9 License][ruby-1.9]: Dual-licensed under either itself or the BSD 2-Clause License, defer to BSD 2-Clause.
diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md
index e15daab0fdb..89a05f91a9e 100644
--- a/doc/development/sidekiq_style_guide.md
+++ b/doc/development/sidekiq_style_guide.md
@@ -121,7 +121,30 @@ NOTE: **Note:**
Note that a cop will fail if the worker class is not marked as idempotent.
Consider skipping the cop if you're not confident your job can safely run multiple times.
-## Latency Sensitive Jobs
+## Job urgency
+
+Jobs can have an `urgency` attribute set, which can be `:high`,
+`:default`, or `:none`. These have the below targets:
+
+| **Urgency** | **Queue Scheduling Target** | **Execution Latency Requirement** |
+|-------------|-----------------------------|------------------------------------|
+| `:high` | 100 milliseconds | p50 of 1 second, p99 of 10 seconds |
+| `:default` | 1 minute | Maximum run time of 1 hour |
+| `:none` | None | Maximum run time of 1 hour |
+
+To set a job's urgency, use the `urgency` class method:
+
+```ruby
+class HighUrgencyWorker
+ include ApplicationWorker
+
+ urgency :high
+
+ # ...
+end
+```
+
+### Latency sensitive jobs
If a large number of background jobs get scheduled at once, queueing of jobs may
occur while jobs wait for a worker node to be become available. This is normal
@@ -140,7 +163,7 @@ of these jobs include:
When these jobs are delayed, the user may perceive the delay as a bug: for
example, they may push a branch and then attempt to create a merge request for
that branch, but be told in the UI that the branch does not exist. We deem these
-jobs to be `latency_sensitive`.
+jobs to be `urgency :high`.
Extra effort is made to ensure that these jobs are started within a very short
period of time after being scheduled. However, in order to ensure throughput,
@@ -150,31 +173,11 @@ these jobs also have very strict execution duration requirements:
1. 99% of jobs should complete within 10 seconds.
If a worker cannot meet these expectations, then it cannot be treated as a
-`latency_sensitive` worker: consider redesigning the worker, or splitting the
-work between two different workers, one with `latency_sensitive` code that
-executes quickly, and the other with non-`latency_sensitive`, which has no
+`urgency :high` worker: consider redesigning the worker, or splitting the
+work between two different workers, one with `urgency :high` code that
+executes quickly, and the other with `urgency :default`, which has no
execution latency requirements (but also has lower scheduling targets).
-This can be summed up in the following table:
-
-| **Latency Sensitivity** | **Queue Scheduling Target** | **Execution Latency Requirement** |
-|-------------------------|-----------------------------|-------------------------------------|
-| Not `latency_sensitive` | 1 minute | Maximum run time of 1 hour |
-| `latency_sensitive` | 100 milliseconds | p50 of 1 second, p99 of 10 seconds |
-
-To mark a worker as being `latency_sensitive`, use the
-`latency_sensitive_worker!` attribute, as shown in this example:
-
-```ruby
-class LatencySensitiveWorker
- include ApplicationWorker
-
- latency_sensitive_worker!
-
- # ...
-end
-```
-
## Jobs with External Dependencies
Most background jobs in the GitLab application communicate with other GitLab
@@ -194,7 +197,7 @@ the background processing cluster in several ways:
therefore we cannot guarantee the execution latencies on these jobs. Since we
cannot guarantee execution latency, we cannot ensure throughput and
therefore, in high-traffic environments, we need to ensure that jobs with
- external dependencies are separated from `latency_sensitive` jobs, to ensure
+ external dependencies are separated from high urgency jobs, to ensure
throughput on those queues.
1. Errors in jobs with external dependencies have higher alerting thresholds as
there is a likelihood that the cause of the error is external.
@@ -212,7 +215,7 @@ class ExternalDependencyWorker
end
```
-NOTE: **Note:** Note that a job cannot be both latency sensitive and have
+NOTE: **Note:** Note that a job cannot be both high urgency and have
external dependencies.
## CPU-bound and Memory-bound Workers
@@ -246,14 +249,14 @@ bespoke low concurrency, high memory fleet.
Note that memory-bound workers create heavy GC workloads, with pauses of
10-50ms. This will have an impact on the latency requirements for the
-worker. For this reason, `memory` bound, `latency_sensitive` jobs are not
+worker. For this reason, `memory` bound, `urgency :high` jobs are not
permitted and will fail CI. In general, `memory` bound workers are
discouraged, and alternative approaches to processing the work should be
considered.
-If a worker needs large amounts of both memory and CPU time, it should be marked as
-memory-bound, due to the above restrction on latency-sensitive memory-bound
-workers.
+If a worker needs large amounts of both memory and CPU time, it should
+be marked as memory-bound, due to the above restrction on high urgency
+memory-bound workers.
## Declaring a Job as CPU-bound
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index f39151cfa59..ef8676ddf32 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -44,9 +44,8 @@ On our CI, we use [rspec-retry] to automatically retry a failing example a few
times (see [`spec/spec_helper.rb`] for the precise retries count).
We also use a home-made `RspecFlaky::Listener` listener which records flaky
-examples in a JSON report file on `master` (`retrieve-tests-metadata` and `update-tests-metadata` jobs), and warns when a new flaky example
-is detected in any other branch (`flaky-examples-check` job). In the future, the
-`flaky-examples-check` job will not be allowed to fail.
+examples in a JSON report file on `master` (`retrieve-tests-metadata` and
+`update-tests-metadata` jobs).
This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021>.