Add latest changes from gitlab-org/gitlab@13-0-stable-ee

1 files changed, 49 insertions, 2 deletions

diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md
index 0ca25f43345..a5d0eecdc7b 100644
--- a/doc/development/sidekiq_style_guide.md
+++ b/doc/development/sidekiq_style_guide.md
@@ -82,6 +82,11 @@ As a general rule, a worker can be considered idempotent if:
 
 A good example of that would be a cache expiration worker.
 
+NOTE: **Note:**
+A job scheduled for an idempotent worker will automatically be
+[deduplicated](#deduplication) when an unstarted job with the same
+arguments is already in the queue.
+
 ### Ensuring a worker is idempotent
 
 Make sure the worker tests pass using the following shared example:
@@ -118,8 +123,50 @@ It's encouraged to only have the `idempotent!` call in the top-most worker class
 the `perform` method is defined in another class or module.
 
 NOTE: **Note:**
-Note that a cop will fail if the worker class is not marked as idempotent.
-Consider skipping the cop if you're not confident your job can safely run multiple times.
+If the worker class is not marked as idempotent, a cop will fail.
+Consider skipping the cop if you're not confident your job can safely
+run multiple times.
+
+### Deduplication
+
+When a job for an idempotent worker is enqueued while another
+unstarted job is already in the queue, GitLab drops the second
+job. The work is skipped because the same work would be
+done by the job that was scheduled first; by the time the second
+job executed, the first job would do nothing.
+
+For example, `AuthorizedProjectsWorker` takes a user ID. When the
+worker runs, it recalculates a user's authorizations. GitLab schedules
+this job each time an action potentially changes a user's
+authorizations. If the same user is added to two projects at the
+same time, the second job can be skipped if the first job hasn't
+begun, because when the first job runs, it creates the
+authorizations for both projects.
+
+GitLab doesn't skip jobs scheduled in the future, as we assume that
+the state will have changed by the time the job is scheduled to
+execute.
+
+More [deduplication strategies have been suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/195). If you are implementing a worker that
+could benefit from a different strategy, please comment in the issue.
+
+If the automatic deduplication were to cause issues in certain
+queues. This can be temporarily disabled by enabling a feature flag
+named `disable_<queue name>_deduplication`. For example to disable
+deduplication for the `AuthorizedProjectsWorker`, we would enable the
+feature flag `disable_authorized_projects_deduplication`.
+
+From chatops:
+
+```shell
+/chatops run feature set disable_authorized_projects_deduplication true
+```
+
+From the rails console:
+
+```ruby
+Feature.enable!(:disable_authorized_projects_deduplication)
+```
 
 ## Job urgency