diff options
Diffstat (limited to 'doc/development/sidekiq_style_guide.md')
| -rw-r--r-- | doc/development/sidekiq_style_guide.md | 51 |
1 files changed, 49 insertions, 2 deletions
diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 0ca25f43345..a5d0eecdc7b 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -82,6 +82,11 @@ As a general rule, a worker can be considered idempotent if: A good example of that would be a cache expiration worker. +NOTE: **Note:** +A job scheduled for an idempotent worker will automatically be +[deduplicated](#deduplication) when an unstarted job with the same +arguments is already in the queue. + ### Ensuring a worker is idempotent Make sure the worker tests pass using the following shared example: @@ -118,8 +123,50 @@ It's encouraged to only have the `idempotent!` call in the top-most worker class the `perform` method is defined in another class or module. NOTE: **Note:** -Note that a cop will fail if the worker class is not marked as idempotent. -Consider skipping the cop if you're not confident your job can safely run multiple times. +If the worker class is not marked as idempotent, a cop will fail. +Consider skipping the cop if you're not confident your job can safely +run multiple times. + +### Deduplication + +When a job for an idempotent worker is enqueued while another +unstarted job is already in the queue, GitLab drops the second +job. The work is skipped because the same work would be +done by the job that was scheduled first; by the time the second +job executed, the first job would do nothing. + +For example, `AuthorizedProjectsWorker` takes a user ID. When the +worker runs, it recalculates a user's authorizations. GitLab schedules +this job each time an action potentially changes a user's +authorizations. If the same user is added to two projects at the +same time, the second job can be skipped if the first job hasn't +begun, because when the first job runs, it creates the +authorizations for both projects. + +GitLab doesn't skip jobs scheduled in the future, as we assume that +the state will have changed by the time the job is scheduled to +execute. + +More [deduplication strategies have been suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/195). If you are implementing a worker that +could benefit from a different strategy, please comment in the issue. + +If the automatic deduplication were to cause issues in certain +queues. This can be temporarily disabled by enabling a feature flag +named `disable_<queue name>_deduplication`. For example to disable +deduplication for the `AuthorizedProjectsWorker`, we would enable the +feature flag `disable_authorized_projects_deduplication`. + +From chatops: + +```shell +/chatops run feature set disable_authorized_projects_deduplication true +``` + +From the rails console: + +```ruby +Feature.enable!(:disable_authorized_projects_deduplication) +``` ## Job urgency |