diff options
| author | Alistair Coles <alistairncoles@gmail.com> | 2022-01-10 11:52:49 +0000 |
|---|---|---|
| committer | Alistair Coles <alistairncoles@gmail.com> | 2022-02-21 10:56:23 +0000 |
| commit | 51da2543ca532204b5b141948dde3a6216c41cf8 (patch) | |
| tree | a929a31c81b4771575756ee6711a5c9ad6d53a9e /etc | |
| parent | de888629817a2a326b6e8dc66edb0ce3168818a7 (diff) | |
| download | swift-51da2543ca532204b5b141948dde3a6216c41cf8.tar.gz | |
object-updater: defer ratelimited updates
Previously, objects updates that could not be sent immediately due to
per-container/bucket ratelimiting [1] would be skipped and re-tried
during the next updater cycle. There could potentially be a period of
time at the end of a cycle when the updater slept, having completed a
sweep of the on-disk async pending files, despite having skipped
updates during the cycle. Skipped updates would then be read from disk
again during the next cycle.
With this change the updater will defer skipped updates to an
in-memory queue (up to a configurable maximum number) until the sweep
of async pending files has completed, and then trickle out deferred
updates until the cycle's interval expires. This increases the useful
work done in the current cycle and reduces the amount of repeated disk
IO during the next cycle.
The deferrals queue is bounded in size and will evict least recently
read updates in order to accept more recently read updates. This
reduces the probablility that a deferred update has been made obsolete
by newer on-disk async pending files while waiting in the deferrals
queue.
The deferrals queue is implemented as a collection of per-bucket
queues so that updates can be drained from the queues in the order
that buckets cease to be ratelimited.
[1] Related-Change: Idef25cd6026b02c1b5c10a9816c8c6cbe505e7ed
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I95e58df9f15c5f9d552b8f4c4989a474f52262f4
Diffstat (limited to 'etc')
| -rw-r--r-- | etc/object-server.conf-sample | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/etc/object-server.conf-sample b/etc/object-server.conf-sample index 5329bca8f..9f9f64515 100644 --- a/etc/object-server.conf-sample +++ b/etc/object-server.conf-sample @@ -483,6 +483,12 @@ use = egg:swift#recon # be an integer greater than 0. # per_container_ratelimit_buckets = 1000 # +# Updates that cannot be sent due to per-container rate-limiting may be +# deferred and re-tried at the end of the updater cycle. This option constrains +# the size of the in-memory data structure used to store deferred updates. +# Must be an integer value greater than or equal to 0. +# max_deferred_updates = 10000 +# # slowdown will sleep that amount between objects. Deprecated; use # objects_per_second instead. # slowdown = 0.01 |