diff options
| author | James E. Blair <jim@acmegating.com> | 2021-12-14 16:30:05 -0800 |
|---|---|---|
| committer | James E. Blair <jim@acmegating.com> | 2021-12-14 16:38:57 -0800 |
| commit | 793a3de2b81b2b917c3ce31b7170b7b54916b76f (patch) | |
| tree | d420d5056e7c752b3ea908df9e0b63f38d97557e /tests/fixtures/layouts | |
| parent | 38fb68ef50aad5009ced49476f46de9352043d22 (diff) | |
| download | zuul-793a3de2b81b2b917c3ce31b7170b7b54916b76f.tar.gz | |
Fix erroneous semaphore leak cleanup
When a scheduler starts, we run some cleanup routines in case
this is the first scheduler start after an unclean shutdown. One
of the routines detectes and releases leaked semaphores. There
were two issues with this which caused us to erroneously detect
a semaphore as leaked and delete it. Fixing either alone would
have been sufficient to avoid the issue; fixing both is correct.
1) The pipeline state object was not serializing its layout uuid.
This means that every time a new pipeline state object is created,
it is assumed that its layout is newer than what is in zookeeper,
and a re-enqueue must be performed. All existing queue items are
moved to the old_queues attribute, and a scheduler (possibly a
different one) will re-enqueue them.
2) The semaphore leak detection method did not ask for "old" queue
items. This means that if a re-enqueue is in progress, it
will only see the queue items which have been re-enqueued.
These two issues combined cause us to fairly reliably move all of
the queue items out of the active queue and then look at only the
active queue to determine which semaphores are in use (which is to
say, none) and then release those semaphores.
This patch corrects both issues.
Change-Id: Ibf5fca03bb3bd33fefdf1982b8245be0e09df567
Diffstat (limited to 'tests/fixtures/layouts')
| -rw-r--r-- | tests/fixtures/layouts/semaphore.yaml | 37 |
1 files changed, 37 insertions, 0 deletions
diff --git a/tests/fixtures/layouts/semaphore.yaml b/tests/fixtures/layouts/semaphore.yaml new file mode 100644 index 000000000..900898d6d --- /dev/null +++ b/tests/fixtures/layouts/semaphore.yaml @@ -0,0 +1,37 @@ +- pipeline: + name: check + manager: independent + trigger: + gerrit: + - event: patchset-created + success: + gerrit: + Verified: 1 + failure: + gerrit: + Verified: -1 + +- job: + name: base + parent: null + run: playbooks/base.yaml + +- job: + name: test1 + run: playbooks/check.yaml + semaphore: test-semaphore + +- job: + name: test2 + run: playbooks/check.yaml + semaphore: test-semaphore + +- semaphore: + name: test-semaphore + +- project: + name: org/project + check: + jobs: + - test1 + - test2 |