summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
* | | | | | Merge "Add Gerrit Verified/Workflow submit reqs to tutorial"Zuul2022-12-011-0/+10
|\ \ \ \ \ \
| * | | | | | Add Gerrit Verified/Workflow submit reqs to tutorialJames E. Blair2022-11-281-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Gerrit 3.7 forbids setting the MaxWithBlock function on labels, so replace that with these equivalent submit requirements in our tutorial example configuration. Change-Id: Iadca7d36a342eb1a890a7b83a5e08938a2b52e20
* | | | | | | Merge "Cleanup leaked git index.lock files on checkout"Zuul2022-12-013-34/+21
|\ \ \ \ \ \ \
| * | | | | | | Cleanup leaked git index.lock files on checkoutSimon Westphahl2022-11-153-34/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the git command crashes or is aborted due to a timeout we might end up with a leaked index.lock file in the affected repository. This has the effect that all subsequent git operations that try to create the lock will fail. Since Zuul maintains a separate lock for serializing operations on a repositotry, we can be sure that the lock file was leaked in a previous operation and can be removed safely. Unable to checkout 8a87ff7cc0d0c73ac14217b653f9773a7cfce3a7 Traceback (most recent call last): File "/opt/zuul/lib/python3.10/site-packages/zuul/merger/merger.py", line 1045, in _mergeChange repo.checkout(ref, zuul_event_id=zuul_event_id) File "/opt/zuul/lib/python3.10/site-packages/zuul/merger/merger.py", line 561, in checkout repo.head.reset(working_tree=True) File "/opt/zuul/lib/python3.10/site-packages/git/refs/head.py", line 82, in reset self.repo.git.reset(mode, commit, '--', paths, **kwargs) File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 542, in <lambda> return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 1005, in _call_process return self.execute(call, **exec_kwargs) File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 822, in execute raise GitCommandError(command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git reset --hard HEAD -- stderr: 'fatal: Unable to create '/var/lib/zuul/merger-git/github/foo/foo%2Fbar/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue.' Change-Id: I97334383df476809c39e0d03b1af50cb59ee0cc7
* | | | | | | | Merge "Add an upgrading document"Zuul2022-12-012-0/+46
|\ \ \ \ \ \ \ \ | |_|_|/ / / / / |/| | | | | | |
| * | | | | | | Add an upgrading documentJames E. Blair2022-11-072-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This outlines our baseline assumptions regarding upgrading. Change-Id: I2f63b3b3d643fc2aae4f35de8206d4081afa1494
* | | | | | | | Merge "Unpin JWT and use integer IAT values"Zuul2022-11-296-49/+49
|\ \ \ \ \ \ \ \
| * | | | | | | | Unpin JWT and use integer IAT valuesJames E. Blair2022-11-156-49/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | PyJWT 2.6.0 began performing validation of iat (issued at) claims in https://github.com/jpadilla/pyjwt/commit/9cb9401cc579f11dbb17181e8713f061f8e40ed4 I believe the intent of RFC7519 is to support any numeric values (including floating point) for iat, nbf, and exp, however, the PyJWT library has made the assumption that the values should be integers, and therefore when we supply an iat with decimal seconds, PyJWT will round down when validating the value. In our unit tests, this can cause validation errors. In order to avoid any issues, we will round down the times that we supply when generating JWT tokens and supply them as integers in accordance with the robustness principle. Change-Id: Ia8341b4d5de827e2df8878f11f2d1f52a1243cd4
* | | | | | | | | Merge "Update following changes logging"Zuul2022-11-292-1/+4
|\ \ \ \ \ \ \ \ \
| * | | | | | | | | Update following changes loggingClark Boylan2022-11-222-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A user pointed out that approving a parent change [0] did not enqueue a child change [1] that had all necessary votes. On inspection of the logs [2] it isn't clear why this happened as at least one other change was found. One thing I noticed is that 'Depends-on' instead of 'Depends-On' was used but both Gerrit and zuul seem to treat this string case insensitively so that shouldn't matter. Update our logging to aid in further debugging should this happen again. In particular log already known following changes which comes from our change cache and newly discovered following changes which comes from the code review system. Hopefully this will help identify where we're not finding the info we expect. Additionally record the number of possible depending changes found by Gerrit. This should give us an indication for whether or not Gerrit returning the information we expect. [0] https://review.opendev.org/c/openstack/devstack/+/860795 [1] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/857031 [2] https://paste.opendev.org/show/b6P1Uy2VjAMZOqtBbwyz/ Change-Id: Iad7fbbc3476ef1ddfd366e93b8505ed0fbdc1dc0
* | | | | | | | | | Merge "Handle Github changed files errors more broadly"Zuul2022-11-291-2/+2
|\ \ \ \ \ \ \ \ \ \
| * | | | | | | | | | Handle Github changed files errors more broadlySimon Westphahl2022-11-151-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In more recent versions Github seems to return different/additional exceptions when it can't generate the diff for changed files fast enough. To make this more robust we will catch all response errors from Github and fall back to getting the list of changed files via the mergers in those cases. 2022-11-14 15:52:02,718 ERROR zuul.GithubEventProcessor: [e: 43745d90-6434-11ed-9182-8080263aeb92] Exception when processing event: Traceback (most recent call last): File "/opt/zuul/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 351, in run self._process_event() File "/opt/zuul/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 413, in _process_event change = self.connection._getChange(change_key, File "/opt/zuul/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1428, in _getChange pull = self.getPull(change.project.name, change.number, File "/opt/zuul/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1823, in getPull for pr_file in probj.files(): File "/opt/zuul/lib/python3.10/site-packages/github3/structs.py", line 98, in __iter__ json = self._get_json(response) File "/opt/zuul/lib/python3.10/site-packages/github3/structs.py", line 145, in _get_json return self._json(response, 200) File "/opt/zuul/lib/python3.10/site-packages/github3/models.py", line 161, in _json raise exceptions.error_for(response) github3.exceptions.UnprocessableEntity: 422 Server Error: Sorry, this diff is taking too long to generate. Change-Id: Idba1775d9d727fcccb7dc5b3a595a4875b9a4ec1
* | | | | | | | | | | Merge "Avoid replacing timer apscheduler jobs"Zuul2022-11-292-23/+62
|\ \ \ \ \ \ \ \ \ \ \ | |_|_|_|_|_|_|_|/ / / |/| | | | | | | | | |
| * | | | | | | | | | Avoid replacing timer apscheduler jobsJames E. Blair2022-11-212-23/+62
| | |_|_|_|_|_|_|_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a timer trigger is configured with a large jitter and a reconfiguration happens within the jitter time, it is possible to miss an expected scheduled trigger because the act of reconfiguration removes and re-adds all of a tenant's timer trigger apscheduler jobs. To avoid this situation, we will try to preserve any jobs with identical configurations. Change-Id: I5d3a4d7be891fcb4b9a3f268ee347f2069aaded3
* | | | | | | | | | tutorial: remove warn parameter from 'shell' ansible moduleGuillaume CHAUVEL2022-11-242-2/+0
| |_|_|_|_|_|/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This parameter is removed in ansible-core 2.14 [1][2] and an error is raised during tutorial run. FAILED! => {"changed": false, "msg": "Unsupported parameters for (ansible.legacy.command) module: warn. [1] https://github.com/ansible/ansible/issues/77394 [2] https://github.com/ansible/ansible/pull/77411 Change-Id: I7ee86f019eeac14ddb22abc7924d0a10b051750e
* | | | | | | | | tutorial: Update gerrit configuration to gerrit 3.7Guillaume CHAUVEL2022-11-241-4/+2
| |_|/ / / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | docker.io/gerritcodereview/gerrit:latest has been updated to gerrit 3.7 it introduces breaking changes [1][2] that require to update how the configuration is modified during tutorial run. ERROR: ... Parameter 'label.Code-Review.copyMinScore' is deprecated and cannot be set, use 'is:MIN' in 'label.Code-Review.copyCondition' instead. ERROR: ... Parameter 'label.Code-Review.copyAllScoresOnTrivialRebase' is deprecated and cannot be set, use 'changekind:TRIVIAL_REBASE' in 'label.Code-Review.copyCondition' instead. ERROR: ... Value 'MaxWithBlock' of 'label.Verified.function' is not allowed and cannot be set. Label functions can only be set to {NO_BLOCK, NO_OP, PATCH_SET_LOCK}. Use submit requirements instead of label functions. [1] https://www.gerritcodereview.com/3.7.html#breaking-changes [2] https://gerrit-review.googlesource.com/c/gerrit/+/334467 Change-Id: I2f27d0e99470c3baee82abc126cac72132a2da48
* | | | | | | | Merge "docs: Clarify zuul_console behaviour for Kubernetes job nodes"Zuul2022-11-221-0/+7
|\ \ \ \ \ \ \ \
| * | | | | | | | docs: Clarify zuul_console behaviour for Kubernetes job nodesMichael Kelly2022-11-211-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change is related to Id3669418189f1083a2fb690ada0b60043a77b1d6 and clarifies the zuul_console connectivity behaviour when dealing with Kubernetes-based job nodes. Change-Id: I7191631dc54071d158657816a8cc10335e122df5
* | | | | | | | | Merge "docs: remove orphaned howto pages"Zuul2022-11-2212-879/+35
|\ \ \ \ \ \ \ \ \ | |/ / / / / / / / | | / / / / / / / | |/ / / / / / / |/| | | | | | |
| * | | | | | | docs: remove orphaned howto pagesIan Wienand2022-11-1112-879/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The zuul-from-scratch page was removed with I3c6327c9bc1a924f076ded06afc0afc4e3024384, but all these files it linked to were left behind. At first glance this seemed a bit odd, because sphinx should warn when pages aren't linked to from a TOC. It took me a while to realise these pages were already marked with :orphan: at the top which stopped this happening. So they really are orphans now, but we haven't noticed. This appears to go back well before the zuul-from-scratch removal to some of the original organisation several years ago in I206a2acf09eb8a2871ec61a00226654c798bb3eb -- it's not clear if this flag was intended to be left in the files or was a temporary step; but it seems that as we've gone on we've copied it into all the other files as they got created too. Most of this is all old and part of the bit-rot as described in the original zuul-from-scratch removal. The only recent part is some console streaming docs added with I5bfb61323bf3219168d4d014cbb9703eed230e71 -- upon reevaluating this I've moved it into the executor docs where it seems to fit. The other orphaned files are removed. Change-Id: Id3669418189f1083a2fb690ada0b60043a77b1d6
* | | | | | | | Check if Github detected a merge conflict for a PRSimon Westphahl2022-11-188-4/+37
| |_|/ / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Github uses libgit2 to compute merges without requiring a worktree [0]. In some cases this can lead to Github detecting a merge conflict while for Zuul the PR merges fine. To prevent such changes from entering dependent pipelines and e.g. cause a gate reset, we'll also check if Github detected any merge conflicts. [0] https://github.blog/2022-10-03-highlights-from-git-2-38/ Change-Id: I22275f24c903a8548bb0ef6c32a2e15ba9eadac8
* | | | | | | Merge "Detect errors with non-permitted parent jobs"Zuul2022-11-155-41/+155
|\ \ \ \ \ \ \ | |_|/ / / / / |/| | | | | |
| * | | | | | Detect errors with non-permitted parent jobsJames E. Blair2022-10-265-41/+155
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We currently only detect some errors with job parents when freezing the job graph. This is due to the vagaries of job variants, where it is possible for a variant on one branch to be okay while one on another branch is an error. If the erroneous job doesn't match, then there is no harm. However, in the typical case where there is only one variant or multiple variants are identical, it is possible for us to detect during config loading a situation where we know the job graph generation will later fail. This change adds that analysis and raises errors early. This can save users quite a bit of time, and since variants are typically added one at a time, may even prevent users from getting into abiguous situations which could only be detected when freezing the job graph. Change-Id: Ie8b9ee7758c94788ee7bc05947ddd97d9fa8e075
* | | | | | | Merge "executor: Skip line mapping for special Gerrit files"Zuul2022-11-143-6/+28
|\ \ \ \ \ \ \
| * | | | | | | executor: Skip line mapping for special Gerrit filesJoshua Watt2022-11-033-6/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the file being comment on is the Gerrit special file starting with "/" (e.g. /COMMIT_MSG), no line mapping transformation should be done, otherwise strange errors like: Job: unable to map line for file comments: stderr: 'fatal: '/COMMIT_MSG' is outside repository at '...' will show up after the job has run. Change-Id: Id89041dc7d8bf3f6c956d85b38355053ff0fd707
* | | | | | | | Merge "Refactor merge mode name lookup"Zuul2022-11-144-12/+14
|\ \ \ \ \ \ \ \ | |_|_|_|_|/ / / |/| | | | | | / | | |_|_|_|_|/ | |/| | | | |
| * | | | | | Refactor merge mode name lookupJames E. Blair2022-11-104-12/+14
| | |_|/ / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | This is repeated in a few places, centralize it. Change-Id: I7bbed1f5f9faad31affa71ef17fbfc1740c54db8
* | | | | | Smooth out upgrades to playbook semaphoresJeremy Stanley2022-11-121-1/+3
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If running a scheduler on code prior to change I90f5e0f570ef6c4b0986b0143318a78ddc27bbde when upgrading an executor to a version which includes it, playbooks will lack a semaphores list. Assume it's empty in that case, in order to avoid causing transient build failures during a rolling upgrade of Zuul services. Change-Id: I17d08f40e9feba62923c6f12a1d4f8b2a05f9cae
* | | | | Merge "Add extra logging around tenant reconfiguration events"Zuul2022-11-101-1/+5
|\ \ \ \ \
| * | | | | Add extra logging around tenant reconfiguration eventsJames E. Blair2022-10-031-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While tracking down a problem, I observed a tenant reconfiguration event apparently go missing. This change adds log messages that might aid debugging in such a situation. Change-Id: Ia975f83b6bba8d48994cd8c39427be2db3d73262
* | | | | | Merge "Parallelize some pipeline refresh ops"Zuul2022-11-1016-284/+424
|\ \ \ \ \ \
| * | | | | | Parallelize some pipeline refresh opsJames E. Blair2022-11-0916-284/+424
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We may be able to speed up pipeline refreshes in cases where there are large numbers of items or jobs/builds by parallelizing ZK reads. Quick refresher: the ZK protocol is async, and kazoo uses a queue to send operations to a single thread which manages IO. We typically call synchronous kazoo client methods which wait for the async result before returning. Since this is all thread-safe, we can attempt to fill the kazoo pipe by having multiple threads call the synchronous kazoo methods. If kazoo is waiting on IO for an earlier call, it will be able to start a later request simultaneously. Quick aside: it would be difficult for us to use the async methods directly since our overall code structure is still ordered and effectively single threaded (we need to load a QueueItem before we can load the BuildSet and the Builds, etc). Thus it makes the most sense for us to retain our ordering by using a ThreadPoolExecutor to run some operations in parallel. This change parallelizes loading QueueItems within a ChangeQueue, and also Builds/Jobs within a BuildSet. These are the points in a pipeline refresh tree which potentially have the largest number of children and could benefit the most from the change, especially if the ZK server has some measurable latency. Change-Id: I0871cc05a2d13e4ddc4ac284bd67e5e3003200ad
* | | | | | | Merge "Add playbook semaphores"Zuul2022-11-1019-156/+788
|\ \ \ \ \ \ \ | | |_|_|_|/ / | |/| | | | |
| * | | | | | Add playbook semaphoresJames E. Blair2022-11-0719-156/+788
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds the ability to specify that the Zuul executor should acquire a semaphore before running an individual playbook. This is useful for long running jobs which need exclusive access to a resources for only a small amount of time. Change-Id: I90f5e0f570ef6c4b0986b0143318a78ddc27bbde
* | | | | | | Merge "Update README to point to Matrix, not IRC"Zuul2022-11-091-3/+4
|\ \ \ \ \ \ \
| * | | | | | | Update README to point to Matrix, not IRCTim Burke2022-11-081-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Change-Id: I30344782fe1d65edd7f2985097ab040c51820097
* | | | | | | | Merge "Fix race in merger shutdown"Zuul2022-11-092-6/+13
|\ \ \ \ \ \ \ \ | | |_|/ / / / / | |/| | | | | |
| * | | | | | | Fix race in merger shutdownJames E. Blair2022-11-082-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can disconnect from ZK while the merger is still running which can have some adverse effects and cause tests to never exit. This moves the zk disconnect in the merger to the join method so that we ensure that we have exited the main loop. It also adds some improved logging so that not everything just says "Stopped". Change-Id: I459af85ac70ecf1f61645466d0eddc63c7e61ff9
* | | | | | | | Merge "Don't refresh the pipeline state during reconfig"Zuul2022-11-094-21/+50
|\ \ \ \ \ \ \ \ | |/ / / / / / / | | / / / / / / | |/ / / / / / |/| | | | | |
| * | | | | | Don't refresh the pipeline state during reconfigSimon Westphahl2022-10-284-21/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When resetting the pipeline state after a reconfig, we will no longer refresh the whole pipeline structure and just update the necessary fields of the top-level pipeline state. This can speed up reconfigurations significantly for tenants that have a lot of active items especially with large jobs graphs. We also add a log message right before we load the tenant after we've aquired the tenant write lock. This change also fixes a small bug where the znode version is initialized to None instead of -1 when no zstat exists. Change-Id: I7aa38569cc2f8a28758a108d4b4d8b86967da985
* | | | | | | Merge "Update help text for zkprofile command"Zuul2022-11-091-1/+1
|\ \ \ \ \ \ \
| * | | | | | | Update help text for zkprofile commandJames E. Blair2022-11-081-1/+1
| |/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The help text for this command should indicate that it toggles rather than simply enables profiling. Change-Id: I65cc16b745f086450463d44fe62c874471349708
* | | | | | | Merge "Add a zkprofile scheduler debug command"Zuul2022-11-093-0/+77
|\ \ \ \ \ \ \ | |/ / / / / /
| * | | | | | Add a zkprofile scheduler debug commandJames E. Blair2022-10-273-0/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds a temporary debug command to the zuul-scheduler process. It allows an operator to enable detailed ZK request profiling for a given tenant-pipeline. This will be used to identify opportunities for further optimization. Change-Id: Id6e0ee3ffc78874e91ebcdbfe14269c93af958cd
* | | | | | | Merge "Remove legacy semaphore handling"Zuul2022-11-072-125/+5
|\ \ \ \ \ \ \ | | |/ / / / / | |/| | | | |
| * | | | | | Remove legacy semaphore handlingJames E. Blair2022-10-312-125/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This model api revision was added after 5.0.0, which means the backwards compatability code is safe to remove afte 6.0.0. Since we're at 7.0.0+ now, remove this code to simplify it in preparation for upcoming changes. Change-Id: Icb84e7ea4c0f5a3eef9970fda4810664739c9725
* | | | | | | Merge "Initialize tracing module in model tests"Zuul2022-11-071-0/+4
|\ \ \ \ \ \ \ | |/ / / / / /
| * | | | | | Initialize tracing module in model testsJames E. Blair2022-10-311-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The model tests exercise some tracing code (by creating buildsets, etc) but do not have the tracing module initialized because they are not full Zuul system tests and do not inherit from ZuulTestCase. Without initialization, we get different behavior from opentelemetry (we have not provided our global tracer object, etc), which makes these tests perform differently than they would in production. They also fail when run in isolation because of these differences. They likely only succeed when we run the whole test suite because of leakage of the opentelemetry module initialization. To correct this, initialize the tracing library (but with no configuration, so we still are not configuring it to emit traces). Change-Id: Iea5a8890f545c4e4ddf0a796b3a2fcfc7078a50e
* | | | | | | Fix config-errors dedicated page8.0.1Clark Boylan2022-11-021-1/+1
| |_|_|/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There was a type update to our configErrors state from a list of errors to a dict containing a list of errors and a ready state flag. This broke the map() call on configErrors in the dedicated config errors page because you can't map an object only an Array. We fix this by mapping the objects errors attribute into the page as the config errors allowing us to keep mapping over it. Change-Id: Ib84ca9241fb456f9493a07d1c7e1220f5fa925c8
* | | | | | Fix skipped builds filter in web uiJames E. Blair2022-11-013-4/+9
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The change to automatically filter out skipped builds inadvertently affected the buildsets tab as well since they share the query builder. This caused the buildsets tab to automatically append a filter which is invalid for buildsets and therefore return a 404. This change adds an argument to the query builder to indicate whether it should include the automatic filter. Additionally, the query builder had an error which caused it to append the query in all cases (which affects the otherwise working builds tab). This change corrects that as well. Change-Id: Id03eee51b3551c4b32ae3c6b784b0993c4a3779c