summaryrefslogtreecommitdiff
path: root/zuul
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Drop support for ansible 2.6"Zuul2020-05-18203-857/+0
|\
| * Drop support for ansible 2.6Tobias Henkel2020-05-15203-857/+0
| | | | | | | | | | | | | | Ansible 2.6 has been unmaintained since several months now so remove support for it. Change-Id: Ifb604eb5cb86fd0210c1dfd8418f069273e302b6
* | Merge "Default to Ansible 2.9"Zuul2020-05-181-1/+1
|\ \ | |/
| * Default to Ansible 2.9Tobias Henkel2020-05-151-1/+1
| | | | | | | | | | | | | | | | Ansible 2.9 is there since several months now and had numerous bugfix releases. Thus we should default to it since it is the newest and best maintained release. Change-Id: If6687514a6237bb9816b54745aac5ec24383c8bf
* | Merge "Report dequeued changes via Github checks API"Zuul2020-05-165-8/+47
|\ \
| * | Report dequeued changes via Github checks APIFelix Edel2020-05-145-8/+47
| | | | | | | | | | | | | | | | | | | | | | | | This patch provides the general functionality to allow reporting of dequeued items and makes use of that in the Github checks API. This reporting will only apply if the item wasn't a success or failure. Change-Id: I1297da4d1708908c6b179110479fe0450e5550fe
* | | Merge "Enables whitelisting and configuring callbacks"Zuul2020-05-161-0/+23
|\ \ \
| * | | Enables whitelisting and configuring callbacksAlbin Vass2020-05-121-0/+23
| | | | | | | | | | | | | | | | Change-Id: Ida7b84795d922b85ec9cc6161ab1203fb82da825
* | | | Merge "Enhance some logging messages"Zuul2020-05-162-4/+4
|\ \ \ \
| * | | | Enhance some logging messagesAntoine Musso2020-04-022-4/+4
| | | | | | | | | | | | | | | | | | | | Change-Id: I257a3592643a716cd88efc255150a603d89d108a
* | | | | Merge "pagure: Improve CI status flag handling"Zuul2020-05-161-11/+35
|\ \ \ \ \
| * | | | | pagure: Improve CI status flag handlingFabien Boucher2020-04-071-11/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Ensure to consider only CI status flags set by the related token account. This is to avoid being triggered by another CI system voting on the same Pull Request. - Re-use the same status CI slot. - Allow to set the display name of the CI system in from of the CI status flag. Change-Id: I12fc761e34413f4efd7c2ad60ed04bf12d5932a3
* | | | | | Fix loading_errors bugJames E. Blair2020-05-141-13/+3
| |_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This corrects an error Error in dynamic layout Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/zuul/manager/__init__.py", line 625, in _loadDynamicLayout relevant_errors = self._findRelevantErrors(item, File "/usr/local/lib/python3.8/site-packages/zuul/manager/__init__.py", line 517, in _findRelevantErrors parent_layout.loading_errors.error_keys) or AttributeError: 'NoneType' object has no attribute 'loading_errors' which can be caused when the following conditions are met: * Tenant A is listed first. * Tenant A holds projects A and B. * Project B references a config object defined in project A. * Tenant A loads without errors. * Tenant B holds projects B and C. * Tenant B has a standing configuration error due to the unknown reference to an object in project A from project B. * Two or more changes to project C which cause dynamic configurations to be created are enqueued in a pipeline. * The merge job for the second change finishes before the first. This is because: We cache configuration objects if they load without error in their first tenant; that means that they can show up as errors in later tenants, but as long as those other tenants aren't proposing changes to that repo, it doesn't matter. But it does mean that every dynamic reconfiguration in this tenant will see errors and will execute the code path that compares the new dynamic configuration to the previous one to see if they are relevant. If a merge job for a dynamic config change arrives out of order, we will compare it to the previous configuration to determine if they are relevant, but since the previous layout had not been calculated yet, the exception above was hit. The solution is to indicate that the layout for the later change is not ready until the layout for the previous change is. Change-Id: Ibe1392a494d42b65080ab7e42f116db3869548ff
* | | | | Merge "pagure: Make use of the new project webhook/token endpoint"Zuul2020-05-131-4/+3
|\ \ \ \ \
| * | | | | pagure: Make use of the new project webhook/token endpointFabien Boucher2020-04-071-4/+3
| |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This endpoint is available in Pagure 5.9. It allows a collaborator to read the webhook token. Thanks to it, Zuul no longer requires the *admin* access level to read that per project webhook token to validate web hook payloads. The *ticket* access level is now the minimal required access level. This patch also removes some unnecessary instructions in the doc. Change-Id: Ia1c602f2b034e809de1a28cca8ef34f2a1c22f74
* | | | | Deprecate ansible 2.7Tobias Henkel2020-05-131-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ansible 2.7 only gets security fixes and no bugfixes anymore. Further it will be end of live as soon as Ansible 2.10 has been released. Thus deprecate it. Change-Id: If6fdabac4ed13be77699ec2a5f9626cdf426ceba
* | | | | Merge "Remove unused function prune in merger"Zuul2020-05-121-9/+0
|\ \ \ \ \
| * | | | | Remove unused function prune in mergerTobias Henkel2020-04-091-9/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This seems unused so it can be removed. Change-Id: Ic6a1fa76c632e05749d48405c953235cfd99bc31
* | | | | | Merge "Detach head before reset"Zuul2020-05-121-13/+13
|\ \ \ \ \ \
| * | | | | | Detach head before resetTobias Henkel2020-04-201-13/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During repo reset we currently try to reset the head to the HEAD of the remote repo. In some cases this doesn't work so simply reset the current HEAD revision as a commit to detach the head upfront. Change-Id: I167e6ce92680c40d2067939cedaa5ed98b950515
* | | | | | | Merge "Allow check runs to be configured as required status in pipeline config"Zuul2020-05-121-0/+11
|\ \ \ \ \ \ \
| * | | | | | | Allow check runs to be configured as required status in pipeline configFelix Edel2020-05-121-0/+11
| | |_|_|_|_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Although commit checks and statuses can only be retrieved via their respective APIs, Github does not differentiate between both in terms of branch protection and in the status section (below the comments of a PR). To mimic this behaviour in Zuul, one can now configure a check run as required status in the pipeline config. Change-Id: Ia5757c476bcee6082991928ab7c1743d0200d04a
* | | | | | | Merge "Replace preexec_fn argument in executor popen call"Zuul2020-05-121-1/+1
|\ \ \ \ \ \ \ | |/ / / / / / |/| | | | | |
| * | | | | | Replace preexec_fn argument in executor popen callSimon Westphahl2020-05-121-1/+1
| | |_|_|_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | According to the Python documentation the preexec_fn argument is not thread-safe and can cause a deadlock in the child process. The use of preexec_fn=os.setsid can be avoided by using start_new_session=True in the popen() call which is available in Python >= 3.2 DEBUG zuul.stack_dump: Thread: 139775071643392 build-e584032fcc424259be5b0feddfc3f0b4 d: True File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/opt/zuul/lib/python3.6/site-packages/zuul/executor/server.py", line 931, in execute self._execute() File "/opt/zuul/lib/python3.6/site-packages/zuul/executor/server.py", line 1154, in _execute result = self.runPlaybooks(args) File "/opt/zuul/lib/python3.6/site-packages/zuul/executor/server.py", line 1437, in runPlaybooks phase='post', index=index) File "/opt/zuul/lib/python3.6/site-packages/zuul/executor/server.py", line 2464, in runAnsiblePlaybook cleanup=phase == 'cleanup') File "/opt/zuul/lib/python3.6/site-packages/zuul/executor/server.py", line 2210, in runAnsible env=env_copy, File "/opt/zuul/lib/python3.6/site-packages/zuul/driver/bubblewrap/__init__.py", line 55, in __call__ proc = psutil.Popen(args, *sub_args, **kwargs) File "/opt/zuul/lib/python3.6/site-packages/psutil/__init__.py", line 1322, in __init__ self.__subproc = subprocess.Popen(*args, **kwargs) File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1318, in _execute_child part = os.read(errpipe_read, 50000) Change-Id: I0b5110e0b29025f9bb121be26a678fcbe130ba73
* | | | | | Fix unused argument triggering flake8Albin Vass2020-05-121-1/+1
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Strangely this doesn't occur on master but did here: https://review.opendev.org/#/c/727089/3 Change-Id: Ie2ff53d8ed2db03f70bd2ab7789028b5b7a5e91d
* | | | | Merge "executor: Catch error when reading cpu_times"Zuul2020-05-101-11/+17
|\ \ \ \ \
| * | | | | executor: Catch error when reading cpu_timesJustus Rossmeier2020-05-091-11/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On older Linux Kernels (especially 3.10 shipped with CentOS 7), having the mount option hidepid=2 set for /proc causes all zombie-processes to vanish from /proc for non-root users. We rely on /proc to get ansible's cpu_times however. Currently a fatal psutil.NoSuchProcess exception is thrown in the given case. Now, the error is catched and a warning is printed instead. This comes with the drawback that cpu_time measuring does not work anymore (it will always print 0s). Another possible solution for this would be to read the cpu_times of /proc/self before and after the execution of the ansible subprocess. This is the behaviour that is present in GNU time for example. For the described edge-case with an old kernel, this is considered overkill. Change-Id: I776f8588e1b0f6f3d145b0c6c54c8970fe864251
* | | | | | Merge "Don't reconfigure the tenant on tag creation"Zuul2020-05-101-1/+6
|\ \ \ \ \ \ | |/ / / / / |/| | | | |
| * | | | | Don't reconfigure the tenant on tag creationTobias Henkel2020-05-071-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We currently reconfigure the tenant also when new tags are created. Since we don't load config from tags it can be a huge performance impact if a large tenant frequently pushes new tags. Fix this by correctly filter for real branches. Change-Id: I0707d8cf8cea37bc4fe2239cf225d0871d993495
* | | | | | Merge "Validate ansible extra packages"Zuul2020-05-081-18/+43
|\ \ \ \ \ \
| * | | | | | Validate ansible extra packagesAlbin Vass2020-05-081-18/+43
| |/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently when validating the ansible installation, zuul only checks if ansible is installed and not any packages that would have been installed with ANSIBLE_EXTRA_PACKAGES. Since the executor image has ansible pre installed the ANSIBLE_EXTRA_PACKAGES environment variable has no effect unless ansible is removed. This adds a check to make sure packages specified with ANSIBLE_EXTRA_PACKAGES are installed as well. Change-Id: I7ee4125d6716db718bb355b837e90dbcfce9b857
* | | | | | Ignore source_context and description in job changesJames E. Blair2020-05-071-1/+2
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When detecting whether a job with file matchers has changed and should be run, ignore the source_context and description fields since they should not materially affect the job. Change-Id: Idd8707a40022a1ca8434b1bfdee0b178aa3cfe3e
* | | | | Merge "Add serial pipeline manager"Zuul2020-05-075-67/+141
|\ \ \ \ \
| * | | | | Add serial pipeline managerJames E. Blair2020-04-295-67/+141
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is designed to handle the case where we want: * The pipeline to be triggered by changes so results report back to the change. * Triggered on change merged. * Jobs with file matchers so that if a subsystem is changed, only the deployment job for that subsystem is run. * Every change is processed in strict sequence. This is designed to accomodate a deployment pipeline with the above constraints. The pipeline manager hierarchy is getting complicated; mark the base class as abstract, and also move the shared-queue methods into an intermediate abstract class. These are shared by both serial and dependent managers. Change-Id: I3c5f3b2f6298292c5c25665923e3a10b07be5419
* | | | | | Merge "Add some debug lines to help with the loading_errors bug"Zuul2020-05-061-0/+13
|\ \ \ \ \ \
| * | | | | | Add some debug lines to help with the loading_errors bugJames E. Blair2020-05-061-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I suspect that we're seeing this bug when a merge job for change B arrives before change A, and change B creates a dynamic config. However, a config error is still needed in order to trigger the code path, and I've been unable to identify a likely candidate. It would be useful to see what config errors are present in the dynamic layout to learn how the bug is being triggered. Change-Id: I2985fd8d69a971c959987896dae1da60533700eb
* | | | | | | Merge "merger: warn about invalid object type"Zuul2020-05-051-0/+4
|\ \ \ \ \ \ \ | |/ / / / / / |/| | | | | |
| * | | | | | merger: warn about invalid object typeTristan Cacqueray2020-05-041-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change helps diagnose issue when an zuul configuration object is not the excepted type, for example when a zuul.yaml reference is a directory. Change-Id: I94c236cde5224070d4cfcdd343b33bd5a51ecedc
* | | | | | | Merge "Don't retry after Gerrit HTTP 409s"Zuul2020-05-041-3/+22
|\ \ \ \ \ \ \
| * | | | | | | Don't retry after Gerrit HTTP 409sClark Boylan2020-04-271-3/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Gerrit will return an HTTP status 409 when you try to modify changes that are already merged or abandoned. For example if you try to vote on a change in these closed states. In those cases we don't actually want to retry the request as doing so is a waste of effort. Update the gerritconnection drive to handle HTTP 409s as a special case that should not be retried. Change-Id: I4bd7b8e5aa512bf0ba1d1bf7dd9a39c3843e6ff6
* | | | | | | | Don't exit zuul-web when signals are receivedClark Boylan2020-05-042-7/+4
| |/ / / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | One of the last things the primary thread in zuul-web does after configuring and starting the service is to call signal.pause(). This causes it to wait for any signal to arrive and once a signal has arrived and been handled we stopped zuul-web and killed the process. This is a problem because SIGUSR2 is expected to toggle preformance and debug tooling in the process. But since we exit after signal.pause() returns we can't do that properly. Fix this by removing the signal.pause() loop entirely and instead wait for the web command thread to die. This command thread is the last thing stopped by ZuulWeb.stop() so we can join() on it to wait for a complete stop of the ZuulWeb service. Change-Id: I66fbb86c25479823faeded8ede9c6204846c35c1
* | | | | | | Bump mypy for py3.8 supportClark Boylan2020-04-271-2/+3
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mypy depends on typed-ast which relatively recently added python 3.8 support. Unfortunately the mypy we were using requires an older typed-ast and fails under python 3.8. Fix this by bumping mypy up to a level where minimal code changes are necessary. The code changes we made ignore type conflicts due to redefinition on conditional imports. Comments include link back to a bug describing this issue and workaround in mypy. Change-Id: Iec69a27b16a1e09eb6bfbcf8d68deb1ae68d42a3
* | | | | | Revert "Tune automatic garbage collection of git repos"James E. Blair2020-04-251-36/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 79e50c04103098632de2ca47b18e08931925963b. We suspect this is causing this error: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/zuul/merger/merger.py", line 841, in _mergeChange zuul_event_id=zuul_event_id) File "/usr/local/lib/python3.5/dist-packages/zuul/merger/merger.py", line 525, in merge repo.git.merge(*args) File "/usr/local/lib/python3.5/dist-packages/git/cmd.py", line 542, in <lambda> return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/git/cmd.py", line 1005, in _call_process return self.execute(call, **exec_kwargs) File "/usr/local/lib/python3.5/dist-packages/git/cmd.py", line 822, in execute raise GitCommandError(command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git merge -s resolve FETCH_HEAD stderr: 'fatal: not something we can merge in .git/FETCH_HEAD: 3298c8f88b7b3fd49f35b0342ebb98f052f8358f 'refs/changes/83/723083/2' of ssh://review.opendev.org:29418/vexxhost/openstack-operator' Change-Id: I2e917710110a3ad85690d1cae53122ada005e4e8
* | | | | | merger: ensure .git/refs is not removed as a leaked dirsTristan Cacqueray2020-04-251-1/+1
|/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change ensures the .git/refs directory is not removed as this prevents further git operation. Change-Id: Ia314103d6807b755de3cf11eb7e7d4151626326f
* | | | | Tune automatic garbage collection of git reposTobias Henkel2020-04-231-0/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Git has an automatic garbage collection mechanism that is intended to keep operations efficient when working with a git repository. However the default settings are tuned to work best with repos that are direcly used by humans. In zuul we see a slowly degrading performance of job startup times over time which we fixed by pruning caches every two weeks. We found out that instead of pruning the caches we can have a similar effect by tuning the automatic garbage collection settings. The effect was that after a restart with optimized settings our worst case job startup times dropped from around 15 minutes to about 9 minutes. Change-Id: I594ceff9073f7ca14221d4f37716c76b462fc3d9
* | | | | Merge "Don't try to stream from winrm connections"Zuul2020-04-221-0/+3
|\ \ \ \ \
| * | | | | Don't try to stream from winrm connectionsTobias Henkel2020-04-071-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Connections via winrm don't have support for streaming yet as this needs a powershell based zuul_stream module. Change-Id: Ibc0804ec0466fa83e37a4e453028833ce737da8e
* | | | | | Merge "Log alembic migrations by default"Zuul2020-04-201-0/+4
|\ \ \ \ \ \
| * | | | | | Log alembic migrations by defaultJames E. Blair2020-04-201-0/+4
| | |/ / / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Update the default server logging config to log alembic migration info. Sometimes these can take a while, and the admin would like to know what's up. Change-Id: Ic7f87ea571ed2d4bdf84ef146e19fc2671e7cdc0
* | | | | | Merge "Filter secret ZUUL_ env variables from ansible env"Zuul2020-04-161-1/+3
|\ \ \ \ \ \ | |/ / / / / |/| | | | |
View Lorry Controller Status