| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
| |
The latest release 8.0.0 of elasticsearch breaks Zuul when using elasticsearch connection
Trace:
...
File "/usr/local/lib/python3.9/dist-packages/zuul/driver/elasticsearch/con>
self.es = Elasticsearch(
TypeError: __init__() got an unexpected keyword argument 'use_ssl'
Change-Id: Idd4cf9d67f60374fa1e8b9a816845829de31e9f1
|
|
|
|
|
|
|
|
|
| |
Since [1] was solved by the 3.1.0 release of github3.py, we can relax
the dependency constraint accordingly.
[1] https://github.com/sigmavirus24/github3.py/issues/1057
Change-Id: Ia31cb4eff50964fe077f9d1d538ca51b4e64c51c
|
|
|
|
|
|
|
| |
The gearman server is no longer required. Remove it from tests and
the scheduler.
Change-Id: I34eda003889305dadec471930ab277e31d78d9fe
|
|
|
|
|
|
|
|
|
|
| |
Apscheduler requires tzlocal/pytz, and they have introduced a warning
which we can avoid by pinning to an earlier version. This is not likely
to be fixed in pytz 3.x, but will be in 4.x.
See https://github.com/agronholm/apscheduler/discussions/570
Change-Id: I9c0555ef107d411b8e2fac9dabc7547459e5ffa7
|
|
|
|
|
|
|
|
|
| |
The latest release 3.0.0 of Github3.py breaks Zuul when using Github
Enterprise <3.1 [1].
[1] https://github.com/sigmavirus24/github3.py/issues/1057
Change-Id: I1c199cd43985458f88137b218916b156972494ad
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A recent update of CacheControl broke parts of the github api in zuul
which resulted in empty bodies in the response [1] when requests were
cached.
[1] Trace:
GET https://(...)/api/v3/(...)/files?per_page=100 result: 200, size: 0, duration: 67
ERROR zuul.GithubEventProcessor: Exception when processing event:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.8/site-packages/github3/models.py", line 168, in _json
ret = response.json()
File "/opt/zuul/lib/python3.8/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 346, in run
self._process_event()
File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 404, in _process_event
change = self.connection._getChange(
File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1334, in _getChange
pull = self.getPull(change.project.name, change.number,
File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1706, in getPull
pr['files'] = [f.filename for f in probj.files()]
File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1706, in <listcomp>
pr['files'] = [f.filename for f in probj.files()]
File "/opt/zuul/lib/python3.8/site-packages/github3/structs.py", line 90, in __iter__
json = self._get_json(response)
File "/opt/zuul/lib/python3.8/site-packages/github3/structs.py", line 137, in _get_json
return self._json(response, 200)
File "/opt/zuul/lib/python3.8/site-packages/github3/models.py", line 170, in _json
raise exceptions.UnexpectedResponse(response)
github3.exceptions.UnexpectedResponse: 200 [No message]
Change-Id: I5921e98d4252d4c96d116fe514b8637fad86a7fb
|
|
|
|
|
|
|
|
| |
Gear is ready to make a release, but we're about to remove it.
Pin to less than the upcoming release to retain stability while
we finish up removal.
Change-Id: I4822c212f0d1fc018e92d01d162868e274dbd6ba
|
|
|
|
|
|
|
|
| |
This change adds a new prometheus_port option to start a metric server
to be scrapped by a prometheus service. By default, the server exposes
process informations.
Change-Id: Ie329df6adc69768dfdb158d00283161f8b70f07a
|
|
|
|
|
|
|
|
| |
An SQL database is an additional requirement so we should also include
the client libraries for it.
Change-Id: Id3a8b55928d9ce9e8764f18a64a42bb2f14f5ecb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| | |
The release of pyjwt 2.0.0 changed the behavior of some functions, which
caused errors. Fix the errors, use pyjwt 2.0.0's better handling of JWKS,
and pin requirement to 2.X to avoid future potential API breaking changes.
Change-Id: Ibef736e0f635dfaf4477cc2a90a22665da9f1959
|
|\ \ |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
It has the capability to index build and buildset results.
With the help of tools like Kibana, advanced analytics
dashboard could be built on top of the Zuul Elasticsearch
index.
Optionally job's variables and zuul_return data can be
exported along with build results under the job_vars and
job_returned_vars fields.
Change-Id: I5315483c55c10de63a3cd995ef681d0b64b98513
|
| |/
|/|
| |
| |
| |
| |
| |
| | |
Removes outdated workaround on importlib-resources.
This fixes issue with py39+, where stdlib already provides it.
Change-Id: I82264c293ff9072e970e81417658fc934cccea2d
|
| |
| |
| |
| |
| |
| |
| | |
Release 2.0.0 on December 22, 2020 brought breaking changes. Pin to the
last working version until we can fix the breaks properly.
Change-Id: I2e6cc36d373f8c774eda5f5caec9868f74023729
|
| |
| |
| |
| |
| |
| |
| | |
This reverts commit a58b29c6bf38ff47b12b5583dfc61c50ce083f2a. The
dependency is uncapped in zuul-client now.
Change-Id: I9d31326c444cf225442c0b46e040312be9a2ff73
|
|/
|
|
|
|
|
| |
The reasoning is explained inline. We can remove this when
zuul-client is fixed.
Change-Id: I431d9202cabcc9bf7b60fd643c55b557f6505185
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
These various exceptions are raised as certain parts of the
configuration are parsed. Although they give you a lot of information
about the issue, they don't give you any context for what part of your
configuration caused the error to be raised.
They are all working on a particular item of the configuration that I
think it would be very useful to show in the error output.
This was inspired by me putting playbooks in the .zuul.d directory,
where they were read as config files. In hindsight it was obvious,
but it took me a while to figure out what was going on. With this it
would have reported something like:
Zuul encountered a syntax error while parsing its configuration in
the repo org/project1 on branch master. The error was:
Configuration item has more than one key. Each zuul.yaml
configuration file must be a list of dictionaries with a
single key, for example:
- job:
name: foo
... blah ...
The incorrect values are:
hosts: all
tasks:
- debug:
msg: blah
name: foo
which would have clued me in immediately.
This truncates the message if it is too long; which might be the case
if dumped a playbook or foreign config file. However I think
practically most errors are typo level on jobs/templates/etc. that
will show nicely.
The YAML version is updated to support non-sorted dumping of the keys.
This is particularly helpful with Python 3.7+ and default ordered
dicts.
Change-Id: I4851a5a796fa452a023e0e6193fc724ae0967a44
|
|/
|
|
|
|
|
|
|
|
|
| |
Branch protection rules in github are fn patterns which are currently
matched locally in zuul. This is error prone and can lead in edge
cases to wrong matches resulting in wrong enqueue decisions into gate
pipelines. When requesting branch protection rules in github we also
can request the matching refs along with the rules. This is much safer
since we can plain text match them against the change branch.
Change-Id: Ic995d4b2e16a5d741f0209fa9236959d8f4d10b9
|
|
|
|
|
|
| |
This has a fix required for using TLS.
Change-Id: I6d29ca3bf31b1f50cebc245d3db2095ab65921c4
|
|
|
|
|
|
|
| |
Cheroot 8.3.1 have been released with the https://github.com/cherrypy/cheroot/issues/263
fix.
Change-Id: I83aad30a6145ed2c7689d6c25c624c453c07ed2b
|
|
|
|
|
|
|
| |
Zuul is using a dateutil provided transitively. This change
makes the requirements explicit.
Change-Id: I672d409a1bb241e5d199c77acdd51a8260d99ee2
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
I installed Zuul 18.0 on a CentOS 7 with Python 3.6 from RH SCL repos
which comes with virtualenv==15.1.0. This is effectively too old to
provision the virtual environments for Ansible, likely because that
version of virtualenv hardcodes PIP 9. During the installation,
zuul-executor dies with this error:
Installing collected packages: MarkupSafe, jinja2, PyYAML, pycparser, cffi, six, pynacl, cryptography, bcrypt, paramiko, ansible, XStatic-Bootstrap-SCSS, pyfakefs, XStatic-Patternfly-Bootstrap-Treeview, wcwidth, pyparsing, pyperclip, cmd2, pbr, stevedore, PrettyTable, cliff, XStatic, XStatic-Patternfly, wrapt, debtcollector, msgpack, pytz, netaddr, iso8601, netifaces, Babel, oslo.i18n, oslo.utils, oslo.serialization, itsdangerous, Werkzeug, click, Flask, SQLAlchemy, Flask-SQLAlchemy, XStatic-jQuery, Mako, python-editor, python-dateutil, alembic, Flask-Migrate, pygments, junit-xml, Flask-Script, Frozen-Flask, extras, linecache2, traceback2, argparse, unittest2, fixtures, python-mimeparse, testtools, python-subunit, XStatic-DataTables, ara, os-service-types, urllib3, chardet, idna, certifi, requests, keystoneauth1, jsonpointer, jsonpatch, appdirs, munch, decorator, jmespath, requestsexceptions, dogpile.cache, openstacksdk, dictdiffer, websocket-client, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, oauthlib, requests-oauthlib, kubernetes, python-string-utils, ruamel.yaml.clib, ruamel.yaml, openshift, google-resumable-media, protobuf, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-storage
Running setup.py install for PyYAML: started
Running setup.py install for PyYAML: finished with status 'done'
Running setup.py install for ansible: started
Running setup.py install for ansible: finished with status 'done'
Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: started
Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: finished with status 'error'
Complete output from command /var/lib/zuul/ansible-bin/2.6/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-24z_rrjk/XStatic-Patternfly-Bootstrap-Treeview/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6ie_p40b-record/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/zuul/ansible-bin/2.6/include/site/python3.6/XStatic-Patternfly-Bootstrap-Treeview:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/__init__.py", line 10, in <module>
from setuptools.extern.six.moves import filter, filterfalse, map
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/extern/__init__.py", line 1, in <module>
from pkg_resources.extern import VendorImporter
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3017, in <module>
@_call_aside
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
f(*args, **kwargs)
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3045, in _initialize_master_working_set
dist.activate(replace=False)
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2577, in activate
declare_namespace(pkg)
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2151, in declare_namespace
_handle_ns(packageName, path_item)
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2091, in _handle_ns
_rebuild_mod_path(path, packageName, module)
File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2120, in _rebuild_mod_path
orig_path.sort(key=position_in_sys_path)
AttributeError: '_NamespacePath' object has no attribute 'sort'
This points to https://github.com/pypa/pip/issues/4216, so the fix is
apparently to make sure that there's new enough PIP + setuptools in the
created virtual env, and these versions are determined by host's
virtualenv module (thanks to webknjaz on IRC for explaining this to me).
After a restart zuul-executor runs normally, but as a side effect not
all required Python modules are installed for Ansible's usage. For me,
this meant that (e.g.) `upload-logs-swift` won't work ebcause there's no
"openstack" module.
I don't know how to make zuul-executor error persistent, but let's at
least make the life easier for the next poor sod installing this SW on
a system where "virtualenv" is "too old" -- which is also the latest
available Ubuntu if I'm checking their revision numbers correctly.
Require virtualenv >= 20 while preserving the blacklist from commit
7c6503aba. Given that Zuul's own CI uses PIP to pull the latest
virtualenv for the test suite, make sure that the users get what was
tested.
Change-Id: I2b53e532b63a9ae20e817d9599c46cccf613e425
|
|/
|
|
|
|
|
|
|
|
|
|
|
| |
Current set of dependencies brings latest cherrypy which is
incompatible with cheroot we need. This pins cherrypy to last
version that is compatible. The current verision of pip will
still install cherrypy, but the upcoming depsolver is unable to
resolve the conflict.
We also add a fail-safe `pip check` that runs the new depsolver
and should prevent a similar regressions from appearing.
Change-Id: I2b0fa43b906d0919854d30da0ee825002444b201
|
|
|
|
|
|
|
|
|
|
|
|
| |
Replace our tox-py37 job with tox-py38, extend the list of trove
classifiers for Python versions in package metadata, and replace the
"py35" in the tox.ini envlist with just "py3" so that folks running
`tox` unqualified on their systems will use whatever python3
interpreter they have on hand (odds are it's in our supported range
these days). Also uncap python-daemon so we use a version compatible
with Python >=3.8.
Change-Id: I4b382bb89847d3ae2b20292d923c3c2211dfb254
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The canMerge check is executed whenever zuul tests if a change can
enter a gate pipeline. This is part of the critical path in the event
handling of the scheduler and therefore must be as fast as
possible. Currently this takes five requests for doing its work and
also transfers large amounts of data that is unneeded:
* get pull request
* get branch protection settings
* get commits
* get status of latest commit
* get check runs of latest commit
Especially when Github is busy this can slow down zuul's event
processing considerably. This can be optimized using graphql to only
query the data we need with a single request. This reduces requests
and load on Github and speeds up event processing in the scheduler.
Since this is the first usage of graphql this also sets up needed
testing infrastructure using graphene to mock the github api with real
test data.
Change-Id: I77be4f16cf7eb5c8035ce0312f792f4e8d4c3e10
|
|\ \ |
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| | |
GitPython was capped to <3.0.0 due to increase test times. According
to the upstream issue [1] this should, be fixed upstream. See if we
can uncap it now.
[1] https://github.com/gitpython-developers/GitPython/issues/906
Change-Id: I62a2731c4045f72060a67b8ad3eb89f6b07a0e97
|
|/
|
|
|
|
|
|
|
|
|
|
| |
The global tox installation on our test nodes is affected by an
upstream issue[1]. Additionally, the virtualenv created by tox
under those conditions is also affected. To work around this,
pin the version of importlib-resources in the global tox install,
as well as in Zuul's own requirements.
[1] https://gitlab.com/python-devs/importlib_resources/issues/83
Change-Id: I31ed50185a71d867a2ad512ef9b526c5b607ed5c
|
|
|
|
|
|
|
| |
There may have been a behavior change with polling that causes a
surplus of log lines. Avoid that version until we understand more.
Change-Id: I246acda9352267eb6b9be6d1b06fb87b133709ec
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
virtualenv==20.0.0 introduced new symlinking of "seed" packages that
broke within a bwrap context. We fixed this by forcing those seed
packages to be installed by pip avoiding any symlinking. Since then
upstream virtualenv has realize this was bad default behavior and
virtualenv==20.0.2 does not symlink by default.
Take advantage of this and exclude 20.0.0 and 20.0.1 as they need
special handling. This allows other versions (new and old) to function.
Change-Id: I3b993e72b52e2234e046a657280b008887c535a3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New virtualenv has changed behavior and cli arguments. In particular the
seed packages are by default installed in a common lib location and not
updated. These packages are then symlinked to in the virtualenv. This
causes problems for zuul when we run ansible in bwrap and that common
lib path isn't bind mounted in the container.
We can fix this by telling virtualenv to install the seed packages uses
pip and not its internal bundled app data. This is preferable for Zuul
because then we don't have to manage permissions and locations of a
separate seed package library location.
Note that we force virtualenv to be >=20.0.0 as old virtualenv does not
have this new seeder option.
Change-Id: Ib9b2e0b51e6acd744d1c23b4a27067f0202a679d
|
|
|
|
|
|
| |
The new virtualenv release 20.0.0 breaks tests so cap it for now.
Change-Id: I40660f31c17ddae13e05a75552baed125235fa05
|
|
|
|
|
|
|
|
|
|
|
| |
CherryPy uses cheroot underneath as http server. The release 8.1.0
introduced a regression which leads to occasional connection resets
while handling concurrent requests [1]. Cap it until there is a
release available that fixes this issue.
[1] https://github.com/cherrypy/cheroot/issues/263
Change-Id: I39716e9c203cc42ab320cb47e090fbc8f67320c9
|
|
|
|
|
|
|
| |
Instead of preventing newer version which would create conflicts, we
just avoid the known broken versions.
Change-Id: I7e04bd6efddaab9847c01c48a2e6ef4be1a41e4e
|
|
|
|
|
|
|
| |
This is broken with Gerrit due to
https://github.com/urllib3/urllib3/pull/1684
Change-Id: Ie2c817bb91463cecc64e3022e11330898b11062c
|
|
|
|
|
|
|
|
| |
This looks like it might be extending our unit test runtime by
50% (from 30 mins to 45, thefore pushing the overall job from
45 to 60+ mins).
Change-Id: I333cc1dbe5b2223e7d6f86ff7cdb6d5f063bda39
|
|
|
|
|
|
|
|
|
|
| |
Allow an operator to define authorization rules.
Allow an operator to add authorization rules to a tenant.
Add a rule parser and a rule registry.
The authZ engine is not plugged in yet.
Change-Id: I3a86c6c7d62ad2bce68a98dbd2fff18549b94fb9
|
|
|
|
|
|
| |
Don't use system virtualenv. Install it from source/pypi.
Change-Id: Idea7291cb6bff9d23c87fd40b3bb45f32115b7ef
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.
The actual usage of multiple ansible versions will be done in
follow-ups.
For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.
Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927
|
|
|
|
|
|
|
|
|
|
|
|
| |
This updates the github driver to cache PRs by sha using cachetool's
LRUCache.
We make this change because we need to cache closed PRs so can't rely on
the action of closing a PR to remove the PR from the cache. Since we
don't have a good method of evicting entries we fall back to LRU with
a reasonable cache size (2k commits).
Change-Id: I5fb6c8b33f9eed221a8b84e537f02e7dccf2d2df
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the gearman server vanishes (e.g. due to a VM crash) some clients
like the merger may not notice that it is gone. They just wait forever
for data to be received on an inactive connection. In our case the VM
containing the zuul-scheduler crashed and after the restart of the
scheduler all mergers were waiting for data on the stale connection
which blocked a successful scheduler restart. Using tcp keepalive we
can detect that situation and let broken inactive connections be
killed by the kernel.
Depends-On: I8589cd45450245a25539c051355b38d16ee9f4b9
Change-Id: I30049d59d873d64f3b69c5587c775827e3545854
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Our custom command.py Ansible module is updated to match the
version from 2.5, plus our additions.
strip_internal_keys() is moved within Ansible yet again.
Change-Id: Iab951c11b23a24757cf5334b36bc8f7d12e19db0
Depends-On: https://review.openstack.org/567007
|
|\ \
| |/ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This updates both the dependency to ansible 2.4 and also ports in the
needed changes to the command module.
Version 2.4.0 definitely does not work for us because YAML
hosts file parsing is broken, but 2.4.1 and greater should
be fine.
Change-Id: I63f72b45ecb9533eac5ba9eb0eef426beec905e3
|
| |
| |
| |
| | |
Change-Id: I480385a8b0e85266fdd77d251d0b748f1be028b0
|