| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
| |
This reverts commit 1c188ad440e82bb356832497c387aace71653dc8.
Reason for revert: This change increases test failures; investigation is ongoing.
Change-Id: I3a101656ee0904e1cf6c856dba845385c333260f
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I'm not sure if this is a good idea or not yet. Testing with MySQL
actually exposed the alembic transaction thing that is fixed in a
grandparent change. That said much of the world seems to be moving to
MariaDB so it might be a good idea for us to test that more explicitly?
We update bindep.txt which will install mariadb during CI job runs. We
also update our local test setup tooling to roughly match what will be
used in CI. For example the mariadb:10.6 docker image is chosen because
this is roughly equivalent to what Jammy packages.
Another good reason for this change is that Debian ARM64 does not
package MySQL and only packages MariaDB. This should give us better
compatibility for testing on architectures other than x86-64.
Change-Id: I9d00557ca5823da34278c3e9ecda2daaf13440b7
|
|
|
|
|
|
|
|
|
|
|
|
| |
MySQL 8 no longer supports implicitly creating a user using the GRANT
statement. Use a separate CREATE USER statement instead.
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the
right syntax to use near 'identified by 'openstack_citest' WITH GRANT
OPTION' at line 1
Change-Id: I4cab4c1855d1ba97cbfc9dd0835b3d302d73aa62
|
|
|
|
|
|
|
|
|
|
| |
Recently I debugged an issue that reproduced on mysql 8.0 on Jammy but
not my local system. It took quite some time for me to realize that
there was a difference in database versions which ended up being
an important detail. Update our docker-compose to better match what we
get in CI.
Change-Id: I7de268acb81680f3e6b7d3b1aa057e7babd3fa62
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Tox v4 behaves significantly differently than v3, and some of the
more complex things we do with tox would need an overhaul to
continue to use it. Meanwhile, nox is much simpler and more
flexible, so let's try using it.
This adds a noxfile which should be equivalent to our tox.ini file.
We still need to update the docs build (which involves changes to
base jobs) before we can completely remove tox.
Depends-On: https://review.opendev.org/868134
Change-Id: Ibebb0988d2702d310e46c437e58917db3f091382
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Zuul unittest jobs were timing out which prompted me to take a look
at what might be taking extra time. One thing I noticed is that we're
running the yarn build (which runs react-scripts) multiple times when we
really only need to run it once. The reason for that is our check to see
if zuul web has already built is looking for a file that the builds no
longer produce. Update that check to look for a current file and we'll
save a bit of time in our jobs and when running things locally.
Change-Id: Iae3604fbaf072d53895db850cfc989a832b12b27
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When not using a ROOTCMD when running the test-setup-docker.sh the
script will fail with the following error message:
++ id -u
+ USER_ID=1000 docker-compose up -d
./test-setup-docker.sh: line 50: USER_ID=1000: command not found
Due to Bash's simple command expansion[0] variable assignments will be
interpreted as the command when no ROOTCMD is given. To work around this
we use default ROOTCMD to the `env` command.
[0]: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Simple-Command-Expansion
Change-Id: Id696069a4eec2b2c2513ac449098b0ef73dc3906
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update the TESTING.rst file to reflect that just installing and
starting zookeeperd is no longer sufficient now that we require TLS
and auth for the connection, as well as running database servers.
Suggest the container-based setup script instead. Also improve that
script to allow it to be invoked as a normal user, with root command
escalation tool choice (e.g. "sudo") supplied through a ROOTCMD
environment variable, so that things created inside the git worktree
like the CA don't end up root-owned.
Related, the tox-docker plugin previously suggested in the document
is no longer a viable option for the same reasons, so clean up the
plumbing for it in tox.ini as well.
Change-Id: Iac32799425a5bd4b1bdbf56f34a2310241ac4499
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.
We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.
Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.
Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.
Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This was deprecated a while ago. This adds a script to help users
identify where it still needs to be updated.
See https://zuul-ci.org/docs/zuul/latest/releasenotes.html#relnotes-4-1-0-deprecation-notes
Change-Id: Ib2102b9dfa048015d9fbbb1f91bc50f0c1909d53
|
|/
|
|
|
|
|
|
| |
A couple of locations continue to reference actiongeneral which has been
removed. Update these locations to use action as the current location
for these plugins.
Change-Id: I71c03d2c0a84592be66fa0d84bc684684a392a27
|
|\ |
|
| |
| |
| |
| |
| |
| | |
Make the Github debugging script work with the evolved API.
Change-Id: I60e229fce5329215417bc52a1adc15d69f7495c8
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The zookeeper containers entrypoint will try to switch to the user
'zookeeper' if the container is started with uid '0'. This can cause issues
when the uid for the zookeeper user inside the container and the
users uid outside the container differ since it will lead to zookeeper
getting access denied when trying to read the certificates.
This also adds logging configuration to make it easier to debug
zookeeper in the future.
Change-Id: I51db53fe093a294e804148f682053123f54adfe1
|
|/
|
|
|
|
|
|
|
|
|
|
|
| |
We had been using version 14 which is the previous LTS. Now there are
problems running:
npx browserslist@latest --update-db
running out of memory. Update to the current nodejs LTS version to
ensure we are running an up to date runtime that hopefully performs more
consistently with the browserslist command.
Change-Id: Ib20c1090ea0f30b7dac2780b6ed963dd6e4b6f77
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since the pipeline state is stored in ZooKeeper, there could be cases
where the change_queues key is missing in the status json. This makes
API requests fail:
2022-02-22 17:56:18,390 ERROR cherrypy.error.139989033522128: [22/Feb/2022:17:56:18] HTTP
Traceback (most recent call last):
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 638, in respond
self._do_respond(path_info)
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond
response.body = self.handler()
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/lib/encoding.py", line 223, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/lib/jsontools.py", line 59, in json_handler
value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/opt/zuul/lib/python3.8/site-packages/zuul/web/__init__.py", line 1050, in status_change
return result_filter.filterPayload(payload)
File "/opt/zuul/lib/python3.8/site-packages/zuul/web/__init__.py", line 193, in filterPayload
for change_queue in pipeline['change_queues']:
KeyError: 'change_queues'
Fix this by using a .get() call rather than directly accessing the
dictionary key by name.
A similar issue was already fixed in [1].
[1]: https://review.opendev.org/c/zuul/zuul/+/829018
Change-Id: I947f58f02c3da7dad35d1fc186c7026800f7cbdd
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Change Ib4037da2f7a0f803aca24ce880dbc262375db6a4 introduced 2 extra
parameters for the zuul.model.SourceContext constructor. These
parameters don't really matter for decrypt_secret.py. We just pass the
None value.
Change-Id: Ic4f09558865349c8c479defe1b314b0c4c92dd0f
|
|\ \ |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This is like the real zk-shell except:
1) It supports SSL
2) It automatically decompresses znodes if necessary
3) It handles sharded data
4) It doesn't have many commands
Change-Id: I2124f216f08f68ca63b763cd6f6adce452c07f79
|
|\ \ \
| |/ / |
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
These may be useful for zuul developers to understand issues with the
ZK data storage.
zk-dump will dump an approximation of the contents of ZK to the
filesystem for manual examination.
zk-analyze will perform some analysis on the tree to identify objects
which may be execessively large.
Change-Id: I1a90cce42da719eee0a5e50242034390722d518e
|
| |
| |
| |
| |
| |
| |
| |
| | |
This can be used immediately to manually build old docs with
updated themes and URLs. We can iterate on this to make it a
periodic job so it happens automatically.
Change-Id: I021e236ffcfd436e1169e9e804efee013cbe10aa
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
| |
In case for whatever reason there is already another
MySQL server running on the standard port developers
may want to run the MySQL test-server on a different
port. This can be done by specifying an environment
variable "ZUUL_MYSQL_PORT":
export ZUUL_MYSQL_PORT=3307
tools/test-setup-docker.sh
tox -e py3
Change-Id: Id32bac1a87619803d973b066762d525bd9a4f913
|
|
|
|
|
|
|
|
| |
The gearman RPC methods are being removed, so this needs a change
to work with one of the web-based methods. The most simpla end
forward-looking method is zuul-client, so update it to use that.
Change-Id: If3f6ca4bae2b2beddb3bb71b36fdcba112722186
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When executing the unit tests with multiple schedulers in parallel a lot
of them are failing becasue there are too many open SQL connections.
However, executing those tests one after another doesn't end up in the
connection limit being exhausted. So this doesn't look like a shutdown
problem in Zuul.
Having a look at the MySQL server when it's under load during the tests
revealed that the default connection limit of 151 on the server side is
exhausted very quickly when a lot of tests are running in parallel (each
test running with two schedulers).
Therefore, this change increases the default MySQL connection limit to
300. We should keep in mind that this connection limit has to go in hand
with the number of schedulers used in the unit tests and the number of
tests executed in parallel. Maybe on the long term it might make sense
to cap the latter parameter somehow in the tox.ini file.
The same applies to the ZooKeeper connection limit which is configured
in tools/zoo.cfg.
Change-Id: Iff76e99ec82edc8e8bc110a22a096bb689d8dd1f
|
|
|
|
|
|
|
|
| |
Debian bullseye includes skopeo, so we can drop the kubic repository
(which as I write this is having synchronization errors) in favor of
just using the version from the underlying OS.
Change-Id: Ifde147c8c690dd6a421b0133dbabcff7dd9b9649
|
|
|
|
|
|
|
| |
This appears to output something like 145,000 lines in our build
logs, so let's be less verbose.
Change-Id: Ie41c1f21a090de84cb51331c00668b9b7f06379a
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a modified kazoo.DataWatch class which does not set
watches on paths that don't exist. This is mostly so that when
a DataWatched path is deleted, we don't leave a watch in place.
Without that, the executor api will leak watches on the ZK cluster.
We vendor the entire watches.py file from kazoo (ASL2 licensed) to
avoid any issues with API changes. Separately we will see if they
are interested in this class upstream.
This updates the tests to use the wchp 4lw to test that we don't
leak watches.
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/799334
Change-Id: Ie4491eef03b58d858d95c78ba9454839d169cff1
|
|\ |
|
| |
| |
| |
| |
| |
| | |
This is mostly a debugging aid.
Change-Id: I843c9a993299c86b66fa5e6c11aed08769d79980
|
| |
| |
| |
| |
| |
| | |
The set -e was subverting the yarn retry loops in tools/pip.sh.
Change-Id: Ibee12479336670142fdc6733e744ab5273cb8c5b
|
|/
|
|
|
|
|
|
|
| |
We're seeing occasional npm install failures in jobs; retry the
"yarn install" command if it fails.
Also, pass the YARN_REGISTRY env var in the tox remote job.
Change-Id: Ic80dccf0869ce805368f678eafcfdf48f8bf9bea
|
|
|
|
|
|
|
|
| |
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests. We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.
Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
"maxClientCnxns".
Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.
Also the zookeeper log is copied to the "zuul_output_dir".
Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19
Story: 2007192
|
|
|
|
|
|
|
|
|
|
| |
Now that zuul-client's encrypt subcommand covers the same
functionalities as encrypt_secret.py, add a deprecation
message when running the script. Document the zuul-client
encrypt command in the doc section about secrets.
Change-Id: Id5437ffbb688cb80b2744db3beeaa28c97080d90
Depends-On: https://review.opendev.org/765313
|
|
|
|
| |
Change-Id: Ia7041423eb0bd5b688c149e70fd06373251e9ced
|
|
|
|
| |
Change-Id: Id1a167e1f722f51b50dd2f0f1bda3a7e6a32d182
|
|
|
|
|
|
|
| |
The kubic repo has changed it signature key so update it to unbreak
container builds.
Change-Id: I3edc9bebfc6807fb2ed76e615fc9eeff48c03911
|
|
|
|
|
|
|
|
|
|
|
|
| |
Opendev runs a caching mirror for npm packages. Use it when running
yarn install during tox tests. Due to the way yarn.lock files work at
the moment we need to patch the lockfile on the fly to inject the
mirror urls [1]. This can be handlel inside the pip.sh where we patch
the lockfile, run yarn and reset it afterwards.
[1] https://github.com/yarnpkg/yarn/issues/2566
Change-Id: Id1f2daac8327f3110e6beeb2d6391baccfd3ce34
|
|
|
|
|
|
|
|
|
|
| |
This change adds a script to check if a zuul deployment is
using untrusted command on the executor host.
Operator needs to take action on the affected job to avoid
failure after upgrading to zuul v3.19.1.
Change-Id: I5d18df3fd120e8f4c189ce0b59aa89553e9df816
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Zuul was designed to block local code execution in untrusted
environments to not only rely on bwrap to contain a job. This got
broken since the creation of a command plugin that injects the
zuul_job_id which is required for log streaming. However this plugin
doesn't do a check if the task is a localhost task. Further it is
required in trusted and untrusted environments due to log
streaming. Thus we need to fork this plugin and restrict the variant
that is used in untrusted environments.
We do this by moving actiongeneral/command.py back to action/*. We
further introduce a new catecory actiontrusted which gets the
unrestricted version of this plugin.
Change-Id: If81cc46bcae466f4c071badf09a8a88469ae6779
Story: 2007935
Task: 40391
|
|
|
|
|
|
| |
This change removes problematic language in the test environment.
Change-Id: I6f81ab68314902f0239babf887de04f8e30d8954
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The js content tarball creation was broken. Instead of leaving the
symlink to a non-existing directory which gets created during python
setup we should just remove that symlink as well and create the
symlink and the static dir during the python setup. This way nothing
will be in the way of the javascript content generation.
This reverts commit eb7b18b38e01688b5b05e5dfdb359ff5d42d1ed8.
Change-Id: I5f8bfa62cd2d4d9823b86dbcda14885230847a82
|
| |
| |
| |
| |
| |
| |
| |
| | |
Starting from mysql version 8 it's not possible to create a user
implicitely when using GRANT.
This patch makes the behavior compatible with that.
Change-Id: I1dc25194de384bf14607aff4e0f230e394f00582
|
|/
|
|
|
|
|
|
|
|
|
| |
The zk-ca.sh tool for generating certificates fails with
and error that ./tools/openssl.cnf cannot be opened if a
relative path is supplied on ubuntu-bionic. So expand the
path to an absolute path instead.
http://paste.openstack.org/show/794091/
Change-Id: Ib30e9754d4520b956e83cae014c1be181ae44631
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
v14 is the latest lts. Let's use it.
Also - rename the jobs to make it clear what they're doing, and
add a dashboard job that points at opendev's multi-tenat api too.
There are new jobs that default to latest node LTS and auto-detect
yarn vs npm. Update to use them.
Depends-On: https://review.opendev.org/728097
Depends-On: https://review.opendev.org/726547
Change-Id: I5717edea2cd09acc5bce673c38bbe7fa8057a376
|
|\ |
|