summaryrefslogtreecommitdiff
path: root/tools
Commit message (Collapse)AuthorAgeFilesLines
* Revert "Test with MariaDB instead of MySQL"James E. Blair2023-04-132-4/+4
| | | | | | | | This reverts commit 1c188ad440e82bb356832497c387aace71653dc8. Reason for revert: This change increases test failures; investigation is ongoing. Change-Id: I3a101656ee0904e1cf6c856dba845385c333260f
* Test with MariaDB instead of MySQLClark Boylan2023-04-052-4/+4
| | | | | | | | | | | | | | | | | | I'm not sure if this is a good idea or not yet. Testing with MySQL actually exposed the alembic transaction thing that is fixed in a grandparent change. That said much of the world seems to be moving to MariaDB so it might be a good idea for us to test that more explicitly? We update bindep.txt which will install mariadb during CI job runs. We also update our local test setup tooling to roughly match what will be used in CI. For example the mariadb:10.6 docker image is chosen because this is roughly equivalent to what Jammy packages. Another good reason for this change is that Debian ARM64 does not package MySQL and only packages MariaDB. This should give us better compatibility for testing on architectures other than x86-64. Change-Id: I9d00557ca5823da34278c3e9ecda2daaf13440b7
* Fix user creation in Docker test setup scriptSimon Westphahl2023-02-211-1/+2
| | | | | | | | | | | | MySQL 8 no longer supports implicitly creating a user using the GRANT statement. Use a separate CREATE USER statement instead. ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by 'openstack_citest' WITH GRANT OPTION' at line 1 Change-Id: I4cab4c1855d1ba97cbfc9dd0835b3d302d73aa62
* Switch our local testing docker-compose to mysql 8.0Clark Boylan2023-01-311-1/+1
| | | | | | | | | | Recently I debugged an issue that reproduced on mysql 8.0 on Jammy but not my local system. It took quite some time for me to realize that there was a difference in database versions which ended up being an important detail. Update our docker-compose to better match what we get in CI. Change-Id: I7de268acb81680f3e6b7d3b1aa057e7babd3fa62
* Add noxfile and switch to noxJames E. Blair2022-12-201-0/+73
| | | | | | | | | | | | | | Tox v4 behaves significantly differently than v3, and some of the more complex things we do with tox would need an overhaul to continue to use it. Meanwhile, nox is much simpler and more flexible, so let's try using it. This adds a noxfile which should be equivalent to our tox.ini file. We still need to update the docs build (which involves changes to base jobs) before we can completely remove tox. Depends-On: https://review.opendev.org/868134 Change-Id: Ibebb0988d2702d310e46c437e58917db3f091382
* Only run yarn build when necessary in tools/pip.shClark Boylan2022-10-071-1/+1
| | | | | | | | | | | | The Zuul unittest jobs were timing out which prompted me to take a look at what might be taking extra time. One thing I noticed is that we're running the yarn build (which runs react-scripts) multiple times when we really only need to run it once. The reason for that is our check to see if zuul web has already built is looking for a file that the builds no longer produce. Update that check to look for a current file and we'll save a bit of time in our jobs and when running things locally. Change-Id: Iae3604fbaf072d53895db850cfc989a832b12b27
* Run docker-compose up test setup script with envSimon Westphahl2022-08-231-1/+5
| | | | | | | | | | | | | | | | | When not using a ROOTCMD when running the test-setup-docker.sh the script will fail with the following error message: ++ id -u + USER_ID=1000 docker-compose up -d ./test-setup-docker.sh: line 50: USER_ID=1000: command not found Due to Bash's simple command expansion[0] variable assignments will be interpreted as the command when no ROOTCMD is given. To work around this we use default ROOTCMD to the `env` command. [0]: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Simple-Command-Expansion Change-Id: Id696069a4eec2b2c2513ac449098b0ef73dc3906
* Update unit test container setup and instructionsJeremy Stanley2022-08-051-8/+15
| | | | | | | | | | | | | | | | | Update the TESTING.rst file to reflect that just installing and starting zookeeperd is no longer sufficient now that we require TLS and auth for the connection, as well as running database servers. Suggest the container-based setup script instead. Also improve that script to allow it to be invoked as a normal user, with root command escalation tool choice (e.g. "sudo") supplied through a ROOTCMD environment variable, so that things created inside the git worktree like the CA don't end up root-owned. Related, the tox-docker plugin previously suggested in the document is no longer a viable option for the same reasons, so clean up the plumbing for it in tox.ini as well. Change-Id: Iac32799425a5bd4b1bdbf56f34a2310241ac4499
* Merge "Bump up to python3.10"Zuul2022-07-051-2/+2
|\
| * Bump up to python3.10Clark Boylan2022-07-011-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds python3.10 testing on Jammy and switches the docker images to python3.10 from 3.8. We run sudo for postgres with -Hi to avoid non fatal errors when postres' client attempts to write command history to Zuul's homedir (it is running as the postgres user which can't write to zuul's homedir). We also need to update the libffi package version for jammy to 8 in bindep.txt. Finally, python_version values need to be quoted as "3.10" is different than 3.10 which is equivalent to 3.1 when serialized by yaml as a float. Force setuptools to use stdlib (shipped by the distro) distutils to avoid problems with virtualenvs not actually being virtualenvs. Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy as the default python there is 3.10. Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
* | Add a script to find deprecated queue syntaxJames E. Blair2022-05-051-0/+61
| | | | | | | | | | | | | | | | | | This was deprecated a while ago. This adds a script to help users identify where it still needs to be updated. See https://zuul-ci.org/docs/zuul/latest/releasenotes.html#relnotes-4-1-0-deprecation-notes Change-Id: Ib2102b9dfa048015d9fbbb1f91bc50f0c1909d53
* | Cleanup actiongeneral usageClark Boylan2022-05-041-5/+5
|/ | | | | | | | A couple of locations continue to reference actiongeneral which has been removed. Update these locations to use action as the current location for these plugins. Change-Id: I71c03d2c0a84592be66fa0d84bc684684a392a27
* Merge "Fix Github debugging script"Zuul2022-04-291-7/+19
|\
| * Fix Github debugging scriptSimon Westphahl2022-03-081-7/+19
| | | | | | | | | | | | Make the Github debugging script work with the evolved API. Change-Id: I60e229fce5329215417bc52a1adc15d69f7495c8
* | Start zookeeper with users uid in test-setup-docker.shAlbin Vass2022-03-252-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | The zookeeper containers entrypoint will try to switch to the user 'zookeeper' if the container is started with uid '0'. This can cause issues when the uid for the zookeeper user inside the container and the users uid outside the container differ since it will lead to zookeeper getting access denied when trying to read the certificates. This also adds logging configuration to make it easier to debug zookeeper in the future. Change-Id: I51db53fe093a294e804148f682053123f54adfe1
* | Update nodejs runtime to version 16Clark Boylan2022-03-142-2/+2
|/ | | | | | | | | | | | | We had been using version 14 which is the previous LTS. Now there are problems running: npx browserslist@latest --update-db running out of memory. Update to the current nodejs LTS version to ensure we are running an up to date runtime that hopefully performs more consistently with the browserslist command. Change-Id: Ib20c1090ea0f30b7dac2780b6ed963dd6e4b6f77
* Don't fail on missing change_queues key in status jsonFelix Edel2022-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since the pipeline state is stored in ZooKeeper, there could be cases where the change_queues key is missing in the status json. This makes API requests fail: 2022-02-22 17:56:18,390 ERROR cherrypy.error.139989033522128: [22/Feb/2022:17:56:18] HTTP Traceback (most recent call last): File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/opt/zuul/lib/python3.8/site-packages/cherrypy/lib/encoding.py", line 223, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/opt/zuul/lib/python3.8/site-packages/cherrypy/lib/jsontools.py", line 59, in json_handler value = cherrypy.serving.request._json_inner_handler(*args, **kwargs) File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/opt/zuul/lib/python3.8/site-packages/zuul/web/__init__.py", line 1050, in status_change return result_filter.filterPayload(payload) File "/opt/zuul/lib/python3.8/site-packages/zuul/web/__init__.py", line 193, in filterPayload for change_queue in pipeline['change_queues']: KeyError: 'change_queues' Fix this by using a .get() call rather than directly accessing the dictionary key by name. A similar issue was already fixed in [1]. [1]: https://review.opendev.org/c/zuul/zuul/+/829018 Change-Id: I947f58f02c3da7dad35d1fc186c7026800f7cbdd
* Merge "decrypt_secret.py: properly init SourceContext class"Zuul2022-02-011-1/+2
|\
| * decrypt_secret.py: properly init SourceContext classGonéri Le Bouder2021-12-171-1/+2
| | | | | | | | | | | | | | | | | | Change Ib4037da2f7a0f803aca24ce880dbc262375db6a4 introduced 2 extra parameters for the zuul.model.SourceContext constructor. These parameters don't really matter for decrypt_secret.py. We just pass the None value. Change-Id: Ic4f09558865349c8c479defe1b314b0c4c92dd0f
* | Merge "Add a zk-shell debug script"Zuul2022-01-271-0/+179
|\ \
| * | Add a zk-shell debug scriptJames E. Blair2022-01-271-0/+179
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is like the real zk-shell except: 1) It supports SSL 2) It automatically decompresses znodes if necessary 3) It handles sharded data 4) It doesn't have many commands Change-Id: I2124f216f08f68ca63b763cd6f6adce452c07f79
* | | Merge "Add some ZK debug scripts"Zuul2022-01-272-0/+539
|\ \ \ | |/ /
| * | Add some ZK debug scriptsJames E. Blair2022-01-262-0/+539
| |/ | | | | | | | | | | | | | | | | | | | | | | | | These may be useful for zuul developers to understand issues with the ZK data storage. zk-dump will dump an approximation of the contents of ZK to the filesystem for manual examination. zk-analyze will perform some analysis on the tree to identify objects which may be execessively large. Change-Id: I1a90cce42da719eee0a5e50242034390722d518e
* | Add a script to rebuild old docsJames E. Blair2022-01-201-0/+78
| | | | | | | | | | | | | | | | This can be used immediately to manually build old docs with updated themes and URLs. We can iterate on this to make it a periodic job so it happens automatically. Change-Id: I021e236ffcfd436e1169e9e804efee013cbe10aa
* | Allow to specify MySQL port for test environmentdjager2022-01-191-1/+1
|/ | | | | | | | | | | | | | In case for whatever reason there is already another MySQL server running on the standard port developers may want to run the MySQL test-server on a different port. This can be done by specifying an environment variable "ZUUL_MYSQL_PORT": export ZUUL_MYSQL_PORT=3307 tools/test-setup-docker.sh tox -e py3 Change-Id: Id32bac1a87619803d973b066762d525bd9a4f913
* Update zuul-changes for use with zuul-clientJames E. Blair2021-12-141-5/+14
| | | | | | | | The gearman RPC methods are being removed, so this needs a change to work with one of the web-based methods. The most simpla end forward-looking method is zuul-client, so update it to use that. Change-Id: If3f6ca4bae2b2beddb3bb71b36fdcba112722186
* Increase MySQL connection limit in testsFelix Edel2021-11-242-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | When executing the unit tests with multiple schedulers in parallel a lot of them are failing becasue there are too many open SQL connections. However, executing those tests one after another doesn't end up in the connection limit being exhausted. So this doesn't look like a shutdown problem in Zuul. Having a look at the MySQL server when it's under load during the tests revealed that the default connection limit of 151 on the server side is exhausted very quickly when a lot of tests are running in parallel (each test running with two schedulers). Therefore, this change increases the default MySQL connection limit to 300. We should keep in mind that this connection limit has to go in hand with the number of schedulers used in the unit tests and the number of tests executed in parallel. Maybe on the long term it might make sense to cap the latter parameter somehow in the tox.ini file. The same applies to the ZooKeeper connection limit which is configured in tools/zoo.cfg. Change-Id: Iff76e99ec82edc8e8bc110a22a096bb689d8dd1f
* Rely on bullseye image for skopeo in container imageJames E. Blair2021-10-271-21/+0
| | | | | | | | Debian bullseye includes skopeo, so we can drop the kubic repository (which as I write this is having synchronization errors) in favor of just using the version from the underlying OS. Change-Id: Ifde147c8c690dd6a421b0133dbabcff7dd9b9649
* Don't use --verbose with yarnJames E. Blair2021-10-211-1/+1
| | | | | | | This appears to output something like 145,000 lines in our build logs, so let's be less verbose. Change-Id: Ie41c1f21a090de84cb51331c00668b9b7f06379a
* Add ExistingDataWatch classJames E. Blair2021-07-021-0/+1
| | | | | | | | | | | | | | | | | This adds a modified kazoo.DataWatch class which does not set watches on paths that don't exist. This is mostly so that when a DataWatched path is deleted, we don't leave a watch in place. Without that, the executor api will leak watches on the ZK cluster. We vendor the entire watches.py file from kazoo (ASL2 licensed) to avoid any issues with API changes. Separately we will see if they are interested in this class upstream. This updates the tests to use the wchp 4lw to test that we don't leak watches. Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/799334 Change-Id: Ie4491eef03b58d858d95c78ba9454839d169cff1
* Merge "Add a tool to decrypt a secret"Zuul2021-05-281-0/+49
|\
| * Add a tool to decrypt a secretJames E. Blair2021-04-121-0/+49
| | | | | | | | | | | | This is mostly a debugging aid. Change-Id: I843c9a993299c86b66fa5e6c11aed08769d79980
* | Really retry yarn installsJames E. Blair2021-05-051-0/+4
| | | | | | | | | | | | The set -e was subverting the yarn retry loops in tools/pip.sh. Change-Id: Ibee12479336670142fdc6733e744ab5273cb8c5b
* | Retry yarn package installsJames E. Blair2021-05-031-1/+12
|/ | | | | | | | | We're seeing occasional npm install failures in jobs; retry the "yarn install" command if it fails. Also, pass the YARN_REGISTRY env var in the tox remote job. Change-Id: Ic80dccf0869ce805368f678eafcfdf48f8bf9bea
* Use ZooKeeper TLS in testsJames E. Blair2021-03-084-10/+27
| | | | | | | | This mirrors the configuration in Nodepool for using TLS-enabled ZooKeeper in tests. We use the ensure-zookeeper role in order to get a newer ZooKeeper than is supplied in bionic. Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
* Prepare Zookeeper for scale-out schedulerJan Kubovy2021-02-151-0/+2
| | | | | | | | | | | | | | | | | | This change is a common root for other Zookeeper related changed regarding scale-out-scheduler. Zookeeper becoming a central component requires to increase "maxClientCnxns". Since the ZooKeeper class is expected to grow significantly (ZooKeeper is becoming a central part of Zuul) a split of the ZooKeeper class (zk.py) into zk module is done here to avoid the current god-class. Also the zookeeper log is copied to the "zuul_output_dir". Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19 Story: 2007192
* tools: Deprecate encrypt_secret.py, document zuul-client encryptMatthieu Huin2020-12-091-0/+5
| | | | | | | | | | Now that zuul-client's encrypt subcommand covers the same functionalities as encrypt_secret.py, add a deprecation message when running the script. Document the zuul-client encrypt command in the doc section about secrets. Change-Id: Id5437ffbb688cb80b2744db3beeaa28c97080d90 Depends-On: https://review.opendev.org/765313
* install-js-tools: add support for manjaroAlbin Vass2020-11-071-0/+2
| | | | Change-Id: Ia7041423eb0bd5b688c149e70fd06373251e9ced
* tools: use python3 for encrypt_secretMohammed Naser2020-10-291-1/+1
| | | | Change-Id: Id1a167e1f722f51b50dd2f0f1bda3a7e6a32d182
* Update kubic signatureTobias Henkel2020-10-121-9/+9
| | | | | | | The kubic repo has changed it signature key so update it to unbreak container builds. Change-Id: I3edc9bebfc6807fb2ed76e615fc9eeff48c03911
* Use npm mirrors in tox jobsTobias Henkel2020-07-281-1/+11
| | | | | | | | | | | | Opendev runs a caching mirror for npm packages. Use it when running yarn install during tox tests. Due to the way yarn.lock files work at the moment we need to patch the lockfile on the fly to inject the mirror urls [1]. This can be handlel inside the pip.sh where we patch the lockfile, run yarn and reset it afterwards. [1] https://github.com/yarnpkg/yarn/issues/2566 Change-Id: Id1f2daac8327f3110e6beeb2d6391baccfd3ce34
* Add a script to find untrusted execution tasksJames E. Blair2020-07-221-0/+173
| | | | | | | | | | This change adds a script to check if a zuul deployment is using untrusted command on the executor host. Operator needs to take action on the affected job to avoid failure after upgrading to zuul v3.19.1. Change-Id: I5d18df3fd120e8f4c189ce0b59aa89553e9df816
* Block localhost shell tasks in untrusted playbooksTobias Henkel2020-07-211-1/+1
| | | | | | | | | | | | | | | | | | | Zuul was designed to block local code execution in untrusted environments to not only rely on bwrap to contain a job. This got broken since the creation of a command plugin that injects the zuul_job_id which is required for log streaming. However this plugin doesn't do a check if the task is a localhost task. Further it is required in trusted and untrusted environments due to log streaming. Thus we need to fork this plugin and restrict the variant that is used in untrusted environments. We do this by moving actiongeneral/command.py back to action/*. We further introduce a new catecory actiontrusted which gets the unrestricted version of this plugin. Change-Id: If81cc46bcae466f4c071badf09a8a88469ae6779 Story: 2007935 Task: 40391
* tests: use a more inclusive word for the default passwordTristan Cacqueray2020-07-093-3/+3
| | | | | | This change removes problematic language in the test environment. Change-Id: I6f81ab68314902f0239babf887de04f8e30d8954
* Merge "Revert "Revert "Create zuul/web/static on demand"""Zuul2020-07-081-0/+2
|\
| * Revert "Revert "Create zuul/web/static on demand""Tobias Henkel2020-07-071-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | The js content tarball creation was broken. Instead of leaving the symlink to a non-existing directory which gets created during python setup we should just remove that symlink as well and create the symlink and the static dir during the python setup. This way nothing will be in the way of the javascript content generation. This reverts commit eb7b18b38e01688b5b05e5dfdb359ff5d42d1ed8. Change-Id: I5f8bfa62cd2d4d9823b86dbcda14885230847a82
* | Correct syntax to be compatible with mysql8Riccardo Pittau2020-06-111-2/+2
| | | | | | | | | | | | | | | | Starting from mysql version 8 it's not possible to create a user implicitely when using GRANT. This patch makes the behavior compatible with that. Change-Id: I1dc25194de384bf14607aff4e0f230e394f00582
* | Use Absolute path in zk-ca.sh for openssl configAlbin Vass2020-05-281-1/+2
|/ | | | | | | | | | | The zk-ca.sh tool for generating certificates fails with and error that ./tools/openssl.cnf cannot be opened if a relative path is supplied on ubuntu-bionic. So expand the path to an absolute path instead. http://paste.openstack.org/show/794091/ Change-Id: Ib30e9754d4520b956e83cae014c1be181ae44631
* Update node to v14 and update to new jobsMonty Taylor2020-05-201-2/+1
| | | | | | | | | | | | | | v14 is the latest lts. Let's use it. Also - rename the jobs to make it clear what they're doing, and add a dashboard job that points at opendev's multi-tenat api too. There are new jobs that default to latest node LTS and auto-detect yarn vs npm. Update to use them. Depends-On: https://review.opendev.org/728097 Depends-On: https://review.opendev.org/726547 Change-Id: I5717edea2cd09acc5bce673c38bbe7fa8057a376
* Merge "Retry nodeenv creation"Zuul2020-05-161-1/+9
|\
View Lorry Controller Status