summaryrefslogtreecommitdiff
path: root/tools
Commit message (Collapse)AuthorAgeFilesLines
* CI: fix backport validator for new branch namingElod Illes2023-05-111-1/+1
| | | | | | | | validate-backport job started to fail as only old stable branch naming is accepted. This patch extends the script to allow numbers and dot as well in the branch names (like stable/2023.1). Change-Id: Icbdcd5d124717e195d55d9e42530611ed812fadd
* Test setting the nova job to centos-9-streamAde Lee2022-07-131-1/+7
| | | | | | Depends-On: https://review.opendev.org/c/openstack/tempest/+/831607 Change-Id: Ic8da6ee0313a911d742190ea5b0d4362cb6aef2f
* tools: Ignore bot-generated branch creation patchesStephen Finucane2021-09-271-0/+5
| | | | | | | | | Don't force infra to change their bot because of our weird script. If it's from the OpenStack Release Bot, it's probably good and we can skip the backport check. Change-Id: I682bd198a74fc252e47e8bdfb33bd21782b63ff9 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* db: Final cleanupsStephen Finucane2021-08-173-481/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | Some things that were missed in previous patches and are thrown together here: - Add alembic as an explicit dependency (we were getting it transitively from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x release, which is the minimum supported by our chosen version of alembic (more on this below) - Remove tooling related to the old migrations - Fix the tox whitelisting of the flaky MySQL tests On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13 we see errors like the following in some tests: sqlalchemy.exc.InvalidRequestError: Entity namespace for "count(instance_mappings.id)" has no property "queued_for_delete" There's nothing specific about this in the release notes for 1.4.13 [1] but it definitely fixes things. [1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13 Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* db: Move remaining 'nova.db.sqlalchemy' modulesStephen Finucane2021-08-091-1/+1
| | | | | | | | The two remaining modules, 'api_models' and 'api_migrations', are moved to the new 'nova.db.api' module. Change-Id: I138670fe36b07546db5518f78c657197780c5040 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* db: Unify 'nova.db.api', 'nova.db.sqlalchemy.api'Stephen Finucane2021-08-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | Merge these, removing an unnecessary layer of abstraction, and place them in the new 'nova.db.main' directory. The resulting change is huge, but it's mainly the result of 's/sqlalchemy import api/main import api/' and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We also need to rework how we do the blocking of API calls since we no longer have a 'DBAPI' object that we can monkey patch as we were doing before. This is now done via a global variable that is set by the 'main' function of 'nova.cmd.compute'. The main impact of this change is that it's no longer possible to set '[database] use_db_reconnect' and have all APIs automatically wrapped in a DB retry. Seeing as this behavior is experimental, isn't applied to any of the API DB methods (which don't use oslo.db's 'DBAPI' helper), and is used explicitly in what would appear to be the critical cases (via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't seem like a huge loss. Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* db: Move main DB migrationsStephen Finucane2021-07-052-2/+2
| | | | | | | | We place these in a 'legacy_migrations' directory, as we will soon be adding alembic-based migrations in a 'migration' directory. Change-Id: Ib927e4c48f59a467a913875111ffbf64ffe0de90 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* tests: Allow bindep and test-setup.sh to run on EL distrosLee Yarwood2021-06-181-0/+35
| | | | | | | | | Only install mariadb on EL based hosts. Also when using mariadb and postgresql on EL based distros we need to ensure each service is configured and actually started before using either. Co-Authored-By: Ade Lee <alee@redhat.com> Change-Id: I7122933d85bd7d0333c2c35e0f1a8414c1baa6d5
* Move 'check-cherry-picks' test to gate, n-v checkStephen Finucane2021-06-161-5/+0
| | | | | | | | | | | | | This currently runs in the 'check' pipeline, as part of the pep8 job, which causes otherwise perfectly valid backports to report as failing CI. There's no reason a stable core shouldn't be encouraged to review these patches: we simply want to prevent them *merging* before their parent(s). Resolve this conflict by moving the check to separate voting job in the 'gate' pipeline as well as a non-voting job in the 'check' pipeline to catch more obvious issues. Change-Id: Id3e4452883f6a3cf44ff58b39ded82e882e28c23 Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* Add generate schemas toolStephen Finucane2021-01-181-0/+123
| | | | | | | | | This tool was used while building the compact-db-migrations-wallaby series during Wallaby [1]. It might be helpful in the future, so commit it now. Change-Id: Idd556fac96ce621ad095862597c4f11851852fed Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* tools: Allow check-cherry-picks.sh to be disabled by an env varLee Yarwood2020-12-071-0/+5
| | | | | | | | The checks performed by this script aren't always useful to downstream consumers of the repo so allow them to disable the script without having to make changes to tox.ini. Change-Id: I4f551dc4b57905cab8aa005c5680223ad1b57639
* Follow up for cherry-pick check for merge patchmelanie witt2020-10-081-1/+1
| | | | | | | | | | | | | This is a follow up to change I8e4e5afc773d53dee9c1c24951bb07a45ddc2f1a which fixed an issue with validation when the topmost patch after a Zuul rebase is a merge patch. We need to also use the $commit_hash variable for the check for stable-only patches, else it will incorrectly fail because it is checking the merge patch's commit message. Change-Id: Ia725346b65dd5e2f16aa049c74b45d99e22b3524
* tools: Remove xenserver toolingStephen Finucane2020-08-316-882/+0
| | | | | | | These will not be used in a world without the XenAPI driver. Change-Id: I5bc3c7855b817c4ce2b8919c76be80cceabeec9e Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* Merge "Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)"Zuul2020-07-021-2/+2
|\
| * Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)Ghanshyam Mann2020-06-301-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In Ubuntu Bionic (18.04) mysql 5.7 version used to create the user implicitly when using using the GRANT. Ubuntu Focal (20.04) has mysql 8.0 and with mysql 8.0 there is no implicit user creation with GRANT. We need to create the user first before using GRANT command. Nova unit and functional tests job using tools/test-setup.sh script start failing when running on Ubuntu Focal https://zuul.opendev.org/t/openstack/build/8b0f4fcc21854655a638c413b6fe1a91 Error log: mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 1064 (42000) at line 4: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by 'openstack_citest' WITH GRANT OPTION' at line 2 Story: #2007865 Task: #40200 Change-Id: I97b0dcbb88c6ef7c22e3c55970211bed792bbd0d Closes-Bug: #1885825
* | Fix cherry-pick check for merge patchElod Illes2020-06-261-1/+11
|/ | | | | | | | | | | | Cherry-pick check script validates the proposed patch's commit message. If a patch is not on top of the given branch then Zuul rebases it to the top and the patch becomes a merge patch. In this case the script validates the merge patch's commit message instead of the original patch's commit message and fails. This fix selects the parent of the patch if it is a merge patch. Change-Id: I8e4e5afc773d53dee9c1c24951bb07a45ddc2f1a
* Check cherry-pick hashes in pep8 tox targetDan Smith2020-06-151-0/+32
| | | | | | | | | | This adds a tools/ script that checks any cherry-picked hashes on the current commit (or a provided commit) to make sure that all the hashes exist on at least master or stable/.* branches. This should help avoid accidentally merging stable backports where one of the hashes along the line has changed due to conflicts. Change-Id: I4afaa0808b75cc31a8dd14663912c162281a1a42
* tox: Integrate mypyStephen Finucane2020-05-151-0/+22
| | | | | | | | | | | | | | | | | | mypy is an experimental optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typing. While still in development, most features are supported and it's already being used by real world projects like Sphinx. Let's start small by integrating it into some of the interfaces that nova exposes. We can further build upon this if it works out. This change sets up the boilerplate necessary to use mypy in nova. Type annotations are not included for any module - these will be added separately. We're calling mypy by way of a script, as this allows us to store a list of files that we have converted while we're in the process of adding type annotations where necessary. Change-Id: I75ab46a6768c4ca2050fdde2b7f8eeb90724c8c6 Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
* Remove future importsStephen Finucane2020-03-241-2/+0
| | | | | | | These particular imports are no longer needed in a Python 3-only world. Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* Keep pre-commit inline with hacking and fix whitespaceMonty Taylor2019-12-121-18/+18
| | | | | | | | | | | | | | | | | | The flake8 hook for pre-commit installs its own flake8 which is not what hacking is pinned to. This results in a bunch of errors. By doing a local hook we can cause these to be the same. There are flake8 exclusions in tox.ini that need to be used in pre-commit. Also we should exclude svg files from the whitespace modifications, because who cares. Once those are fixed, running pre-commit -a fixes some whitespace issues. While the normal use isn't to run -a - if someone does make a patch that touches any of these they'll get erroneous unrelated errors compared to their lower work - so fix all of the existing issues, which are mostly space-at-end-of-line issues it seems. Change-Id: I78e753492e99c03b30a0a691db3bd75ee3d289c9
* Integrate 'pre-commit'Stephen Finucane2019-11-281-0/+7
| | | | | | | | | | | | | | | | | | | | This nifty tool provides a extensible way to configure pre-commit hooks to do everything from running linters to spellchecking [1] and has already been adopted by a few OpenStack projects [2]. Let's start using it as a faster, more automatic alternative to the 'fast8' tox target and iterate on it as we find more things to use it for. The 'fast8' target is not deprecated though a note is included for end-users to alert them to this great new tool. Note that the 'check-executables-have-shebangs' actually failed and highlighted that the 'contributor/development-environment.rst' document was marked as executable when it shouldn't have been. This is corrected. [1] https://pre-commit.com/ [2] https://review.opendev.org/#/q/file:.pre-commit-config.yaml Change-Id: I256e75eceeb3495dce04885ca42d16adb4160cf3 Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
* Make it easier to run a selection of tests relevant to ongoing workAdam Spiers2019-08-192-0/+132
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During development of a new git commit, locally running a whole unit or functional test suite to check every minor code change is prohibitively expensive. For maximum developer productivity and happiness, it's generally desirable to make the feedback loop of the traditional red/green cycle as quick as possible. So add run-tests-for-diff.sh and run-tests.py to the tools/ subdirectory, using a few tricks as explained below to help with this. run-tests.py takes a list of files on STDIN, filters the list for tests which can be run in the current tox virtualenv, and then runs them with the correct stestr options. run-tests-for-diff.sh is a simple wrapper around run-tests.py which determines which tests to run using output from "git diff". This allows running only the test files changed/added in the working tree: tools/run-tests-for-diff.sh or by a single commit: tools/run-tests-for-diff.sh mybranch^! or a range of commits, e.g. a branch containing a whole patch series for a blueprint: tools/run-tests-for-diff.sh gerrit/master..bp/my-blueprint It supports the same "-HEAD" invocation syntax as flake8wrap.sh (as used by the "fast8" tox environment): tools/run-tests-for-diff.sh -HEAD run-tests.py uses two tricks to make test runs as quick as possible: 1. It's (already) possible to speed up running of tests by source'ing the "activate" file for the desired tox virtualenv, e.g. source .tox/py36/bin/activate and then running stestr directly. This saves a few seconds by skipping the overhead introduced by running tox. 2. When only one test file needs to be run, specifying the -n option to stestr will skip the costly test discovery phase, saving several more valuable seconds. Future commits could build on top of this work, harnessing a framework such as watchdog / watchmedo[0] or Guard[1] in order to automatically run relevant tests every time your editor saves changes to a .py file. [0] https://github.com/gorakhargosh/watchdog - Python-based [1] https://guardgem.org - probably best in class, but Ruby-based so maybe unacceptable for use within Nova. Change-Id: I9a9bda5d29bbb8d8d77f769cd1abf7c42a18c36b
* Replace git.openstack.org URLs with opendev.org URLsZhongShengping2019-04-241-7/+7
| | | | | | | Thorough replacement of git.openstack.org URLs with their opendev.org counterparts. Change-Id: I3e0af55e0707f04428a422b973d016ad30c82a12
* Make Xen code py3-compatibleMatthew Edmonds2018-08-101-6/+6
| | | | | | | | | | | | | | | | | | | A couple places in the Xen code have been using syntax that is python2-specific. This replaces those instances with code that will work in both python2 and python3. PEP 3108 [1] moves urllib.urlopen to urllib.request.urlopen. The six module is used in order to work in both python2 and python3 [2]. PEP 3110 [3] changed the grammar for except clauses, such that the 'as' keyword is now required in place of a comma when specifying an exception variable. [1] https://www.python.org/dev/peps/pep-3108/ [2] https://pythonhosted.org/six/#module-six.moves.urllib.request [3] https://www.python.org/dev/peps/pep-3110/ Change-Id: I1235d767718a4207f4cef3e5b140319d003ad7b0
* Convert 'placement_api_docs' into a Sphinx extensionStephen Finucane2018-07-021-71/+0
| | | | | | | This ensures it will get run in the gate. Change-Id: I923c39d3115b595b8de94e85d2977eb7782d98e2 Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
* trivial: Remove 'tools/releasenotes_tox.sh'Stephen Finucane2018-06-281-28/+0
| | | | | | | | This is not needed since reno 2.1.0 (commit f8fc8f97) [1]. [1] https://github.com/openstack/reno/commit/f8fc8f97ff20026582742e3e7838cdd0ed5cad68 Change-Id: I66ba0e0835970b965acce00ba69d143bd4471037
* Remove deprecated monkey_patch config optionsMatt Riedemann2018-05-161-2/+0
| | | | | | | | | | These were deprecated in Queens: Id793ac2c5cdc5dc473f95eac53b77617a1e389da And can now be removed. Change-Id: I27518d1154ec09c57700905bc58ae6a3ce1d8049
* Finish stestr migrationMatthew Treinish2017-11-241-15/+0
| | | | | | | | | | | Nova switched to using stestr for all it's in tree testing except for the coverage job. This commit updates that job so it also uses stestr and cleans up the last bits of testr setup left in tree. As part of this change this moves all the one off per job configuration into the tox job definitions instead of hiding it in either the .stestr.conf or in a wrapper script. Change-Id: I36e0e791d485b44641a1ed8770eca8e95aca19be
* xenapi: cached images should be cleaned up by timenaichuans2017-10-231-2/+8
| | | | | | | | | For xenapi driver, there needs to be some way to delete cached images based on when they were created. add an optional arg to control delete operation. Change-Id: I24fc45e989aa951aab55a261fce77f7e3667d988 Closes-bug: 1481689
* Make setenv consistent for unit, func, and api-samplesmelanie witt2017-10-051-2/+2
| | | | | | | | | | | | Since ostestr switched to running stestr under the covers, we lost the old magic setting of the environment variables via .testr.conf for capturing stderr/stdout and the test timeout. This makes the unit, functional, and api-samples envs consistent with the py27 env that was already updated to set those variables. This also updates the pretty_tox3.sh script to run stestr directly. Change-Id: I27fa9b7e25c1a1dc921653eec84864423f898a85
* Merge "trivial: Remove files from 'tools'"Jenkins2017-09-062-144/+0
|\
| * trivial: Remove files from 'tools'Stephen Finucane2017-08-072-144/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two files are removed. 'ebtables.workaround' was required for libvirt < 1.2.11 for the reasons described in the file itself. We no longer support this version of libvirt, thus, the workaround can be removed. 'regression_tester' is a tool for running tests in a patch against old code, to ensure the tests validate what the patch is supposed to be fixing. This is unmaintained and is not referenced anywhere, and should therefore be removed as it's likely bitrotted (or will do so). Change-Id: Iac7f16ff4c178b5e71b786c332ca46a78bef5e3c TrivialFix
* | Prevent blank line at start of migration placeholdersChris Dent2017-08-251-2/+2
| | | | | | | | | | | | Because those blank lines sure are annoying. Change-Id: I7b73536a89d18e0752651c198018eff6cc8b0e57
* | [placement] Make placement_api_docs.py failingAndrey Volkov2017-08-101-1/+3
| | | | | | | | | | | | | | If there is no api reference for some placement API method tox -e placement-api-ref will fail now. Change-Id: Ifefedcd51f1f8d5e741ef6d02b7a0879a185e05f
* | tools/xenserver: Remove 'cleanup_sm_locks'Stephen Finucane2017-08-071-123/+0
|/ | | | | | | | | XenServer 6.0 has long since been replaced and the bug was fixed in 6.1 [1]. No need to keep this legacy script around. [1] https://bugs.launchpad.net/nova/+bug/1103158/comments/1 Change-Id: I8c41e88281aaf4250859622142a2f03f4452e1be
* [placement] Fix placement-api-ref check toolAndrey Volkov2017-05-241-6/+11
| | | | | | | | | | | Some time ago the placement-api-ref source structure was changed (Ia2fd62ae7f401cad34ee7c2b355c9a5ab1c93f6b) to use inc-files istead of putting all in index.rst, but that change broke placement_api_docs.py. This change modifies placement_api_docs.py to go through all *.inc files in placement-api-ref directory and scan for the "rest method" header. Change-Id: I60150ab51173934f848662499dcf6af18625697e
* Merge "Structure for simply managing placement-api-ref"Jenkins2017-03-241-0/+64
|\
| * Structure for simply managing placement-api-refChris Dent2017-03-241-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This consists of a duplicate of the [nova-]api-ref setup and conf.py along with tooling to fail the tox -edocs target when a route that is defined in nova.api.openstack.placement.handler.ROUTE_DECLARATIONS is not present in placement-api-ref/source/index.rst. tools/placement_api_docs.py will report which routes are missing. Though completely gameable (as demonstrated in the current lame index.rst) it's better than nothing and provides some useful structuring on what to do next. It's also the case that the 'docs' target in tox is not part of gating. The response for GET / is in place with the necessary parameters.yaml for it to be correctly described. The 'get-root.json' file provides the JSON of the expected response. The expectation is that later commits will add information for other urls and their JSON files will be named method-path-separated-by-dash.json with a request/response qualifier as necessary. Followup patches will add other routes. A new parameters.yaml is used instead of reusing the one from api-ref as there isn't a lot of expected overlap and having a separate file will ease eventual extraction. Running tox -eplacement-api-ref will generate the docs for review, with output in placement-api-ref/build/html/index.html. This will be hooked up with CI to deploy the generated docs, eventually. Change-Id: Ifb4d91d39db0e49b55952e37cdfc9f63dcd37aa3
* | Merge "More usage of ostestr and cleanup an unused dependency"Jenkins2017-03-201-16/+0
|\ \
| * | More usage of ostestr and cleanup an unused dependencyLuigi Toscano2017-02-231-16/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use ostestr as runner instead of the custome pretty_tox.sh script (pretty much removed everywhere). pretty_tox3.sh is just a wrapper for ostestr but it could be probably removed as well going forward. Also remove the tempest-lib dependency, which originally provided subunit-trace (now from os-testr). Change-Id: I4390964ce83167d634b80f696e82a0e199582ce6
* | | Remove usage of config option verboseChangBo Guo(gcb)2017-02-272-23/+9
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | oslo.log deprecate config option verbose since aug 1, 2015 in review: in https://review.openstack.org/#/c/206437/ That was a long time ago so it should be possible to remove it now. This was already merged once but had to be reverted because some projects were still relying on it. Oslo team plan to remove it, so we need clean up its usage before removing it. Closes-Bug: #1663570 Change-Id: I072e1078486f50cebea92695ff46c1af879eb23d
* | Merge "tools: Remove 'colorizer'"Jenkins2017-02-231-326/+0
|\ \
| * | tools: Remove 'colorizer'Stephen Finucane2017-01-041-326/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was used by the 'run_tests.sh' script, which has long since been removed. There appears to be something similar built into ostestr [1] should people care about this (I suspect they don't). [1] https://github.com/openstack/os-testr/blob/0.8.0/os_testr/utils/colorizer.py Change-Id: I5634b3f75fa2b392a2f49a61b7b46a299bbcd2fd
* | | Merge "tools: Remove 'with_venv'"Jenkins2017-02-231-7/+0
|\ \ \ | |/ /
| * | tools: Remove 'with_venv'Stephen Finucane2017-01-041-7/+0
| | | | | | | | | | | | | | | | | | Nothing appears to use this any more. No point in keeping it around. Change-Id: I7150c78f55fd2eb639ed85c868d38461219fda12
* | | Merge "tools: Remove 'install_venv', 'install_venv_common'"Jenkins2017-02-232-245/+0
|\ \ \ | |/ /
| * | tools: Remove 'install_venv', 'install_venv_common'Stephen Finucane2017-01-042-245/+0
| | | | | | | | | | | | | | | | | | | | | tox has long since usurped this as the preferred approach to development and testing. Change-Id: I4baa1688b88135b71e1e04b4aede6d6d0197504b
* | | Merge "tools: Remove 'clean-vlans'"Jenkins2017-02-231-24/+0
|\ \ \ | |/ /
| * | tools: Remove 'clean-vlans'Stephen Finucane2017-01-041-24/+0
| | | | | | | | | | | | | | | | | | | | | Whatever function this script served in the past, it has long outlived its usefulness in the age of neutron. Change-Id: I0e5b0e0a49ee26bddbfce456aa91ffe78f62f34d
* | | Merge "tools: Remove 'enable-pre-commit-hook'"Jenkins2017-02-231-42/+0
|\ \ \ | |/ /
View Lorry Controller Status