From 828734a1204b80dbbf799f08749660bbb0a1a5f4 Mon Sep 17 00:00:00 2001 From: Matthew Treinish Date: Mon, 6 Jul 2015 15:43:46 -0400 Subject: Update the Tempest README for new run workflow This commit updates the tempest README to document the new workflow for running and interacting with tempest. The quickstart section is used for this because that is where we were previously explaining how to run tempest. The previous instructions for running tempest will still work (and should always continue to work, we do not want to break that usage model) so instead of being removed a new section at the bottom of the README is added to document this, despite not being the currently recommended workflow. Right now tempest run is excluded from this doc, since more work needs to be done to figure out how that will implemented. Partially Implements: bp tempest-cli-improvements Change-Id: Iced98b74b7ff651423fd1efa205dc0a4aaa28d5f --- README.rst | 137 ++++++++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 100 insertions(+), 37 deletions(-) (limited to 'README.rst') diff --git a/README.rst b/README.rst index 431be7c3f..1b2e474e8 100644 --- a/README.rst +++ b/README.rst @@ -37,57 +37,61 @@ Tempest Design Principles that we strive to live by. Quickstart ---------- -To run Tempest, you first need to create a configuration file that -will tell Tempest where to find the various OpenStack services and -other testing behavior switches. +To run Tempest, you first need to create a configuration file that will tell +Tempest where to find the various OpenStack services and other testing behavior +switches. Where the configuration file lives and how you interact with it +depends on how you'll be running Tempest. There are 2 methods of using Tempest. +The first, which is a newer and recommended workflow treats Tempest as a system +installed program. The second older method is to run Tempest assuming your +working dir is the actually Tempest source repo, and there are a number of +assumptions related to that. For this section we'll only cover the newer method +as it is simpler, and quicker to work with. -The easiest way to create a configuration file is to generate a sample -in the ``etc/`` directory :: +#. You first need to install Tempest this is done with pip, after you check out + the Tempest repo you simply run something like:: - $> cd $TEMPEST_ROOT_DIR - $> oslo-config-generator --config-file \ - tools/config/config-generator.tempest.conf \ - --output-file etc/tempest.conf - -After that, open up the ``etc/tempest.conf`` file and edit the -configuration variables to match valid data in your environment. -This includes your Keystone endpoint, a valid user and credentials, -and reference data to be used in testing. - -.. note:: - - If you have a running devstack environment, Tempest will be - automatically configured and placed in ``/opt/stack/tempest``. It - will have a configuration file already set up to work with your - devstack installation. + $ pip install tempest -Tempest is not tied to any single test runner, but `testr`_ is the most commonly -used tool. Also, the nosetests test runner is **not** recommended to run Tempest. + This can be done within a venv, but the assumption for this guide is that + the Tempest cli entry point will be in your shell's PATH. -After setting up your configuration file, you can execute the set of Tempest -tests by using ``testr`` :: +#. Installing Tempest will create a /etc/tempest dir which will contain the + sample config file packaged with Tempest. The contents of /etc/tempest will + be copied to all local working dirs, so if there is any common configuration + you'd like to be shared between anyone setting up local Tempest working dirs + it's recommended that you copy or rename tempest.conf.sample to tempest.conf + and make those changes to that file in /etc/tempest - $> testr run --parallel +#. Setup a local working Tempest dir. This is done using the tempest init + command:: -.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html + tempest init cloud-01 -To run one single test serially :: + works the same as:: - $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server + mkdir cloud-01 && cd cloud-01 && tempest init -Alternatively, you can use the run_tempest.sh script which will create a venv -and run the tests or use tox to do the same. Tox also contains several existing -job configurations. For example:: + This will create a new directory for running a single Tempest configuration. + If you'd like to run Tempest against multiple OpenStack deployments the idea + is that you'll create a new working directory for each to maintain separate + configuration files and local artifact storage for each. - $> tox -efull +#. Then cd into the newly created working dir and also modify the local + config files located in the etc/ subdir created by the ``tempest init`` + command. Tempest is expecting a tempest.conf file in etc/ so if only a + sample exists you must rename or copy it to tempest.conf before making + any changes to it otherwise Tempest will not know how to load it. -which will run the same set of tests as the OpenStack gate. (it's exactly how -the gate invokes Tempest) Or:: +#. Once the configuration is done you're now ready to run Tempest. This can + be done with testr directly or any `testr`_ based test runner, like + `ostestr`_. For example, from the working dir running:: - $> tox -esmoke + $ ostestr --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.(api|scenario|thirdparty))' -to run the tests tagged as smoke. + will run the same set of tests as the default gate jobs. +.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html +.. _ostestr: http://docs.openstack.org/developer/os-testr/ Configuration ------------- @@ -146,3 +150,62 @@ work on Python 3.4. However, because large parts of Tempest are self verifying there might be uncaught issues running on Python 3.4. So until there is a gating job which does a full Tempest run using Python 3.4 there isn't any guarantee that running Tempest under Python 3.4 is bug free. + +Legacy run method +----------------- + +The legacy method of running Tempest is to just treat the Tempest source code +as a python unittest repository and run directly from the source repo. When +running in this way you still start with a Tempest config file and the steps +are basically the same except that it expects you know where the Tempest code +lives on your system and requires a bit more manual interaction to get Tempest +running. For example, when running Tempest this way things like a lock file +directory do not get generated automatically and the burden is on the user to +create and configure that. + +To start you need to create a configuration file. The easiest way to create a +configuration file is to generate a sample in the ``etc/`` directory :: + + $> cd $TEMPEST_ROOT_DIR + $> oslo-config-generator --config-file \ + tools/config/config-generator.tempest.conf \ + --output-file etc/tempest.conf + +After that, open up the ``etc/tempest.conf`` file and edit the +configuration variables to match valid data in your environment. +This includes your Keystone endpoint, a valid user and credentials, +and reference data to be used in testing. + +.. note:: + + If you have a running devstack environment, Tempest will be + automatically configured and placed in ``/opt/stack/tempest``. It + will have a configuration file already set up to work with your + devstack installation. + +Tempest is not tied to any single test runner, but `testr`_ is the most commonly +used tool. Also, the nosetests test runner is **not** recommended to run Tempest. + +After setting up your configuration file, you can execute the set of Tempest +tests by using ``testr`` :: + + $> testr run --parallel + +.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html + +To run one single test serially :: + + $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server + +Alternatively, you can use the run_tempest.sh script which will create a venv +and run the tests or use tox to do the same. Tox also contains several existing +job configurations. For example:: + + $> tox -efull + +which will run the same set of tests as the OpenStack gate. (it's exactly how +the gate invokes Tempest) Or:: + + $> tox -esmoke + +to run the tests tagged as smoke. -- cgit v1.2.1