Merge "Update the Tempest README for new run workflow"

1 files changed, 100 insertions, 37 deletions

diff --git a/README.rst b/README.rst
index 7a7dfa619..d94fbdd6d 100644
--- a/README.rst
+++ b/README.rst
@@ -37,57 +37,61 @@ Tempest Design Principles that we strive to live by.
 Quickstart
 ----------
 
-To run Tempest, you first need to create a configuration file that
-will tell Tempest where to find the various OpenStack services and
-other testing behavior switches.
+To run Tempest, you first need to create a configuration file that will tell
+Tempest where to find the various OpenStack services and other testing behavior
+switches. Where the configuration file lives and how you interact with it
+depends on how you'll be running Tempest. There are 2 methods of using Tempest.
+The first, which is a newer and recommended workflow treats Tempest as a system
+installed program. The second older method is to run Tempest assuming your
+working dir is the actually Tempest source repo, and there are a number of
+assumptions related to that. For this section we'll only cover the newer method
+as it is simpler, and quicker to work with.
 
-The easiest way to create a configuration file is to generate a sample
-in the ``etc/`` directory ::
+#. You first need to install Tempest this is done with pip, after you check out
+   the Tempest repo you simply run something like::
 
-    $> cd $TEMPEST_ROOT_DIR
-    $> oslo-config-generator --config-file \
-        tools/config/config-generator.tempest.conf \
-        --output-file etc/tempest.conf
-
-After that, open up the ``etc/tempest.conf`` file and edit the
-configuration variables to match valid data in your environment.
-This includes your Keystone endpoint, a valid user and credentials,
-and reference data to be used in testing.
-
-.. note::
-
-    If you have a running devstack environment, Tempest will be
-    automatically configured and placed in ``/opt/stack/tempest``. It
-    will have a configuration file already set up to work with your
-    devstack installation.
+    $ pip install tempest
 
-Tempest is not tied to any single test runner, but `testr`_ is the most commonly
-used tool. Also, the nosetests test runner is **not** recommended to run Tempest.
+   This can be done within a venv, but the assumption for this guide is that
+   the Tempest cli entry point will be in your shell's PATH.
 
-After setting up your configuration file, you can execute the set of Tempest
-tests by using ``testr`` ::
+#. Installing Tempest will create a /etc/tempest dir which will contain the
+   sample config file packaged with Tempest. The contents of /etc/tempest will
+   be copied to all local working dirs, so if there is any common configuration
+   you'd like to be shared between anyone setting up local Tempest working dirs
+   it's recommended that you copy or rename tempest.conf.sample to tempest.conf
+   and make those changes to that file in /etc/tempest
 
-    $> testr run --parallel
+#. Setup a local working Tempest dir. This is done using the tempest init
+   command::
 
-.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html
+    tempest init cloud-01
 
-To run one single test serially ::
+   works the same as::
 
-    $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
+    mkdir cloud-01 && cd cloud-01 && tempest init
 
-Alternatively, you can use the run_tempest.sh script which will create a venv
-and run the tests or use tox to do the same. Tox also contains several existing
-job configurations. For example::
+   This will create a new directory for running a single Tempest configuration.
+   If you'd like to run Tempest against multiple OpenStack deployments the idea
+   is that you'll create a new working directory for each to maintain separate
+   configuration files and local artifact storage for each.
 
-   $> tox -efull
+#. Then cd into the newly created working dir and also modify the local
+   config files located in the etc/ subdir created by the ``tempest init``
+   command. Tempest is expecting a tempest.conf file in etc/ so if only a
+   sample exists you must rename or copy it to tempest.conf before making
+   any changes to it otherwise Tempest will not know how to load it.
 
-which will run the same set of tests as the OpenStack gate. (it's exactly how
-the gate invokes Tempest) Or::
+#. Once the configuration is done you're now ready to run Tempest. This can
+   be done with testr directly or any `testr`_ based test runner, like
+   `ostestr`_. For example, from the working dir running::
 
-  $> tox -esmoke
+     $ ostestr --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.(api|scenario|thirdparty))'
 
-to run the tests tagged as smoke.
+   will run the same set of tests as the default gate jobs.
 
+.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html
+.. _ostestr: http://docs.openstack.org/developer/os-testr/
 
 Configuration
 -------------
@@ -146,3 +150,62 @@ work on Python 3.4. However, because large parts of Tempest are self-verifying
 there might be uncaught issues running on Python 3.4. So until there is a gating
 job which does a full Tempest run using Python 3.4 there isn't any guarantee
 that running Tempest under Python 3.4 is bug free.
+
+Legacy run method
+-----------------
+
+The legacy method of running Tempest is to just treat the Tempest source code
+as a python unittest repository and run directly from the source repo. When
+running in this way you still start with a Tempest config file and the steps
+are basically the same except that it expects you know where the Tempest code
+lives on your system and requires a bit more manual interaction to get Tempest
+running. For example, when running Tempest this way things like a lock file
+directory do not get generated automatically and the burden is on the user to
+create and configure that.
+
+To start you need to create a configuration file. The easiest way to create a
+configuration file is to generate a sample in the ``etc/`` directory ::
+
+    $> cd $TEMPEST_ROOT_DIR
+    $> oslo-config-generator --config-file \
+        tools/config/config-generator.tempest.conf \
+        --output-file etc/tempest.conf
+
+After that, open up the ``etc/tempest.conf`` file and edit the
+configuration variables to match valid data in your environment.
+This includes your Keystone endpoint, a valid user and credentials,
+and reference data to be used in testing.
+
+.. note::
+
+    If you have a running devstack environment, Tempest will be
+    automatically configured and placed in ``/opt/stack/tempest``. It
+    will have a configuration file already set up to work with your
+    devstack installation.
+
+Tempest is not tied to any single test runner, but `testr`_ is the most commonly
+used tool. Also, the nosetests test runner is **not** recommended to run Tempest.
+
+After setting up your configuration file, you can execute the set of Tempest
+tests by using ``testr`` ::
+
+    $> testr run --parallel
+
+.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html
+
+To run one single test serially ::
+
+    $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
+
+Alternatively, you can use the run_tempest.sh script which will create a venv
+and run the tests or use tox to do the same. Tox also contains several existing
+job configurations. For example::
+
+   $> tox -efull
+
+which will run the same set of tests as the OpenStack gate. (it's exactly how
+the gate invokes Tempest) Or::
+
+  $> tox -esmoke
+
+to run the tests tagged as smoke.