Restructured documentation

Moved information out of intro.rst and into more specific doc files.
Reorganised index.rst accordingly, and deleted unused files.

Change-Id: I660c5e5e597baddbd1d2d5f60871687b5306834c

6 files changed, 259 insertions, 429 deletions

diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
deleted file mode 100644
index 5bba790..0000000
--- a/doc/source/configuration.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-:title: Configuration
-
-
-Configuration
-=============
-
-It is required that debug logging be configured for Turbo Hispter, as we use
-the python logging framework to capture log messages from the task plugin
-code. Configuration is as simple as setting the debug_log configuration
-setting in config.json.
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 25c0f30..c9ebbd0 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -12,9 +12,7 @@ Contents:
    :maxdepth: 2
 
    intro
-   structure
    installation
-   configuration
    running
 
 
diff --git a/doc/source/installation.rst b/doc/source/installation.rst
index a0a0732..7c3de7f 100644
--- a/doc/source/installation.rst
+++ b/doc/source/installation.rst
@@ -6,142 +6,111 @@
 Installation
 ============
 
-Download
---------
-
-Grab a copy from https://github.com/rcbau/turbo-hipster
-
-
-Install
--------
-
-turbo-hipster is configured to use setup tools for installation if
-you would like to install it to your site-packages use::
-
-    sudo python setup.py install
-
-
-Copy config
------------
-
-Place the configuration where you are comfortable managing it. For
-example::
-
-    cp -R etc/turbo-hipster /etc/
-
-
-Edit config
------------
-
-Turbo-hipsters configuration is currently stored in json format.
-Modify the config.json appropriately::
-
-    **zuul_server**
-        A dictionary containing details about how to communicate
-        with zuul
-            **git_url**
-                The publicly accessible protocol and URI from where
-                to clone projects and zuul_ references from. For
-                example::
-                    http://review.openstack.org/p/
-                or::
-                    git://review.example.org
-            **gearman_host**
-                The host of gearman_. zuul talks to its workers via
-                the gearman protocol and while it comes with a built-
-                in gearman server you can use a separate one.
-            **gearman_port**
-                The port that gearman listens on.
-    **debug_log**
-        A path to the debug log. Turbo-hipster will attempt to create
-        the file but must have write permissions.
-    **jobs_working_dir**
-        Each job will likely need to write out log and debug
-        information. This defines where turbo-hipster will do that.
-    **git_working_dir**
-        turbo-hipster needs to take a copy of the git tree of a
-        project to work from. This is the path it'll clone into and
-        work from (if needed).
-    **pip_download_cache**
-        Some of turbo-hipsters task plugins download requirements
-        for projects. This is the cache directory used by pip.
-    **plugins**
-        A list of enabled plugins and their settings in a dictionary.
-        The only required parameters are *name*, which should be the
-        same as the folder containing the plugin module, and
-        *function*, which is the function registered with zuul.
-        Any other parameters are specified by the plugin themselves
-        as required.
-    **publish_logs**
-        Log results from plugins can be published using multiple
-        methods. Currently only a local copy is fully implemented.
-            **type**
-                The type of protocol to copy the log to. eg 'local'
-            **path**
-                A type specific parameter defining the local location
-                destination.
-            **prepend_url**
-                What to prepend to the path when sending the result
-                URL back to zuul. This can be useful as you may want
-                to use a script to authenticate against a swift
-                account or to use *laughing_spice* to format the logs
-                etc.
-
-
-Set up
-------
-
-You probably want to create a user to run turbo-hipster as. This user
-will need to write to all of the directories specified in
-config.json.
-
-Make sure the required directories as defined by the config.json
-exist and are writeable by your turbo-hipster user::
-
-    mkdir -p /var/log/turbo-hipster/
-    chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
-
-    mkdir -p /var/lib/turbo-hipster/jobs
-    chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
-
-    mkdir -p /var/lib/turbo-hipster/git
-    chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
-
-    mkdir -p /var/cache/pip
-    chown turbo-hipster:turbo-hipster /var/cache/pip
-
-Edit MySQL's log rotate to ensure it is other writable::
-
-    vim /etc/logrotate.d/mysql-server
-    # edit create 640 to 644.
-
-
-Start turbo-hipster
--------------------
-
-turbo hipster can be ran by executing::
-
-    ./turbo-hipster/worker_server.py
-
-and optionally takes the following parameters::
-
-    ./turbo_hipster/worker_server.py --help
-    usage: worker_server.py [-h] [-c CONFIG] [-b] [-p PIDFILE]
-
-    optional arguments:
-      -h, --help            show this help message and exit
-      -c CONFIG, --config CONFIG
-                            Path to json config file.
-      -b, --background      Run as a daemon in the background.
-      -p PIDFILE, --pidfile PIDFILE
-                            PID file to lock during daemonization.
-
-By default turbo-hipster will look for
-*/etc/turbo-hipster/config.json*
-
-Alternatively turbo-hipster can be launched by init.d using the
-included etc/init.d/turbo-hipster script::
-
-    sudo cp etc/init.d/turbo-hipster /etc/init.d/
-    sudo update-rc.d turbo-hipster defaults
-    sudo service turbo-hipster start
+Turbo-hipster is installed directly into your Python ``site-packages``
+directory, and is then run as a service. It is managed using a configuration
+file, which is in .json format.
+
+Installing turbo-hipster
+------------------------
+
+1. Turbo-Hipster can be installed directly to your Python ``site-packages``
+directory::
+
+ $ sudo python setup.py install
+
+2. Copy the configuration file to a convenient location. By default,
+turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
+
+ $ cp -R etc/turbo-hipster /etc/
+
+3. The Turbo-Hipster configuration file is in .json format. Open the
+``config.json`` configuration file in your preferred editor and modify it
+for your environment::
+
+  **zuul_server**
+    A dictionary containing details about how to communicate
+    with zuul
+       **git_url**
+           The publicly accessible protocol and URI from where
+           to clone projects and zuul_ references from. For
+           example::
+               http://review.openstack.org/p/
+           or::
+               git://review.example.org
+       **gearman_host**
+           The host of gearman_. zuul talks to its workers via
+           the gearman protocol and while it comes with a built-
+           in gearman server you can use a separate one.
+       **gearman_port**
+           The port that gearman listens on.
+  **debug_log**
+    A path to the debug log. Turbo-hipster will attempt to create
+    the file but must have write permissions.
+  **jobs_working_dir**
+    Each job will likely need to write out log and debug
+    information. This defines where turbo-hipster will do that.
+  **git_working_dir**
+    turbo-hipster needs to take a copy of the git tree of a
+    project to work from. This is the path it'll clone into and
+    work from (if needed).
+  **pip_download_cache**
+    Some of turbo-hipsters task plugins download requirements
+    for projects. This is the cache directory used by pip.
+  **plugins**
+    A list of enabled plugins and their settings in a dictionary.
+    The only required parameters are *name*, which should be the
+    same as the folder containing the plugin module, and
+    *function*, which is the function registered with zuul.
+    Any other parameters are specified by the plugin themselves
+    as required.
+  **publish_logs**
+    Log results from plugins can be published using multiple
+    methods. Currently only a local copy is fully implemented.
+       **type**
+           The type of protocol to copy the log to. eg 'local'
+       **path**
+           A type specific parameter defining the local location
+           destination.
+       **prepend_url**
+           What to prepend to the path when sending the result
+           URL back to zuul. This can be useful as you may want
+           to use a script to authenticate against a swift
+           account or to use *laughing_spice* to format the logs
+           etc.
+
+4. Create a turbo-hipster user:
+
+ $ useradd turbo-hipster
+
+5. Create the directories listed in the configuration file, and give the
+``turbo-hipster`` user write access::
+
+ $ mkdir -p /var/log/turbo-hipster/
+ $ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
+
+ $ mkdir -p /var/lib/turbo-hipster/jobs
+ $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
+
+ $ mkdir -p /var/lib/turbo-hipster/git
+ $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
+
+ $ mkdir -p /var/cache/pip
+ $ chown turbo-hipster:turbo-hipster /var/cache/pip
+
+6. Open the MySQL log rotation configuration file in your preferred text
+editor, and edit it to ensure it is writable by ``other``::
+
+ $ vim /etc/logrotate.d/mysql-server
+ # edit create 640 to 644.
+
+.. note::
+  The turbo-hipster source code is also available for download from
+  the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_
+
+  $ git clone https://github.com/rcbau/turbo-hipster
+
+.. note::
+ Debug logging must be configured for turbo-hipster, as it uses the Python
+ logging framework to capture log messages from the task plugin code.
+ To configure debug logging, set the ``debug_log`` configuration
+ setting in the ``config.json`` configuration file.
\ No newline at end of file
diff --git a/doc/source/intro.rst b/doc/source/intro.rst
index 5ec60bc..d2c0045 100644
--- a/doc/source/intro.rst
+++ b/doc/source/intro.rst
@@ -92,260 +92,4 @@ Typical workflow diagram
 
    }
 
-Installation
-============
 
-Turbo-hipster is installed directly into your Python ``site-packages``
-directory, and is then run as a service. It is managed using a configuration
-file, which is in .json format.
-
-Installing turbo-hipster
-------------------------
-
-1. Turbo-Hipster can be installed directly to your Python ``site-packages``
-directory::
-
- $ sudo python setup.py install
-
-2. Copy the configuration file to a convenient location. By default,
-turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
-
- $ cp -R etc/turbo-hipster /etc/
-
-3. The Turbo-Hipster configuration file is in .json format. Open the
-``config.json`` configuration file in your preferred editor and modify it
-for your environment::
-
-  **zuul_server**
-    A dictionary containing details about how to communicate
-    with zuul
-       **git_url**
-           The publicly accessible protocol and URI from where
-           to clone projects and zuul_ references from. For
-           example::
-               http://review.openstack.org/p/
-           or::
-               git://review.example.org
-       **gearman_host**
-           The host of gearman_. zuul talks to its workers via
-           the gearman protocol and while it comes with a built-
-           in gearman server you can use a separate one.
-       **gearman_port**
-           The port that gearman listens on.
-  **debug_log**
-    A path to the debug log. Turbo-hipster will attempt to create
-    the file but must have write permissions.
-  **jobs_working_dir**
-    Each job will likely need to write out log and debug
-    information. This defines where turbo-hipster will do that.
-  **git_working_dir**
-    turbo-hipster needs to take a copy of the git tree of a
-    project to work from. This is the path it'll clone into and
-    work from (if needed).
-  **pip_download_cache**
-    Some of turbo-hipsters task plugins download requirements
-    for projects. This is the cache directory used by pip.
-  **plugins**
-    A list of enabled plugins and their settings in a dictionary.
-    The only required parameters are *name*, which should be the
-    same as the folder containing the plugin module, and
-    *function*, which is the function registered with zuul.
-    Any other parameters are specified by the plugin themselves
-    as required.
-  **publish_logs**
-    Log results from plugins can be published using multiple
-    methods. Currently only a local copy is fully implemented.
-       **type**
-           The type of protocol to copy the log to. eg 'local'
-       **path**
-           A type specific parameter defining the local location
-           destination.
-       **prepend_url**
-           What to prepend to the path when sending the result
-           URL back to zuul. This can be useful as you may want
-           to use a script to authenticate against a swift
-           account or to use *laughing_spice* to format the logs
-           etc.
-
-4. Create a turbo-hipster user:
-
- $ useradd turbo-hipster
-
-5. Create the directories listed in the configuration file, and give the
-``turbo-hipster`` user write access::
-
- $ mkdir -p /var/log/turbo-hipster/
- $ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
-
- $ mkdir -p /var/lib/turbo-hipster/jobs
- $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
-
- $ mkdir -p /var/lib/turbo-hipster/git
- $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
-
- $ mkdir -p /var/cache/pip
- $ chown turbo-hipster:turbo-hipster /var/cache/pip
-
-6. Open the MySQL log rotation configuration file in your preferred text
-editor, and edit it to ensure it is writable by ``other``::
-
- $ vim /etc/logrotate.d/mysql-server
- # edit create 640 to 644.
-
-.. note::
-  The turbo-hipster source code is also available for download from
-  the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_ 
-
-  $ git clone https://github.com/rcbau/turbo-hipster
-
-Starting turbo-hipster
-----------------------
-
-Turbo-hipster can be run from the command line::
-
- $ ./turbo-hipster/worker_server.py
-
-This option allows you to pass parameters to turbo-hipster. Use the --help
-parameter to see a full list.
-
-+-------+--------------+--------------------------------------------------------+
-| Short |    Long      | Description                                            |
-+=======+==============+========================================================+
-|  -c   | --config     | Print the path to the configuration file and exit      |
-+-------+--------------+--------------------------------------------------------+
-|  -b   | --background | Run as a daemon in the background                      |
-+-------+--------------+--------------------------------------------------------+
-|  -p   | --pidfile    | Specify the PID file to lock while running as a daemon |
-+-------+--------------+--------------------------------------------------------+
-
-Alternatively, you can start turbo-hipster as a service.
-
-1. Copy the turbo-hipster init.d script to /etc/init.d/:
-
- $ sudo cp etc/init.d/turbo-hipster /etc/init.d/
-
-2. Reload the script with the default configuration:
-
- $ sudo update-rc.d turbo-hipster defaults
-
-3. Start the service:
-
- $ sudo service turbo-hipster start
-
-Plugins
-=======
-
-Plugins can be used to extend turbo-hipster's capabilities.
-
-.. note::
- Currently, the only available plugin for turbo-hipster is the
- Database Migration plugin, ``gate_real_db_upgrade``, which tests code
- against a variety of real-world databases.
-
-Installing plugins
-------------------
-
-Turbo-hipster plugins are responsible for handling the jobs that are passed
-to it. They must successfully build reports and publish them according to
-their configuration. They must also be able to communicate test results back
-to Zuul using Gearman.
-
-Plugins must take a standard format in order to be able to work correctly
-with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
-class.
-
-Once you have created a turbo-hipster plugin, you need to configure it in
-the ``config.json`` configuration file.
-
-.. FIXME More config information required here
-
-Plugin: Database migration with ``gate_real_db_upgrade``
---------------------------------------------------------
-
-The database migration plugin, ``gate_real_db_upgrade``, is used to test
-datasets against real-world, anonymized, databases.
-
-Migrating a database
---------------------
-
-In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
-need to set up the databases to test against, and point to the plugin in
-turbo-hipster's configuration file.
-
-1. Create a directory for the datasets:
-
- $ mkdir -p /var/lib/turbo-hipster/datasets
- 
-2. Copy the json dataset to the directory you created:
-
- $ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
-
-3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
-editor, locate the plugins section, and add this line::
-
-  **plugins**
-   gate_real_db_upgrade
-
-Testing with turbo-hipster
-==========================
-
-When turbo-hipster completes a test, it sends the result of the test back to
-Gearman. These results contain a link to a compiled logfile for the test.
-
-If the test fails, or takes too long to complete, turbo-hipster will add a
-review to your patchset that looks like this:
-
-.. image:: ../images/THTestResult.png
-
-Reading test reports
---------------------
-
-An example of a standard log file:
-http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
-
-An example of the same logfile, using the javascript logviewer:
-http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
-
-Test failure codes
-------------------
-
-This section gives a list of failure codes, including some steps you can
-take for troubleshooting errors:
-
- FAILURE - Did not find the end of a migration after a start
-
-If you look at the log you should find that a migration began but never
-finished. Hopefully there'll be a traceroute for you to follow through to
-get some hints about why it failed.
-
- WARNING - Migration %s took too long
-
-In this case your migration took a long time to run against one of the test
-datasets. You should reconsider what operations your migration is performing
-and see if there are any optimizations you can make, or if it is really
-necessary. If there is no way to speed up your migration you can email us at
-rcbau@rcbops.com for an exception.
-
- FAILURE - Final schema version does not match expectation
-
-Somewhere along the line the migrations stopped and did not reach the
-expected version. Our datasets start at previous releases and have to
-upgrade all the way through to the most current release. If you see this,
-inspect the log for traceroutes or other hints about the failure.
-
- FAILURE - Could not setup seed database.
- FAILURE - Could not find seed database.
-
-These errors are internal errors. If you see either of these, contact us at
-rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
-
- FAILURE - Could not import required module.
-
-This error probably shouldn't happen as Jenkins should catch it in the unit
-tests before Turbo-Hipster launches. If you see this, please contact us at
-rcbau@rcbops.com and let us know.
-
-If you receive an error that you think is a false positive, leave a comment
-on the review with the sole contents of "recheck migrations".
-
-If you have any questions/problems please contact us at rcbau@rcbops.com.
diff --git a/doc/source/running.rst b/doc/source/running.rst
index b0903db..9a60686 100644
--- a/doc/source/running.rst
+++ b/doc/source/running.rst
@@ -4,4 +4,154 @@
 Running
 =======
 
-todo...
\ No newline at end of file
+Starting turbo-hipster
+----------------------
+
+Turbo-hipster can be run from the command line::
+
+ $ ./turbo-hipster/worker_server.py
+
+This option allows you to pass parameters to turbo-hipster. Use the --help
+parameter to see a full list.
+
++-------+--------------+--------------------------------------------------------+
+| Short |    Long      | Description                                            |
++=======+==============+========================================================+
+|  -c   | --config     | Print the path to the configuration file and exit      |
++-------+--------------+--------------------------------------------------------+
+|  -b   | --background | Run as a daemon in the background                      |
++-------+--------------+--------------------------------------------------------+
+|  -p   | --pidfile    | Specify the PID file to lock while running as a daemon |
++-------+--------------+--------------------------------------------------------+
+
+Alternatively, you can start turbo-hipster as a service.
+
+1. Copy the turbo-hipster init.d script to /etc/init.d/:
+
+ $ sudo cp etc/init.d/turbo-hipster /etc/init.d/
+
+2. Reload the script with the default configuration:
+
+ $ sudo update-rc.d turbo-hipster defaults
+
+3. Start the service:
+
+ $ sudo service turbo-hipster start
+
+Plugins
+=======
+
+Plugins can be used to extend turbo-hipster's capabilities.
+
+.. note::
+ Currently, the only available plugin for turbo-hipster is the
+ Database Migration plugin, ``gate_real_db_upgrade``, which tests code
+ against a variety of real-world databases.
+
+Installing plugins
+------------------
+
+Turbo-hipster plugins are responsible for handling the jobs that are passed
+to it. They must successfully build reports and publish them according to
+their configuration. They must also be able to communicate test results back
+to Zuul using Gearman.
+
+Plugins must take a standard format in order to be able to work correctly
+with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
+class.
+
+Once you have created a turbo-hipster plugin, you need to configure it in
+the ``config.json`` configuration file.
+
+.. FIXME More config information required here
+
+Plugin: Database migration with ``gate_real_db_upgrade``
+--------------------------------------------------------
+
+The database migration plugin, ``gate_real_db_upgrade``, is used to test
+datasets against real-world, anonymized, databases.
+
+Migrating a database
+--------------------
+
+In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
+need to set up the databases to test against, and point to the plugin in
+turbo-hipster's configuration file.
+
+1. Create a directory for the datasets:
+
+ $ mkdir -p /var/lib/turbo-hipster/datasets
+
+2. Copy the json dataset to the directory you created:
+
+ $ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
+
+3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
+editor, locate the plugins section, and add this line::
+
+  **plugins**
+   gate_real_db_upgrade
+
+Testing with turbo-hipster
+==========================
+
+When turbo-hipster completes a test, it sends the result of the test back to
+Gearman. These results contain a link to a compiled logfile for the test.
+
+If the test fails, or takes too long to complete, turbo-hipster will add a
+review to your patchset that looks like this:
+
+.. image:: ../images/THTestResult.png
+
+Reading test reports
+--------------------
+
+An example of a standard log file:
+http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
+
+An example of the same logfile, using the javascript logviewer:
+http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
+
+Test failure codes
+------------------
+
+This section gives a list of failure codes, including some steps you can
+take for troubleshooting errors:
+
+ FAILURE - Did not find the end of a migration after a start
+
+If you look at the log you should find that a migration began but never
+finished. Hopefully there'll be a traceroute for you to follow through to
+get some hints about why it failed.
+
+ WARNING - Migration %s took too long
+
+In this case your migration took a long time to run against one of the test
+datasets. You should reconsider what operations your migration is performing
+and see if there are any optimizations you can make, or if it is really
+necessary. If there is no way to speed up your migration you can email us at
+rcbau@rcbops.com for an exception.
+
+ FAILURE - Final schema version does not match expectation
+
+Somewhere along the line the migrations stopped and did not reach the
+expected version. Our datasets start at previous releases and have to
+upgrade all the way through to the most current release. If you see this,
+inspect the log for traceroutes or other hints about the failure.
+
+ FAILURE - Could not setup seed database.
+ FAILURE - Could not find seed database.
+
+These errors are internal errors. If you see either of these, contact us at
+rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
+
+ FAILURE - Could not import required module.
+
+This error probably shouldn't happen as Jenkins should catch it in the unit
+tests before Turbo-Hipster launches. If you see this, please contact us at
+rcbau@rcbops.com and let us know.
+
+If you receive an error that you think is a false positive, leave a comment
+on the review with the sole contents of "recheck migrations".
+
+If you have any questions/problems please contact us at rcbau@rcbops.com.
diff --git a/doc/source/structure.rst b/doc/source/structure.rst
deleted file mode 100644
index c203466..0000000
--- a/doc/source/structure.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-:title:Turbo Hipster Structure
-
-Structure
-=======================
-
-Plugins
--------
-
-A little bit about plugins to come here soon.
-
- - You need to configure it in config.json
- - Folder name must be the same as 'name'
- - You probably want to specify a gate in the config
-    - see something (introduction?) about gates and how turbo-hipster
-      uses them
- - Each plugin has a task.py with a Runner class
-    - Responsible for registering functions
-    - handling jobs
-    - checking for errors
-    - building reports
-    - publishing them according to configuration