path: root/TESTING

How the test suites work in Gitano
==================================

Gitano's test suite is a functional black-box suite built in the `yarn` testing
language.  The testing stuff all lives in the testing directory at the top
level of the Gitano git repository.

The test suite can run in both an SSH mode and an HTTP mode.  In the former the
accesses to the Gitano instances created during the tests are made by simulated
SSH access.  In the latter they are made by simulating HTTP access to the CGIs
which form the Gitano interface.

To run the test suite, simply:

    make test

If you want to run a specific scenario from the test suite:

    make test SCENARIO="whatever scenario name"

If you are making changes and just want to 'smoke test' Gitano to ensure that
you have not made any fundamental compiler errors etc, then run:

    make basictest

The basictest run will only run in SSH mode by default, you can switch that
to HTTP mode if you are changing the CGI code by running:

    make basictest TEST_PROTO=http

The protocol being used during a test run is exposed in the GTT_PROTO
environment variable passed to implementations.

Yarn implementations
--------------------

The original (and primary) Yarn is part of the `cmdtest` suite by Lars
Wirzenius, is written in Python 2, and can be found in the `cmdtest` Debian
package or at <http://git.liw.fi/cgi-bin/cgit/cgit.cgi/cmdtest/> at the time of
writing this document.

The Gitano test suite is also compatible with `rsyarn` by Daniel Silverstone,
is written in Rust (needs 1.13 or better), and can be found at
<https://git.gitano.org.uk/rsyarn.git> at the time of writing this document.

Other Yarn implementations may be compatible, but the Makefile may need
tweaking if they do not expose the same cmdline arguments.  Patches are welcome
to support other Yarn implementations' cmdline syntax.

You may select a different Yarn tool by setting the 'YARN' make variable on
the command line, and you may pass additional arguments as may be necessary by
setting the 'YARN_ARGS' variable on the command line, for example if you wish
to use `rsyarn` and have it run five tests in parallel, you might run:

    make test YARN=rsyarn YARN_ARGS=-j5

This can be a significant speed boost when your computer is capable of running
multiple of the scenarios in parallel, for example on the author's slow old
laptop, the full yarn suite takes over two minutes to run using a single
threaded approach, or only 40 seconds if five scenarios are run in parallel.

Note: The test suite was originally written with single-threaded execution in
mind, and though the Yarns allows for arbitrary numbers of scenarios to run in
parallel, you should be careful not to overload your system when running the
test suite.

HTTP mode test dependencies
---------------------------

HTTP tests work via password access rather than key access, so the `htpasswd`
binary must be in your `PATH`.  In Debian (and derived distributions) this can
be accomplised by installing the `apache2-utils` package.

HTTP tests require starting an HTTP server on an unused port.  The test suite
implements this using `lighttpd`, so that binary must be in your `PATH`.  In
Debian (and derived distributions) this can be accomplished by installing the
`lighttpd` package.  This package may start the server by default, so you are
likely to want to run `systemctl disable --now lighttpd.service` to stop it.

The tests default to running servers on ports starting at 8080 and walking up
to find a free one (up to a maximum of ten times).  If there is a better port
to use set `HTTP_FIRST_TEST_PORT`.  This limitation means that if your `yarn`
implementation parallelises scenarios, you must limit the parallelism to ten or
else the HTTP tests will not run successfully.

NOTE: If you are definitely not going to use Gitano in HTTP mode, and you do
not want to run the HTTP tests, then you can disable them by passing in
'SKIP_HTTP_TESTS=1' on the make command line.  Do not do this before submitting
a patch for review as all tests will be run on potential merges.

The fundamental parts of the `testing/` directory are
=====================================================

* `gitano-test-tool.in`: This is a tool which helps the test suite to do
  complex operations including simulating different UNIX users, setting up a
  Gitano instance, etc.

* `http-unwrap`: This tool helps gitano-test-tool to unwrap HTTP responses
  generated by the test suite when running in HTTP mode.

* `keys/`: This directory contains a set of keys which are used in various
  parts of the test suite

* `admin-patches/`: This directory contains patches to the gitano-admin.git
  which can be applied during tests.  If you change the skeleton ruleset (or
  otherwise) for Gitano, you may need to refresh these patches.

* `library.yarn`: This contains the implementations for the test suite yarn
  statements.

* `01_*.yarn`: These are considered the basic tests

* `02_*.yarn`: These tests are focussed around validating the commands
   specifically.  The test might not use only the command in question, but that
   is their goal.

* `03_*.yarn` These tests are more integration tests which look to simulate
  more complex user use-cases.

Formatting Yarn tests
=====================

Yarns are indented code blocks surrounded by documentation.
For the sake of readability, this codebase also has the following rules:

1.  Make use of `AND` rather than repeating.

    This cuts down on the noise.

2.  Don't use `AND` at the beginning of a code block.

    This makes it difficult to recall which type the statement is.

    When reading the scenario in-flow you end up needing to backtrack
    to the previous code-block to see where you continue from.

3.  Indent everything to 4 spaces.

    Markdown supports tabs as indentation too,
    but some editors normalise tabs to spaces
    and others may normalise spaces to tabs.

    When they disagree with how wide a tab should be
    they make the indentation super weird.

    In a perfect world everyone's editor would be correctly configured,
    but in our imperfect world, we use spaces for indentation.

4.  In each code block, right-justify the first word.

    This makes the text line up in a way that is aesthetically pleasing
    and easier to scan visually.

    So if you have a `GIVEN`, `WHEN`, `THEN` and `AND`, align as:

        GIVEN foo exists
         WHEN foo is barred
         THEN foo is bazzed
          AND foo is quxed

5.  Justification doesn't span multiple blocks.

    If a block contains no GIVEN then do not indent to 5 spaces
    to align with other code blocks.

    This adds unnecessary padding,
    and if the explanatory text between the blocks grows
    then there becomes no need to align with an earlier block.

    So if you have two blocks:

        GIVEN foo exists
        WHEN foo is barred
        THEN foo is bazzed

        WHEN foo is poked
        THEN foo is notified

    Do not justify as:

        GIVEN foo exists
         WHEN foo is barred
         THEN foo is bazzed

         WHEN foo is poked
         THEN foo is notified

    Justify as:

        GIVEN foo exists
         WHEN foo is barred
         THEN foo is bazzed

        WHEN foo is poked
        THEN foo is notified

5.  Put `SCENARIO`, `ASSUMING` and `FINALLY` in a separate code-block.

    Yarn will appropriately merge these into the normal flow,
    and if they were kept in the same code-block
    they would introduce unnecessary padding for everything else.

    `SCENARIO`, `ASSUMING` and `FINALLY` may go in the same code blocks.

6.  Put `FINALLY` at the end of the scenario.

    Yarn will always run `FINALLY` at the end,
    and while it may be appealing to put the `FINALLY` with the `GIVEN`
    that requires cleaning up,
    this often reads strangely and causes unnecessary padding.

7.  In non-story scenarios
    the `GIVEN` may be part of the `SCENARIO` and `ASSUMING` block.

    There's two rough classifications of scenarios:

    1.  Those that set up one thing (one `GIVEN` block) and test that,
        somewhat like unit tests.
    2.  Those that set things up multiple times (many `GIVEN` blocks)
        and test each of those, structured as a story.

    In the non-story scenarios there will be only one `GIVEN` block,
    so it may be more aesthetically pleasing to group it with the
    `SCENARIO` and `ASSUMING`s which also only happen once, at the start.

Tip and tricks when writing tests for Gitano
============================================

* Until this point is taken out of this document, testing explicitly against
  stderr is only guaranteed to work in ssh-only tests.  Guard those scenarios
  with an 'ASSUMING' which checks.  For example, 'ASSUMING rsync is possible'
  checks this.

* Where there are checks for 'stdout' or 'stderr' you can also write the same
  kinds of statements for 'the output' which will consider a concatenation of
  the two.

* Remember that the stdout/stderr of a step is not retained by default by all
  yarn implementations, even in --snapshot mode.  If you need the output of
  a shell command for debugging purposes, be sure to write it to a file.

* There is a utility "IMPLEMENTS THEN failure ensues" which always fails but
  also dumps a whole bunch of information to its output which can be a quick
  way to debug a scenario without needing to --snapshot which slows things
  down.

* Remember that access to Gitano behaves differently over SSH and over HTTP.
  If your test assumes the behaviour of one or the other you must either guard
  as in the first point or write your IMPLEMENTS to cope.

* As of the time of writing, Gitano's Yarns are written in shell.  You should
  only assume POSIX shell functionality.  We default to `/bin/sh` which on all
  sensible systems is a simple shell such as `dash`.  If your `/bin/sh` is,
  however, `bash` then you should take extra care not to write bashisms into
  your test implementations.

Analysing the variables used in lace contexts
=============================================

The set of predicates and variables available to lace contexts
depends on the operation, and in many cases the data in the repository.

This makes it difficult to know exactly what variables are available.

To aid with this uncomment the block in `lib/gitano/lace.lua`
and if `GITANO_DUMP_VARIABLE_FILE` is set in the environment
it will write a table of variables and the operations they exist in
to the file path in `GITANO_DUMP_VARIABLE_FILE`.

To make use of this in the test suite add
`--env GITANO_DUMP_VARIABLE_FILE=$file_path` to `YARN_ARGS`.

This is not compatible with running yarns in parallel
and depends on the test suite coverage to produce sufficient results.

Making use of this also requires the penlight library to be installed.

Obtaining coverage statistics
=============================

When writing tests, one often wishes to know that ones test has improved the
coverage of the codebase.  To that end, the build system for Gitano has support
for using luacov to generate and report coverage statistics.

In order to use this, you must first ensure that you have the submodules
present:

    $ git submodule init
    $ git submodule update

Next, to gather coverage data run:

    $ make COVERAGE=yes test

You can specify `SCENARIO=blahfoo`, or `YARN_ARGS=...` etc, on the command line
if you wish.  If you do not specify `COVERAGE=yes` then the tests will be run
without augmenting the coverage data.

Coverage data is aggregated across multiple runs.  This allows you to run only
specific scenarios, and still see aggregated coverage across the codebase.

To generate the coverage report, run:

    $ make coverage-report

This will merge all the aggregated coverage data into a single file, and then
will run the luacov tool to generate `luacov.report.out` which contains the
report.  Finally it will display a summary of the coverage for the relevant
parts of the codebase.  Some of the codebase, as well as anything in
`/usr/share` will be ignored by the report generator.

If you find that something which should be included is not, or something which
should be excluded is not, then adjust the `.luacov` file which is the
configuration for the coverage tool.  This is also loaded by the internal
in-tree-only module `lib/gitano/coverage.lua` which is generated at build time
from `lib/gitano/coverage.lua.in`.

If you are trying to get coverage for the `gitano-test-tool` binary in the
`testing` directory then you will need to **also** pass `COVER_GTT=yes` when
running the test suite.  This distinction is present since around two thirds of
all invocations of coverable code in the test suite uses the test tool to
perform some related task, meaning that covering it causes a significant
slowdown in testing, and increases the cost of generating the coverage report.