How the test suites work in Gitano ================================== Gitano's test suite is a functional black-box suite built in the `yarn` testing language. The testing stuff all lives in the testing directory at the top level of the Gitano git repository. The test suite can run in both an SSH mode and an HTTP mode. In the former the accesses to the Gitano instances created during the tests are made by simulated SSH access. In the latter they are made by simulating HTTP access to the CGIs which form the Gitano interface. To run the test suite, simply: make test If you want to run a specific scenario from the test suite: make test SCENARIO="whatever scenario name" If you are making changes and just want to 'smoke test' Gitano to ensure that you have not made any fundamental compiler errors etc, then run: make basictest The basictest run will only run in SSH mode by default, you can switch that to HTTP mode if you are changing the CGI code by running: make basictest TEST_PROTO=http The protocol being used during a test run is exposed in the GTT_PROTO environment variable passed to implementations. Yarn implementations -------------------- The original (and primary) Yarn is part of the `cmdtest` suite by Lars Wirzenius, is written in Python 2, and can be found in the `cmdtest` Debian package or at at the time of writing this document. The Gitano test suite is also compatible with `rsyarn` by Daniel Silverstone, is written in Rust (needs 1.13 or better), and can be found at at the time of writing this document. Other Yarn implementations may be compatible, but the Makefile may need tweaking if they do not expose the same cmdline arguments. Patches are welcome to support other Yarn implementations' cmdline syntax. You may select a different Yarn tool by setting the 'YARN' make variable on the command line, and you may pass additional arguments as may be necessary by setting the 'YARN_ARGS' variable on the command line, for example if you wish to use `rsyarn` and have it run five tests in parallel, you might run: make test YARN=rsyarn YARN_ARGS=-j5 This can be a significant speed boost when your computer is capable of running multiple of the scenarios in parallel, for example on the author's slow old laptop, the full yarn suite takes over two minutes to run using a single threaded approach, or only 40 seconds if five scenarios are run in parallel. Note: The test suite was originally written with single-threaded execution in mind, and though the Yarns allows for arbitrary numbers of scenarios to run in parallel, you should be careful not to overload your system when running the test suite. HTTP mode test dependencies --------------------------- HTTP tests work via password access rather than key access, so the `htpasswd` binary must be in your `PATH`. In Debian (and derived distributions) this can be accomplised by installing the `apache2-utils` package. HTTP tests require starting an HTTP server on an unused port. The test suite implements this using `lighttpd`, so that binary must be in your `PATH`. In Debian (and derived distributions) this can be accomplished by installing the `lighttpd` package. This package may start the server by default, so you are likely to want to run `systemctl disable --now lighttpd.service` to stop it. The tests default to running servers on ports starting at 8080 and walking up to find a free one (up to a maximum of ten times). If there is a better port to use set `HTTP_FIRST_TEST_PORT`. This limitation means that if your `yarn` implementation parallelises scenarios, you must limit the parallelism to ten or else the HTTP tests will not run successfully. NOTE: If you are definitely not going to use Gitano in HTTP mode, and you do not want to run the HTTP tests, then you can disable them by passing in 'SKIP_HTTP_TESTS=1' on the make command line. Do not do this before submitting a patch for review as all tests will be run on potential merges. The fundamental parts of the `testing/` directory are ===================================================== * `gitano-test-tool.in`: This is a tool which helps the test suite to do complex operations including simulating different UNIX users, setting up a Gitano instance, etc. * `http-unwrap`: This tool helps gitano-test-tool to unwrap HTTP responses generated by the test suite when running in HTTP mode. * `keys/`: This directory contains a set of keys which are used in various parts of the test suite * `admin-patches/`: This directory contains patches to the gitano-admin.git which can be applied during tests. If you change the skeleton ruleset (or otherwise) for Gitano, you may need to refresh these patches. * `library.yarn`: This contains the implementations for the test suite yarn statements. * `01_*.yarn`: These are considered the basic tests * `02_*.yarn`: These tests are focussed around validating the commands specifically. The test might not use only the command in question, but that is their goal. * `03_*.yarn` These tests are more integration tests which look to simulate more complex user use-cases. Formatting Yarn tests ===================== Yarns are indented code blocks surrounded by documentation. For the sake of readability, this codebase also has the following rules: 1. Make use of `AND` rather than repeating. This cuts down on the noise. 2. Don't use `AND` at the beginning of a code block. This makes it difficult to recall which type the statement is. When reading the scenario in-flow you end up needing to backtrack to the previous code-block to see where you continue from. 3. Indent everything to 4 spaces. Markdown supports tabs as indentation too, but some editors normalise tabs to spaces and others may normalise spaces to tabs. When they disagree with how wide a tab should be they make the indentation super weird. In a perfect world everyone's editor would be correctly configured, but in our imperfect world, we use spaces for indentation. 4. In each code block, right-justify the first word. This makes the text line up in a way that is aesthetically pleasing and easier to scan visually. So if you have a `GIVEN`, `WHEN`, `THEN` and `AND`, align as: GIVEN foo exists WHEN foo is barred THEN foo is bazzed AND foo is quxed 5. Justification doesn't span multiple blocks. If a block contains no GIVEN then do not indent to 5 spaces to align with other code blocks. This adds unnecessary padding, and if the explanatory text between the blocks grows then there becomes no need to align with an earlier block. So if you have two blocks: GIVEN foo exists WHEN foo is barred THEN foo is bazzed WHEN foo is poked THEN foo is notified Do not justify as: GIVEN foo exists WHEN foo is barred THEN foo is bazzed WHEN foo is poked THEN foo is notified Justify as: GIVEN foo exists WHEN foo is barred THEN foo is bazzed WHEN foo is poked THEN foo is notified 5. Put `SCENARIO`, `ASSUMING` and `FINALLY` in a separate code-block. Yarn will appropriately merge these into the normal flow, and if they were kept in the same code-block they would introduce unnecessary padding for everything else. `SCENARIO`, `ASSUMING` and `FINALLY` may go in the same code blocks. 6. Put `FINALLY` at the end of the scenario. Yarn will always run `FINALLY` at the end, and while it may be appealing to put the `FINALLY` with the `GIVEN` that requires cleaning up, this often reads strangely and causes unnecessary padding. 7. In non-story scenarios the `GIVEN` may be part of the `SCENARIO` and `ASSUMING` block. There's two rough classifications of scenarios: 1. Those that set up one thing (one `GIVEN` block) and test that, somewhat like unit tests. 2. Those that set things up multiple times (many `GIVEN` blocks) and test each of those, structured as a story. In the non-story scenarios there will be only one `GIVEN` block, so it may be more aesthetically pleasing to group it with the `SCENARIO` and `ASSUMING`s which also only happen once, at the start. Tip and tricks when writing tests for Gitano ============================================ * Until this point is taken out of this document, testing explicitly against stderr is only guaranteed to work in ssh-only tests. Guard those scenarios with an 'ASSUMING' which checks. For example, 'ASSUMING rsync is possible' checks this. * Where there are checks for 'stdout' or 'stderr' you can also write the same kinds of statements for 'the output' which will consider a concatenation of the two. * Remember that the stdout/stderr of a step is not retained by default by all yarn implementations, even in --snapshot mode. If you need the output of a shell command for debugging purposes, be sure to write it to a file. * There is a utility "IMPLEMENTS THEN failure ensues" which always fails but also dumps a whole bunch of information to its output which can be a quick way to debug a scenario without needing to --snapshot which slows things down. * Remember that access to Gitano behaves differently over SSH and over HTTP. If your test assumes the behaviour of one or the other you must either guard as in the first point or write your IMPLEMENTS to cope. * As of the time of writing, Gitano's Yarns are written in shell. You should only assume POSIX shell functionality. We default to `/bin/sh` which on all sensible systems is a simple shell such as `dash`. If your `/bin/sh` is, however, `bash` then you should take extra care not to write bashisms into your test implementations. Analysing the variables used in lace contexts ============================================= The set of predicates and variables available to lace contexts depends on the operation, and in many cases the data in the repository. This makes it difficult to know exactly what variables are available. To aid with this uncomment the block in `lib/gitano/lace.lua` and if `GITANO_DUMP_VARIABLE_FILE` is set in the environment it will write a table of variables and the operations they exist in to the file path in `GITANO_DUMP_VARIABLE_FILE`. To make use of this in the test suite add `--env GITANO_DUMP_VARIABLE_FILE=$file_path` to `YARN_ARGS`. This is not compatible with running yarns in parallel and depends on the test suite coverage to produce sufficient results. Making use of this also requires the penlight library to be installed. Obtaining coverage statistics ============================= When writing tests, one often wishes to know that ones test has improved the coverage of the codebase. To that end, the build system for Gitano has support for using luacov to generate and report coverage statistics. In order to use this, you must first ensure that you have the submodules present: $ git submodule init $ git submodule update Next, to gather coverage data run: $ make COVERAGE=yes test You can specify `SCENARIO=blahfoo`, or `YARN_ARGS=...` etc, on the command line if you wish. If you do not specify `COVERAGE=yes` then the tests will be run without augmenting the coverage data. Coverage data is aggregated across multiple runs. This allows you to run only specific scenarios, and still see aggregated coverage across the codebase. To generate the coverage report, run: $ make coverage-report This will merge all the aggregated coverage data into a single file, and then will run the luacov tool to generate `luacov.report.out` which contains the report. Finally it will display a summary of the coverage for the relevant parts of the codebase. Some of the codebase, as well as anything in `/usr/share` will be ignored by the report generator. If you find that something which should be included is not, or something which should be excluded is not, then adjust the `.luacov` file which is the configuration for the coverage tool. This is also loaded by the internal in-tree-only module `lib/gitano/coverage.lua` which is generated at build time from `lib/gitano/coverage.lua.in`. If you are trying to get coverage for the `gitano-test-tool` binary in the `testing` directory then you will need to **also** pass `COVER_GTT=yes` when running the test suite. This distinction is present since around two thirds of all invocations of coverable code in the test suite uses the test tool to perform some related task, meaning that covering it causes a significant slowdown in testing, and increases the cost of generating the coverage report.