Automated testing
=================

Introduction
------------
The bash-completion package contains an automated test suite.  Running the
tests should help verifying that bash-completion works as expected.  The tests
are also very helpful in uncovering software regressions at an early stage.

The original, "legacy" bash-completion test suite is written on top of the
http://www.gnu.org/software/dejagnu/[DejaGnu] testing framework.  DejaGnu is
written in http://expect.nist.gov[Expect], which in turn uses
http://tcl.sourceforge.net[Tcl] -- Tool command language.

Most of the test framework has been ported over to use
https://pytest.org/[pytest] and https://pexpect.readthedocs.io/[pexpect].
Eventually, all of it should be ported.


Coding Style Guide
------------------

For the Python part, all of it is formatted using
https://github.com/ambv/black[Black], and we also run
http://flake8.pycqa.org/[Flake8] on it.

The legacy test suite tries to adhere to this
http://wiki.tcl.tk/708[Tcl Style Guide].


Installing dependencies
-----------------------

Installing dependencies should be easy using your local package manager or
`pip`. Python 3.4 or newer is required, and the rest of the Python package
dependencies are specified in the `test/requirements.txt` file. If using `pip`,
this file can be fed directly to it, e.g. like:
------------------------------------
pip install -r test/requirements.txt
------------------------------------


Debian/Ubuntu
~~~~~~~~~~~~~

On Debian/Ubuntu you can use `apt-get`:
-------------
sudo apt-get install python3-pytest python3-pexpect dejagnu tcllib
-------------
This should also install the necessary dependencies. Only Debian testing
(buster) and Ubuntu 18.10 (cosmic) and later have an appropriate version
of pytest in the repositories.

Fedora/RHEL/CentOS
~~~~~~~~~~~~~~~~~~

On Fedora and RHEL/CentOS (with EPEL) you can try `yum` or `dnf`:
-------------
sudo yum install python3-pytest python3-pexpect dejagnu tcllib
-------------
This should also install the necessary dependencies. At time of writing, only
Fedora 29 comes with recent enough pytest.



Structure
---------

Pytest tests are in the `t/` subdirectory, with `t/test_\*.py` being
completion tests, and `t/unit/test_unit_\*.py` unit tests.

Legacy tests are grouped into different areas, called _tool_ in DejaGnu:

*completion*::
  Functional tests per completion.
*install*::
  Functional tests for installation and caching of the main bash-completion
  package.
*unit*::
  Unit tests for bash-completion helper functions.


Running the tests
-----------------

Python based tests are run by calling `pytest` on the desired test directories
or individual files, for example in the project root directory:
-----------------------
pytest test/t
-----------------------

Legacy tests are run by calling `runtest` command in the test directory:
-----------------------
runtest --outdir log --tool completion
runtest --outdir log --tool install
runtest --outdir log --tool unit
-----------------------
The commands above are already wrapped up in shell scripts within the `test`
directory:
-----------------------
./runCompletion
./runInstall
./runUnit
-----------------------
To run a particular test, specify file name of your test as an argument to
`runCompletion` script:
-----------------------
./runCompletion ssh.exp
-----------------------
That will run `test/completion/ssh.exp`.

See `test/docker/docker-script.sh` for how and what we run and test in CI.


Specifying bash binary
~~~~~~~~~~~~~~~~~~~~~~

The test suite standard uses `bash` as found in PATH. Export the
`bashcomp_bash` environment variable with a path to another bash executable if
you want to test against something else.


Maintenance
-----------


Adding a completion test
~~~~~~~~~~~~~~~~~~~~~~~~

You can run `cd test && ./generate cmd` to add a test for the `cmd`
command. Additional arguments will be passed to the first generated test case.
This will add the `test/t/test_cmd.py` file with a very basic test, and add it
to `test/t/Makefile.am`. Add additional tests to the generated file.

Rationale
---------


Naming conventions
~~~~~~~~~~~~~~~~~~

Test suite or testsuite
^^^^^^^^^^^^^^^^^^^^^^^
The primary Wikipedia page is called
http://en.wikipedia.org/wiki/Test_suite[test suite] and not testsuite, so
that's what this document sticks to.

script/generate
^^^^^^^^^^^^^^^
The name and location of this code generation script come from Ruby on Rails'
http://en.wikibooks.org/wiki/Ruby_on_Rails/Tools/Generators[script/generate].