From 1d7436935f00379d00682c9ec1059b3a1c279fce Mon Sep 17 00:00:00 2001 From: Matt Clay Date: Tue, 6 Dec 2022 14:29:45 -0800 Subject: [stable-2.13] ansible-test - Improve container management. (#79538) * ansible-test - More flexible become support. (cherry picked from commit 5666c6d6a3207f279cbb3ce0b0bd070ab5a9ecd4) * ansible-test - Add support for more remotes. (cherry picked from commit 24d91f552cad2a485f286f3c34cbba2005599ab4) * ansible-test - Enable ACLs on FreeBSD remotes. This allows integration tests to switch from one unprivileged user to another, sharing files between the users using ACLs. (cherry picked from commit b63812bc08fd00fd772c28a2604f77f487d23104) * ansible-test - Enable ACL support on more remotes. (#78299) (cherry picked from commit 8bb20fe06b458e6d7c4290dcb86cced3bce9d825) * ansible-test - Install `acl` on Alpine remotes. (#78303) (cherry picked from commit 8565deaae393a136ada071c27a8b7791d3640e41) * ansible-test - Use sudo for all remotes. (#78310) (cherry picked from commit f3f4ad93325725b91902984b419502e8b4b97188) * ansible-test - Fix Ubuntu 20.04 bootstrapping. (#78551) (cherry picked from commit d683c5bd212375c9ffc785ab65ee6e68ea913c8b) * ansible-test - Verify executables are executable. (#78606) (cherry picked from commit ece85abbc46e087187caf6e05b1515b97c578531) * ansible-test - Use --tmpfs to run containers. (#78605) (cherry picked from commit 4187707f035a5dde9d02e99e5dec40d71b06d5d1) * ansible-test - Remove Fedora 35 support. (#78720) (cherry picked from commit 38a82a5cc43ed4b48cf5fbc3addc5d2993c37eef) * ansible-test - Fix container error handling. (#78739) (cherry picked from commit 79f67ed56116be11b1c992fade04acf06d9208d1) * ansible-test - Improve container management. (#78550) See changelogs/fragments/ansible-test-container-management.yml for details. (cherry picked from commit cda16cc5e9aa8703fb4e1ac0a0be6b631d9076cc) * ansible-test - Fix container detection. (#79530) (cherry picked from commit 80d2f8da02052f64396da6b8caaf820eedbf18e2) * Update test matrix symlinks. (#78737) * Add remote platform symlinks. These were already supported by ansible-test. This change adds the symlinks for use in the CI test matrix. * Remove obsolete AIX test symlink. AIX was already removed from ansible-test. (cherry picked from commit 62221a3be214082610ba33c62e82bc3a778be7fb) * Use patched containers without VOLUME instruction. * Make test suite compatible with older ansible. * Fix compatibility with Python 3.8. --- .../rst/community/create_pr_quick_start.rst | 8 +- .../testing/sanity/integration-aliases.rst | 1 + .../rst/dev_guide/testing_running_locally.rst | 345 +++++++++++++++++++-- 3 files changed, 317 insertions(+), 37 deletions(-) (limited to 'docs') diff --git a/docs/docsite/rst/community/create_pr_quick_start.rst b/docs/docsite/rst/community/create_pr_quick_start.rst index d06f58ddd7..d645e8b940 100644 --- a/docs/docsite/rst/community/create_pr_quick_start.rst +++ b/docs/docsite/rst/community/create_pr_quick_start.rst @@ -16,7 +16,7 @@ Prepare your environment These steps assume a Linux work environment with ``git`` installed. -1. Install and start ``docker`` or ``podman`` with the ``docker`` executable shim. This insures tests run properly isolated and in the exact environments as in CI. The latest ``ansible-core`` development version also supports the ``podman`` CLI program. +1. Install and start ``docker`` or ``podman``. This ensures tests run properly isolated and in the same environment as in CI. 2. :ref:`Install Ansible or ansible-core `. You need the ``ansible-test`` utility which is provided by either of these packages. @@ -157,7 +157,7 @@ Test your changes 1. Install ``flake8`` (``pip install flake8``, or install the corresponding package on your operating system). -1. Run ``flake8`` against a changed file: +2. Run ``flake8`` against a changed file: .. code-block:: bash @@ -167,7 +167,7 @@ Test your changes This shows unused imports, which is not shown by sanity tests, as well as other common issues. Optionally, you can use the ``--max-line-length=160`` command-line argument. -2. Run sanity tests: +3. Run sanity tests: .. code-block:: bash @@ -176,7 +176,7 @@ Test your changes If they failed, look at the output carefully - it is informative and helps to identify a problem line quickly. Sanity failings usually relate to incorrect code and documentation formatting. -3. Run integration tests: +4. Run integration tests: .. code-block:: bash diff --git a/docs/docsite/rst/dev_guide/testing/sanity/integration-aliases.rst b/docs/docsite/rst/dev_guide/testing/sanity/integration-aliases.rst index 39a03126a8..e14188d6cc 100644 --- a/docs/docsite/rst/dev_guide/testing/sanity/integration-aliases.rst +++ b/docs/docsite/rst/dev_guide/testing/sanity/integration-aliases.rst @@ -96,6 +96,7 @@ There are several other aliases available as well: - ``destructive`` - Requires ``--allow-destructive`` to run without ``--docker`` or ``--remote``. - ``hidden`` - Target is ignored. Usable as a dependency. Automatic for ``setup_`` and ``prepare_`` prefixed targets. +- ``retry/never`` - Target is excluded from retries enabled by the ``--retry-on-error`` option. Unstable -------- diff --git a/docs/docsite/rst/dev_guide/testing_running_locally.rst b/docs/docsite/rst/dev_guide/testing_running_locally.rst index dcf7e6d9f7..40910a87ec 100644 --- a/docs/docsite/rst/dev_guide/testing_running_locally.rst +++ b/docs/docsite/rst/dev_guide/testing_running_locally.rst @@ -2,44 +2,329 @@ .. _testing_running_locally: -*************** -Testing Ansible -*************** +******************************* +Testing Ansible and Collections +******************************* -This document describes how to: - -* Run tests locally using ``ansible-test`` -* Extend +This document describes how to run tests using ``ansible-test``. .. contents:: :local: -Requirements -============ +Setup +===== -There are no special requirements for running ``ansible-test`` on Python 2.7 or later. -The ``argparse`` package is required for Python 2.6. -The requirements for each ``ansible-test`` command are covered later. +Before running ``ansible-test``, set up your environment for :ref:`Testing an Ansible Collection` or +:ref:`Testing ansible-core`, depending on which scenario applies to you. +.. warning:: -Test Environments -================= + If you use ``git`` for version control, make sure the files you are working with are not ignored by ``git``. + If they are, ``ansible-test`` will ignore them as well. + +Testing an Ansible Collection +----------------------------- + +If you are testing an Ansible Collection, you need a copy of the collection, preferably a git clone. +For example, to work with the ``community.windows`` collection, follow these steps: + +1. Clone the collection you want to test into a valid collection root: + + .. code-block:: shell + + git clone https://github.com/ansible-collections/community.windows ~/dev/ansible_collections/community/windows + + .. important:: + + The path must end with ``/ansible_collections/{collection_namespace}/{collection_name}`` where + ``{collection_namespace}`` is the namespace of the collection and ``{collection_name}`` is the collection name. + +2. Clone any collections on which the collection depends: + + .. code-block:: shell + + git clone https://github.com/ansible-collections/ansible.windows ~/dev/ansible_collections/ansible/windows + + .. important:: + + If your collection has any dependencies on other collections, they must be in the same collection root, since + ``ansible-test`` will not use your configured collection roots (or other Ansible configuration). + + .. note:: + + See the collection's ``galaxy.yml`` for a list of possible dependencies. + +3. Switch to the directory where the collection to test resides: + + .. code-block:: shell + + cd ~/dev/ansible_collections/community/windows + +Testing ``ansible-core`` +------------------------ + +If you are testing ``ansible-core`` itself, you need a copy of the ``ansible-core`` source code, preferably a git clone. +Having an installed copy of ``ansible-core`` is not sufficient or required. +For example, to work with the ``ansible-core`` source cloned from GitHub, follow these steps: + +1. Clone the ``ansible-core`` repository: + + .. code-block:: shell + + git clone https://github.com/ansible/ansible ~/dev/ansible + +2. Switch to the directory where the ``ansible-core`` source resides: + + .. code-block:: shell + + cd ~/dev/ansible + +3. Add ``ansible-core`` programs to your ``PATH``: + + .. code-block:: shell + + source hacking/env-setup + + .. note:: + + You can skip this step if you only need to run ``ansible-test``, and not other ``ansible-core`` programs. + In that case, simply run ``bin/ansible-test`` from the root of the ``ansible-core`` source. + + .. caution:: + + If you have an installed version of ``ansible-core`` and are trying to run ``ansible-test`` from your ``PATH``, + make sure the program found by your shell is the one from the ``ansible-core`` source: + + .. code-block:: shell + + which ansible-test + +Commands +======== + +The most commonly used test commands are: + +* ``ansible-test sanity`` - Run sanity tests (mostly linters and static analysis). +* ``ansible-test integration`` - Run integration tests. +* ``ansible-test units`` - Run unit tests. + +Run ``ansible-test --help`` to see a complete list of available commands. + +.. note:: + + For detailed help on a specific command, add the ``--help`` option after the command. + +Environments +============ Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing. +Containers +---------- + +Containers are recommended for running sanity, unit and integration tests, since they provide consistent environments. +Unit tests will be run with network isolation, which avoids unintentional dependencies on network resources. + +The ``--docker`` option runs tests in a container using either Docker or Podman. + +.. note:: + + If both Docker and Podman are installed, Docker will be used. + To override this, set the environment variable ``ANSIBLE_TEST_PREFER_PODMAN`` to any non-empty value. + +Choosing a container +^^^^^^^^^^^^^^^^^^^^ + +Without an additional argument, the ``--docker`` option uses the ``default`` container. +To use another container, specify it immediately after the ``--docker`` option. + +.. note:: + + The ``default`` container is recommended for all sanity and unit tests. + +To see the list of supported containers, use the ``--help`` option with the ``ansible-test`` command you want to use. + +.. note:: + + The list of available containers is dependent on the ``ansible-test`` command you are using. + +You can also specify your own container. +When doing so, you will need to indicate the Python version in the container with the ``--python`` option. + +Custom containers +""""""""""""""""" + +When building custom containers, keep in mind the following requirements: + +* The ``USER`` should be ``root``. +* Use an ``init`` process, such as ``systemd``. +* Include ``sshd`` and accept connections on the default port of ``22``. +* Include a POSIX compatible ``sh`` shell which can be found on ``PATH``. +* Include a ``sleep`` utility which runs as a subprocess. +* Include a supported version of Python. +* Avoid using the ``VOLUME`` statement. + +Docker and SELinux +^^^^^^^^^^^^^^^^^^ + +Using Docker on a host with SELinux may require setting the system in permissive mode. +Consider using Podman instead. + +Docker Desktop with WSL2 +^^^^^^^^^^^^^^^^^^^^^^^^ + +These instructions explain how to use ``ansible-test`` with WSL2 and Docker Desktop *without* ``systemd`` support. -Remote ------- +.. note:: -The ``--remote`` option runs tests in a cloud hosted environment. -An API key is required to use this feature. + If your WSL2 environment includes ``systemd`` support, these steps are not required. - Recommended for integration tests. +Configuration requirements +"""""""""""""""""""""""""" -See the `list of supported platforms and versions `_ for additional details. +1. Open Docker Desktop and go to the **Settings** screen. +2. On the the **General** tab: -Environment Variables ---------------------- + a. Uncheck the **Start Docker Desktop when you log in** checkbox. + b. Check the **Use the WSL 2 based engine** checkbox. + +3. On the **Resources** tab under the **WSL Integration** section: + + a. Enable distros you want to use under the **Enable integration with additional distros** section. + +4. Click **Apply and restart** if changes were made. + +Setup instructions +"""""""""""""""""" + +.. note:: + + If all WSL instances have been stopped, these changes will need to be re-applied. + +1. Verify Docker Desktop is properly configured (see :ref:`Configuration requirements`). +2. Quit Docker Desktop if it is running: + + a. Right click the **Docker Desktop** taskbar icon. + b. Click the **Quit Docker Desktop** option. + +3. Stop any running WSL instances with the command: + + .. code-block:: shell + + wsl --shutdown + +4. Verify all WSL instances have stopped with the command: + + .. code-block:: shell + + wsl -l -v + +5. Start a WSL instance and perform the following steps as ``root``: + + a. Verify the ``systemd`` subsystem is not registered: + + a. Check for the ``systemd`` cgroup hierarchy with the following command: + + .. code-block:: shell + + grep systemd /proc/self/cgroup + + b. If any matches are found, re-check the :ref:`Configuration requirements` and follow the + :ref:`Setup instructions` again. + + b. Mount the ``systemd`` cgroup hierarchy with the following commands: + + .. code-block:: shell + + mkdir /sys/fs/cgroup/systemd + mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr + +6. Start Docker Desktop. + +You should now be able to use ``ansible-test`` with the ``--docker`` option. + +Linux cgroup configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + These changes will need to be re-applied each time the container host is booted. + +For certain container hosts and container combinations, additional setup on the container host may be required. +In these situations ``ansible-test`` will report an error and provide additional instructions to run as ``root``: + +.. code-block:: shell + + mkdir /sys/fs/cgroup/systemd + mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr + +If you are using rootless Podman, an additional command must be run, also as ``root``. +Make sure to substitute your user and group for ``{user}`` and ``{group}`` respectively: + +.. code-block:: shell + + chown -R {user}:{group} /sys/fs/cgroup/systemd + +Podman +"""""" + +When using Podman, you may need to stop existing Podman processes after following the :ref:`Linux cgroup configuration` +instructions. Otherwise Podman may be unable to see the new mount point. + +You can check to see if Podman is running by looking for ``podman`` and ``catatonit`` processes. + +Remote virtual machines +----------------------- + +Remote virtual machines are recommended for running integration tests not suitable for execution in containers. + +The ``--remote`` option runs tests in a cloud hosted ephemeral virtual machine. + +.. note:: + + An API key is required to use this feature, unless running under an approved Azure Pipelines organization. + +To see the list of supported systems, use the ``--help`` option with the ``ansible-test`` command you want to use. + +.. note:: + + The list of available systems is dependent on the ``ansible-test`` command you are using. + +Python virtual environments +--------------------------- + +Python virtual environments provide a simple way to achieve isolation from the system and user Python environments. +They are recommended for unit and integration tests when the ``--docker`` and ``--remote`` options cannot be used. + +The ``--venv`` option runs tests in a virtual environment managed by ``ansible-test``. +Requirements are automatically installed before tests are run. + +Composite environment arguments +------------------------------- + +The environment arguments covered in this document are sufficient for most use cases. +However, some scenarios may require the additional flexibility offered by composite environment arguments. + +The ``--controller`` and ``--target`` options are alternatives to the ``--docker``, ``--remote`` and ``--venv`` options. + +.. note:: + + When using the ``shell`` command, the ``--target`` option is replaced by three platform specific options. + +Add the ``--help`` option to your ``ansible-test`` command to learn more about the composite environment arguments. + +Additional Requirements +======================= + +Some ``ansible-test`` commands have additional requirements. +You can use the ``--requirements`` option to automatically install them. + +.. note:: + + When using a test environment managed by ``ansible-test`` the ``--requirements`` option is usually unnecessary. + +Environment variables +===================== When using environment variables to manipulate tests there some limitations to keep in mind. Environment variables are: @@ -51,16 +336,15 @@ When using environment variables to manipulate tests there some limitations to k and the tests executed. This is useful for debugging tests inside a container by following the :ref:`Debugging AnsibleModule-based modules ` instructions. -Interactive Shell +Interactive shell ================= Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples: * ``ansible-test shell --docker`` - Open a shell in the default docker container. -* ``ansible-test shell --venv --python 3.6`` - Open a shell in a Python 3.6 virtual environment. - +* ``ansible-test shell --venv --python 3.10`` - Open a shell in a Python 3.10 virtual environment. -Code Coverage +Code coverage ============= Code coverage reports make it easy to identify untested code for which more tests should @@ -72,22 +356,17 @@ aren't using the ``--venv`` or ``--docker`` options which create an isolated pyt environment then you may have to use the ``--requirements`` option to ensure that the correct version of the coverage module is installed: -.. code-block:: shell-session +.. code-block:: shell ansible-test coverage erase ansible-test units --coverage apt ansible-test integration --coverage aws_lambda ansible-test coverage html - Reports can be generated in several different formats: * ``ansible-test coverage report`` - Console report. * ``ansible-test coverage html`` - HTML report. * ``ansible-test coverage xml`` - XML report. -To clear data between test runs, use the ``ansible-test coverage erase`` command. For a full list of features see the online help: - -.. code-block:: shell-session - - ansible-test coverage --help +To clear data between test runs, use the ``ansible-test coverage erase`` command. -- cgit v1.2.1