[stable-2.13] ansible-test - Improve container management. (#79538)

* ansible-test - More flexible become support.

(cherry picked from commit 5666c6d6a3207f279cbb3ce0b0bd070ab5a9ecd4)

* ansible-test - Add support for more remotes.

(cherry picked from commit 24d91f552cad2a485f286f3c34cbba2005599ab4)

* ansible-test - Enable ACLs on FreeBSD remotes.

This allows integration tests to switch from one unprivileged user
to another, sharing files between the users using ACLs.

(cherry picked from commit b63812bc08fd00fd772c28a2604f77f487d23104)

* ansible-test - Enable ACL support on more remotes. (#78299)

(cherry picked from commit 8bb20fe06b458e6d7c4290dcb86cced3bce9d825)

* ansible-test - Install `acl` on Alpine remotes. (#78303)

(cherry picked from commit 8565deaae393a136ada071c27a8b7791d3640e41)

* ansible-test - Use sudo for all remotes. (#78310)

(cherry picked from commit f3f4ad93325725b91902984b419502e8b4b97188)

* ansible-test - Fix Ubuntu 20.04 bootstrapping. (#78551)

(cherry picked from commit d683c5bd212375c9ffc785ab65ee6e68ea913c8b)

* ansible-test - Verify executables are executable. (#78606)

(cherry picked from commit ece85abbc46e087187caf6e05b1515b97c578531)

* ansible-test - Use --tmpfs to run containers. (#78605)

(cherry picked from commit 4187707f035a5dde9d02e99e5dec40d71b06d5d1)

* ansible-test - Remove Fedora 35 support. (#78720)

(cherry picked from commit 38a82a5cc43ed4b48cf5fbc3addc5d2993c37eef)

* ansible-test - Fix container error handling. (#78739)

(cherry picked from commit 79f67ed56116be11b1c992fade04acf06d9208d1)

* ansible-test - Improve container management. (#78550)

See changelogs/fragments/ansible-test-container-management.yml for details.

(cherry picked from commit cda16cc5e9aa8703fb4e1ac0a0be6b631d9076cc)

* ansible-test - Fix container detection. (#79530)

(cherry picked from commit 80d2f8da02052f64396da6b8caaf820eedbf18e2)

* Update test matrix symlinks. (#78737)

* Add remote platform symlinks.

These were already supported by ansible-test.
This change adds the symlinks for use in the CI test matrix.

* Remove obsolete AIX test symlink.

AIX was already removed from ansible-test.

(cherry picked from commit 62221a3be214082610ba33c62e82bc3a778be7fb)

* Use patched containers without VOLUME instruction.

* Make test suite compatible with older ansible.

* Fix compatibility with Python 3.8.

9 files changed, 94 insertions, 0 deletions

diff --git a/changelogs/fragments/ansible-test-container-management.yml b/changelogs/fragments/ansible-test-container-management.yml
new file mode 100644
index 0000000000..04961b98ee
--- /dev/null
+++ b/changelogs/fragments/ansible-test-container-management.yml
@@ -0,0 +1,63 @@
+major_changes:
+  - ansible-test - Docker and Podman are now supported on hosts with cgroup v2 unified. 
+                   Previously only cgroup v1 and cgroup v2 hybrid were supported.
+  - ansible-test - Docker Desktop on WSL2 is now supported (additional configuration required).
+  - ansible-test - Podman on WSL2 is now supported.
+  - ansible-test - Podman now works on container hosts without systemd.
+                   Previously only some containers worked, while others required rootfull or rootless Podman,
+                   but would not work with both. Some containers did not work at all.
+  - ansible-test - When additional cgroup setup is required on the container host, this will be automatically detected.
+                   Instructions on how to configure the host will be provided in the error message shown.
+minor_changes:
+  - ansible-test - When using Podman, ansible-test will detect if the loginuid used in containers is incorrect.
+                   When this occurs a warning is displayed and the container is run with the AUDIT_CONTROL capability.
+                   Previously containers would fail under this situation, with no useful warnings or errors given.
+  - ansible-test - Failure to connect to a container over SSH now results in a clear error.
+                   Previously tests would be attempted even after initial connection attempts failed.
+  - ansible-test - Warnings are now shown when using containers that were built with VOLUME instructions.
+  - ansible-test - Unit tests now support network disconnect by default when running under Podman.
+                   Previously this feature only worked by default under Docker.
+  - ansible-test - Additional log details are shown when containers fail to start or SSH connections to containers fail.
+  - ansible-test - Containers included with ansible-test no longer disable seccomp by default.
+  - ansible-test - A new ``cgroup`` option is available when running custom containers.
+                   This option can be used to indicate a container requires cgroup v1 or that it does not use cgroup.
+                   The default behavior assumes the container works with cgroup v2 (as well as v1).
+  - ansible-test - A new ``audit`` option is available when running custom containers.
+                   This option can be used to indicate whether a container requires the AUDIT_WRITE capability.
+                   The default is ``required``, which most containers will need when using Podman.
+                   If necessary, the ``none`` option can be used to opt-out of the capability.
+                   This has no effect on Docker, which always provides the capability.
+  - ansible-test - More details are provided about an instance when provisioning fails.
+  - ansible-test - Connection failures to remote provisioned hosts now show failure details as a warning.
+  - ansible-test - When setting the max open files for containers, the container host's limit will be checked.
+                   If the host limit is lower than the preferred value, it will be used and a warning will be shown.
+  - ansible-test - Use ``stop --time 0`` followed by ``rm`` to remove ephemeral containers instead of ``rm -f``.
+                   This speeds up teardown of ephemeral containers.
+  - ansible-test - Reduce the polling limit for SSHD startup in containers from 60 retries to 10.
+                   The one second delay between retries remains in place.
+  - ansible-test - Integration tests can be excluded from retries triggered by the ``--retry-on-error`` option by
+                   adding the ``retry/never`` alias. This is useful for tests that cannot pass on a retry or are too
+                   slow to make retries useful.
+  - ansible-test - The ``ansible-test env`` command now detects and reports the container ID if running in a container.
+bugfixes:
+  - ansible-test - Multiple containers now work under Podman without specifying the ``--docker-network`` option.
+  - ansible-test - Prevent concurrent / repeat pulls of the same container image.
+  - ansible-test - Prevent concurrent / repeat inspections of the same container image.
+  - ansible-test - Prevent concurrent execution of cached methods.
+  - ansible-test - Handle server errors when executing the ``docker info`` command.
+  - ansible-test - Show the exception type when reporting errors during instance provisioning.
+  - ansible-test - Pass the ``XDG_RUNTIME_DIR`` environment variable through to container commands.
+  - ansible-test - Connection attempts to managed remote instances no longer abort on ``Permission denied`` errors.
+  - ansible-test - Detection for running in a Podman or Docker container has been fixed to detect more scenarios.
+                   The new detection relies on ``/proc/self/mountinfo`` instead of ``/proc/self/cpuset``.
+                   Detection now works with custom cgroups and private cgroup namespaces.
+known_issues:
+  - ansible-test - Using Docker on systems with SELinux may require setting SELinux to permissive mode.
+                   Podman should work with SELinux in enforcing mode.
+  - ansible-test - Additional configuration may be required for certain container host and container combinations.
+                   Further details are available in the testing documentation.
+  - ansible-test - Systems with Podman networking issues may be unable to run containers, when previously the issue
+                   went unreported. Correct the networking issues to continue using ``ansible-test`` with Podman.
+  - ansible-test - Custom containers with ``VOLUME`` instructions may be unable to start, when previously the containers
+                   started correctly. Remove the ``VOLUME`` instructions to resolve the issue. Containers with this
+                   condition will cause ``ansible-test`` to emit a warning.
diff --git a/changelogs/fragments/ansible-test-container-tmpfs.yml b/changelogs/fragments/ansible-test-container-tmpfs.yml
new file mode 100644
index 0000000000..678cd0770d
--- /dev/null
+++ b/changelogs/fragments/ansible-test-container-tmpfs.yml
@@ -0,0 +1,5 @@
+bugfixes:
+  - ansible-test - Test containers are now run with the ``--tmpfs`` option for ``/tmp``, ``/run`` and ``/run/lock``.
+                   This allows use of containers built without the ``VOLUME`` instruction.
+                   Additionally, containers with those volumes defined no longer create anonymous volumes for them.
+                   This avoids leaving behind volumes on the container host after the container is stopped and deleted.
diff --git a/changelogs/fragments/ansible-test-generalize-become.yml b/changelogs/fragments/ansible-test-generalize-become.yml
new file mode 100644
index 0000000000..1831c05228
--- /dev/null
+++ b/changelogs/fragments/ansible-test-generalize-become.yml
@@ -0,0 +1,2 @@
+minor_changes:
+  - ansible-test - Become support for remote instance provisioning is no longer tied to a fixed list of platforms.
diff --git a/changelogs/fragments/ansible-test-more-remotes.yml b/changelogs/fragments/ansible-test-more-remotes.yml
new file mode 100644
index 0000000000..7eb1615011
--- /dev/null
+++ b/changelogs/fragments/ansible-test-more-remotes.yml
@@ -0,0 +1,5 @@
+minor_changes:
+  - ansible-test - Add support for provisioning remotes which require ``doas`` for become.
+  - ansible-test - Add support for provisioning Ubuntu 20.04 remote instances.
+  - ansible-test - Add support for provisioning Alpine 3.16 remote instances.
+  - ansible-test - Add support for provisioning Fedora 36 remote instances.
diff --git a/changelogs/fragments/ansible-test-podman-create-retry.yml b/changelogs/fragments/ansible-test-podman-create-retry.yml
new file mode 100644
index 0000000000..7416e89f79
--- /dev/null
+++ b/changelogs/fragments/ansible-test-podman-create-retry.yml
@@ -0,0 +1,3 @@
+bugfixes:
+  - ansible-test - Always remove containers after failing to create/run them.
+                   This avoids leaving behind created containers when using podman.
diff --git a/changelogs/fragments/ansible-test-remote-acl.yml b/changelogs/fragments/ansible-test-remote-acl.yml
new file mode 100644
index 0000000000..79ff7e5148
--- /dev/null
+++ b/changelogs/fragments/ansible-test-remote-acl.yml
@@ -0,0 +1,5 @@
+minor_changes:
+  - ansible-test - Remote FreeBSD instances now have ACLs enabled on the root filesystem.
+  - ansible-test - Remote Fedora instances now have the ``acl`` package installed.
+  - ansible-test - Remote Ubuntu instances now have the ``acl`` package installed.
+  - ansible-test - Remote Alpine instances now have the ``acl`` package installed.
diff --git a/changelogs/fragments/ansible-test-remote-become.yml b/changelogs/fragments/ansible-test-remote-become.yml
new file mode 100644
index 0000000000..031cac34ba
--- /dev/null
+++ b/changelogs/fragments/ansible-test-remote-become.yml
@@ -0,0 +1,3 @@
+minor_changes:
+  - ansible-test - Alpine remotes now use ``sudo`` for tests, using ``doas`` only for bootstrapping.
+  - ansible-test - FreeBSD remotes now use ``sudo`` for tests, using ``su`` only for bootstrapping.
diff --git a/changelogs/fragments/ansible-test-ubuntu-bootstrap-fix.yml b/changelogs/fragments/ansible-test-ubuntu-bootstrap-fix.yml
new file mode 100644
index 0000000000..92666bed73
--- /dev/null
+++ b/changelogs/fragments/ansible-test-ubuntu-bootstrap-fix.yml
@@ -0,0 +1,2 @@
+bugfixes:
+  - ansible-test - Fix bootstrapping of Python 3.9 on Ubuntu 20.04 remotes.
diff --git a/changelogs/fragments/ansible-test-verify-executables.yml b/changelogs/fragments/ansible-test-verify-executables.yml
new file mode 100644
index 0000000000..a1eff95d09
--- /dev/null
+++ b/changelogs/fragments/ansible-test-verify-executables.yml
@@ -0,0 +1,6 @@
+bugfixes:
+  - ansible-test - Temporary executables are now verified as executable after creation.
+                   Without this check, path injected scripts may not be found,
+                   typically on systems with ``/tmp`` mounted using the "noexec" option.
+                   This can manifest as a missing Python interpreter, or use of the wrong Python interpreter, as well
+                   as other error conditions.