diff options
| -rw-r--r-- | TESTING.rst | 6 | ||||
| -rw-r--r-- | doc/source/contributor/testing/index.rst | 1 | ||||
| -rw-r--r-- | doc/source/contributor/testing/tempest.rst | 269 |
3 files changed, 276 insertions, 0 deletions
diff --git a/TESTING.rst b/TESTING.rst index 4a928dd05f..311654b9ee 100644 --- a/TESTING.rst +++ b/TESTING.rst @@ -236,6 +236,12 @@ reproducible way to verify code while you're still writing it. More details can be found in :ref:`FullStack Testing<fullstack_testing>` guide. +Integration - tempest tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tempest is the integration test suit of Openstack, more details can be found in +:ref:`Tempest testing<tempest_testing>` + API Tests ~~~~~~~~~ diff --git a/doc/source/contributor/testing/index.rst b/doc/source/contributor/testing/index.rst index e09fea4d67..12a235aa3b 100644 --- a/doc/source/contributor/testing/index.rst +++ b/doc/source/contributor/testing/index.rst @@ -37,3 +37,4 @@ Testing db_transient_failure_injection ci_scenario_jobs ovn_devstack + tempest diff --git a/doc/source/contributor/testing/tempest.rst b/doc/source/contributor/testing/tempest.rst new file mode 100644 index 0000000000..6e92013fa3 --- /dev/null +++ b/doc/source/contributor/testing/tempest.rst @@ -0,0 +1,269 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + + Convention for heading levels in Neutron devref: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + (Avoid deeper levels because they do not render well.) + +.. _tempest_testing: + +Tempest testing +=============== + +Tempest basics in Networking projects +------------------------------------- +Tempest is the integration test suite of Openstack, for details see +`Tempest Testing Project <https://docs.openstack.org/tempest/latest/>`_. + +Tempest makes it possible to add project-specific plugins, and for networking +this is `neutron-tempest-plugin <https://opendev.org/openstack/neutron-tempest-plugin>`_. + +neutron-tempest-plugin covers API and scenario tests not just for core Neutron +functionality, but for stadium projects as well. +For reference please read `Testing Neutron\'s related sections <testing.html#api-tests>`_ + +API Tests +~~~~~~~~~ + +API tests (neutron-tempest-plugin/neutron_tempest_plugin/api/) are +intended to ensure the function +and stability of the Neutron API. As much as possible, changes to +this path should not be made at the same time as changes to the code +to limit the potential for introducing backwards-incompatible changes, +although the same patch that introduces a new API should include an API +test. + +Since API tests target a deployed Neutron daemon that is not test-managed, +they should not depend on controlling the runtime configuration +of the target daemon. API tests should be black-box - no assumptions should +be made about implementation. Only the contract defined by Neutron's REST API +should be validated, and all interaction with the daemon should be via +a REST client. + +The neutron-tempest-plugin/neutron_tempest_plugin directory was copied from the +Tempest project around the Kilo timeframe. At the time, there was an overlap of tests +between the Tempest and Neutron repositories. This overlap was then eliminated by carving +out a subset of resources that belong to Tempest, with the rest in Neutron. + +API tests that belong to Tempest deal with a subset of Neutron's resources: + +* Port +* Network +* Subnet +* Security Group +* Router +* Floating IP + +These resources were chosen for their ubiquity. They are found in most +Neutron deployments regardless of plugin, and are directly involved in the +networking and security of an instance. Together, they form the bare minimum +needed by Neutron. + +This is excluding extensions to these resources (For example: Extra DHCP +options to subnets, or snat_gateway mode to routers) that are not mandatory +in the majority of cases. + +Tests for other resources should be contributed to the neutron-tempest-plugin +repository. Scenario tests should be similarly split up between Tempest and +Neutron according to the API they're targeting. + +To create an API test, the testing class must at least inherit from +neutron_tempest_plugin.api.base.BaseNetworkTest base class. As some of tests +may require certain extensions to be enabled, the base class provides +``required_extensions`` class attribute which can be used by subclasses to +define a list of required extensions for a particular test class. + +Scenario Tests +~~~~~~~~~~~~~~ + +Scenario tests (neutron-tempest-plugin/neutron_tempest_plugin/scenario), like API tests, +use the Tempest test infrastructure and have the same requirements. Guidelines for +writing a good scenario test may be found at the Tempest developer guide: +https://docs.openstack.org/tempest/latest/field_guide/scenario.html + +Scenario tests, like API tests, are split between the Tempest and Neutron +repositories according to the Neutron API the test is targeting. + +Some scenario tests require advanced ``Glance`` images (for example, ``Ubuntu`` +or ``CentOS``) in order to pass. Those tests are skipped by default. To enable +them, include the following in ``tempest.conf``: + +.. code-block:: ini + + [compute] + image_ref = <uuid of advanced image> + [neutron_plugin_options] + default_image_is_advanced = True + +To use the ``advanced image`` only for the tests that really need it and +``cirros`` for the rest to keep test execution as fast as possible: + +.. code-block:: ini + + [compute] + image_ref = <uuid of cirros image> + [neutron_plugin_options] + advanced_image_ref = <uuid of advanced image> + advanced_image_flavor_ref = <suitable flavor for the advance image> + advanced_image_ssh_user = <username for the advanced image> + +Specific test requirements for advanced images are: + +#. ``test_trunk`` requires ``802.11q`` kernel module loaded. +#. ``test_metadata`` requires capabilty to run ``curl`` for IPv6 addresses. +#. ``test_multicast`` needs to execute python script on the VM to open + multicast sockets. +#. ``test_mtu`` requires ping to be able to send packets with specific ``mtu``. + +Zuul basics & job structure +--------------------------- +Zuul is the gating system behind Openstack, for details see: +`Zuul - A Project Gating System <https://zuul-ci.org/docs/zuul/>`_. + +Zuul job definitions are in yaml, ansible in the depths. The job definitions can be +inherited. Networking projects job definitions parents are coming from +`devstack zuul job config <https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml>`_ +and from `tempest <https://opendev.org/openstack/tempest/src/branch/master/tempest>`_ +and defined in `neutron-tempest-plugin zuul.d folder <https://opendev.org/openstack/neutron-tempest-plugin/src/branch/master/zuul.d>`_ +and in `neutron zuul.d folder <https://opendev.org/openstack/neutron/src/branch/master/zuul.d>`_ . + +Where to look +------------- + +Debugging zuul results +~~~~~~~~~~~~~~~~~~~~~~ +Tempest executed with different configurations, +for details check this page +:ref:`Tempest jobs running in Neutron CI<ci_jobs>` + +When zuul reports back job results to a review it gives links to the results +as well. + +The logs can be checked online if you select ``Logs`` tab on the logs page. + +* ``job-output.txt`` is the full log which contains not just test execution + logs, but devstack console output. +* ``test_results.html`` is the clickable html test report. +* ``controller`` and ``compute`` (in case of multinode job) are a dictionary + tree containing the relevant files (configuration files, logs etc) + created in the job. For example under controller/logs/etc/neutron/ you can + check how Neutron services were configured, or in the file + controller/logs/tempest_conf.txt you can check tempest configuration file. +* services' log files are the in files ``controller/logs/screen-`*`.txt``, + so for example neutron l2 agent logs are in the file + controller/logs/screen-q-agt.txt. + +Downloading logs +++++++++++++++++ +There is a possibility to download all logs related to a job. + +If you choose this on the zuul logs page select ``Artifacts`` tab on the +logs page and click on ``Download all logs``. This will download a script +``download-logs.sh``, which when executed downloads all the logs for the job +under ``/tmp/``: + +.. code-block:: shell + + $ chmod +x download-logs.sh + $ ./download-logs.sh + 2020-12-07T18:12:09+01:00 | Querying https://zuul.opendev.org/api/tenant/openstack/build/8caed05f5ba441b4be2b061d1d421e4e for manifest + 2020-12-07T18:12:11+01:00 | Saving logs to /tmp/zuul-logs.c8ZhLM + 2020-12-07T18:12:11+01:00 | Getting logs from https://3612101d6c142bf9c77a-c96c299047b55dcdeaefef8e344ceab6.ssl.cf1.rackcdn.com/694539/11/check/tempest-slow-py3/8caed05/ + 2020-12-07T18:12:11+01:00 | compute1/logs/apache/access_log.txt [ 0001/0337 ] + ... + + $ ls /tmp/zuul-logs.c8ZhLM/ + compute1 + controller + +Executing tempest locally +~~~~~~~~~~~~~~~~~~~~~~~~~ +For executing tempest locally you need a working devstack, to make it worse +if you have to debug a test executed in a multinode job you need a multinode +setup as well. + +For devstack documentation please refer to this page: +`DevStack <https://docs.openstack.org/devstack/latest/>`_ + +To have tempest installed and have a proper configuration file for it in your +local.conf file enable tempest as service: + +.. code-block:: ini + + ENABLED_SERVICES+=tempest + +or + +.. code-block:: ini + + enable_service tempest + +To use specific config options for tempest you can add those as well to +local.conf: + +.. code-block:: ini + + [[test-config|/opt/stack/tempest/etc/tempest.conf]] + [network-feature-enabled] + qos_placement_physnet=physnet1 + +To make devstack setup neutron and neutron-tempest-plugin as well enable their +devstack plugin: + +.. code-block:: ini + + enable_plugin neutron https://opendev.org/openstack/neutron + enable_plugin neutron-tempest-plugin https://opendev.org/openstack/neutron-tempest-plugin + +If you need a special image for the tests you can set that too in local.conf: + +.. code-block:: ini + + IMAGE_URLS="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-i386-disk.img,https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.img" + ADVANCED_IMAGE_NAME=ubuntu-18.04-server-cloudimg-amd64 + ADVANCED_INSTANCE_TYPE=ds512M + ADVANCED_INSTANCE_USER=ubuntu + +If devstack succeeds you can find tempest and neutron-tempest-plugin under +``/opt/stack/`` directory (with all other project folders which are set to be +installed from git). + +Tempest's configuration file is under ``/opt/stack/tempest/etc/`` folder, you +can check there if everything is as expected. + +You can check if neutron-tempest-plugin is known as a tempest plugin by +tempest: + +.. code-block:: shell + + $ tempest list-plugins + +---------------------------------+------------------------------------------------------------+ + | Name | EntryPoint | + +---------------------------------+------------------------------------------------------------+ + | neutron_tests | neutron_tempest_plugin.plugin:NeutronTempestPlugin | + +---------------------------------+------------------------------------------------------------+ + +To execute a given test or group of tests you can use a regex, or you can use +the idempotent id of a test or the tag associated with the test: + +.. code-block:: shell + + tempest run --config etc/tempest.conf --regex tempest.scenario + tempest run --config etc/tempest.conf --regex neutron_tempest_plugin.scenario + tempest run --config etc/tempest.conf smoke + tempest run --config etc/tempest.conf ab40fc48-ca8d-41a0-b2a3-f6679c847bfe |