diff options
| -rw-r--r-- | doc/source/developer/ansible.rst | 71 | ||||
| -rw-r--r-- | doc/source/howtos/nodepool_static.rst | 32 | ||||
| -rw-r--r-- | doc/source/installation.rst | 14 |
3 files changed, 89 insertions, 28 deletions
diff --git a/doc/source/developer/ansible.rst b/doc/source/developer/ansible.rst index 415c47df7..c3135debe 100644 --- a/doc/source/developer/ansible.rst +++ b/doc/source/developer/ansible.rst @@ -4,19 +4,11 @@ Ansible Integration Zuul contains Ansible modules and plugins to control the execution of Ansible Job content. -Build Log Support ------------------ +Zuul provides realtime build log streaming to end users so that users +can watch long-running jobs in progress. -Zuul provides realtime build log streaming to end users so that users can -watch long-running jobs in progress. As jobs may be written that execute a -shell script that could run for a long time, additional effort is expended -to stream stdout and stderr of shell tasks as they happen rather than waiting -for the command to finish. - -Zuul contains a modified version of the :ansible:module:`command` -that starts a log streaming daemon on the build node. - -.. automodule:: zuul.ansible.base.library.command +Streaming job output +-------------------- All jobs run with the :py:mod:`zuul.ansible.base.callback.zuul_stream` callback plugin enabled, which writes the build log to a file so that the @@ -35,10 +27,55 @@ exposes that log stream over a websocket connection as part of In addition to real-time streaming, Zuul also installs another callback module, :py:mod:`zuul.ansible.base.callback.zuul_json.CallbackModule` that collects all of the information about a given run into a json file which is written to the -work dir so that it can be published along with build logs. Since the streaming -log is by necessity a single text stream, choices have to be made for -readability about what data is shown and what is not shown. The json log file -is intended to allow for a richer more interactive set of data to be displayed -to the user. +work dir so that it can be published along with build logs. .. autoclass:: zuul.ansible.base.callback.zuul_json.CallbackModule + +Since the streaming log is by necessity a single text stream, choices +have to be made for readability about what data is shown and what is +not shown. The json log file is intended to allow for a richer more +interactive set of data to be displayed to the user. + +.. _zuul_console_streaming: + +Capturing live command output +----------------------------- + +As jobs may execute long-running shell scripts or other commands, +additional effort is expended to stream ``stdout`` and ``stderr`` of +shell tasks as they happen rather than waiting for the command to +finish. + +The global job configuration should run the ``zuul_console`` task as a +very early prerequisite step. + +.. automodule:: zuul.ansible.base.library.zuul_console + +This will start a daemon that listens on TCP port 19885 on the testing +node. This daemon can be queried to stream back the output of shell +tasks as described below. + +Zuul contains a modified version of Ansible's +:ansible:module:`command` module that overrides the default +implementation. + +.. automodule:: zuul.ansible.base.library.command + +This library will capture the output of the running +command and write it to a temporary file on the host the command is +running on. These files are named in the format +``/tmp/console-<uuid>-<task_id>-<host>.log`` + +The ``zuul_stream`` callback mentioned above will send a request to +the remote ``zuul_console`` daemon, providing the uuid and task id of +the task it is currently processing. The ``zuul_console`` daemon will +then read the logfile from disk and stream the data back as it +appears, which ``zuul_stream`` will then present as described above. + +The ``zuul_stream`` callback will indicate to the ``zuul_console`` +daemon when it has finished reading the task, which prompts the remote +side to remove the temporary streaming output files. In some cases, +aborting the Ansible process may not give the ``zuul_stream`` callback +the chance to send this notice, leaking the temporary files. If nodes +are ephemeral this makes little difference, but these files may be +visible on static nodes. diff --git a/doc/source/howtos/nodepool_static.rst b/doc/source/howtos/nodepool_static.rst index ff2d35d6a..c10672e7b 100644 --- a/doc/source/howtos/nodepool_static.rst +++ b/doc/source/howtos/nodepool_static.rst @@ -15,9 +15,9 @@ the following requirements: * Must be reachable by Zuul executors and have SSH access enabled. * Must have a user that Zuul can use for SSH. -* Must have Python 2 installed for Ansible. -* Must be reachable by Zuul executors over TCP port 19885 (console log - streaming). +* Must have an Ansible supported Python installed +* Must be reachable by Zuul executors over TCP port 19885 for console + log streaming. See :ref:`nodepool_console_streaming` When setting up your nodepool.yaml file, you will need the host keys for each node for the ``host-key`` value. This can be obtained with @@ -40,7 +40,7 @@ nodes. Place this file in ``/etc/nodepool/nodepool.yaml``: - host: localhost labels: - - name: ubuntu-xenial + - name: ubuntu-jammy providers: - name: static-vms @@ -49,14 +49,34 @@ nodes. Place this file in ``/etc/nodepool/nodepool.yaml``: - name: main nodes: - name: 192.168.1.10 - labels: ubuntu-xenial + labels: ubuntu-jammy host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ4qE47o5gb" username: zuul - name: 192.168.1.11 - labels: ubuntu-xenial + labels: ubuntu-jammy host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ5sE47o5gc" username: zuul EOF" Make sure that ``username``, ``host-key``, IP addresses and label names are customized for your environment. + +.. _nodepool_console_streaming: + +Log streaming +------------- + +The log streaming service enables Zuul to show the live status of +long-running ``shell`` or ``command`` tasks. The server side is setup +by the ``zuul_console:`` task built-in to Zuul's Ansible installation. +The executor requires the ability to communicate with the job nodes on +port 19885 for this to work. + +The log streaming service may leave files on the static node in the +format ``/tmp/console-<uuid>-<task_id>-<host>.log`` if jobs are +interrupted. These may be safely removed after a short period of +inactivity with a command such as + +.. code-block:: shell + + find /tmp -maxdepth 1 -name 'console-*-*-<host>.log' -mtime +2 -delete diff --git a/doc/source/installation.rst b/doc/source/installation.rst index a9a526f13..17665ca76 100644 --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -10,11 +10,15 @@ Nodepool ~~~~~~~~ In order to run all but the simplest jobs, Zuul uses a companion -program, Nodepool, to supply the nodes (whether dynamic cloud -instances or static hardware) used by jobs. Before starting Zuul, -ensure you have Nodepool installed and any images you require built. -Zuul only makes one requirement of these nodes: that it be able to log -in given a username and ssh private key. +program `Nodepool <https://opendev.org/zuul/nodepool>`__ to supply the +nodes (whether dynamic cloud instances or static hardware) used by +jobs. Before starting Zuul, ensure you have Nodepool installed and +any images you require built. + +Zuul must be able to log into the nodes provisioned by Nodepool with a +given username and SSH private key. Executors should also be able to +talk to nodes on TCP port 19885 for log streaming; see +:ref:`nodepool_console_streaming`. ZooKeeper ~~~~~~~~~ |