zuul-stream: implement a protocol and version

A refresher on how this works, to the best of my knowledge

 1 Firstly, Zuul's Ansible has a library task "zuul_console:" which is
   run against the remote node; this forks a console daemon, listening
   on a default port.

 2 We have a action plugin that runs for each task, and if that task
   is a command/shell task, assigns it a unique id

 3 We then override with library/command.py (which backs command/shell
   tasks) with a version that forks and runs the process on the target
   node as usual, but also saves the stdout/stderr output to a
   temporary file named per the unique uuid from the step above.

 4 At the same time we have the callback plugin zuul_stream.py, which
   Ansible is calling as it moves through starting, running and
   finishing the tasks.  This looks at the task, and if it has a UUID
   [2], sends a request to the zuul_console [1], which opens the
   temporary file [3] and starts streaming it back.

 5 We loop reading this until the connection is closed by [1],
   eventually outputting each line.

In this way, the console log is effectively streamed and saved into
our job output.

We have established that we expect the console [1] is updated
asynchronously to the command/streaming [3,4] in situation such as
static nodes.  This poses a problem if we ever want to update either
part -- for example we can not change the file-name that the
command.py file logs to, because an old zuul_console: will not know to
open the new file.  You could imagine other fantasy things you might
like to do; e.g. negotiate compression etc. that would have similar
issues.

To provide the flexibility for these types of changes, implement a
simple protocol where the zuul_stream and zuul_console sides exchange
their respective version numbers before sending the log files.  This
way they can both decide what operations are compatible both ways.

Luckily the extant protocol, which is really just sending a plain
uuid, can be adapted to this.  When an old zuul_console server gets
the protocol request it will just look like an invalid log file, which
zuul_stream can handle and thus assume the remote end doesn't know
about protocols.

This bumps the testing timeout; it seems that the extra calls make for
random failures.  The failures are random and not in the same place,
I've run this separately in 850719 several times and not seen any
problems with the higher timeout.  This test is already has a settle
timeout slightly higher so I think it must have just been on the edge.

Change-Id: Ief366c092e05fb88351782f6d9cd280bfae96237

1 files changed, 6 insertions, 0 deletions

diff --git a/playbooks/zuul-stream/functional.yaml b/playbooks/zuul-stream/functional.yaml
index 3b79e7b40..2f2e9a059 100644
--- a/playbooks/zuul-stream/functional.yaml
+++ b/playbooks/zuul-stream/functional.yaml
@@ -19,12 +19,18 @@
         ZUUL_JOB_LOG_CONFIG: "{{ ansible_user_dir}}/logging.json"
         ZUUL_JOBDIR: "{{ ansible_user_dir}}"
         PYTHONPATH: "{{ python_path }}"
+      register: _success_output
 
     - name: Save output
       shell: |
           mv job-output.txt job-output-success-19887.txt
           mv job-output.json job-output-success-19887.json
 
+    - name: Check protocol version
+      assert:
+        that:
+          - "'[node1] Reports streaming version: 1' in _success_output.stdout"
+
     # NOTE(ianw) 2022-07 : we deliberatly have this second step to run
     # against the console setup by the infrastructure executor in the
     # job pre playbooks as a backwards compatability sanity check.