zephyr: zmake: separate zmake test execution out of zmake

Putting execution of "pytest" inside of zmake testall itself is
problematic:  The path this code runs relies on a lot of zmake's
features, and if there were a bug in zmake that prevented that code
from being reached, the code could never get executed.  Additionally,
the log output is impossible to read as it gets mixed in with all the
build output.

Separate it into a simple shell script, which greatly simplifies the
code, and makes it easier to add further style tests down the
road (isort, flake8, black).

BUG=b:192389533
BRANCH=none
TEST=run firmware_builder.py

Signed-off-by: Jack Rosenthal <jrosenth@chromium.org>
Change-Id: I8795ba0c0225183b597f3738575c790ba73610f1
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/3002837
Commit-Queue: Yuval Peress <peress@chromium.org>
Reviewed-by: Yuval Peress <peress@chromium.org>

3 files changed, 27 insertions, 55 deletions

diff --git a/zephyr/zmake/run_tests.sh b/zephyr/zmake/run_tests.sh
new file mode 100755
index 0000000000..2dd1a21939
--- /dev/null
+++ b/zephyr/zmake/run_tests.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Copyright 2021 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Run tests for zmake itself (not including Zephyr builds).
+
+# Show commands being run.
+set -x
+
+# Exit if any command exits non-zero.
+set -e
+
+# cd to the directory containing this script.
+cd "$(dirname "$(realpath -e "${BASH_SOURCE[0]}")")"
+
+# Test the copy in-tree, instead of what setuptools or the ebuild
+# installed.
+export PYTHONPATH="${PWD}"
+
+# Run pytest.
+# TODO(jrosenth): --hypothesis-profile=cq is very likely to be
+# unnecessary, as this was only needed when we were heavily taxing the
+# CPU by running pytest alongside all the ninjas, which no longer
+# happens.  Remove this flag.
+pytest --hypothesis-profile=cq .
diff --git a/zephyr/zmake/zmake/__main__.py b/zephyr/zmake/zmake/__main__.py
index 45712cc591..389d5f09b6 100644
--- a/zephyr/zmake/zmake/__main__.py
+++ b/zephyr/zmake/zmake/__main__.py
@@ -77,10 +77,6 @@ def main(argv=None):
                         'checkout.')
     parser.add_argument('--zephyr-base', type=pathlib.Path,
                         help='Path to Zephyr OS repository')
-    parser.add_argument('--cq', action='store_true',
-                        default=False,
-                        help='Run this pass from CQ (turns off some flaky '
-                             'features)')
 
     sub = parser.add_subparsers(dest='subcommand', help='Subcommand')
     sub.required = True
diff --git a/zephyr/zmake/zmake/zmake.py b/zephyr/zmake/zmake/zmake.py
index e102e865a1..c535ebc656 100644
--- a/zephyr/zmake/zmake/zmake.py
+++ b/zephyr/zmake/zmake/zmake.py
@@ -146,11 +146,10 @@ class Zmake:
             before launching more, False to just do this after all jobs complete
     """
     def __init__(self, checkout=None, jobserver=None, jobs=0, modules_dir=None,
-                 zephyr_base=None, cq=False):
+                 zephyr_base=None):
         zmake.multiproc.reset()
         self._checkout = checkout
         self._zephyr_base = zephyr_base
-        self._is_cq = cq
 
         if modules_dir:
             self.module_paths = zmake.modules.locate_from_directory(modules_dir)
@@ -437,51 +436,6 @@ class Zmake:
                 raise OSError(get_process_failure_msg(proc))
         return 0
 
-    def _run_pytest(self, directory):
-        """Run pytest on a given directory.
-
-        This is a utility function to help parallelize running pytest on
-        multiple directories.
-
-        Args:
-            directory: The directory that we should search for tests in.
-        """
-        def get_log_level(line, current_log_level):
-            matches = [
-                ('PASSED', logging.INFO),
-                ('FAILED', logging.ERROR),
-                ('warnings summary', logging.WARNING),
-            ]
-
-            for text, lvl in matches:
-                if text in line:
-                    return lvl
-
-            return current_log_level
-
-        def run_test(test_file):
-            with self.jobserver.get_job():
-                proc_args = ['pytest', '--verbose']
-                if self._is_cq:
-                    proc_args.append('--hypothesis-profile=cq')
-                proc = self.jobserver.popen(
-                    proc_args + [test_file],
-                    stdout=subprocess.PIPE,
-                    stderr=subprocess.STDOUT,
-                    encoding='utf-8',
-                    errors='replace')
-                zmake.multiproc.log_output(
-                    self.logger, logging.DEBUG,
-                    proc.stdout, log_level_override_func=get_log_level,
-                    job_id=os.path.basename(test_file),)
-                rv = proc.wait()
-                if rv:
-                    self.logger.error(get_process_failure_msg(proc))
-                return rv
-
-        for test_file in directory.glob('test_*.py'):
-            self.executor.append(func=lambda f=test_file: run_test(f))
-
     def testall(self):
         """Test all the valid test targets"""
         tmp_dirs = []
@@ -501,10 +455,6 @@ class Zmake:
                     build_after_configure=True,
                     test_after_configure=is_test))
 
-        # Run pytest on platform/ec/zephyr/zmake/tests.
-        self._run_pytest(
-            self.module_paths['ec'] / 'zephyr' / 'zmake' / 'tests')
-
         rv = self.executor.wait()
         for tmpdir in tmp_dirs:
             shutil.rmtree(tmpdir)