summaryrefslogtreecommitdiff
path: root/runtest.py
Commit message (Collapse)AuthorAgeFilesLines
* Use pathlib in runtestMats Wichmann2023-05-081-64/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the past, there have been some mismatches between how tests are specified and how they are found. testlist files, excludelist files and command-line specifications should be agnostic to operating system conventions. For example, typing "runtest.py foo/bar" on windows will produce paths like foo/bar\test.py, which is hard to match and painful to read, it should obviously match discovered foo\bar\test.py. Test information should be output using the native path separator for consistency. Using pathlib lets these be normalized - stored in a common format and output in the expected format. Adding this normalization of course broke some tests, which either intentionally or through omission expected some portion of a path to be UNIX-style. Specifically these five: test\runtest\baseline\fail.py test\runtest\baseline\no_result.py test\runtest\simple\fail.py test\runtest\simple\no_result.py test\runtest\simple\pass.py test\runtest\testargv.py This was fixed and a general cleanup/reformat performed on the runtest tests. Signed-off-by: Mats Wichmann <mats@linux.com>
* Fix handling of posix paths containing backslash, and update testsRyan Saunders2022-08-141-4/+3
|
* Remove unnecessary escaping from runtest.pyRyan Saunders2022-08-141-11/+1
|
* test runner: accept -j 0 to mean detect cpu countMats Wichmann2022-02-041-8/+23
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* Oops, attributed a contribution to the wrong person!Mats Wichmann2021-11-261-1/+1
| | | | | | Eryk Sun provided the stanza used to check for attribution. Signed-off-by: Mats Wichmann <mats@linux.com>
* Fix sider complaints and add missed CHANGES.txt updateMats Wichmann2021-11-261-1/+1
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* Fix tests to not hang on Windows with bad .py assocMats Wichmann2021-11-261-24/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For systems where the association for .py files is not to an actual Python interpreter, those few cases where we need to run a Python script directly as a program don't work. This could be because the association was never set up, or because some other program (e.g. Visual Studio Code) has taken it over. In some cases may appear to "hang" because the alternate program is waiting for user interaction runtest.py now has a mechanism to check (thanks to Brett Cannon for providing this incantation). It isn't super precise (looks for the substring "py" in the queried association), but should work out. It sets an environment variable which the test framework can read and as a result set a flag which individual tests can read. Two tests in scons-time which had previously been set to skip-if-win32 now look at this flag instead. Three tests in sconsign now also look at this flag. This allows a clean run on my dev box with VS Code having taken over the .py association. Various things can break if the environment used to fire off Windows processes doesn't contain %UserProfile%. Added this to the short list of passthrough env vars. Apparently an environment without this value is now considered invalid (it blew up the erroneously launched VS Code, but we've apparently been lucky it hasn't blown up more things - believe there was also a report of a problem with the Visual Studio setup scripts). A little extra cleanup: - a couple of Py2-isms were cleaned out (Script/Main.py and in the test framework) - The paths to look for site-scons were rewritten (part of this was another Py2-ism), and the system path changed a bit - the old path is still checked, and the manpage updated to reflect this. - runtest.py dropped the unused whereis functions. - the three sconsign tests now use f-string formatting, mostly as an experiment to see how easy it is to convert. Fixes #4053 Signed-off-by: Mats Wichmann <mats@linux.com>
* Some adjustments to runtest.pyMats Wichmann2021-11-101-9/+25
| | | | | | | | | | * Close a hole where if logging (-o) *and* multi-job (-j), the feedback from the tests was not logged. (-j) * Clear list of failed tests if no tests failed. * add a flush method to the Tee class, there are error conditions where the lack of a flush threw an exception. Signed-off-by: Mats Wichmann <mats@linux.com>
* fix --no-ignore-skipsWilliam Deegan2021-11-011-1/+0
|
* fix --no-ignore-skipsWilliam Deegan2021-11-011-4/+5
|
* Added --no-ignore-skips and changed default to only exit status 2 if this ↵William Deegan2021-11-011-1/+6
| | | | flag is specified to runtest.py and some tests are skipped. Otherwise assuming no failures will exit 0
* [PR #3965] add TODO per review commentMats Wichmann2021-06-301-5/+8
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3965] fix a couple of typo errorsMats Wichmann2021-06-291-1/+1
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* Add AddOption tests and fix runtest.pyMats Wichmann2021-06-291-3/+11
| | | | | | | | | | | The two tests originally proposed in #3436, now withdrawn, are added along with a new .exclude_tests to not run them, since they'd currently be in a failing state. Needed to fix runtest.py, wasn't completely correctly processing .exclude_tests files. Signed-off-by: Mats Wichmann <mats@linux.com>
* Fix (and simplify) starting tests in debugger [ci skip]Mats Wichmann2021-05-061-6/+3
| | | | | | | | | A mis-indented break statement caused the "-d" option to runtest.py to start a test in the debugger to be inoperative. It's not necessary to search for the path to the debugger module, so this can be simplified to just supply "-m pdb". Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3822] fix typo, reword commentMats Wichmann2020-11-081-4/+4
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* runtest now writes a log of failsMats Wichmann2020-11-071-10/+32
| | | | | | | | | | | | | | | failed_tests.log is created if there are any fails (on by default). --retry option added to rerun tests from that file, or use -f listfile to use a file of a different name. --faillog=FILE allows saving the fails to a non-default name (in case don't want to overwrite the existing one, perhaps); --no-faillog disables the writing of the log. Two unneeded tests relating to qmtest were dropped: fallback.py didn't really test anything, and noqmtest.py was a duplicate of simple/combined.py after the qmtest specifics had been stripped out. Added tests for the added three options (git thinks two were renames). Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3851] fix cut-n-paste error in previous fixMats Wichmann2020-11-051-2/+4
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3821] fix editing errorMats Wichmann2020-11-051-2/+0
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3821] accomodate Py3.5 in pathlib useMats Wichmann2020-11-051-2/+10
| | | | | | | | the resolve() method of a Path objects takes a "strict" argument which was not defined until Py3.6. But the older behavior was "strict", so just add a version check here. Signed-off-by: Mats Wichmann <mats@linux.com>
* Convert test runner to use argparseMats Wichmann2020-11-041-331/+286
| | | | | | | | | | | | Formerly used optparse + homegrown, with a comment stating migration should be done (argprse now has superseded optparse). Cleaned up a few stray bits that weren't used any longer. Collects the changes intended for os.environ into a local dict, which is then used to make a single update. Signed-off-by: Mats Wichmann <mats@linux.com>
* Fix occasional test failures when running multiple jobsAdam Gross2020-10-131-14/+23
| | | | | | | | | | | | | | | | The way runtest.py passes the list of fixture directories is racy because it sets it in os.environ['FIXTURE_DIRS'] and then spawns the subprocess, counting on Python to start the subprocess before that list is overwritten when spawning the next directory. At least on Windows, the environment is not copied in subprocess.run so runtest.py may overwrite the list of fixture directories with the list for test #2 while the subprocess module is still kicking off test #1. I was able to easily reproduce this by running the command: `python runtest.py -j 2 test\MSVC\VSWHERE.py test\AS\ASPPFLAGS.py` a few times in a row. However, with this fix, that command repeatedly succeeds. To validate ths fix, I also ran that command with "--xml a.xml" and "--xml a.xml --nopipefiles" to validate that those other executors worked correctly.
* Further runtest tweaksMats Wichmann2020-10-041-65/+36
| | | | | | | | | Drop unused package and builddir options. Extract the reading of a list of tests to a common routine. Move the e2e-only and unit-only to a separate postprocessing step, and combine with exclude processing. Signed-off-by: Mats Wichmann <mats@linux.com>
* Add selection options to test runnerMats Wichmann2020-10-021-28/+56
| | | | | | | runtest.py can now take --unit-only and --e2e-only selection options to limit a test run to tests from one or other category. Signed-off-by: Mats Wichmann <mats@linux.com>
* correct copyright statement per mwichmann's feedbackWilliam Deegan2020-09-211-1/+1
|
* Fix runtest.py to check SCons dir instead of src dir. Missed this change ↵William Deegan2020-05-101-5/+5
| | | | during repro directory refactor.
* Reorganize the repo. Moved src/engine/SCons to ./SCons to be more in line ↵William Deegan2020-05-061-1/+1
| | | | with current python packaging practices
* Address github review notes and sider issuesWilliam Deegan2020-04-111-20/+2
|
* drop runntest.py's run from packages. define SCONS_TOOLS_DIR and use that ↵William Deegan2020-04-091-78/+38
| | | | for scons-time tests
* Fix scripts dirname typoWilliam Deegan2020-04-091-1/+1
|
* update runtest.py to find scons in scriptsWilliam Deegan2020-04-091-1/+1
|
* Merge branch 'master' into runtest-devmodeWilliam Deegan2020-02-251-147/+166
|\
| * rutest cleanups and subprocess updateMats Wichmann2020-02-241-148/+162
| | | | | | | | | | | | | | | | | | | | | | | | | | - Clean up some things suggested by checkers. - Add a few docsstrings. - Make RuntestBase an abstract class - Eliminate Py2 remnants. - Use subprocess.run for a slightly cleaner interface - don't have to separately do wait or communicate, don't need to set up context manager, and everything returns cleanly in a subprocess.CompletedProcess. Signed-off-by: Mats Wichmann <mats@linux.com>
| * Fix runtest no-exec modeMats Wichmann2020-02-241-1/+4
| | | | | | | | | | | | | | | | Failure to default some values in class initializer caused a downstream problem if running no-exec mode, as one field was added only if running in execute mode. Signed-off-by: Mats Wichmann <mats@linux.com>
* | Change devmode check to not exclude 3.7.0Mats Wichmann2020-02-201-2/+2
| | | | | | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* | Let test runner use new Dev. ModeMats Wichmann2020-02-201-4/+9
|/ | | | | | | Python Development Mode imposes more checks (since Python 3.7). Allow it to optionally be enabled while doing a test run. Signed-off-by: Mats Wichmann <mats@linux.com>
* rm-py2: Remove "from __future__" from more placesMats Wichmann2020-02-201-2/+0
| | | | | | | | | | | | | | | A couple of minor reformats along the way, most prominently, in tests, if being edited anyway, make sure the docstring most tests have is actually the docstring (sometimes the __revision__ line came before, which makes the string not be the docstring). Snuck in some minor framework changes that were orphaned when another draft PR was not needed: this almost all docstring changes, the functional part is using casefold instead of lower in a match func - a slightly better approach which is now possible that Py2 compatibility is not needed. Signed-off-by: Mats Wichmann <mats@linux.com>
* Fix runtest.py to only run *Tests.py files under testing as well as under ↵William Deegan2020-02-181-1/+1
| | | | src (unittests)
* Get rid of remaining mktemp callsMats Wichmann2019-12-301-1/+0
| | | | | | | | | | | | Except for Platform/win32.py, the engine code no longer called insecure/deprecated tempfile.mktemp. That location is now also changed to use better methods. A couple of vestiges of old ways (like in comments) cleaned out. A couple of unit tests also converted - these don't have any impact on the scons engine, of course. Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3411] fix typo spotted by sider CIMats Wichmann2019-08-021-4/+4
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* Document and polish runtest.pyMats Wichmann2019-08-021-44/+51
| | | | | | | | This is a minor change to runtest, some slight rearrangements, adding a docstring, clarifying a few comments. The subclass of Thread is now more precise in following the Thread constructor. Signed-off-by: Mats Wichmann <mats@linux.com>
* [PR #3330] do not use shell in subproc callsMats Wichmann2019-03-291-14/+16
| | | | | | | | | | The various instances of subprocess.Popen call with shell=True, but this is not necessary, the test command lines don't depend on any shell processing and are complete as constructed. With shell=False, send a list of command words instead of a single string. Signed-off-by: Mats Wichmann <mats@linux.com>
* Runtest improvements for multi-job runsMats Wichmann2019-03-291-160/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Main fix: if duing a multi-job run (-j flag is present), collect the test output in runtest and keep the test heading together with the test output, so the information about a given test will not separated in the output. The test number is recorded in the test instance instead of kept as a global. Tests may appear out of order now, but since they may be run "out of order" depending on how fast the various threads complete the jobs they pull off the queue, that just reflects reality. The help message is now the script's docstring, and help prints the docstring along with some other info. This avoids having the options described twice (and going out of sync) - once in the inital comment, once in the help message. A mention of the -j multi-job option is added, was previously missing. Also: one less worker than requested was created due to use of range(1, n) (may have been intentional?). changed to number requested. Also: the lock is taken from the RunTest instance, not from the global lock variable. This must have been intended as the lock is stored in the instance on initialization. The lock was not really declared as a global, it was just assigned in global scope. Also: the test result reporting is now split off into its own function, makes a bit cleaner separation (and facilitates possible changes to the multi-job model in future). Signed-off-by: Mats Wichmann <mats@linux.com>
* Clean up some file opens, regex stringsMats Wichmann2019-03-071-7/+13
| | | | | | | | | | | | | | | | | | Most recent Python (3.8 alpha) spews warnings aplenty about two subjects: unclosed files and strings which look like they have embedded escapes that Python does not recognize. The latter are usually regexes, and it provides a reminder that regular expressions should normally be specified as raw strings, so Python does not attempt to interpret them. Irritating is that even docstrings are flagged, it's not obvious what the right answer is for a docstring which contains, say, a Windows-style path with backslashes. This converts a bunch of opens that are not closed into context manager usage and regex patterns into raw strings. This eliminate about 4000 warnings spewed by Py3.8 (9200 remain). Signed-off-by: Mats Wichmann <mats@linux.com>
* [WIP] for #3304: drop use of -tt if py3Mats Wichmann2019-03-011-0/+2
| | | | | | | | | | As noted in issue #3304, the Python flag to error on inconsistent tabs/spaces has been dropped for Python 3 interpreters; while CPython still accepts it, silently ignoring it, PyPy3 errors out. This change adds the flag throughout the testsuite run only if using a Python2 interpreter. Signed-off-by: Mats Wichmann <mats@linux.com>
* Drop -tt flag in runtest.pyMats Wichmann2019-03-011-1/+1
| | | | | | | | | | Test runs launch Python with the -tt flag to error on inconsistent tab usage. That flag is no longer part of Python 3, though it is silently accepted and ignored in cpython. In PyPy3, it errors. We have other ways to detect such things now anyway, so just drop. Signed-off-by: Mats Wichmann <mats@linux.com>
* More deletion of _JAVA_OPTIONS to runtest.py from travis startup files.William Deegan2018-08-301-0/+4
|
* Part of previous change restoredMats Wichmann2018-08-301-1/+1
| | | | | somehow the change from if test to try block got partly lost during commit. changing else->except
* Rework the time.clock -> time.perf_counter changeMats Wichmann2018-08-301-3/+2
| | | | Signed-off-by: Mats Wichmann <mats@linux.com>
* Do not use time.clock for Py3 in runtestMats Wichmann2018-08-301-4/+8
| | | | | | | | | | | | | | | | | | | | | time.clock is deprecated since Python 3.3 and will be removed in Python 3.8. 3.7 started issuing DeprecationWarning, which fails about 16 tests in AppVeyor CI (and running manually), since the warning appears in stderr stream and so real vs expected does not match any longer. Arguably the tests could be fixed to do a different check (contains rather than exactly-equal), but a change needs to be made anyway. use time.perf_counter as the time function for Python 3. This works for Windows and non-Windows; however since this function did not exist in 2.7 (added for 3.3), the Py2 case is left, which selects between time.clock and time.time depending on OS. Note this addresses runtest.py, but not code in bench/, which need attention also. Suggest that be a separate change since it's not failing tests. Signed-off-by: Mats Wichmann <mats@linux.com>
View Lorry Controller Status