summaryrefslogtreecommitdiff
path: root/testsuite/driver/runtests.py
Commit message (Collapse)AuthorAgeFilesLines
* ci: test in-tree compiler in hadrianZubin Duggal2021-10-131-3/+3
|
* hadrian, testsuite: Teach Hadrian to query the testsuite driver for dependenciesZubin Duggal2021-10-131-0/+10
| | | | Issues #19072, #17728, #20176
* testsuite: Clean up dynlib support predicatesBen Gamari2021-10-121-0/+1
| | | | | | | | | | | | | | | | | | Previously it was unclear whether req_shared_libs should require: * that the platform supports dynamic library loading, * that GHC supports dynamic linking of Haskell code, or * that the dyn way libraries were built Clarify by splitting the predicate into two: * `req_dynamic_lib_support` demands that the platform support dynamic linking * `req_dynamic_hs` demands that the GHC support dynamic linking of Haskell code on the target platform Naturally `req_dynamic_hs` cannot be true unless `req_dynamic_lib_support` is also true.
* [darwin] stop the DYLD_LIBRARY_PATH madnessMoritz Angermann2021-05-071-5/+2
| | | | | | | | this causes *significant* slowdown on macOS as the linker ends up looking through all the paths. Slowdown can be as bad as 100% or more. (cherry picked from commit 820b0766984d42c06c977a6c32da75c429106f7f)
* tests: Allow --skip-perf-tests/--only-perf-tests to be used with ↵Matthew Pickering2021-04-101-1/+1
| | | | --ignore-perf-failures
* Revert "[ci/arm/darwin/testsuite] Forwards ports from GHC-8.10"Ben Gamari2021-04-051-2/+5
| | | | This reverts commit 0cbdba2768d84a0f6832ae5cf9ea1e98efd739da.
* [ci/arm/darwin/testsuite] Forwards ports from GHC-8.10Moritz Angermann2021-03-211-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a set of forward ports (cherry-picks) from 8.10 - a7d22795ed [ci] Add support for building on aarch64-darwin - 5109e87e13 [testlib/driver] denoise - 307d34945b [ci] default value for CONFIGURE_ARGS - 10a18cb4e0 [testsuite] mark ghci056 as fragile - 16c13d5acf [ci] Default value for MAKE_ARGS - ab571457b9 [ci/build] Copy config.sub around - 251892b98f [ci/darwin] bump nixpkgs rev - 5a6c36ecb4 [testsuite/darwin] fix conc059 - aae95ef0c9 [ci] add timing info - 3592d1104c [Aarch64] No div-by-zero; disable test. - 57671071ad [Darwin] mark stdc++ tests as broken - 33c4d49754 [testsuite] filter out superfluous dylib warnings - 4bea83afec [ci/nix-shell] Add Foundation and Security - 6345530062 [testsuite/json2] Fix failure with LLVM backends - c3944bc89d [ci/nix-shell] [Darwin] Stop the ld warnings about libiconv. - b821fcc714 [testsuite] static001 is not broken anymore. - f7062e1b0c [testsuite/arm64] fix section_alignment - 820b076698 [darwin] stop the DYLD_LIBRARY_PATH madness - 07b1af0362 [ci/nix-shell] uniquify NIX_LDFLAGS{_FOR_TARGET} As well as a few additional fixups needed to make this block compile: - Fixup all.T - Set CROSS_TARGET, BROKEN_TESTS, XZ, RUNTEST_ARGS, default value. - [ci] shell.nix bump happy
* testsuite: Make --ignore-perf-tests more expressiveBen Gamari2021-03-191-2/+9
| | | | Allow skipping of only increases/decreases.
* testsuite: Introduce flag to ignore performance failuresBen Gamari2021-02-241-0/+2
| | | | Needed by #19025.
* Reduce default test verbosityMatthew Pickering2021-01-281-1/+1
|
* Testsuite: Support for user supplied package dbsAndreas Klebinger2020-11-041-0/+4
| | | | | | | | | We can now supply additional package dbs to the testsuite. For make the package db can be supplied by passing PACKAGE_DB=/path/to/db. In the testsuite driver it's passed via the --test-package-db argument.
* testsuite: Add --top flag to driverGHC GitLab CI2020-11-021-1/+6
| | | | | This allows us to make `config.top` a proper Path. Previously it was a str, which caused the Ghostscript detection logic to break.
* testsuite: Sort metrics by metric typeBen Gamari2020-10-161-1/+15
| | | | Closes #18838.
* Enhance metrics outputSylvain Henry2020-09-151-5/+6
|
* testsuite: Output performance test results in tabular formatDaishi Nakajima2020-09-081-19/+48
| | | | | | | | | | | | | | | | | this was suggested in #18417. Change the print format of the values. * Shorten commit hash * Reduce precision of the "Value" field * Shorten metrics name * e.g. runtime/bytes allocated -> run/alloc * Shorten "MetricsChange" * e.g. unchanged -> unch, increased -> incr And, print the baseline environment if there are baselines that were measured in a different environment than the current environment. If all "Baseline commit" are the same, print it once.
* testsuite: Allow baseline commit to be set explicitlyBen Gamari2020-08-181-8/+12
|
* Sort deterministically metric outputKrzysztof Gogolewski2020-05-211-1/+1
| | | | | | Previously, we sorted according to the test name and way, but the metrics (max_bytes_used/peak_megabytes_allocated etc.) were appearing in nondeterministic order.
* testsuite: Print sign of performance changesBen Gamari2020-05-131-2/+2
| | | | | Executes the minor formatting change in the tabulated performance changes suggested in #18135.
* testsuite: Make hasMetricsFile RHS more descriptiveBen Gamari2020-02-261-1/+1
|
* testsuite: Allow tests to be marked as broken on the command lineBen Gamari2020-02-261-0/+4
| | | | This allows us to work-around distribution-specific breakage easily.
* testsuite: Use colors more consistentlyBen Gamari2019-12-051-8/+1
|
* testsuite: Make performance metric summary more readableBen Gamari2019-12-051-15/+35
| | | | Along with some refactoring.
* testsuite: Factor out terminal coloringBen Gamari2019-12-051-0/+3
|
* CI: Always dump performance metrics.David Eichmann2019-10-221-3/+19
|
* testsuite: More type checking fixesBen Gamari2019-07-181-2/+2
|
* testsuite: A major revamp of the driverBen Gamari2019-06-251-11/+20
| | | | | | | | | This tries to put the testsuite driver into a slightly more maintainable condition: * Add type annotations where easily done * Use pathlib.Path instead of str paths * Make it pass the mypy typechecker
* gitlab-ci: Lint testsuite for framework failuresBen Gamari2019-06-141-1/+1
| | | | | This introduces a new lint job checking for framework failures and listing broken tests.
* Update terminal title while running test-suiteOleg Grenrus2019-05-141-1/+18
| | | | | Useful progress indicator even when `make test VERBOSE=1`, and when you do something else, but have terminal title visible.
* Exit with exit code 1 when tests unexpectedly passMatthew Pickering2019-02-231-0/+1
| | | | | | This was causing gitlab to not report from builds as failing. It also highlighted a problem with the LLVM tests where some of the external interpreter tests are failing.
* Fix test runner crash when not in a git repoDavid Eichmann2019-02-211-6/+5
| | | | Respect `inside_git_repo()` when checking performance stats.
* Fix and Reapply "Performance tests: recover a baseline from ancestor commits ↵David Eichmann2019-02-161-8/+27
| | | | and CI results."
* Revert "Performance tests: recover a baseline from ancestor commits and CI ↵Ben Gamari2019-01-311-28/+8
| | | | | | | | | results." Unfortunately this has broken all future commits due to spurious(?) performance changes which I have been unable to work around. This reverts commit cc2261d42f6a954d88e355aaad41f001f65c95da.
* Performance tests: recover a baseline from ancestor commits and CI results.David Eichmann2019-01-301-8/+28
| | | | gitlab-ci: push performance metrics as git notes to the "GHC Performance Notes" repository.
* Revert "Batch merge"Ben Gamari2019-01-301-28/+8
| | | | This reverts commit 76c8fd674435a652c75a96c85abbf26f1f221876.
* Batch mergeBen Gamari2019-01-301-8/+28
|
* testsuite: Add predicate for CPU feature availabilityBen Gamari2019-01-271-0/+7
| | | | | | | Previously testing code-generation for ISA extensions was nearly impossible since we had no ability to determine whether the host supports the needed extension. Here we fix this by introducing a simple /proc/cpuinfo-based testsuite predicate. We really ought to
* testsuite: Print which ways we are going to runBen Gamari2018-12-121-1/+4
|
* Do not save performance test results if worktree is dirty.David Eichmann2018-12-111-2/+7
| | | | | | | | | | | | Reviewers: bgamari, tdammers Reviewed By: bgamari, tdammers Subscribers: rwbarton, carter GHC Trac Issues: #15924 Differential Revision: https://phabricator.haskell.org/D5368
* testsuite: Don't use git status to determine whether we are inside a repoBen Gamari2018-12-011-3/+3
| | | | | Git status is extremely expensive for this task. We instead use `git rev-parse HEAD` and throw away the output to ensure we don't spam the user.
* Skip all performance tests if not in a git repo.David Eichmann2018-11-301-6/+26
| | | | | | | | | | | | Reviewers: bgamari, tdammers, osa1 Reviewed By: tdammers Subscribers: osa1, tdammers, rwbarton, carter GHC Trac Issues: #15923 Differential Revision: https://phabricator.haskell.org/D5367
* testsuite: Save performance metrics in git notes.David Eichmann2018-11-071-11/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes the following improvement: - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files. - On expected metric changes, the programmer need only indicate the direction of change in the git commit message. - Provides a simple python tool "perf_notes.py" to compare metrics over time. Issues: - Using just the previous commit allows performance to drift with each commit. - Currently we allow drift as we have a preference for minimizing false positives. - Some possible alternatives include: - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric) - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics) - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes. - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes. Previously, *.T files contain statements such as: ``` stats_num_field('peak_megabytes_allocated', (2, 1)) compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)]) ``` This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with: ``` collect_stats('peak_megabytes_allocated', 5) collect_compiler_stats('bytes allocated', 10) ``` So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g. ``` # Metric (In|De)crease <metric(s)> <options>: <tests> Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \ (test_env='linux_x86', way='default'): Test012, Test345 Metric Decrease 'bytes allocated': Test678 Metric Increase: Test711 ``` This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711) phabricator will use the message in the description Reviewers: bgamari, hvr Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #12758 Differential Revision: https://phabricator.haskell.org/D5059
* Simplify testsuite driver, part 2Krzysztof Gogolewski2018-08-121-4/+4
| | | | | | | | | | | | | | | | | | Summary: - Avoid import *; this helps tools such as pyflakes. The last occurrence in runtests.py is not easy to remove as it's used by .T files. - Use False/True instead of 0/1. Test Plan: validate Reviewers: bgamari, thomie, simonmar Reviewed By: thomie Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5062
* Simplify testsuite driverKrzysztof Gogolewski2018-08-111-1/+1
| | | | | | | | | | | | | | | | | | | Summary: - remove clean_cmd - framework_failures was undefined - times_file was not used - if_verbose_dump was called only when verbose >= 1; remove the check - simplify normalise_whitespace Test Plan: validate Reviewers: bgamari, thomie Reviewed By: thomie Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5061
* Testsuite driver: fix encoding issue when calling ghc-pkgKrzysztof Gogolewski2018-08-061-1/+1
| | | | | | | | | | | | | | | | Summary: In Python 3, subprocess.communicate() returns a pair of bytes, which need to be decoded. In runtests.py, we were just calling str() instead, which converts b'x' to "b'x'". As a result, the loop that was checking pkginfo for lines starting with 'library-dirs' couldn't work. Reviewers: bgamari, thomie, Phyx Reviewed By: thomie Subscribers: Phyx, rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5046
* Fix typosKrzysztof Gogolewski2018-08-051-1/+1
|
* Remove dead code in testsuite driverKrzysztof Gogolewski2018-07-271-4/+0
| | | | | | | | | | | | Test Plan: validate Reviewers: bgamari, O7 GHC - Testsuite Reviewed By: bgamari Subscribers: rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D4972
* #15387 Fix setting testsuite verbose to zeroAntti Siponen2018-07-161-1/+1
|
* testsuite: Print summary even if interruptedBen Gamari2018-06-141-15/+18
| | | | | | | | | | | | | | Fixes #15265. Reviewers: osa1 Reviewed By: osa1 Subscribers: rwbarton, thomie, carter GHC Trac Issues: #15265 Differential Revision: https://phabricator.haskell.org/D4841
* Windows: fix all failing tests.Tamar Christina2018-01-021-10/+23
| | | | | | | | | | | | | | | | | | | | | | This makes the testsuite pass clean on Windows again. It also fixes the `libstdc++-6.dll` error harbormaster was showing. I'm marking some tests as isolated tests to reduce their flakiness (mostly concurrency tests) when the test system is under heavy load. Updates process submodule. Test Plan: ./validate Reviewers: hvr, bgamari, erikd, simonmar Reviewed By: bgamari Subscribers: rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D4277
* testsuite: Exit with non-zero exit code when tests failBen Gamari2017-12-181-1/+8
|
View Lorry Controller Status