summaryrefslogtreecommitdiff
path: root/testsuite/driver/testglobals.py
Commit message (Collapse)AuthorAgeFilesLines
* testsuite: Make --ignore-perf-tests more expressiveBen Gamari2021-03-191-1/+2
| | | | Allow skipping of only increases/decreases.
* testsuite: Introduce flag to ignore performance failuresBen Gamari2021-02-241-0/+3
| | | | Needed by #19025.
* Fix typosBrian Wignall2021-02-061-1/+1
|
* Reduce default test verbosityMatthew Pickering2021-01-281-1/+1
|
* Merge remote-tracking branch 'origin/wip/tsan/all'Ben Gamari2020-11-081-0/+3
|\
| * testsuite: Mark setnumcapabilities001 as broken with TSANGHC GitLab CI2020-10-241-0/+3
| | | | | | | | Due to #18808.
* | Testsuite: Support for user supplied package dbsAndreas Klebinger2020-11-041-0/+3
| | | | | | | | | | | | | | | | | | We can now supply additional package dbs to the testsuite. For make the package db can be supplied by passing PACKAGE_DB=/path/to/db. In the testsuite driver it's passed via the --test-package-db argument.
* | testsuite: Add --top flag to driverGHC GitLab CI2020-11-021-1/+1
|/ | | | | This allows us to make `config.top` a proper Path. Previously it was a str, which caused the Ghostscript detection logic to break.
* testsuite: Only run llvm ways if llc is availableBen Gamari2020-08-181-0/+3
| | | | | | | | | As noted in #18560, we previously would always run the LLVM ways since `configure` would set `SettingsLlcCommand` to something non-null when it otherwise couldn't find the `llc` executable. Now we rather probe for the existence of the `llc` executable in the testsuite driver. Fixes #18560.
* testsuite: Allow baseline commit to be set explicitlyBen Gamari2020-08-181-1/+4
|
* Add test for #18064Sylvain Henry2020-07-221-0/+3
| | | | It has been fixed by 0effc57d48ace6b719a9f4cbeac67c95ad55010b
* Update testsuiteSylvain Henry2020-06-171-0/+3
| | | | | | | | | | | | | | * support detection of slow ghc-bignum backend (to replace the detection of integer-simple use). There are still some test cases that the native backend doesn't handle efficiently enough. * remove tests for GMP only functions that have been removed from ghc-bignum * fix test results showing dependent packages (e.g. integer-gmp) or showing suggested instances * fix test using Integer/Natural API or showing internal names
* testsuite: Refactor ghostscript detectionBen Gamari2020-06-011-1/+1
| | | | Tamar reported that he saw crashes due to unhandled exceptions.
* testsuite: Allow tests to be marked as broken on the command lineBen Gamari2020-02-261-1/+6
| | | | This allows us to work-around distribution-specific breakage easily.
* testsuite: Make performance metric summary more readableBen Gamari2019-12-051-1/+7
| | | | Along with some refactoring.
* Fix typos, using Wikipedia list of common typosBrian Wignall2019-11-281-1/+1
|
* CI: Always dump performance metrics.David Eichmann2019-10-221-2/+2
|
* testsuite: Rework tracking of fragile testsBen Gamari2019-08-041-1/+2
| | | | | Breaks fragile tests into two groups, allowing us to easily preserve stdout/stderr of failing fragile tests.
* testsuite: More type checking fixesBen Gamari2019-07-181-20/+22
|
* testsuite: More type signatureswip/run-fragile-testsBen Gamari2019-06-261-8/+10
|
* testsuite: Run and report on fragile testsBen Gamari2019-06-261-0/+4
| | | | | | This allows us to run (but ignore the result of) fragile testcases. Hopefully this should allow us to more easily spot when a fragile test becomes un-fragile.
* testsuite: A major revamp of the driverBen Gamari2019-06-251-19/+60
| | | | | | | | | This tries to put the testsuite driver into a slightly more maintainable condition: * Add type annotations where easily done * Use pathlib.Path instead of str paths * Make it pass the mypy typechecker
* testsuite: introduce 'static_stats' testsAlp Mestanogullari2019-05-291-0/+5
| | | | | | | | | | | | | | | | They are a particular type of perf tests. This patch introduces a 'stats_files_dir' configuration field in the testsuite driver where all haddock timing files (and possibly others in the future) are assumed to live. We also change both the Make and Hadrian build systems to pass respectively $(TOP)/testsuite/tests/perf/haddock/ and <build root>/stage1/haddock-timing-files/ as the value of that new configuration field, and to generate the timing files in those directories in the first place while generating documentation with haddock. This new test type can be seen as one dedicated to examining stats files that are generated while building a GHC distribution. This also lets us get rid of the 'extra_files' directives in the all.T entries for haddock.base, haddock.Cabal and haddock.compiler.
* Update terminal title while running test-suiteOleg Grenrus2019-05-141-0/+3
| | | | | Useful progress indicator even when `make test VERBOSE=1`, and when you do something else, but have terminal title visible.
* Testsuite: implement use_specs.Tamar Christina2019-02-171-0/+3
|
* Fix and Reapply "Performance tests: recover a baseline from ancestor commits ↵David Eichmann2019-02-161-3/+13
| | | | and CI results."
* testsuite: Report stdout and stderr in JUnit outputBen Gamari2019-02-101-0/+15
| | | | | | | | | | This patch makes the JUnit output more useful as now we also report the stdout/stderr in the message which can be used to quickly identify why a test is failing without downloading the log. This also introduces TestResult, previously we were simply passing around tuples, making things the implementation rather difficult to follow and harder to extend.
* Revert "Performance tests: recover a baseline from ancestor commits and CI ↵Ben Gamari2019-01-311-9/+3
| | | | | | | | | results." Unfortunately this has broken all future commits due to spurious(?) performance changes which I have been unable to work around. This reverts commit cc2261d42f6a954d88e355aaad41f001f65c95da.
* Performance tests: recover a baseline from ancestor commits and CI results.David Eichmann2019-01-301-3/+9
| | | | gitlab-ci: push performance metrics as git notes to the "GHC Performance Notes" repository.
* Revert "Batch merge"Ben Gamari2019-01-301-9/+3
| | | | This reverts commit 76c8fd674435a652c75a96c85abbf26f1f221876.
* Batch mergeBen Gamari2019-01-301-3/+9
|
* testsuite: Ensure that config.{msys,cygwin} are initializedBen Gamari2019-01-231-0/+2
| | | | | | | | | | Reviewers: monoidal Reviewed By: monoidal Subscribers: monoidal, rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5056
* testsuite: Save performance metrics in git notes.David Eichmann2018-11-071-9/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes the following improvement: - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files. - On expected metric changes, the programmer need only indicate the direction of change in the git commit message. - Provides a simple python tool "perf_notes.py" to compare metrics over time. Issues: - Using just the previous commit allows performance to drift with each commit. - Currently we allow drift as we have a preference for minimizing false positives. - Some possible alternatives include: - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric) - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics) - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes. - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes. Previously, *.T files contain statements such as: ``` stats_num_field('peak_megabytes_allocated', (2, 1)) compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)]) ``` This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with: ``` collect_stats('peak_megabytes_allocated', 5) collect_compiler_stats('bytes allocated', 10) ``` So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g. ``` # Metric (In|De)crease <metric(s)> <options>: <tests> Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \ (test_env='linux_x86', way='default'): Test012, Test345 Metric Decrease 'bytes allocated': Test678 Metric Increase: Test711 ``` This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711) phabricator will use the message in the description Reviewers: bgamari, hvr Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #12758 Differential Revision: https://phabricator.haskell.org/D5059
* testsuite: Use bools for booleans, not intsBen Gamari2018-09-051-11/+11
| | | | | | | | | | | | | | Summary: Just as it says on the tin. Test Plan: Validate Reviewers: bgamari, osa1 Reviewed By: osa1 Subscribers: osa1, monoidal, rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D5010
* Simplify testsuite driverKrzysztof Gogolewski2018-08-111-6/+0
| | | | | | | | | | | | | | | | | | | Summary: - remove clean_cmd - framework_failures was undefined - times_file was not used - if_verbose_dump was called only when verbose >= 1; remove the check - simplify normalise_whitespace Test Plan: validate Reviewers: bgamari, thomie Reviewed By: thomie Subscribers: rwbarton, carter Differential Revision: https://phabricator.haskell.org/D5061
* Correctly add unwinding info in manifestSp and makeFixupBlocksBartosz Nitka2018-05-031-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | In `manifestSp` the unwind info was before the relevant instruction, not after. I added some notes to establish semantics. Also removes redundant annotation in stg_catch_frame. For `makeFixupBlocks` it looks like we were off by `wORD_SIZE dflags`. I'm not sure why, but it lines up with `manifestSp`. In fact it lines up so well so that I can consolidate the Sp unwind logic in `maybeAddUnwind`. I detected the problems with `makeFixupBlocks` by running T14779b after patching D4559. Test Plan: added a new test Reviewers: bgamari, scpmw, simonmar, erikd Reviewed By: bgamari Subscribers: thomie, carter GHC Trac Issues: #14999 Differential Revision: https://phabricator.haskell.org/D4606
* testsuite: allow accepting of fine grained results [skip ci]Tamar Christina2018-03-311-4/+6
| | | | | | | | | | | | | | | | | | | | | | Summary: Sometimes we need to be able to mass accept changes that are platform specific and that can't be normalized away using our string formatters. e.g. differences in I/O manager errors or behaviors. This allows one to do so easier than before and less error prone. I have updated the docs and made it clear this should only be used when a normalizer won't work: https://ghc.haskell.org/trac/ghc/wiki/Building/RunningTests/Updating Test Plan: Manually tested while working on new I/O manager Reviewers: bgamari Subscribers: thomie, carter Differential Revision: https://phabricator.haskell.org/D4549
* Windows: fix all failing tests.Tamar Christina2018-01-021-0/+6
| | | | | | | | | | | | | | | | | | | | | | This makes the testsuite pass clean on Windows again. It also fixes the `libstdc++-6.dll` error harbormaster was showing. I'm marking some tests as isolated tests to reduce their flakiness (mostly concurrency tests) when the test system is under heavy load. Updates process submodule. Test Plan: ./validate Reviewers: hvr, bgamari, erikd, simonmar Reviewed By: bgamari Subscribers: rwbarton, thomie, carter Differential Revision: https://phabricator.haskell.org/D4277
* testsuite: Produce JUnit outputBen Gamari2017-07-281-0/+1
| | | | | | | | | | | | Test Plan: Validate, try ingesting into Jenkins. Reviewers: austin Subscribers: rwbarton, thomie GHC Trac Issues: #13716 Differential Revision: https://phabricator.haskell.org/D3796
* testsuite: Classify missing expected perf numbers as merely warningsBen Gamari2017-04-021-0/+1
| | | | | | Previously these were considered to be framework failures, meaning that validate would fail. For better or worse, Windows lacks a good number of metrics and I don't see this changing any time soon. Let's consider these to be non-fatal.
* The Backpack patch.Edward Z. Yang2016-10-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This patch implements Backpack for GHC. It's a big patch but I've tried quite hard to keep things, by-in-large, self-contained. The user facing specification for Backpack can be found at: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst A guide to the implementation can be found at: https://github.com/ezyang/ghc-proposals/blob/backpack-impl/proposals/0000-backpack-impl.rst Has a submodule update for Cabal, as well as a submodule update for filepath to handle more strict checking of cabal-version. Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu> Test Plan: validate Reviewers: simonpj, austin, simonmar, bgamari, goldfire Subscribers: thomie, mpickering Differential Revision: https://phabricator.haskell.org/D1482
* Testsuite: use ignore_stderr/stdout instead of ignore_outputThomas Miedema2016-06-291-1/+5
| | | | | | | | | | | | | | | | | The problem with ignore_output is that it hides errors for WAY=ghci. GHCi always returns with exit code 0 (unless it is broken itself). For example: ghci015 must have been failing with compile errors for years, but we didn't notice because all output was ignored. Therefore, replace all uses of ignore_output with either ignore_stderr or ignore_stdout. In some cases I opted for adding the expected output. Update submodule hpc and stm. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2367
* Testsuite: cleanup printing of summaryThomas Miedema2016-06-281-14/+9
| | | | | | Just use a simple list of tuples, instead of a nested map. -90 lines of code.
* Testsuite: open/close stdin/stdout/stderr explicitlyThomas Miedema2016-06-281-3/+0
| | | | | | | | | | | | | | | | | | This allows run_command's to contain `|`, and `no_stdin` isn't necessary anymore. Unfortunately it doesn't fix T7037 on Windows which I had hoped it would (testsuite driver tries to read a file that it just created itself, but the OS says it doesn't exist). The only drawback of this commit is that the command that the testsuite prints to the terminal (for debugging purposes) doesn't mention the files that stdout and stderr are redirected to anymore. This is probably ok. Update submodule unix. Differential Revision: https://phabricator.haskell.org/D1234
* Testsuite: delete dead code + cleanupThomas Miedema2016-06-201-7/+0
| | | | | | * Set config settings directly in mk/test.mk, instead of indirectly in config/ghc * passing --hpcdir for WAY=hpc is unnecessary
* Testsuite: validate the tests/stage1 directory with the stage1 compilerThomas Miedema2016-06-181-1/+4
| | | | | | | | | | | * See `Note [Why is there no stage1 setup function?]`. * Move T2632 to the tests/stage1 directory (#10382). Reviewed by: ezyang, nomeata, bgamari Differential Revision: https://phabricator.haskell.org/D2341 GHC Trac Issues: #12197
* Testsuite: run tests in <testdir>.run instead of /tmpThomas Miedema2016-06-181-0/+6
| | | | | | | | | | | | | | | | | | | | As discussed in Phab:D1187, this approach makes it a bit easier to inspect the test directory while working on a new test. The only tests that needed changes are the ones that refer to files in ancestor directories. Those files are now copied directly into the test directory. validate still runs the tests in a temporary directory in /tmp, see `Note [Running tests in /tmp]` in testsuite/driver/runtests.py. Update submodule hpc. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2333 GHC Trac Issues: #11980
* Testsuite: delete check_files_writtenThomas Miedema2016-05-241-3/+0
| | | | | | | | | The CHECK_FILES_WRITTEN feature is no longer necessary, since tests don't write to the source directory anymore (#11980). Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2162
* Testsuite: run tests in /tmp after copying required filesThomas Miedema2016-05-171-0/+3
| | | | | | | | | | | | | | | | | | | | | | | Major change to the testsuite driver. For each TEST: * create a directory `<testdir>` inside `/tmp`. * link/copy all source files that the test needs into `<testdir>`. * run the test inside `<testdir>`. * delete `<testdir>` Extra files are (temporarily) tracked in `testsuite/driver/extra_files.py`, but can also be specified using the `extra_files` setup function. Differential Revision: https://phabricator.haskell.org/D1187 Reviewed by: Rufflewind, bgamari Trac: #11980
* Testsuite: make CLEANUP=1 the default (#9758)Thomas Miedema2016-04-301-3/+0
| | | | | | | | | | Also move the `cleanup` setting from `default_testopts` to `config`. The `cleanup` setting is the same for all tests, hence it belongs in `config`. Reviewed by: austin Differential Revision: https://phabricator.haskell.org/D2148
View Lorry Controller Status