summaryrefslogtreecommitdiff
path: root/testsuite/driver/testlib.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Testsuite: delete *.stderr-ghc-7.0 *.stdout-ghc-7.0Thomas Miedema2015-07-141-4/+3
| | | | No point in pretending the testsuite can be run with older versions of GHC.
* Testsuite: rename *.stderr-ghc to *.stderrThomas Miedema2015-07-141-4/+3
| | | | And *.stdout-ghc to *.stdout. We only have output files for ghc now.
* Testsuite: delete remaining only_compiler_types(['ghc']) setupsThomas Miedema2015-07-141-29/+14
| | | | | No point in pretending other compilers can use the GHC testsuite. This makes the *.T files a bit shorter.
* Testsuite: delete unused with_namebaseThomas Miedema2015-07-141-38/+5
| | | | It was introduced in 39c6c735c216d259854ee31b15ec87ea653f2b5d (2007).
* Testsuite: put extra_run_opts last on command lineThomas Miedema2015-06-301-3/+7
| | | | | Some tests use the format: extra_run_opts('+RTS foo') (without closing -RTS). Make it clear in testlib.py that this should work.
* Testsuite: fix framework failureThomas Miedema2015-06-151-2/+2
| | | | I forgot to rename this in 5ddd90415f307cac72d75d86da58e552b168ee30.
* Testsuite: add function compile_timeout_multiplier (#10345)Thomas Miedema2015-06-131-7/+15
| | | | | | | | | | | And rename timeout_multiplier to run_timeout_multiplier. timeout_multiplier was added in commit a00389794b839971c7d52ead9e8570bfaa25ac55. The name suggested that it would affect any test, but it actually only affected tests that had a run component, and only that run component (as needed by test T367). Differential Revision: https://phabricator.haskell.org/D982
* Testsuite: diff non-whitespace normalised output (#10152)Thomas Miedema2015-06-131-29/+36
| | | | | | | | | | | | | | | | | | | | | On a test failure, we show a diff between the expected and the actual output. The method of how we do this has changed a couple of times: * In 2007: 9951189ccf90b709436fa55ee49eeef031f79f4e "On failure, diff the normalised test outputs" * In 2011: 3019b1e409c129ef7af63e6a7408fb36ec44444b "When the output files differ, present the diffs between the *actual* output, not the normalised output. The latter may have newlines removed, making the diff unreadable." * In 2015 (now): do something in between. - Do apply the normalisers again, to make the diff smaller (only showing the actual problem). - But don't apply normalise_whitespace, as it indeed makes the diff unreadable. Differential Revision: https://phabricator.haskell.org/D984
* Testsuite: fix the little known CHECK_FILES_WRITTEN=1Thomas Miedema2015-06-121-3/+17
| | | | | | | | | | | | | The testsuite driver has a little known feature to check which files each test writes to, whether there are tests that write to same file, and whether the tests leave any files behind when CLEANUP=1. It uses strace under the hood. This commit fixes some bitrot, and filters out some more strace lines that we're not interested in (and are shown as framework failures otherwise). Differential Revision: https://phabricator.haskell.org/D979
* Testsuite: change some expect_fail tests to expect_brokenThomas Miedema2015-06-111-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | Change the following tests from expect_fail to expect_broken: and list the ticket number: * driver/sigof03m/sigof03 (#9252) * driver/static001 (#8127) * partial-sigs/should_compile/EqualityConstraint (#9478) * partial-sigs/should_compile/ExtraNumAMROn (#9478) * partial-sigs/should_compile/PatBind2 (#9478) * partial-sigs/should_fail/TidyClash2 (#9478) * simplCore/should_compile/T8832 (#8832) The following tests are still marked as expect_fail, but it is not clearly documented why so: * gadt/lazypatok * indexed-types/should_fail/SkolemOccursLoop All other expect_fail tests are only expected to fail on either a certain platform/os or for a certain way only. Differential Revision: https://phabricator.haskell.org/D966
* Testsuite Windows: fix T8172 (#8172)Thomas Miedema2015-06-111-0/+4
| | | | | Use the new function `normalise_drive_letter` to change D:\ to C:\ before comparing outputs.
* Revert "The test runner now also works under the msys-native Python."Thomas Miedema2015-06-091-16/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To make the test runner work under msys-native Python... Commit 5258566ee5c89aa757b0cf1433169346319c018f broke the msys testsuite driver (#10441). It changed the quoting of `config.compiler` from single quotes to double quote, which turns out to not be compatible with what the function `passThroughCmd` expected. We could fix `passThroughCmd` to handle the case where `config.compiler` is double quoted, and scatter some notes around to make sure the quoting done in various places of the testsuite driver stay compatible. Instead, this commit reverts 101c62e26286353dd3fac1ef54323529b64c9902, which introdced the function `passThroughCmd` in the first place (#9626). ezyang reports that doing this revert fixes the testsuite driver for him using the the following version of msys2: msys2-keyring r8.3864337-1 msys2-runtime 2.1.0.16351.cd3184b-1 msys2-runtime-devel 2.1.0.16351.cd3184b-1 msys2-w32api-headers 5.0.0.4456.c8b6742-1 msys2-w32api-runtime 5.0.0.4455.32db221-1 Ideally we'd know what minimum version of msys2 we require, but for now this fix is better than nothing. Only gintas ever reported the original problem, and he actually mentioned shortly afterwards: "This may have been fixed by a recent release of msys2, but I am not sure." Differential Revision: https://phabricator.haskell.org/D952
* Testsuite: only show output diff when test is expected to passThomas Miedema2015-06-061-11/+17
| | | | | | | | | | | | | | | | | | | | | | Don't let the output of tests that either have missing libraries or are expected to be broken obscure real failures. This makes it easier to analyse the testlogs. The only consequence is that when a test fails because a certain library isn't installed, you have to check the all.T file in which the test is defined to actually find out _which_ library that is. Before it would print something like Compile failed (status 256) errors were: stm052.hs:10:8: error: Could not find module ‘System.Random’ Use -v to see a list of the files searched for. And now it doesn't. I think this is an acceptable tradeoff. Differential Revision: https://phabricator.haskell.org/D945
* Testsuite: add/fix cleanup for certain testsThomas Miedema2015-06-041-0/+6
| | | | | | | | | | | | | | | | | | | * extra_clean argument should be a list Add an assert to prevent regressions. * properly clean package conf direcories They are directories now, which was causing problems. * properly clean write_interface_* tests We were getting these errors: [Errno 21] Is a directory: './driver/write_interface_oneshot' [Errno 39] Directory not empty: './driver/write_interface_oneshot' [Errno 21] Is a directory: './driver/write_interface_make' [Errno 39] Directory not empty: './driver/write_interface_make' * outputdir() is better than -outputdir, as it knows how to (pre)clean itself.
* Make validate more quietThomas Miedema2015-06-041-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * By default use V=0, and call the testsuite with VERBOSE=2, which we did before only with validate --quiet. This disables printing the test commands it runs. * When --quiet is used, call the testsuite with VERBOSE=1. This disables printing the '====> Scanning' lines, and doesn't print which test is being run. So it only prints something when a test accidentally prints to stdout or when it fails. Don't set this option on Travis, as Travis will cancel a build if it doesn't see any output for more than 10 minutes. * When --quiet is used, set the new test option NO_PRINT_SUMMARY, which skips printing the test summary. Only the list of unexpected failures is printed, if there are any. Note that the full summary can still be found in testsuite_summary.txt * When --quiet is used, don't pass the `-v` flag to `ghc-pkg check` * When --quiet is used, don't print the Oops! header. It shoud be clear from the list of failing tests that something is wrong. This is all done to get the most out of 30 lines of logfile. These changes can be disabled later by simply not passing the --quiet flag to validate. Differential Revision: https://phabricator.haskell.org/D942
* Testdriver: don't use os.popen in config/ghcPhil Ruffwind2015-05-281-42/+10
| | | | | | | | | | | | | | | Rewrite config/ghc to use getStdout (which use subprocess.Popen) instead of os.popen, which is deprecated; this also avoids the use of shell Also: * Move getStdout to driver/testutil.py so both config/ghc and driver/runtests.py can use it * Remove support for Python below 2.4, which doesn't have subprocess Reviewed By: thomie Differential Revision: https://phabricator.haskell.org/D908
* testsuite: handle missing stats files gracefully (#10305)Thomas Miedema2015-05-231-1/+8
| | | | | | | | | | | | | The following tests would result in framework failures when using a ghc build with HADDOCK_DOCS=NO in mk/build.mk or mk/validate.mk: * haddock.Cabal * haddock.base * haddock.compiler Test Plan: run make in tests/perf/haddock Differential Revision: https://phabricator.haskell.org/D899
* Add "error:" prefix to error-messagesKonstantine Rybnikov2015-04-141-0/+12
| | | | | | | | | | | Add "error:" prefix to error-messages, also lowercase "Warning:" message to match GCC behavior closer. Reviewed By: thomie, austin Differential Revision: https://phabricator.haskell.org/D811 GHC Trac Issues: #10021
* Don't `make accept` output of `expect_broken_for` testsThomas Miedema2015-04-031-16/+17
| | | | | | | | | This is a followup to d4cf7051bc17182238b17ba1dc42e190fa5c6f0d, which did the same for `expect_broken` tests. Reviewed By: austin Differential Revision: https://phabricator.haskell.org/D786
* Don't `make accept` output of expect_broken testsThomas Miedema2015-04-011-1/+4
| | | | | | | | | | | | | | | | | | | | | | When running `make accept` on a directory in the testsuite, don't accept the output of tests that are marked as expect_broken. This makes it easier to review `git diff` after running `make accept`. When you change an error message in the compiler that shows up in the output of many tests, you can run `make accept` in the testsuite directory, and all expected test output will be updated. But since your change didn't magically fix all the other bugs in the compiler for which we have an expect_broken test, the output for those tests should probably not be updated. Before, the effect of running `make accept` could be that some tests would end up in the 'unexpected passes' group. [skip ci] Reviewed By: austin Differential Revision: https://phabricator.haskell.org/D781
* Testsuite: redirect stderr to /dev/null when running GS on bad.psThomas Miedema2015-03-311-1/+2
| | | | | | | | This is a followup to a3d0a7a0ba3a1ee458a9883011247561dfe22f4a. Reviewed by: Rufflewind Differential Revision: https://phabricator.haskell.org/D780
* Testsuite: suppress errors when running GS on bad.psPhil Ruffwind2015-03-311-1/+1
| | | | | | | | | Suppress the errors that appear in standard output when running gs on bad.ps since it's expected to fail anyway. Reviewed By: thomie, austin Differential Revision: https://phabricator.haskell.org/D773
* Make testsuite driver Python 2.6 compatible againThomas Miedema2015-03-231-8/+8
| | | | | | | | | | | | | Another bug in the #10164 series. Only Python 2.7 and up allow you to omit the positional argument specifiers in format strings. Test Plan: this fixes the Solaris builders Reviewed By: kgardas Differential Revision: https://phabricator.haskell.org/D750 GHC Trac Issues: #10164
* Fix Windows testsuite driverThomas Miedema2015-03-171-4/+4
| | | | This got broken in commit 5258566.
* Fix testsuite driver for a profiling compilerThomas Miedema2015-03-161-0/+3
| | | | | | | | | This should have been part of commit 5258566ee5c8, to allow expansion of '{hp2ps}' in a command string to `config.hp2ps`. Reviewed by: austin Differential Revision: https://phabricator.haskell.org/D734
* Move the function strip_quotes to testutil.pyThomas Miedema2015-03-131-4/+0
| | | | | | | | | | | | | | | | If one runs the testsuite with a profiling compiler, during the import of `testlib.py`, `testlib.py` sets the global variable `gs_working`. To do so, it executes a few statements which require the function `strip_quotes` to be in scope. But that function only gets defined at the very end of testlib.py. This patch moves the definition of `strip_quotes` to testutil.py, which is imported at the very top of testlib.py. This unbreaks the nightly builders. Reviewed By: austin Differential Revision: https://phabricator.haskell.org/D728
* testsuite: use same flags for ghci way and scriptsThomas Miedema2015-03-111-1/+1
| | | | | | | | | The ghci script tests were using different RTS flags from the normal ghci tests. This commit makes them use the same flags. Reviewers: austin Differential Revision: https://phabricator.haskell.org/D724
* Cleanup test framework string formattingThomas Miedema2015-03-111-48/+44
| | | | | | | | | | | | * Use format strings instead of string concatenation. * Wrap `config.compiler`, `config.hpc` etc. in quotes in `mk/test.mk`, so we don't have to in .T scripts and driver/testlib.py. Update hpc submodule (test cleanup) Reviewers: austin Differential Revision: https://phabricator.haskell.org/D718
* Refactor testsuite with normalise_version()Edward Z. Yang2015-03-101-4/+16
| | | | | | | | | | | | | | | | | | | | Summary: This function generalizes the normaliseBytestringPackage and other similar one-off functions into normalise_version() with takes a package name to normalize against. This JUST manages package versions; we also could use a normalize for keys. In the process, I modified all the normalization functions to be accumulative; I don't think this makes a difference for current test cases but I think it makes things nicer. Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu> Reviewers: austin Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D725
* testsuite: format commands using config dictThomas Miedema2015-03-061-7/+17
| | | | | | | | | | | | | | | | | | | | | | Summary: Allow `cmd_wrapper` to return a format string that can refer to config values. Very useful! This allows for many tests to be defined in pure Python, instead of in an additional script or Makefile. Example: def Thpc(cmd): return(cmd + ' && {hpc} report Thpc.tix') test('Thpc', [cmd_wrapper(Thpc), only_ways['hpc']), compile_and_run, ['']) The `{hpc}` is replaced by the value of `config.hpc`. The result is that the module `Thpc` first gets compiled, then the binary `Thpc` is run, and then the `hpc report` command is run. The output of all of this is redirected (and later appended) to Thpc.run.stdout/stderr as normally. Reviewed By: austin Differential Revision: https://phabricator.haskell.org/D706
* Always ignore user-package-db when running testsThomas Miedema2015-02-231-2/+2
| | | | | | | | | | | | | | | | | | | | Summary: The user package database was already ignored for systems that `have_subprocess`. To [wiki:Debugging/InstallingPackagesInplace install] a package inplace: `cabal install --with-compiler=<inplace-ghc> --with-package-db=<inplace-package-db>` <package> Reviewers: austin Reviewed By: austin Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D668
* The test runner now also works under the msys-native Python.Gintautas Miliauskas2014-11-191-0/+16
| | | | | | | Msys binaries apply heuristics to escape paths in arguments intended for non-msys binaries, which breaks timeout invocations, see #9626. Signed-off-by: Austin Seipp <austin@well-typed.com>
* Split off stat (benchmark) test failures into a separate section in the test ↵Gintautas Miliauskas2014-10-311-8/+20
| | | | | | | | | | | | | | | | | runner summary. Stat tests are generally less reliable than other types of tests, so it's nice to have them in a separate section rather than interspersed with potential... Summary: ...correctness issues. Reviewers: austin Reviewed By: austin Subscribers: thomie, carter, simonmar Differential Revision: https://phabricator.haskell.org/D406
* Fix test driver python3 compatibility issuesMateusz Lenik2014-10-301-5/+5
| | | | | | | | | | | | | | | | | Summary: Fixes python3 compatibility issues by replacing filter with a list comperhension and a potential issue with python2 when override_flags would be an empty list. Reviewers: austin, thomie Reviewed By: austin, thomie Subscribers: thomie, carter, simonmar, mlen Differential Revision: https://phabricator.haskell.org/D399 GHC Trac Issues: #9230
* testlib: Get rid of two_normalisersJoachim Breitner2014-10-281-24/+31
| | | | | And make normalise_fun polyvariadic. After all, this is untyped code, so lets make use of it :-)
* Add n-ary version of `two_normalisers` to testsuite libHerbert Valerio Riedel2014-10-281-2/+25
| | | | This is more readable than nesting `two_normalisers()`-invocations
* Enabled warn on tabs by default (fixes #9230)Mateusz Lenik2014-10-211-7/+55
| | | | | | | | | | | | | | | | | | | | | | Summary: This revision enables -fwarn-tabs by default and add a suppression flag, so that GHC compilation won't fail when some files contain tab characters. Test Plan: Additional test case, T9230, was added to cover that change. Reviewers: austin Reviewed By: austin Subscribers: simonmar, ezyang, carter, thomie, mlen Differential Revision: https://phabricator.haskell.org/D255 GHC Trac Issues: #9230 Conflicts: testsuite/driver/testlib.py
* Python 3 support, second attempt (Trac #9184)Krzysztof Gogolewski2014-10-191-85/+92
| | | | | | | | | | | | | | | | | | | | | | | | Summary: This is a fixup of https://phabricator.haskell.org/D233 The only difference is in findTFiles (first commit), which previously broke Windows runner; now I translated literally instead attempting to improve it, and checked it works. Test Plan: I used validate under 2,3 on Linux and under 2 on msys2. On Windows I've seen a large number of failures, but they don't seem to be connected with the patch. Reviewers: hvr, simonmar, thomie, austin Reviewed By: austin Subscribers: thomie, carter, ezyang, simonmar Differential Revision: https://phabricator.haskell.org/D310 GHC Trac Issues: #9184
* Revert "Basic Python 3 support for testsuite driver (Trac #9184)"Krzysztof Gogolewski2014-10-031-92/+93
| | | | | | This reverts commit 084d241b316bfa12e41fc34cae993ca276bf0730. This is a possible culprit of Windows breakage reported at ghc-devs.
* Basic Python 3 support for testsuite driver (Trac #9184)Krzysztof Gogolewski2014-10-011-93/+92
| | | | | | | | | | | | | | | | | | | | | | | | Summary: Most of the changes is adaptation of old Python 2 only code. My priority was not breaking Python 2, and so I avoided bigger changes to the driver. In particular, under Python 3 the output is a str and buffering cannot be disabled. To test, define PYTHON=python3 in testsuite/mk/boilerplate.mk. Thanks to aspidites <emarshall85@gmail.com> who provided the initial patch. Test Plan: validate under 2 and 3 Reviewers: hvr, simonmar, thomie, austin Reviewed By: thomie, austin Subscribers: aspidites, thomie, simonmar, ezyang, carter Differential Revision: https://phabricator.haskell.org/D233 GHC Trac Issues: #9184
* testsuite: normalise integer library name for T8958Sergei Trofimovich2014-08-301-4/+2
| | | | Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
* testsuite: add signal_exit_code function to the driverKarel Gardas2014-08-071-0/+11
| | | | | | | | | | | | | | | | | | | Summary: New function signal_exit_code hides differences between target platforms handling of fatal error signals and the applications' exit codes. E.g. on Linux the application exit code which receives fatal error signal is encoded as 128 + signal value. On the other hand on Solaris the application exit code is signal value alone. Test Plan: validated on Linux and tested on Solaris Reviewers: austin, simonmar Reviewed By: simonmar Subscribers: phaskell, simonmar, relrod, ezyang, carter Differential Revision: https://phabricator.haskell.org/D122
* Correctly round when calculating the deviationJoachim Breitner2014-07-171-1/+1
|
* Include test case name in performance resultJoachim Breitner2014-07-171-9/+11
| | | | | | | With THREADS=n, for n > 1, it becomes impossible to match the performance numbers to the test case name. Hence include it in the output. This also makes grepping through a bunch of logs for a specific test case much easier, and outweighs the extra verbosity.
* Give performance benchmark deviation also in percentsJoachim Breitner2014-07-141-2/+6
| | | | | this makes it easier to spot a “just over the mark” change (e.g. +5.1%), compared to a more radical jump (e.g. +15%).
* New testsuite verbosity level 4Joachim Breitner2014-07-141-1/+1
| | | | | which makes it print performance numbers even when the test succeeds (good for historic analysis)
* Remove external coreAustin Seipp2014-05-031-95/+0
| | | | Signed-off-by: Austin Seipp <austin@well-typed.com>
* Fix #5435, adding new test config check_stdout.Edward Z. Yang2014-04-091-1/+15
| | | | | | | | | | | check_stdout(f) allows you to override the test framework's diff based output checking with another mechanism. f is a function which takes two arguments: the first is the filename containing the observed stdout, the second is the normaliser that would have been applied (in case you want to read, normalise, and then do something.) Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Fix some Python brainos in testlib (except e is not valid form).Edward Z. Yang2014-02-061-2/+2
| | | | Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Explain skipping compiler-perf tests when debugging is onJoachim Breitner2013-11-281-0/+2
|
View Lorry Controller Status