summaryrefslogtreecommitdiff
path: root/testsuite/driver/testlib.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Testsuite: simplify extra_file handlingThomas Miedema2016-06-281-55/+13
| | | | | | | | | Before, `extra_files(['.hpc/Main.mix'])` meant copy `Main.mix` to `<testdir>/.hpc/Main.mix`. This feature wasn't really necessary, so now it just means copy `Main.mix` to `<testdir>/Main.mix`. This simplifies the implementation. Some small other cleanups as well. -40 lines of code.
* Testsuite: remove one level of indentation [skip ci]Thomas Miedema2016-06-271-133/+129
| | | | Refactoring only. Move try/except out of do_test.
* Testsuite: report duplicate testnames when `make TEST=<name>`Thomas Miedema2016-06-271-9/+9
|
* Testsuite: never pick up .T files in .run directoriesThomas Miedema2016-06-271-13/+8
| | | | | And use os.walk instead of calling os.listdir many times. The testsuite driver should be able to handle backward slashes on Windows now.
* Testsuite: do not copy .hi/.o files to testdir (#12112)Thomas Miedema2016-06-241-3/+6
|
* Testsuite: assume timeout_prog always existsThomas Miedema2016-06-201-86/+28
| | | | | | | | | | | Merge the following functions into one: * rawSystem * rawSystemWithTimeout * runCmd * runCmdFor * runCmdExitCode I don't know why this wasn't done before.
* Testsuite: delete dead code + cleanupThomas Miedema2016-06-201-22/+7
| | | | | | * Set config settings directly in mk/test.mk, instead of indirectly in config/ghc * passing --hpcdir for WAY=hpc is unnecessary
* Testsuite: remove `-Wno-warn-tabs` from default flagsThomas Miedema2016-06-201-40/+9
| | | | This allows the removal of the override_flags stuff in testlib.py.
* Testsuite: remove `-fforce-recomp` from default flags (#11980)Thomas Miedema2016-06-201-24/+10
| | | | | | | | | | | | | | There is no need for this flag anymore, since each test runs in a newly created directory. Removing it cleans up testlib.py a bit. There is a small risk that this renders some tests useless. It's hard to know. Those tests should have specified -fforce-recomp` explicitly anyway, so I'm not going to worry about it. I've fixed the ones that failed without -fforce-recomp. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2346
* Testsuite: fix WAY=ghci when LOCAL=0Thomas Miedema2016-06-201-4/+4
|
* Testsuite: recover from utf8 decoding errorsThomas Miedema2016-06-201-2/+8
|
* Testsuite: validate the tests/stage1 directory with the stage1 compilerThomas Miedema2016-06-181-1/+31
| | | | | | | | | | | * See `Note [Why is there no stage1 setup function?]`. * Move T2632 to the tests/stage1 directory (#10382). Reviewed by: ezyang, nomeata, bgamari Differential Revision: https://phabricator.haskell.org/D2341 GHC Trac Issues: #12197
* Testsuite: write "\n" instead of "\r\n" when using mingw PythonThomas Miedema2016-06-181-11/+32
| | | | | | | | | | | | | | | Mingw style Python uses '\r\n' by default for newlines. This is annoying, because it means that when a GHC developer on Windows uses mingw Python to `make accept` a test, every single line of the .stderr file is touched. This makes it difficult to spot the real changes, and it leads to unnecessary git history bloat. Prevent this from happening by using io.open instead of open. See `Note [Universal newlines]` Reviewed by: Phyx Differential Revision: https://phabricator.haskell.org/D2342
* Testsuite: run tests in <testdir>.run instead of /tmpThomas Miedema2016-06-181-29/+19
| | | | | | | | | | | | | | | | | | | | As discussed in Phab:D1187, this approach makes it a bit easier to inspect the test directory while working on a new test. The only tests that needed changes are the ones that refer to files in ancestor directories. Those files are now copied directly into the test directory. validate still runs the tests in a temporary directory in /tmp, see `Note [Running tests in /tmp]` in testsuite/driver/runtests.py. Update submodule hpc. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2333 GHC Trac Issues: #11980
* Testsuite: delete dead code [skip ci]Thomas Miedema2016-06-091-19/+0
|
* Show sources of cost centers in .profÖmer Sinan Ağacan2016-06-081-15/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the problem with duplicate cost-centre names that was reported a couple of times before. When a module implements a typeclass multiple times for different types, methods of different implementations get same cost-centre names and are reported like this: COST CENTRE MODULE %time %alloc CAF GHC.IO.Handle.FD 0.0 32.8 CAF GHC.Read 0.0 1.0 CAF GHC.IO.Encoding 0.0 1.8 showsPrec Main 0.0 1.2 readPrec Main 0.0 19.4 readPrec Main 0.0 20.5 main Main 0.0 20.2 individual inherited COST CENTRE MODULE no. entries %time %alloc %time %alloc MAIN MAIN 53 0 0.0 0.2 0.0 100.0 CAF Main 105 0 0.0 0.3 0.0 62.5 readPrec Main 109 1 0.0 0.6 0.0 0.6 readPrec Main 107 1 0.0 0.6 0.0 0.6 main Main 106 1 0.0 20.2 0.0 61.0 == Main 114 1 0.0 0.0 0.0 0.0 == Main 113 1 0.0 0.0 0.0 0.0 showsPrec Main 112 2 0.0 1.2 0.0 1.2 showsPrec Main 111 2 0.0 0.9 0.0 0.9 readPrec Main 110 0 0.0 18.8 0.0 18.8 readPrec Main 108 0 0.0 19.9 0.0 19.9 It's not possible to tell from the report which `==` took how long. This patch adds one more column at the cost of making outputs wider. The report now looks like this: COST CENTRE MODULE SRC %time %alloc CAF GHC.IO.Handle.FD <entire-module> 0.0 32.9 CAF GHC.IO.Encoding <entire-module> 0.0 1.8 CAF GHC.Read <entire-module> 0.0 1.0 showsPrec Main Main_1.hs:7:19-22 0.0 1.2 readPrec Main Main_1.hs:7:13-16 0.0 19.5 readPrec Main Main_1.hs:4:13-16 0.0 20.5 main Main Main_1.hs:(10,1)-(20,20) 0.0 20.2 individual inherited COST CENTRE MODULE SRC no. entries %time %alloc %time %alloc MAIN MAIN <built-in> 53 0 0.0 0.2 0.0 100.0 CAF Main <entire-module> 105 0 0.0 0.3 0.0 62.5 readPrec Main Main_1.hs:7:13-16 109 1 0.0 0.6 0.0 0.6 readPrec Main Main_1.hs:4:13-16 107 1 0.0 0.6 0.0 0.6 main Main Main_1.hs:(10,1)-(20,20) 106 1 0.0 20.2 0.0 61.0 == Main Main_1.hs:7:25-26 114 1 0.0 0.0 0.0 0.0 == Main Main_1.hs:4:25-26 113 1 0.0 0.0 0.0 0.0 showsPrec Main Main_1.hs:7:19-22 112 2 0.0 1.2 0.0 1.2 showsPrec Main Main_1.hs:4:19-22 111 2 0.0 0.9 0.0 0.9 readPrec Main Main_1.hs:7:13-16 110 0 0.0 18.8 0.0 18.8 readPrec Main Main_1.hs:4:13-16 108 0 0.0 19.9 0.0 19.9 CAF Text.Read.Lex <entire-module> 102 0 0.0 0.5 0.0 0.5 To fix failing test cases because of different orderings of cost centres (e.g. optimized and non-optimized build printing in different order), with this patch we also start sorting cost centres before printing. The order depends on 1) entries (more entered cost centres come first) 2) names (using strcmp() on cost centre names). Reviewers: simonmar, austin, erikd, bgamari Reviewed By: simonmar, bgamari Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2282 GHC Trac Issues: #11543, #8473, #7105
* Testsuite driver: always quote opts.testdirThomas Miedema2016-06-071-18/+25
| | | | | | | This makes sure the testsuite keeps working when testdir contains backward slashes. Differential Revision: https://phabricator.haskell.org/D2314
* Testsuite: also normalise platform-dependent .stdout/stderrThomas Miedema2016-05-251-26/+12
| | | | | | | | | | | | | | | | | This effectively reverses commit 429f0099ab9adfadc779ca76f3aae1c9c160fb8c (2006). I don't see why platform-dependent .stdout/stderr files should //not// get normalised. It fixes T11223_link_order_a_b_2_fail on Windows, by normalising `ghc-stage2.exe` to `ghc` when comparing stderr with .stderr-mingw32. Reviewed by: Phyx Differential Revision: https://phabricator.haskell.org/D2267 GHC Trac Issues: #12118
* Testsuite: delete check_files_writtenThomas Miedema2016-05-241-179/+1
| | | | | | | | | The CHECK_FILES_WRITTEN feature is no longer necessary, since tests don't write to the source directory anymore (#11980). Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2162
* Testsuite: delete old cleanup code (#11980)Thomas Miedema2016-05-171-146/+13
|
* Testsuite: run tests in /tmp after copying required filesThomas Miedema2016-05-171-14/+137
| | | | | | | | | | | | | | | | | | | | | | | Major change to the testsuite driver. For each TEST: * create a directory `<testdir>` inside `/tmp`. * link/copy all source files that the test needs into `<testdir>`. * run the test inside `<testdir>`. * delete `<testdir>` Extra files are (temporarily) tracked in `testsuite/driver/extra_files.py`, but can also be specified using the `extra_files` setup function. Differential Revision: https://phabricator.haskell.org/D1187 Reviewed by: Rufflewind, bgamari Trac: #11980
* Testsuite: make CLEANUP=1 the default (#9758)Thomas Miedema2016-04-301-1/+1
| | | | | | | | | | Also move the `cleanup` setting from `default_testopts` to `config`. The `cleanup` setting is the same for all tests, hence it belongs in `config`. Reviewed by: austin Differential Revision: https://phabricator.haskell.org/D2148
* Testsuite: delete -fesc testsThomas Miedema2016-04-281-3/+0
| | | | | | | | | | | The -fesc flag does not exist, and has never existed. Also delete now unused config.compiler_tags, and 'Project version' never contains a '-'. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2138
* testsuite: Identify framework failures in testsuite summaryBen Gamari2016-03-261-0/+14
| | | | | | | | | | | | | | | | Currently the testsuite driver tells you how many tests failed due to a framework failure but you need to manually grep through the testsuite output to identify which ones. Test Plan: Validate with, e.g., a timing out testcase Reviewers: austin, thomie Reviewed By: austin, thomie Differential Revision: https://phabricator.haskell.org/D2026 GHC Trac Issues: #11165
* Testsuite: check actual_prof_file only when neededThomas Miedema2016-02-291-10/+11
| | | | | | | | | Might be a little faster. Avoids testing for #6113 (.prof file not written when process is killed with any signal but SIGINT) for tests that don't have a .prof.sample file (which is almost all of them) when running the profiling ways. Tests that were failing because of #6113: T8089, overflow1, overflow2 and overflow3.
* Testsuite: do not write empty files on 'make accept'Thomas Miedema2016-02-251-2/+10
| | | | Also prevent showing '\ No newline at end of file' in diff output.
* Filter out -prof callstacks from test output (#11521)Thomas Miedema2016-02-231-3/+16
|
* Testsuite: delete compiler_lt/le/gt/ge setup functionsThomas Miedema2016-02-171-20/+0
| | | | | | | | Since we're not consisently keeping track of which tests should pass with which compiler versions, there is no point in keeping these functions. Update submodules containers, hpc and stm.
* Testsuite: delete only_compiler_types, assume ghcThomas Miedema2016-02-161-6/+0
| | | | | | Update submodules stm, hpc and unix. Differential Revision: https://phabricator.haskell.org/D1921
* Hide the CallStack implicit parameterEric Seidel2016-02-011-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The implicit parameter isn't actually very relevant to the CallStack machinery, so we hide the implementation details behind a constraint alias ``` type HasCallStack = (?callStack :: CallStack) ``` This has a few benefits: 1. No need to enable `ImplicitParams` in user code. 2. No need to remember the `?callStack` naming convention. 3. Gives us the option to change the implementation details in the future with less user-land breakage. The revised `CallStack` API is exported from `GHC.Stack` and makes no mention of the implicit parameter. Test Plan: ./validate Reviewers: simonpj, austin, hvr, bgamari Reviewed By: simonpj, bgamari Subscribers: thomie Projects: #ghc Differential Revision: https://phabricator.haskell.org/D1818
* Testsuite: fixup req_profiling tests (#11496)Thomas Miedema2016-01-271-1/+1
| | | | | | * T2552 (#10037) is failng for all threaded opt_ways, not for WAY=prof. * TH_spliceE5_prof (#11495) is failing when ghc_dynamic * Rename ghci_dynamic to ghc_dynamic. It's the same thing.
* testsuite: normalise away `ld`-warning on AIXHerbert Valerio Riedel2015-12-301-0/+4
| | | | | | | On AIX, `ld` doesn't support `-x` and ignores it. However, a warning is emitted to stderr which ends up triggering false positives in some of GHC's testsuite tests. So we simply filter out that noise as part of normalising stderr.
* testlib: Make TyCon normalization Python 2.6-compatibleBen Gamari2015-12-291-4/+3
| | | | | | | | | | | D1629 introduced this normalization which was not Python 2.6 compatible due to the use of the `flags` argument of `re.sub`. Fix this. Test Plan: Validate Reviewers: austin, thomie Differential Revision: https://phabricator.haskell.org/D1718
* Don't drop last char of file if -osuf contains dotThomas Miedema2015-12-241-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given: * `file = "foo.a.b"` * `osuf = ".a.b"` -- Note the initial dot. * `new_osuf = "c"` Before (bad, the last character of the filename is dropped): `dropTail (length osuf + 1) file <.> new_osuf == "fo.c"` After (good): `stripExtension osuf file <.> new_osuf` == "foo.c" This regression was introduced in commit c489af73 (#5554). That commit fixed a similar but different bug, and care has been taken to not reintroduce it (using the the newly introduced `System.Filepath.stripExtension`). Given: * `file = "foo.a.b"` * `osuf = "a.b"` * `new_osuf = "c"` Before c489af73 (bad, the full suffix should get replaced): `replaceExtension file new_osuf == "foo.a.c"` After c489af73 (good): `dropTail (length osuf + 1) file <.> new_osuf == "foo.c"` After this commit (still good): `stripExtension osuf file <.> new_osuf == "foo.c"` Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D1692 GHC Trac Issues: #9760
* Fix normalisation of TyCon representationsErik de Castro Lopo2015-12-241-1/+1
| | | | | | | | | | | | Test Plan: run tests on powerpc and x86_64 Reviewers: hvr, austin, thomie, bgamari Reviewed By: thomie, bgamari Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1694
* Make testsuite work again with Py3Herbert Valerio Riedel2015-12-231-3/+8
| | | | | Python 3 support seems to have mildly bitrotten since #9184 was closed. Luckily, only some minor tweaks seem necessary.
* TcTypeable: Don't use bogus fingerprints when suppress-uniques is enabledBen Gamari2015-12-171-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | Previously the Typeable implementation would intentionally create TyCon representations with bogus fingerprints to avoid fingerprints (which may change often) from leaking into test output. As pointed out by Richard in #10376 this is very bad as simply enabling a debug flag, `-dsuppress-uniques`, completely breaks the soundness of `Typeable`! This patch removes this behavior and replaces it with logic in the testsuite driver to filter out spurious changes due to Typeable representations. Test Plan: Validate Reviewers: austin, simonpj, goldfire Reviewed By: goldfire Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1629 GHC Trac Issues: #10376
* Remote GHCi, -fexternal-interpreterSimon Marlow2015-12-171-18/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: (Apologies for the size of this patch, I couldn't make a smaller one that was validate-clean and also made sense independently) (Some of this code is derived from GHCJS.) This commit adds support for running interpreted code (for GHCi and TemplateHaskell) in a separate process. The functionality is experimental, so for now it is off by default and enabled by the flag -fexternal-interpreter. Reaosns we want this: * compiling Template Haskell code with -prof does not require building the code without -prof first * when GHC itself is profiled, it can interpret unprofiled code, and the same applies to dynamic linking. We would no longer need to force -dynamic-too with TemplateHaskell, and we can load ordinary objects into a dynamically-linked GHCi (and vice versa). * An unprofiled GHCi can load and run profiled code, which means it can use the stack-trace functionality provided by profiling without taking the performance hit on the compiler that profiling would entail. Amongst other things; see https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details. Notes on the implementation are in Note [Remote GHCi] in the new module compiler/ghci/GHCi.hs. It probably needs more documenting, feel free to suggest things I could elaborate on. Things that are not currently implemented for -fexternal-interpreter: * The GHCi debugger * :set prog, :set args in GHCi * `recover` in Template Haskell * Redirecting stdin/stdout for the external process These are all doable, I just wanted to get to a working validate-clean patch first. I also haven't done any benchmarking yet. I expect there to be slight hit to link times for byte code and some penalty due to having to serialize/deserialize TH syntax, but I don't expect it to be a serious problem. There's also lots of low-hanging fruit in the byte code generator/linker that we could exploit to speed things up. Test Plan: * validate * I've run parts of the test suite with EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th. There are a few failures due to the things not currently implemented (see above). Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1562
* Rearrange error msgs and add section markers (Trac #11014).Evan Laforge2015-11-241-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This puts the "Relevant bindings" section at the end. It uses a TcErrors.Report Monoid to divide messages by importance and then mappends them together. This is not the most efficient way since there are various intermediate Reports and list appends, but it probably doesn't matter since error messages shouldn't get that large, and are usually prepended. In practice, everything is `important` except `relevantBindings`, which is `supplementary`. ErrMsg's errMsgShortDoc and errMsgExtraInfo were extracted into ErrDoc, which has important, context, and suppelementary fields. Each of those three sections is marked with a bullet character, '•' on unicode terminals and '*' on ascii terminals. Since this breaks tons of tests, I also modified testlib.normalise_errmsg to strip out '•'s. --- Additional notes: To avoid prepending * to an empty doc, I needed to filter empty docs. This seemed less error-prone than trying to modify everyone who produces SDoc to instead produce Maybe SDoc. So I added `Outputable.isEmpty`. Unfortunately it needs a DynFlags, which is kind of bogus, but otherwise I think I'd need another Empty case for SDoc, and then it couldn't be a newtype any more. ErrMsg's errMsgShortString is only used by the Show instance, which is in turn only used by Show HscTypes.SourceError, which is in turn only needed for the Exception instance. So it's probably possible to get rid of errMsgShortString, but that would a be an unrelated cleanup. Fixes #11014. Test Plan: see above Reviewers: austin, simonpj, thomie, bgamari Reviewed By: thomie, bgamari Subscribers: simonpj, nomeata, thomie Differential Revision: https://phabricator.haskell.org/D1427 GHC Trac Issues: #11014
* Testsuite: report and error out on unfound testsThomas Miedema2015-10-291-2/+10
| | | | | | | | | | | | | | | | | | | Users are sometimes confused why their test doesn't run. It is usually because of a misspelled testname, for example using 'TEST=1234' instead of 'TEST=T1234'. After this patch it is hopefully more clear what the problem is, showing: ERROR: tests not found: ['1234'] Instead of: 0 total tests, which gave rise to 0 test cases, of which 0 were skipped Reviewed by: austin, bgamari Differential Revision: https://phabricator.haskell.org/D1388
* Testsuite: make driver python 2.6 compatible againThomas Miedema2015-10-051-6/+6
| | | | | | Reviewed by: kgardas Differential Revision: https://phabricator.haskell.org/D1311
* base: use Show for ErrorCall in uncaughtExceptionHandlerEric Seidel2015-09-211-2/+2
| | | | | | | | | | | | | The default top-level exception handler now uses the `Show` instance for `ErrorCall` when printing exceptions, so it will actually print the out-of-band data (e.g. `CallStack`s) in compiled binaries, instead of just printing the error message. This also updates the hpc submodule to fix the test output. Reviewed By: austin, thomie Differential Revision: https://phabricator.haskell.org/D1217
* Always run explicitly requested ways (extra_ways) for fast runs.Edward Z. Yang2015-09-201-1/+8
| | | | | | | | | | | | | | | | | | | | | | | To keep validates fast, we only one run one way. But I think that it's important for some tests to run them a few ways, just to make sure functionality, e.g. the profiler, is working. This commit changes the logic so that any way specified in extra_ways is always run for fast. The big changes is now profiling tests are run on validate. I also made it so the G1 garbage collector tests only run on slow. Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu> Test Plan: validate Reviewers: austin, thomie, bgamari Reviewed By: austin, thomie, bgamari Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D1251
* Testsuite: normalise slashes in callstack outputThomas Miedema2015-09-121-3/+5
| | | | This fixes some tests on Windows, for example T2120.
* Testsuite: ignore line number differences in call stacks (#10834)Thomas Miedema2015-09-041-0/+8
| | | | Differential Revision: https://phabricator.haskell.org/D1206
* Testsuite: only print msg when timeout kills process unexpectedlyThomas Miedema2015-09-041-0/+4
| | | | Differential Revision: https://phabricator.haskell.org/D1207
* Testsuite: delete dead codeThomas Miedema2015-09-021-3/+0
|
* Testsuite: by default run all tests for a single wayThomas Miedema2015-09-021-4/+5
| | | | | | | | | | | | | | `make test` now runs all tests for a single way only. Use `make slowtest` to get the previous behaviour (i.e. run all tests for all ways). The intention is to use this new `make test` setting for Phabricator, as a reasonable compromise between `make fasttest` (what it previously used) and a fullblown `make slowtest` (which runs all tests for all ways). See Note [validate and testsuite speed] in toplevel Makefile. Differential Revision: https://phabricator.haskell.org/D1178
* Testsuite: refactoring onlyThomas Miedema2015-09-021-53/+58
| | | | | | | | | | | | | | | | * Rename `platform_wordsize_qualify` to `find_expected_file`, and make it return a filename instead of an (absolute) filepath. * Replace most usages of `qualify` by `in_testdir`. Others usage sites will be deleted in a later commit. These changes will be useful in a later commit, when we'll distinguish between files in the source directory and those in a (newly created) test directory. Reviewed by: austin, bgamari Differential Revision: https://phabricator.haskell.org/D1186
* Testsuite: speedup running a single testThomas Miedema2015-08-151-1/+3
| | | | | | Benchmark: in rootdirectory, run `time make test TEST=dummy VERBOSE=0` Before this commit: 2.6s After this commit: 0.7s
View Lorry Controller Status