summaryrefslogtreecommitdiff
path: root/testsuite/driver/testglobals.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Testsuite: validate the tests/stage1 directory with the stage1 compilerThomas Miedema2016-06-181-1/+4
| | | | | | | | | | | * See `Note [Why is there no stage1 setup function?]`. * Move T2632 to the tests/stage1 directory (#10382). Reviewed by: ezyang, nomeata, bgamari Differential Revision: https://phabricator.haskell.org/D2341 GHC Trac Issues: #12197
* Testsuite: run tests in <testdir>.run instead of /tmpThomas Miedema2016-06-181-0/+6
| | | | | | | | | | | | | | | | | | | | As discussed in Phab:D1187, this approach makes it a bit easier to inspect the test directory while working on a new test. The only tests that needed changes are the ones that refer to files in ancestor directories. Those files are now copied directly into the test directory. validate still runs the tests in a temporary directory in /tmp, see `Note [Running tests in /tmp]` in testsuite/driver/runtests.py. Update submodule hpc. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2333 GHC Trac Issues: #11980
* Testsuite: delete check_files_writtenThomas Miedema2016-05-241-3/+0
| | | | | | | | | The CHECK_FILES_WRITTEN feature is no longer necessary, since tests don't write to the source directory anymore (#11980). Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2162
* Testsuite: run tests in /tmp after copying required filesThomas Miedema2016-05-171-0/+3
| | | | | | | | | | | | | | | | | | | | | | | Major change to the testsuite driver. For each TEST: * create a directory `<testdir>` inside `/tmp`. * link/copy all source files that the test needs into `<testdir>`. * run the test inside `<testdir>`. * delete `<testdir>` Extra files are (temporarily) tracked in `testsuite/driver/extra_files.py`, but can also be specified using the `extra_files` setup function. Differential Revision: https://phabricator.haskell.org/D1187 Reviewed by: Rufflewind, bgamari Trac: #11980
* Testsuite: make CLEANUP=1 the default (#9758)Thomas Miedema2016-04-301-3/+0
| | | | | | | | | | Also move the `cleanup` setting from `default_testopts` to `config`. The `cleanup` setting is the same for all tests, hence it belongs in `config`. Reviewed by: austin Differential Revision: https://phabricator.haskell.org/D2148
* Testsuite: delete -fesc testsThomas Miedema2016-04-281-1/+0
| | | | | | | | | | | The -fesc flag does not exist, and has never existed. Also delete now unused config.compiler_tags, and 'Project version' never contains a '-'. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2138
* Filter out -prof callstacks from test output (#11521)Thomas Miedema2016-02-231-0/+3
|
* Testsuite: report and error out on unfound testsThomas Miedema2015-10-291-1/+2
| | | | | | | | | | | | | | | | | | | Users are sometimes confused why their test doesn't run. It is usually because of a misspelled testname, for example using 'TEST=1234' instead of 'TEST=T1234'. After this patch it is hopefully more clear what the problem is, showing: ERROR: tests not found: ['1234'] Instead of: 0 total tests, which gave rise to 0 test cases, of which 0 were skipped Reviewed by: austin, bgamari Differential Revision: https://phabricator.haskell.org/D1388
* Testsuite: by default run all tests for a single wayThomas Miedema2015-09-021-2/+2
| | | | | | | | | | | | | | `make test` now runs all tests for a single way only. Use `make slowtest` to get the previous behaviour (i.e. run all tests for all ways). The intention is to use this new `make test` setting for Phabricator, as a reasonable compromise between `make fasttest` (what it previously used) and a fullblown `make slowtest` (which runs all tests for all ways). See Note [validate and testsuite speed] in toplevel Makefile. Differential Revision: https://phabricator.haskell.org/D1178
* Testsuite: delete remaining only_compiler_types(['ghc']) setupsThomas Miedema2015-07-141-3/+0
| | | | | No point in pretending other compilers can use the GHC testsuite. This makes the *.T files a bit shorter.
* Testsuite: delete unused with_namebaseThomas Miedema2015-07-141-4/+0
| | | | It was introduced in 39c6c735c216d259854ee31b15ec87ea653f2b5d (2007).
* Testsuite: add function compile_timeout_multiplier (#10345)Thomas Miedema2015-06-131-1/+2
| | | | | | | | | | | And rename timeout_multiplier to run_timeout_multiplier. timeout_multiplier was added in commit a00389794b839971c7d52ead9e8570bfaa25ac55. The name suggested that it would affect any test, but it actually only affected tests that had a run component, and only that run component (as needed by test T367). Differential Revision: https://phabricator.haskell.org/D982
* Make validate more quietThomas Miedema2015-06-041-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * By default use V=0, and call the testsuite with VERBOSE=2, which we did before only with validate --quiet. This disables printing the test commands it runs. * When --quiet is used, call the testsuite with VERBOSE=1. This disables printing the '====> Scanning' lines, and doesn't print which test is being run. So it only prints something when a test accidentally prints to stdout or when it fails. Don't set this option on Travis, as Travis will cancel a build if it doesn't see any output for more than 10 minutes. * When --quiet is used, set the new test option NO_PRINT_SUMMARY, which skips printing the test summary. Only the list of unexpected failures is printed, if there are any. Note that the full summary can still be found in testsuite_summary.txt * When --quiet is used, don't pass the `-v` flag to `ghc-pkg check` * When --quiet is used, don't print the Oops! header. It shoud be clear from the list of failing tests that something is wrong. This is all done to get the most out of 30 lines of logfile. These changes can be disabled later by simply not passing the --quiet flag to validate. Differential Revision: https://phabricator.haskell.org/D942
* Split off stat (benchmark) test failures into a separate section in the test ↵Gintautas Miliauskas2014-10-311-0/+2
| | | | | | | | | | | | | | | | | runner summary. Stat tests are generally less reliable than other types of tests, so it's nice to have them in a separate section rather than interspersed with potential... Summary: ...correctness issues. Reviewers: austin Reviewed By: austin Subscribers: thomie, carter, simonmar Differential Revision: https://phabricator.haskell.org/D406
* Fix #5435, adding new test config check_stdout.Edward Z. Yang2014-04-091-0/+5
| | | | | | | | | | | check_stdout(f) allows you to override the test framework's diff based output checking with another mechanism. f is a function which takes two arguments: the first is the filename containing the observed stdout, the second is the normaliser that would have been applied (in case you want to read, normalise, and then do something.) Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
* Flag to test suite: SKIP_PERF_TESTSJoachim Breitner2013-10-051-0/+3
| | | | | | More often than not the output of the performance tests is in the way, rather than helping. This allows the use of `make SKIP_PERF_TESTS=YES` to skip these tests. Fixes #8413
* Test driver: Implement different verbositiesJoachim Breitner2013-09-201-1/+1
| | | | | | | | | Select verbosity with "make VERBOSE=n". Options so far: n=0: No per-test output n=1: Only failing test results n=2: As above, plus progress information (names of all tests) n=3: As aobve, plus commands called. Default currently is n=3, although n=2 might be a nicer default.
* Test driver: Print total time spent at the endJoachim Breitner2013-09-201-1/+1
|
* Tweak the brokens list to include the directory the test is inIan Lynagh2013-02-111-1/+1
|
* Add 'make list_brokens'Ian Lynagh2013-02-111-0/+6
| | | | | | Gives a list of tickets that the testsuite thinks are broken, and what bug it thinks is the reason. This can then be pasted into trac and 'previewed', which will show any closed tickets with strikeout.
* Remove unused *_num_fields bindingsIan Lynagh2013-02-071-11/+0
|
* Add better support for .cmm test files.Austin Seipp2013-01-171-0/+3
| | | | | | Fixes Trac #7573. Signed-off-by: Austin Seipp <mad.one@gmail.com>
* Only run dynamicToo001 if we have both vanilla and dynamic librariesIan Lynagh2012-12-141-0/+3
|
* Testsuite fixes for when we only have dynamic librariesIan Lynagh2012-10-251-0/+3
|
* Add an outputdir test optionIan Lynagh2012-10-111-0/+3
| | | | | This makes it possible to share source files between tests, without having the .o/.hi files overlap
* Add support for per-test timeout adjustment (timeout_multiplier), and ↵Edward Z. Yang2012-09-251-0/+3
| | | | | | tighten up #367 test. Signed-off-by: Edward Z. Yang <ezyang@mit.edu>
* Add combined_output option.Paolo Capriotti2012-04-231-1/+3
| | | | | | | | | Added an option to combine stdout and stderr into a single file. This is useful for ghci scripts that produce interleaved errors and normal output. Also modified check_stderr_ok so that it normalizes stderr in the same way as compile tests.
* add req_smpSimon Marlow2012-02-161-0/+3
|
* Allow perf bounds to be specified as base + percentage deviationSimon Marlow2012-02-161-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | This makes it much easier to update the bounds. Instead of coming up with a suitable (min,max) pair, you just give e.g. (base, 10) to allow 10% deviation from the base figure, which can be pasted from the error. e.g. previously: - # expected value: 458700632 (amd64/Linux): - if_wordsize(64, - compiler_stats_num_field('bytes allocated', 440000000, - 480000000)), now: + if_wordsize(64, # sample from amd64/Linux 15/2/2012 + compiler_stats_range_field('bytes allocated', 360243576, 10)), Note: use stats_range_field rather than stats_num_field. I left support for the old way for now so that we can do a gradual migration. (next I suppose we should make it so that 'make accept' works for perf tests, but that's for another day)
* Make the compiler_always_flags modifiable on a per-test basisSimon Marlow2012-02-161-0/+5
| | | | Motivation: we wanted to disable -dcore-lint for the perf tests
* Count the number of tests that failed due to reqlib()Simon Marlow2012-01-061-0/+2
| | | | | | | | | | | | | | OVERALL SUMMARY for test run started at Fri Jan 6 09:46:46 GMT 2012 82 total tests, which gave rise to 820 test cases, of which 0 caused framework failures 818 were skipped 0 expected passes 2 had missing libraries <--- 0 expected failures 0 unexpected passes 0 unexpected failures
* Fix meaning of the only_ways field (fixed cgrun057(normal) in validate)Simon Marlow2011-11-161-4/+4
| | | | | | | | | | | | | The default setting for only_ways was [], which meant "all ways". However, it's really useful to have [] mean "no ways". In cgrun057 I used only_ways(prof_ways) to mean "only run this in the profiling ways", but if profiling is disabled then prof_ways = [] and this doesn't do what we want. Profiling is normally disabled in validate, but I had it enabled because I was testing profiling. So now only_ways defaults to None, which means "all ways", and [] now means "no ways".
* allow error message normalisation to be customisedSimon Marlow2011-11-091-0/+3
|
* generalise cmd_prefix to a general function to transform the commandSimon Marlow2011-10-181-2/+2
|
* Rejig how performance test stats are storedIan Lynagh2011-09-181-2/+2
| | | | | | In particular, this allows more specific results to be specified, e.g. if_wordsize(64, stats_num_field('foo', 50, 54)), if_platform('x86_64-apple-darwin', stats_num_field('foo', 62, 66)),
* Use True/False rather than 1/0 for opts.aloneIan Lynagh2011-08-081-1/+1
|
* Add support for checking whether files are written by more than one testIan Lynagh2011-08-071-0/+3
| | | | | | | | | | | | | | | | | | | | Work in progress, but largely works. Known issues: * only supported when using the timeout program * 'test.strace' files aren't cleaned, as they end up in the root directory rather than the test's directory * Doesn't yet track what the current directory is, so finds several files like "A.o" being written by multiple tests (and conversely, may be missing writes to the same file from different directories) * Lots of tests write to $HOME/.ghc/ghci_history. We should probably be passing ghci a flag to stop this from happening. * Some .strace lines aren't understood yet, causing framework failures * One .strace file can cause muiltiple framework failures, if it contains lots of lines that aren't understood Performance: Threads fast testsuite time fast testsuite time with checks 1 16:36.14 25:16.07 5 5:33.95 8:04.05
* Testcase for objective-c++ compilation (trac #5150)Peter Wortmann2011-08-061-3/+4
|
* Make and use a req_interp helperIan Lynagh2011-07-131-0/+3
| | | | | Tests can now specify that they require the interpreter (i.e. that they use one of ghci, annotations, TH, etc).
* Tests for trac #5025.Austin Seipp2011-04-031-2/+3
| | | | | Also teach the testsuite how to compile objective-c code with .m as a suffix.
* Fix the ffi002 testIan Lynagh2010-04-281-0/+3
|
* Add clean_cmd to the testsuite, and use it in bug1465Ian Lynagh2009-12-191-0/+3
|
* Add pre-command support to the testsuite, and fix annrun01 by using itIan Lynagh2009-12-191-0/+3
|
* Allow tests to behave differently depending on whether the compiler is in-treeIan Lynagh2009-12-181-0/+3
| | | | | And skip testwsdeque if it is not in-tree, as we rely on some headers from the build tree.
* Gather all tests at once, rather than doing them directory by directoryIan Lynagh2009-11-281-0/+3
| | | | | This increases the parallelism possible, and allows us to track what progress we are making.
* grab the target architecture from GHC, and add an if_arch() testSimon Marlow2009-11-241-0/+1
|
* add config.have_shared_libsSimon Marlow2009-11-041-0/+3
|
* Add support for compiler -t statsIan Lynagh2009-04-041-1/+2
|
* Improve the testsuite driver support for -t stats, and enhance space_leak_001Ian Lynagh2009-04-031-5/+7
|
* Add a space leak test, and some infrastructure for checking space usageIan Lynagh2008-11-231-0/+6
|
View Lorry Controller Status