diff options
| author | Matthew Pickering <matthewtpickering@gmail.com> | 2022-01-25 11:16:07 +0000 |
|---|---|---|
| committer | Marge Bot <ben+marge-bot@smart-cactus.org> | 2022-01-25 17:29:17 -0500 |
| commit | 781323a3076781b5db50bdbeb8f64394add43836 (patch) | |
| tree | 96d856b70d4e24c3760b0b4b0d9d73b5e48eb9c5 | |
| parent | 112e9e9ee7a5af7db3ec29cd3fdfc73aa4125afa (diff) | |
| download | haskell-781323a3076781b5db50bdbeb8f64394add43836.tar.gz | |
Widen T12545 acceptance window
This test has been the scourge of contributors for a long time.
It has caused many failed CI runs and wasted hours debugging a test
which barely does anything. The fact is does nothing is the reason for
the flakiness and it's very sensitive to small changes in initialisation costs,
in particular adding wired-in things can cause this test to fluctuate
quite a bit.
Therefore we admit defeat and just bump the threshold up to 10% to catch
very large regressions but otherwise don't care what this test does.
Fixes #19414
| -rw-r--r-- | testsuite/tests/perf/compiler/all.T | 13 |
1 files changed, 9 insertions, 4 deletions
diff --git a/testsuite/tests/perf/compiler/all.T b/testsuite/tests/perf/compiler/all.T index 25672bf7e7..4136572419 100644 --- a/testsuite/tests/perf/compiler/all.T +++ b/testsuite/tests/perf/compiler/all.T @@ -212,12 +212,17 @@ test('T12234', ['']) # T12545 is sensitive to -dunique-increments changes, see #19414. I've seen -# variations of as much as 4.8% by playing with that parameter, but I think -# it's better to check with T12545.measure.sh that lower and upper bounds of -# allocations indeed haven't changed and then simply accept the metric increase. +# variations of as much as 4.8% by playing with that parameter, +# +# The issue with the test is that it does too little so is very sensitive to +# any small variations during initialisation and in particular populating the +# initial environments with wired-in things. Therefore it has a very high change +# threshold so we catch if it regresses a lot but don't worry if it regresses a little. +# +# You can measure the variance by running T12545.measure.sh. test('T12545', [ only_ways(['normal']), - collect_compiler_stats('bytes allocated', 3), # Don't increase it, run T12545.measure.sh and leave a comment on #19414! + collect_compiler_stats('bytes allocated', 10), # extra_clean(['T12545a.hi', 'T12545a.o']) ], multimod_compile, |