summaryrefslogtreecommitdiff
path: root/testsuite/driver/perf_notes.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix typosBrian Wignall2021-02-061-1/+1
|
* Bump the # of commits searched for perf baselineRichard Eisenberg2020-12-011-1/+1
| | | | | | | | The previous value of 75 meant that a feature branch with more than 75 commits would get spurious CI passes. This affects #18692, but does not fix that ticket, because if a baseline cannot be found, we should fail, not succeed.
* testsuite: Allow whitespace before "Metric (in|de)crease"Ben Gamari2020-10-071-1/+1
| | | | | | | Several people have struggled with metric change annotations in their commit messages not being recognized due to the fact that GitLab's job log inserts a space at the beginning of each line. Teach the regular expression to accept this whitespace.
* Enhance metrics outputSylvain Henry2020-09-151-5/+5
|
* Make sure we can read past perf notesKrzysztof Gogolewski2020-09-121-1/+9
| | | | See #18656.
* testsuite: Output performance test results in tabular formatDaishi Nakajima2020-09-081-16/+24
| | | | | | | | | | | | | | | | | this was suggested in #18417. Change the print format of the values. * Shorten commit hash * Reduce precision of the "Value" field * Shorten metrics name * e.g. runtime/bytes allocated -> run/alloc * Shorten "MetricsChange" * e.g. unchanged -> unch, increased -> incr And, print the baseline environment if there are baselines that were measured in a different environment than the current environment. If all "Baseline commit" are the same, print it once.
* testsuite: Allow baseline commit to be set explicitlyBen Gamari2020-08-181-5/+14
|
* testsuite/perf_notes: Fix ill-typed assignmentsBen Gamari2020-06-011-4/+5
|
* testsuite: Sort test names in expected change outputBen Gamari2020-02-151-3/+3
|
* Fix more typos, via an improved Levenshtein-style correctorBrian Wignall2020-01-121-1/+1
|
* perf_notes: Add --zero-y argumentBen Gamari2019-12-301-12/+25
| | | | | This makes it easier to see the true magnitude of fluctuations. Also do some house-keeping in the argument parsing department.
* testsuite: Simplify and clarify performance test baseline searchBen Gamari2019-12-121-65/+44
| | | | | | | The previous implementation was extremely complicated, seemingly to allow the local and CI namespaces to be searched incrementally. However, it's quite unclear why this is needed and moreover the implementation seems to have had quadratic runtime cost in the search depth(!).
* testsuite: Use colors more consistentlyBen Gamari2019-12-051-1/+11
|
* testsuite: Make performance metric summary more readableBen Gamari2019-12-051-2/+9
| | | | Along with some refactoring.
* Fix typos, using Wikipedia list of common typosBrian Wignall2019-11-281-1/+1
|
* CI: Always dump performance metrics.David Eichmann2019-10-221-5/+11
|
* testsuite: Add more type annotations to perf_notesBen Gamari2019-06-271-54/+98
|
* testsuite: More type signatureswip/run-fragile-testsBen Gamari2019-06-261-1/+2
|
* testsuite: A major revamp of the driverBen Gamari2019-06-251-31/+32
| | | | | | | | | This tries to put the testsuite driver into a slightly more maintainable condition: * Add type annotations where easily done * Use pathlib.Path instead of str paths * Make it pass the mypy typechecker
* TestRunner: Added --chart to display a chart of performance testsDavid Eichmann2019-06-041-120/+234
| | | | | | | | | | | | | This uses the Chart.js javascript library. Everything is put into a standalone .html file and opened with the default browser. I also simplified the text output to use the same data as the chart. You can now use a commit range with git's ".." syntax. The --ci option will use results from CI (you'll need to fetch them first): $ git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf $ python3 testsuite/driver/perf_notes.py --ci --chart --test-env x86_64-darwin --test-name T9630 master~500..master
* Improve test runner logging when calculating performance metric baseline #16662David Eichmann2019-05-201-5/+16
| | | | | | We attempt to get 75 commit hashes via `git log`, but this only gave 10 hashes in a CI run (see #16662). Better logging may help solve this error if it occurs again in the future.
* Test Runner: don't show missing baseline warning for performance tests with ↵David Eichmann2019-03-051-15/+18
| | | | | | expected changes on the current commit. Trac #16359
* Fix parsing of expected performance changes for tests with non-alpha characters.David Eichmann2019-03-011-1/+1
| | | | Python's split() function is used to split on all white space.
* Fix test runner crash when not in a git repoDavid Eichmann2019-02-211-6/+10
| | | | Respect `inside_git_repo()` when checking performance stats.
* Fix and Reapply "Performance tests: recover a baseline from ancestor commits ↵David Eichmann2019-02-161-10/+209
| | | | and CI results."
* Revert "Performance tests: recover a baseline from ancestor commits and CI ↵Ben Gamari2019-01-311-195/+7
| | | | | | | | | results." Unfortunately this has broken all future commits due to spurious(?) performance changes which I have been unable to work around. This reverts commit cc2261d42f6a954d88e355aaad41f001f65c95da.
* Performance tests: recover a baseline from ancestor commits and CI results.David Eichmann2019-01-301-7/+195
| | | | gitlab-ci: push performance metrics as git notes to the "GHC Performance Notes" repository.
* Revert "Batch merge"Ben Gamari2019-01-301-195/+7
| | | | This reverts commit 76c8fd674435a652c75a96c85abbf26f1f221876.
* Batch mergeBen Gamari2019-01-301-7/+195
|
* Do not save performance test results if worktree is dirty.David Eichmann2018-12-111-0/+4
| | | | | | | | | | | | Reviewers: bgamari, tdammers Reviewed By: bgamari, tdammers Subscribers: rwbarton, carter GHC Trac Issues: #15924 Differential Revision: https://phabricator.haskell.org/D5368
* testsuite: Don't use git status to determine whether we are inside a repoBen Gamari2018-12-011-4/+5
| | | | | Git status is extremely expensive for this task. We instead use `git rev-parse HEAD` and throw away the output to ensure we don't spam the user.
* Skip all performance tests if not in a git repo.David Eichmann2018-11-301-0/+9
| | | | | | | | | | | | Reviewers: bgamari, tdammers, osa1 Reviewed By: tdammers Subscribers: osa1, tdammers, rwbarton, carter GHC Trac Issues: #15923 Differential Revision: https://phabricator.haskell.org/D5367
* testsuite: Save performance metrics in git notes.David Eichmann2018-11-071-0/+382
This patch makes the following improvement: - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files. - On expected metric changes, the programmer need only indicate the direction of change in the git commit message. - Provides a simple python tool "perf_notes.py" to compare metrics over time. Issues: - Using just the previous commit allows performance to drift with each commit. - Currently we allow drift as we have a preference for minimizing false positives. - Some possible alternatives include: - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric) - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics) - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes. - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes. Previously, *.T files contain statements such as: ``` stats_num_field('peak_megabytes_allocated', (2, 1)) compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)]) ``` This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with: ``` collect_stats('peak_megabytes_allocated', 5) collect_compiler_stats('bytes allocated', 10) ``` So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g. ``` # Metric (In|De)crease <metric(s)> <options>: <tests> Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \ (test_env='linux_x86', way='default'): Test012, Test345 Metric Decrease 'bytes allocated': Test678 Metric Increase: Test711 ``` This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711) phabricator will use the message in the description Reviewers: bgamari, hvr Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #12758 Differential Revision: https://phabricator.haskell.org/D5059
View Lorry Controller Status