<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/haskell.git/testsuite/driver/testlib.py, branch wip/fix-submodules</title>
<subtitle>gitlab.haskell.org: ghc/ghc.git
</subtitle>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/'/>
<entry>
<title>Introduce NCG config flag and add helper</title>
<updated>2018-12-30T16:32:35+00:00</updated>
<author>
<name>Gabor Greif</name>
<email>ggreif@gmail.com</email>
</author>
<published>2018-12-30T09:57:46+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=ae4f1033cfe131fca9416e2993bda081e1f8c152'/>
<id>ae4f1033cfe131fca9416e2993bda081e1f8c152</id>
<content type='text'>
... for testing presence of NCG

This commit adds a criterion for checking whether we can expect sensible
output from --ddump-asm.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
... for testing presence of NCG

This commit adds a criterion for checking whether we can expect sensible
output from --ddump-asm.
</pre>
</div>
</content>
</entry>
<entry>
<title>testsuite: Fix a variety of issues when building with integer-simple</title>
<updated>2018-12-25T03:25:47+00:00</updated>
<author>
<name>Ben Gamari</name>
<email>ben@smart-cactus.org</email>
</author>
<published>2018-12-23T06:34:00+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=993782073c0b380908e9541c40c6c5849dbacfec'/>
<id>993782073c0b380908e9541c40c6c5849dbacfec</id>
<content type='text'>
 * Mark arith011 as broken with integer-simple

   As noted in #16091, arith011 fails when run against integer-simple with a
   "divide by zero" exception. This suggests that integer-gmp and integer-simple
   are handling division by zero differently.

 * This also fixes broken_without_gmp; the lack of types made the previous
   failure silent, sadly. Improves situation of #16043.

 * Mark several tests implicitly depending upon integer-gmp as broken
   with integer-simple. These expect to see Core coming from integer-gmp,
   which breaks with integer-simple.

 * Increase runtime timeout multiplier of T11627a with integer-simple

   I previously saw that T11627a timed out in all profiling ways when run against
   integer-simple. I suspect this is due to integer-simple's rather verbose heap
   representation. Let's see whether increasing the runtime timeout helps.

   Fixes test for #11627.

This is all in service of fixing #16043.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
 * Mark arith011 as broken with integer-simple

   As noted in #16091, arith011 fails when run against integer-simple with a
   "divide by zero" exception. This suggests that integer-gmp and integer-simple
   are handling division by zero differently.

 * This also fixes broken_without_gmp; the lack of types made the previous
   failure silent, sadly. Improves situation of #16043.

 * Mark several tests implicitly depending upon integer-gmp as broken
   with integer-simple. These expect to see Core coming from integer-gmp,
   which breaks with integer-simple.

 * Increase runtime timeout multiplier of T11627a with integer-simple

   I previously saw that T11627a timed out in all profiling ways when run against
   integer-simple. I suspect this is due to integer-simple's rather verbose heap
   representation. Let's see whether increasing the runtime timeout helps.

   Fixes test for #11627.

This is all in service of fixing #16043.
</pre>
</div>
</content>
</entry>
<entry>
<title>Revert "testsuite: Fix broken_without_gmp"</title>
<updated>2018-12-23T18:45:24+00:00</updated>
<author>
<name>Ben Gamari</name>
<email>ben@smart-cactus.org</email>
</author>
<published>2018-12-23T18:45:24+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=a0b9d42be3ae65860d5187918891344a611ca2ac'/>
<id>a0b9d42be3ae65860d5187918891344a611ca2ac</id>
<content type='text'>
This reverts commit e59439af3222d151918ad1ad2a03942ce9e6a1ff.

This is causing unexpected failures in some test ways. Further proof
that no change is too trivial for CI.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This reverts commit e59439af3222d151918ad1ad2a03942ce9e6a1ff.

This is causing unexpected failures in some test ways. Further proof
that no change is too trivial for CI.
</pre>
</div>
</content>
</entry>
<entry>
<title>testsuite: Fix broken_without_gmp</title>
<updated>2018-12-23T06:36:08+00:00</updated>
<author>
<name>Ben Gamari</name>
<email>ben@smart-cactus.org</email>
</author>
<published>2018-12-23T06:34:00+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=e59439af3222d151918ad1ad2a03942ce9e6a1ff'/>
<id>e59439af3222d151918ad1ad2a03942ce9e6a1ff</id>
<content type='text'>
The lack of types made the previous failure silent, sadly. Improves
situation of #16043.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The lack of types made the previous failure silent, sadly. Improves
situation of #16043.
</pre>
</div>
</content>
</entry>
<entry>
<title>testsuite: Fix typo: integer-gimp /= integer-gmp</title>
<updated>2018-12-14T02:59:20+00:00</updated>
<author>
<name>Ben Gamari</name>
<email>ben@smart-cactus.org</email>
</author>
<published>2018-12-13T16:45:27+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=fa2a6e4fe95c20a21d7722eacc9aea18343c034e'/>
<id>fa2a6e4fe95c20a21d7722eacc9aea18343c034e</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>testsuite: Normalise away spurious differences in out-of-scope instances</title>
<updated>2018-12-13T04:26:34+00:00</updated>
<author>
<name>Ben Gamari</name>
<email>ben@smart-cactus.org</email>
</author>
<published>2018-12-12T21:45:55+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=9d9f4c9a7c04e152b40ae2db7c051cd9e0f5df61'/>
<id>9d9f4c9a7c04e152b40ae2db7c051cd9e0f5df61</id>
<content type='text'>
This fixes a variety of testsuite failures with integer-simple of the form

```
--- typecheck/should_fail/tcfail072.run/tcfail072.stderr.normalised
+++ typecheck/should_fail/tcfail072.run/tcfail072.comp.stderr.normalised
@@ -12,7 +12,7 @@
           -- Defined in ‘integer-&lt;IMPL&gt;-&lt;VERSION&gt;:GHC.Integer.Type’
         instance Ord () -- Defined in ‘GHC.Classes’
         ...plus 21 others
-        ...plus three instances involving out-of-scope types
+        ...plus two instances involving out-of-scope types
         (use -fprint-potential-instances to see them all)
      In the expression: g A
       In an equation for ‘g’: g (B _ _) = g A
```

In service of fixing #16043.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This fixes a variety of testsuite failures with integer-simple of the form

```
--- typecheck/should_fail/tcfail072.run/tcfail072.stderr.normalised
+++ typecheck/should_fail/tcfail072.run/tcfail072.comp.stderr.normalised
@@ -12,7 +12,7 @@
           -- Defined in ‘integer-&lt;IMPL&gt;-&lt;VERSION&gt;:GHC.Integer.Type’
         instance Ord () -- Defined in ‘GHC.Classes’
         ...plus 21 others
-        ...plus three instances involving out-of-scope types
+        ...plus two instances involving out-of-scope types
         (use -fprint-potential-instances to see them all)
      In the expression: g A
       In an equation for ‘g’: g (B _ _) = g A
```

In service of fixing #16043.
</pre>
</div>
</content>
</entry>
<entry>
<title>testsuite: Fix a number of GHCi-related failures due to integer-simple</title>
<updated>2018-12-13T04:25:02+00:00</updated>
<author>
<name>Ben Gamari</name>
<email>ben@smart-cactus.org</email>
</author>
<published>2018-12-12T21:42:37+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=14500dab3e9c24d2701c8be6f5a0fca30531ab80'/>
<id>14500dab3e9c24d2701c8be6f5a0fca30531ab80</id>
<content type='text'>
Towards fixing #16043.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Towards fixing #16043.
</pre>
</div>
</content>
</entry>
<entry>
<title>testuite: update more windows tests outputs</title>
<updated>2018-11-22T17:11:16+00:00</updated>
<author>
<name>Tamar Christina</name>
<email>tamar@zhox.com</email>
</author>
<published>2018-11-22T16:43:37+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=67277e7c783dac1fb66d1b113a8b1f2be784b24b'/>
<id>67277e7c783dac1fb66d1b113a8b1f2be784b24b</id>
<content type='text'>
Test Plan: ./validate

Reviewers: bgamari, simonmar

Subscribers: rwbarton, carter

Differential Revision: https://phabricator.haskell.org/D5362
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Test Plan: ./validate

Reviewers: bgamari, simonmar

Subscribers: rwbarton, carter

Differential Revision: https://phabricator.haskell.org/D5362
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix GhostScript detection (Trac #15856)</title>
<updated>2018-11-22T17:11:16+00:00</updated>
<author>
<name>Krzysztof Gogolewski</name>
<email>krz.gogolewski@gmail.com</email>
</author>
<published>2018-11-22T16:32:55+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=011e39d7fe533ca772beeed8529749c4750d4817'/>
<id>011e39d7fe533ca772beeed8529749c4750d4817</id>
<content type='text'>
The option `confdir` (used in GhostScript test) was set correctly
via `--config` in `test.mk` and incorrectly via `config/ghc`.
AFAICT, some time ago this was working because the
incorrect assignment was done first, and later it broke.

Hardian doesn't pass `confdir`. I removed `confdir` and use
`config.top` to determine the directory of the
`good.ps` and `bad.ps` files. This is simpler.

I also removed some redundant assignments in `config/ghc`.

Test Plan: manually set config.have_profiling and make test

Reviewers: bgamari

Subscribers: rwbarton, carter

GHC Trac Issues: #15856

Differential Revision: https://phabricator.haskell.org/D5298
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The option `confdir` (used in GhostScript test) was set correctly
via `--config` in `test.mk` and incorrectly via `config/ghc`.
AFAICT, some time ago this was working because the
incorrect assignment was done first, and later it broke.

Hardian doesn't pass `confdir`. I removed `confdir` and use
`config.top` to determine the directory of the
`good.ps` and `bad.ps` files. This is simpler.

I also removed some redundant assignments in `config/ghc`.

Test Plan: manually set config.have_profiling and make test

Reviewers: bgamari

Subscribers: rwbarton, carter

GHC Trac Issues: #15856

Differential Revision: https://phabricator.haskell.org/D5298
</pre>
</div>
</content>
</entry>
<entry>
<title>testsuite: Save performance metrics in git notes.</title>
<updated>2018-11-07T17:07:11+00:00</updated>
<author>
<name>David Eichmann</name>
<email>davide@Well-Typed.com</email>
</author>
<published>2018-11-07T17:02:47+00:00</published>
<link rel='alternate' type='text/html' href='http://trove.baserock.org/cgit/delta/haskell.git/commit/?id=932cd41d8c7984c767c1b3b58e05146f69cc5c15'/>
<id>932cd41d8c7984c767c1b3b58e05146f69cc5c15</id>
<content type='text'>
This patch makes the following improvement:
  - Automatically records test metrics (per test environment) so that
    the programmer need not supply nor update expected values in *.T
    files.
    - On expected metric changes, the programmer need only indicate the
      direction of change in the git commit message.
  - Provides a simple python tool "perf_notes.py" to compare metrics
    over time.

Issues:
  - Using just the previous commit allows performance to drift with each
    commit.
    - Currently we allow drift as we have a preference for minimizing
      false positives.
    - Some possible alternatives include:
      - Use metrics from a fixed commit per test: the last commit that
        allowed a change in performance (else the oldest metric)
      - Or use some sort of aggregate since the last commit that allowed
        a change in performance (else all available metrics)
      - These alternatives may result in a performance issue (with the
        test driver) having to heavily search git commits/notes.
  - Run locally, performance tests will trivially pass unless the tests
    were run locally on the previous commit. This is often not the case
    e.g.  after pulling recent changes.

Previously, *.T files contain statements such as:
```
stats_num_field('peak_megabytes_allocated', (2, 1))
compiler_stats_num_field('bytes allocated',
                         [(wordsize(64), 165890392, 10)])
```
This required the programmer to give the expected values and a tolerance
deviation (percentage). With this patch, the above statements are
replaced with:
```
collect_stats('peak_megabytes_allocated', 5)
collect_compiler_stats('bytes allocated', 10)
```
So that programmer must only enter which metrics to test and a tolerance
deviation. No expected value is required. CircleCI will then run the
tests per test environment and record the metrics to a git note for that
commit and push them to the git.haskell.org ghc repo. Metrics will be
compared to the previous commit. If they are different by the tolerance
deviation from the *.T file, then the corresponding test will fail. By
adding to the git commit message e.g.
```
 # Metric (In|De)crease &lt;metric(s)&gt; &lt;options&gt;: &lt;tests&gt;
Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \
         (test_env='linux_x86', way='default'):
    Test012, Test345
Metric Decrease 'bytes allocated':
    Test678
Metric Increase:
    Test711
```
This will allow the noted changes (letting the test pass). Note that by
omitting metrics or options, the change will apply to all possible
metrics/options (i.e. in the above, an increase for all metrics in all
test environments is allowed for Test711)

phabricator will use the message in the description

Reviewers: bgamari, hvr

Reviewed By: bgamari

Subscribers: rwbarton, carter

GHC Trac Issues: #12758

Differential Revision: https://phabricator.haskell.org/D5059
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This patch makes the following improvement:
  - Automatically records test metrics (per test environment) so that
    the programmer need not supply nor update expected values in *.T
    files.
    - On expected metric changes, the programmer need only indicate the
      direction of change in the git commit message.
  - Provides a simple python tool "perf_notes.py" to compare metrics
    over time.

Issues:
  - Using just the previous commit allows performance to drift with each
    commit.
    - Currently we allow drift as we have a preference for minimizing
      false positives.
    - Some possible alternatives include:
      - Use metrics from a fixed commit per test: the last commit that
        allowed a change in performance (else the oldest metric)
      - Or use some sort of aggregate since the last commit that allowed
        a change in performance (else all available metrics)
      - These alternatives may result in a performance issue (with the
        test driver) having to heavily search git commits/notes.
  - Run locally, performance tests will trivially pass unless the tests
    were run locally on the previous commit. This is often not the case
    e.g.  after pulling recent changes.

Previously, *.T files contain statements such as:
```
stats_num_field('peak_megabytes_allocated', (2, 1))
compiler_stats_num_field('bytes allocated',
                         [(wordsize(64), 165890392, 10)])
```
This required the programmer to give the expected values and a tolerance
deviation (percentage). With this patch, the above statements are
replaced with:
```
collect_stats('peak_megabytes_allocated', 5)
collect_compiler_stats('bytes allocated', 10)
```
So that programmer must only enter which metrics to test and a tolerance
deviation. No expected value is required. CircleCI will then run the
tests per test environment and record the metrics to a git note for that
commit and push them to the git.haskell.org ghc repo. Metrics will be
compared to the previous commit. If they are different by the tolerance
deviation from the *.T file, then the corresponding test will fail. By
adding to the git commit message e.g.
```
 # Metric (In|De)crease &lt;metric(s)&gt; &lt;options&gt;: &lt;tests&gt;
Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \
         (test_env='linux_x86', way='default'):
    Test012, Test345
Metric Decrease 'bytes allocated':
    Test678
Metric Increase:
    Test711
```
This will allow the noted changes (letting the test pass). Note that by
omitting metrics or options, the change will apply to all possible
metrics/options (i.e. in the above, an increase for all metrics in all
test environments is allowed for Test711)

phabricator will use the message in the description

Reviewers: bgamari, hvr

Reviewed By: bgamari

Subscribers: rwbarton, carter

GHC Trac Issues: #12758

Differential Revision: https://phabricator.haskell.org/D5059
</pre>
</div>
</content>
</entry>
</feed>
