Make checkFamInstConsistency faster

We've noticed that `checkFamInstConsistency` takes 6% of
overall build time on our codebase.
I've poked around for a bit and most of type family
instances are `Rep` from `Generics`. I think those are
unavoidable, so I don't think we can have less of them.

I also looked at the code and noticed a simple algorithmic
improvement can be made. The algorithm is pretty simple:
we take all the family instances from one module (`M1`)
and test it against another module (`M2`).
The cost of that is dominated by the size of `M1`, because
for each instance in `M1` we look it up in the type family
env from `M2`, and lookup is cheap.
If `M1` is bigger than `M2`, that's suboptimal, so after
my change we always iterate through the smaller set.

This drives down the cost of `checkFamInstConsistency`
to 2%.

Test Plan: harbormaster

Reviewers: simonmar, simonpj, goldfire, rwbarton, bgamari, ezyang, austin

Reviewed By: rwbarton, ezyang

Subscribers: ezyang, thomie

Differential Revision: https://phabricator.haskell.org/D2833

1 files changed, 2 insertions, 2 deletions

diff --git a/testsuite/tests/typecheck/should_fail/T6018fail.stderr b/testsuite/tests/typecheck/should_fail/T6018fail.stderr
index e40cb84d12..2525934a5d 100644
--- a/testsuite/tests/typecheck/should_fail/T6018fail.stderr
+++ b/testsuite/tests/typecheck/should_fail/T6018fail.stderr
@@ -9,10 +9,10 @@ T6018Afail.hs:7:15: error:
       G Char Bool Int = Int -- Defined at T6018Afail.hs:7:15
       G Bool Int Char = Int -- Defined at T6018fail.hs:15:15
 
-T6018Dfail.hs:7:15: error:
+T6018Cfail.hs:8:15: error:
     Type family equations violate injectivity annotation:
-      T6018Bfail.H Bool Int Char = Int -- Defined at T6018Dfail.hs:7:15
       T6018Bfail.H Char Bool Int = Int -- Defined at T6018Cfail.hs:8:15
+      T6018Bfail.H Bool Int Char = Int -- Defined at T6018Dfail.hs:7:15
 
 T6018fail.hs:13:15: error:
     Type family equations violate injectivity annotation: