Control validity-checking of type synonym applications more carefully

Trac #16059 shows that when validity checking applications of type
synonyms, GHC sometimes wasn't checking the expanded type enough.
We must be careful, however, since checking both the expanded type as
well as the arguments to the type synonym can lead to exponential
blowup (see https://ghc.haskell.org/trac/ghc/ticket/16059#comment:4).
Nor can we omit checking either the expanded type or the argument for
correctness reasons.

The solution here is to introduce a new `ExpandMode` data type that
is plumbed through all of the type-validity-checking functions in
`TcValidity`. `ExpandMode` dictates whether we only check the
expanded type (`Expand`), only check the arguments (`NoExpand), or
both (`Both`). Importantly, if we check `Both` in the function for
validity checking type synonym applications, then we switch to
`NoExpand` when checking the arguments so as to avoid exponential
blowup. See `Note [Correctness and performance of type synonym validity
checking]` for the full story.

1 files changed, 6 insertions, 0 deletions

diff --git a/testsuite/tests/typecheck/should_fail/T16059c.stderr b/testsuite/tests/typecheck/should_fail/T16059c.stderr
new file mode 100644
index 0000000000..6c6113fb5f
--- /dev/null
+++ b/testsuite/tests/typecheck/should_fail/T16059c.stderr
@@ -0,0 +1,6 @@
+
+T16059c.hs:5:6: error:
+    • Illegal polymorphic type: forall a1. a1
+      Perhaps you intended to use RankNTypes
+    • In the expansion of type synonym ‘Foo’
+      In the type signature: f :: Foo -> a -> f