Refactor TcDeriv to validity-check less in anyclass/via deriving (#13154)

Due to the way `DerivEnv` is currently structured, there is an
invariant that every derived instance must consist of a class applied
to a non-empty list of argument types, where the last argument *must*
be an application of a type constructor to some arguments. This works
for many cases, but there are also some design patterns in standalone
`anyclass`/`via` deriving that are made impossible due to enforcing
this invariant, as documented in #13154.

This fixes #13154 by refactoring `TcDeriv` and friends to perform
fewer validity checks when using the `anyclass` or `via` strategies.
The highlights are as followed:

* Five fields of `DerivEnv` have been factored out into a new
  `DerivInstTys` data type. These fields only make sense for
  instances that satisfy the invariant mentioned above, so
  `DerivInstTys` is now only used in `stock` and `newtype` deriving,
  but not in other deriving strategies.
* There is now a `Note [DerivEnv and DerivSpecMechanism]` describing
  the bullet point above in more detail, as well as explaining the
  exact requirements that each deriving strategy imposes.
* I've refactored `mkEqnHelp`'s call graph to be slightly less
  complicated. Instead of the previous `mkDataTypeEqn`/`mkNewTypeEqn`
  dichotomy, there is now a single entrypoint `mk_eqn`.
* Various bits of code were tweaked so as not to use fields that are
  specific to `DerivInstTys` so that they may be used by all deriving
  strategies, since not all deriving strategies use `DerivInstTys`.

6 files changed, 123 insertions, 1 deletions

diff --git a/testsuite/tests/deriving/should_compile/T13154b.hs b/testsuite/tests/deriving/should_compile/T13154b.hs
new file mode 100644
index 0000000000..9df828b111
--- /dev/null
+++ b/testsuite/tests/deriving/should_compile/T13154b.hs
@@ -0,0 +1,62 @@
+{-# LANGUAGE DataKinds #-}
+{-# LANGUAGE DeriveAnyClass #-}
+{-# LANGUAGE DerivingStrategies #-}
+{-# LANGUAGE DerivingVia #-}
+{-# LANGUAGE FlexibleInstances #-}
+{-# LANGUAGE MultiParamTypeClasses #-}
+{-# LANGUAGE PartialTypeSignatures #-}
+{-# LANGUAGE PolyKinds #-}
+{-# LANGUAGE StandaloneDeriving #-}
+{-# LANGUAGE TypeFamilies #-}
+{-# LANGUAGE UnboxedSums #-}
+{-# LANGUAGE UnboxedTuples #-}
+{-# OPTIONS_GHC -Wno-partial-type-signatures #-}
+module T13154b where
+
+import Data.Kind
+import Data.Typeable
+import GHC.Exts
+import GHC.TypeLits
+
+class Foo1 (a :: TYPE ('TupleRep '[]))
+deriving instance Foo1 a
+
+class Foo2 (a :: TYPE ('TupleRep '[]))
+deriving instance Foo2 (##)
+
+class Foo3 (a :: TYPE ('SumRep '[ 'LiftedRep, 'LiftedRep ]))
+deriving instance Foo3 a
+
+class Foo4 (a :: TYPE ('SumRep '[ 'LiftedRep, 'LiftedRep ]))
+deriving instance Foo4 (# a | b #)
+
+class Foo5 (a :: Type)
+deriving instance Foo5 a
+
+class Foo6
+deriving instance Foo6
+
+class Foo7 (a :: Nat)
+deriving anyclass instance Foo7 0
+deriving          instance Foo7 1
+
+class Foo8 (a :: Symbol)
+deriving anyclass instance Foo8 "a"
+deriving          instance Foo8 "b"
+
+class Typeable a => Foo9 a
+deriving instance _ => Foo9 (f a)
+
+data family D1 a
+newtype ByBar a = ByBar a
+class Foo10 a where
+  baz :: a -> a
+instance Foo10 (ByBar a) where
+  baz = id
+deriving via ByBar (D1 a) instance Foo10 (D1 a)
+
+data family D2
+data family D3
+class Foo11 a where
+deriving anyclass instance Foo11 D2
+deriving          instance Foo11 D3
diff --git a/testsuite/tests/deriving/should_compile/all.T b/testsuite/tests/deriving/should_compile/all.T
index 55c7d90f09..e29ae0e0b5 100644
--- a/testsuite/tests/deriving/should_compile/all.T
+++ b/testsuite/tests/deriving/should_compile/all.T
@@ -89,6 +89,7 @@ test('T12616', normal, compile, [''])
 test('T12688', normal, compile, [''])
 test('T12814', normal, compile, ['-Wredundant-constraints'])
 test('T13154a', normal, compile, [''])
+test('T13154b', normal, compile, [''])
 test('T13272', normal, compile, [''])
 test('T13272a', normal, compile, [''])
 test('T13297', normal, compile, [''])
diff --git a/testsuite/tests/deriving/should_fail/T13154c.hs b/testsuite/tests/deriving/should_fail/T13154c.hs
new file mode 100644
index 0000000000..342bb9fc48
--- /dev/null
+++ b/testsuite/tests/deriving/should_fail/T13154c.hs
@@ -0,0 +1,23 @@
+{-# LANGUAGE DataKinds #-}
+{-# LANGUAGE DerivingStrategies #-}
+{-# LANGUAGE FlexibleInstances #-}
+{-# LANGUAGE GeneralizedNewtypeDeriving #-}
+{-# LANGUAGE MultiParamTypeClasses #-}
+{-# LANGUAGE PolyKinds #-}
+{-# LANGUAGE StandaloneDeriving #-}
+{-# LANGUAGE UnboxedTuples #-}
+module T13154c where
+
+import GHC.Exts
+
+-- Test some nonsense configurations
+
+class Foo1 (a :: TYPE ('TupleRep '[]))
+deriving stock   instance Foo1 a
+deriving stock   instance Foo1 (##)
+deriving newtype instance Foo1 a
+deriving newtype instance Foo1 (##)
+
+class Foo2
+deriving stock   instance Foo2
+deriving newtype instance Foo2
diff --git a/testsuite/tests/deriving/should_fail/T13154c.stderr b/testsuite/tests/deriving/should_fail/T13154c.stderr
new file mode 100644
index 0000000000..70031a79b2
--- /dev/null
+++ b/testsuite/tests/deriving/should_fail/T13154c.stderr
@@ -0,0 +1,35 @@
+
+T13154c.hs:16:1: error:
+    • Can't make a derived instance of
+        ‘Foo1 a’ with the stock strategy:
+        The last argument of the instance must be a data or newtype application
+    • In the stand-alone deriving instance for ‘Foo1 a’
+
+T13154c.hs:17:1: error:
+    • Can't make a derived instance of
+        ‘Foo1 (# #)’ with the stock strategy:
+        ‘Foo1’ is not a stock derivable class (Eq, Show, etc.)
+    • In the stand-alone deriving instance for ‘Foo1 (# #)’
+
+T13154c.hs:18:1: error:
+    • Can't make a derived instance of
+        ‘Foo1 a’ with the newtype strategy:
+        The last argument of the instance must be a data or newtype application
+    • In the stand-alone deriving instance for ‘Foo1 a’
+
+T13154c.hs:19:1: error:
+    • Can't make a derived instance of
+        ‘Foo1 (# #)’ with the newtype strategy:
+        GeneralizedNewtypeDeriving cannot be used on non-newtypes
+    • In the stand-alone deriving instance for ‘Foo1 (# #)’
+
+T13154c.hs:22:1: error:
+    • Can't make a derived instance of ‘Foo2’ with the stock strategy:
+        Cannot derive instances for nullary classes
+    • In the stand-alone deriving instance for ‘Foo2’
+
+T13154c.hs:23:1: error:
+    • Can't make a derived instance of
+        ‘Foo2’ with the newtype strategy:
+        Cannot derive instances for nullary classes
+    • In the stand-alone deriving instance for ‘Foo2’
diff --git a/testsuite/tests/deriving/should_fail/T7959.stderr b/testsuite/tests/deriving/should_fail/T7959.stderr
index 254cfedacb..0ba77ffb8b 100644
--- a/testsuite/tests/deriving/should_fail/T7959.stderr
+++ b/testsuite/tests/deriving/should_fail/T7959.stderr
@@ -1,6 +1,6 @@
 
 T7959.hs:5:1: error:
-    • Cannot derive instances for nullary classes
+    • Can't make a derived instance of ‘A’: Try enabling DeriveAnyClass
     • In the stand-alone deriving instance for ‘A’
 
 T7959.hs:6:17: error:
diff --git a/testsuite/tests/deriving/should_fail/all.T b/testsuite/tests/deriving/should_fail/all.T
index bd2c55983a..d195a08691 100644
--- a/testsuite/tests/deriving/should_fail/all.T
+++ b/testsuite/tests/deriving/should_fail/all.T
@@ -66,6 +66,7 @@ test('T11509_1', [when(doing_ghci(), extra_hc_opts('-fobject-code'))],
 test('T12163', normal, compile_fail, [''])
 test('T12512', omit_ways(['ghci']), compile_fail, [''])
 test('T12801', normal, compile_fail, [''])
+test('T13154c', normal, compile_fail, [''])
 test('T14365', [extra_files(['T14365B.hs','T14365B.hs-boot'])],
                multimod_compile_fail, ['T14365A',''])
 test('T14728a', normal, compile_fail, [''])