Driver Rework Patch

This patch comprises of four different but closely related ideas. The
net result is fixing a large number of open issues with the driver
whilst making it simpler to understand.

1. Use the hash of the source file to determine whether the source file
has changed or not. This makes the recompilation checking more robust to
modern build systems which are liable to copy files around changing
their modification times.

2. Remove the concept of a "stable module", a stable module was one
where the object file was older than the source file, and all transitive
dependencies were also stable. Now we don't rely on the modification
time of the source file, the notion of stability is moot.

3. Fix TH/plugin recompilation after the removal of stable modules. The
TH recompilation check used to rely on stable modules. Now there is a
uniform and simple way, we directly track the linkables which were
loaded into the interpreter whilst compiling a module. This is an
over-approximation but more robust wrt package dependencies changing.

4. Fix recompilation checking for dynamic object files. Now we actually
check if the dynamic object file exists when compiling with -dynamic-too

Fixes #19774 #19771 #19758 #17434 #11556 #9121 #8211 #16495 #7277 #16093

20 files changed, 176 insertions, 0 deletions

diff --git a/testsuite/tests/driver/recompPluginPackage/Makefile b/testsuite/tests/driver/recompPluginPackage/Makefile
new file mode 100644
index 0000000000..1748c86481
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/Makefile
@@ -0,0 +1,30 @@
+TOP=../../..
+include $(TOP)/mk/boilerplate.mk
+include $(TOP)/mk/test.mk
+
+SETUP='$(PWD)/Setup' -v0
+CONFIGURE=$(SETUP) configure $(CABAL_MINIMAL_BUILD) --with-ghc='$(TEST_HC)' --ghc-options='$(filter-out -rtsopts,$(TEST_HC_OPTS))' --package-db='$(PWD)/tmp.d' --prefix='$(PWD)/inst' $(VANILLA) $(PROF) $(DYN)
+
+recompPluginPackage:
+	'$(GHC_PKG)' init tmp.d
+	'$(TEST_HC)' $(TEST_HC_OPTS) -v0 --make Setup
+	# build p
+	cp p/Lib1.hs p/Lib.hs
+	rm -rf p/dist
+	(cd p; $(CONFIGURE) --ipid "p-0.1")
+	(cd p; $(SETUP) build)
+	(cd p; $(SETUP) copy)
+	(cd p; $(SETUP) register)
+	# build q
+	rm -rf q/dist
+	(cd q; $(CONFIGURE) --ipid "q-0.1")
+	(cd q; $(SETUP) build)
+	(cd q; $(SETUP) copy)
+	(cd q; $(SETUP) register)
+	cp p/Lib2.hs p/Lib.hs
+	(cd p; $(SETUP) build)
+	(cd p; $(SETUP) copy)
+	(cd p; $(SETUP) register)
+	(cd q; $(SETUP) build)
+	(cd q; $(SETUP) copy)
+	(cd q; $(SETUP) register)
diff --git a/testsuite/tests/driver/recompPluginPackage/README.md b/testsuite/tests/driver/recompPluginPackage/README.md
new file mode 100644
index 0000000000..45d4b1d817
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/README.md
@@ -0,0 +1,9 @@
+Modifying the definition of `p` in `Lib.hs` does not cause `q` to be recompiled.
+
+```
+> cabal new-run q
+0
+-- Change p = [| 1 |]
+> cabal new-run q
+-- Prints 1.. used to print 0
+1
diff --git a/testsuite/tests/driver/recompPluginPackage/Setup.hs b/testsuite/tests/driver/recompPluginPackage/Setup.hs
new file mode 100644
index 0000000000..9a994af677
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/Setup.hs
@@ -0,0 +1,2 @@
+import Distribution.Simple
+main = defaultMain
diff --git a/testsuite/tests/driver/recompPluginPackage/all.T b/testsuite/tests/driver/recompPluginPackage/all.T
new file mode 100644
index 0000000000..ea6cce4baa
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/all.T
@@ -0,0 +1,20 @@
+if config.have_vanilla:
+    vanilla = '--enable-library-vanilla'
+else:
+    vanilla = '--disable-library-vanilla'
+
+if config.have_profiling:
+    prof = '--enable-library-profiling'
+else:
+    prof = '--disable-library-profiling'
+
+if not config.compiler_profiled and config.have_shared_libs:
+    dyn = '--enable-shared'
+else:
+    dyn = '--disable-shared'
+
+test('recompPluginPackage', [extra_files(['p', 'q', 'Setup.hs']),
+                   when(opsys('mingw32'), fragile(16405)),
+                   when(fast(), skip)],
+     run_command,
+     ['$MAKE -s --no-print-directory recompPluginPackage VANILLA=' + vanilla + ' PROF=' + prof + ' DYN=' + dyn])
diff --git a/testsuite/tests/driver/recompPluginPackage/cabal.project b/testsuite/tests/driver/recompPluginPackage/cabal.project
new file mode 100644
index 0000000000..838e7f9a75
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/cabal.project
@@ -0,0 +1,2 @@
+packages: p q
+
diff --git a/testsuite/tests/driver/recompPluginPackage/p/CHANGELOG.md b/testsuite/tests/driver/recompPluginPackage/p/CHANGELOG.md
new file mode 100644
index 0000000000..9ede8b27d4
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/CHANGELOG.md
@@ -0,0 +1,5 @@
+# Revision history for p
+
+## 0.1.0.0 -- YYYY-mm-dd
+
+* First version. Released on an unsuspecting world.
diff --git a/testsuite/tests/driver/recompPluginPackage/p/LICENSE b/testsuite/tests/driver/recompPluginPackage/p/LICENSE
new file mode 100644
index 0000000000..05bef1a55a
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/LICENSE
@@ -0,0 +1 @@
+L
diff --git a/testsuite/tests/driver/recompPluginPackage/p/Lib1.hs b/testsuite/tests/driver/recompPluginPackage/p/Lib1.hs
new file mode 100644
index 0000000000..942c536874
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/Lib1.hs
@@ -0,0 +1,5 @@
+module Lib where
+
+{-# NOINLINE p #-}
+p = 1
+
diff --git a/testsuite/tests/driver/recompPluginPackage/p/Lib2.hs b/testsuite/tests/driver/recompPluginPackage/p/Lib2.hs
new file mode 100644
index 0000000000..259ae46461
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/Lib2.hs
@@ -0,0 +1,5 @@
+module Lib where
+
+{-# NOINLINE p #-}
+p = 2
+
diff --git a/testsuite/tests/driver/recompPluginPackage/p/Main.hs b/testsuite/tests/driver/recompPluginPackage/p/Main.hs
new file mode 100644
index 0000000000..65ae4a05d5
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/Main.hs
@@ -0,0 +1,4 @@
+module Main where
+
+main :: IO ()
+main = putStrLn "Hello, Haskell!"
diff --git a/testsuite/tests/driver/recompPluginPackage/p/Setup.hs b/testsuite/tests/driver/recompPluginPackage/p/Setup.hs
new file mode 100644
index 0000000000..9a994af677
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/Setup.hs
@@ -0,0 +1,2 @@
+import Distribution.Simple
+main = defaultMain
diff --git a/testsuite/tests/driver/recompPluginPackage/p/p.cabal b/testsuite/tests/driver/recompPluginPackage/p/p.cabal
new file mode 100644
index 0000000000..ebf8f3dc5c
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/p/p.cabal
@@ -0,0 +1,24 @@
+cabal-version:       >=1.10
+-- Initial package description 'p.cabal' generated by 'cabal init'.  For
+-- further documentation, see http://haskell.org/cabal/users-guide/
+
+name:                p
+version:             0.1.0.0
+-- synopsis:
+-- description:
+-- bug-reports:
+-- license:
+license-file:        LICENSE
+author:              Matthew Pickering
+maintainer:          matthewtpickering@gmail.com
+-- copyright:
+-- category:
+build-type:          Simple
+extra-source-files:  CHANGELOG.md
+
+library
+  exposed-modules: Lib
+  -- other-extensions:
+  build-depends:       base >=4.14 && < 5
+  -- hs-source-dirs:
+  default-language:    Haskell2010
diff --git a/testsuite/tests/driver/recompPluginPackage/q/CHANGELOG.md b/testsuite/tests/driver/recompPluginPackage/q/CHANGELOG.md
new file mode 100644
index 0000000000..62632c5376
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/q/CHANGELOG.md
@@ -0,0 +1,5 @@
+# Revision history for q
+
+## 0.1.0.0 -- YYYY-mm-dd
+
+* First version. Released on an unsuspecting world.
diff --git a/testsuite/tests/driver/recompPluginPackage/q/LICENSE b/testsuite/tests/driver/recompPluginPackage/q/LICENSE
new file mode 100644
index 0000000000..bca70f3531
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/q/LICENSE
@@ -0,0 +1 @@
+q
diff --git a/testsuite/tests/driver/recompPluginPackage/q/Main.hs b/testsuite/tests/driver/recompPluginPackage/q/Main.hs
new file mode 100644
index 0000000000..60fd0cc6f5
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/q/Main.hs
@@ -0,0 +1,4 @@
+{-# OPTIONS_GHC -fplugin=Plugin #-}
+module Main where
+
+main = print ()
diff --git a/testsuite/tests/driver/recompPluginPackage/q/Plugin.hs b/testsuite/tests/driver/recompPluginPackage/q/Plugin.hs
new file mode 100644
index 0000000000..32ce1bd96b
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/q/Plugin.hs
@@ -0,0 +1,15 @@
+module Plugin where
+
+import GHC.Plugins
+import Lib
+
+plugin :: Plugin
+plugin = defaultPlugin {
+  installCoreToDos = install
+  , pluginRecompile = purePlugin
+  }
+
+install :: [CommandLineOption] -> [CoreToDo] -> CoreM [CoreToDo]
+install _ todo = do
+  liftIO (print p)
+  return todo
diff --git a/testsuite/tests/driver/recompPluginPackage/q/Setup.hs b/testsuite/tests/driver/recompPluginPackage/q/Setup.hs
new file mode 100644
index 0000000000..9a994af677
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/q/Setup.hs
@@ -0,0 +1,2 @@
+import Distribution.Simple
+main = defaultMain
diff --git a/testsuite/tests/driver/recompPluginPackage/q/q.cabal b/testsuite/tests/driver/recompPluginPackage/q/q.cabal
new file mode 100644
index 0000000000..8216113fce
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/q/q.cabal
@@ -0,0 +1,26 @@
+cabal-version:       >=1.10
+-- Initial package description 'q.cabal' generated by 'cabal init'.  For
+-- further documentation, see http://haskell.org/cabal/users-guide/
+
+name:                q
+version:             0.1.0.0
+-- synopsis:
+-- description:
+-- bug-reports:
+-- license:
+license-file:        LICENSE
+author:              Matthew Pickering
+maintainer:          matthewtpickering@gmail.com
+-- copyright:
+-- category:
+build-type:          Simple
+extra-source-files:  CHANGELOG.md
+
+executable q
+  main-is:             Main.hs
+  other-modules:       Plugin
+  -- other-extensions:
+  build-depends:       base >=4.14 && <5, p, ghc
+  -- hs-source-dirs:
+  ghc-options: -dynamic-too
+  default-language:    Haskell2010
diff --git a/testsuite/tests/driver/recompPluginPackage/recompPluginPackage.stderr-mingw32 b/testsuite/tests/driver/recompPluginPackage/recompPluginPackage.stderr-mingw32
new file mode 100644
index 0000000000..46497c6618
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/recompPluginPackage.stderr-mingw32
@@ -0,0 +1,12 @@
+
+when making flags consistent: warning:
+    -dynamic-too is not supported on Windows
+
+when making flags consistent: warning:
+    -dynamic-too is not supported on Windows
+
+when making flags consistent: warning:
+    -dynamic-too is not supported on Windows
+
+when making flags consistent: warning:
+    -dynamic-too is not supported on Windows
diff --git a/testsuite/tests/driver/recompPluginPackage/recompPluginPackage.stdout b/testsuite/tests/driver/recompPluginPackage/recompPluginPackage.stdout
new file mode 100644
index 0000000000..1191247b6d
--- /dev/null
+++ b/testsuite/tests/driver/recompPluginPackage/recompPluginPackage.stdout
@@ -0,0 +1,2 @@
+1
+2