Remote GHCi, -fexternal-interpreter

Summary:
(Apologies for the size of this patch, I couldn't make a smaller one
that was validate-clean and also made sense independently)

(Some of this code is derived from GHCJS.)

This commit adds support for running interpreted code (for GHCi and
TemplateHaskell) in a separate process.  The functionality is
experimental, so for now it is off by default and enabled by the flag
-fexternal-interpreter.

Reaosns we want this:

* compiling Template Haskell code with -prof does not require
  building the code without -prof first

* when GHC itself is profiled, it can interpret unprofiled code, and
  the same applies to dynamic linking.  We would no longer need to
  force -dynamic-too with TemplateHaskell, and we can load ordinary
  objects into a dynamically-linked GHCi (and vice versa).

* An unprofiled GHCi can load and run profiled code, which means it
  can use the stack-trace functionality provided by profiling without
  taking the performance hit on the compiler that profiling would
  entail.

Amongst other things; see
https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details.

Notes on the implementation are in Note [Remote GHCi] in the new
module compiler/ghci/GHCi.hs.  It probably needs more documenting,
feel free to suggest things I could elaborate on.

Things that are not currently implemented for -fexternal-interpreter:

* The GHCi debugger
* :set prog, :set args in GHCi
* `recover` in Template Haskell
* Redirecting stdin/stdout for the external process

These are all doable, I just wanted to get to a working validate-clean
patch first.

I also haven't done any benchmarking yet.  I expect there to be slight hit
to link times for byte code and some penalty due to having to
serialize/deserialize TH syntax, but I don't expect it to be a serious
problem.  There's also lots of low-hanging fruit in the byte code
generator/linker that we could exploit to speed things up.

Test Plan:
* validate
* I've run parts of the test suite with
EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th.
There are a few failures due to the things not currently implemented
(see above).

Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1562

8 files changed, 56 insertions, 2 deletions

diff --git a/testsuite/tests/th/Makefile b/testsuite/tests/th/Makefile
index 5d2be1eb34..b759a81b67 100644
--- a/testsuite/tests/th/Makefile
+++ b/testsuite/tests/th/Makefile
@@ -20,6 +20,13 @@ TH_spliceE5_prof::
 	'$(TEST_HC)' $(TEST_HC_OPTS) $(HC_OPTS) --make -v0 TH_spliceE5_prof.hs -prof -auto-all -osuf p.o -o $@
 	./$@
 
+# With -fexternal-interpreter, we don't have to build the non-profiled
+# objects first.
+TH_spliceE5_prof_ext::
+	$(RM) TH_spliceE5_prof_ext*.o TH_spliceE5_prof_ext*.hi TH_spliceE5_prof_ext*.p.o
+	'$(TEST_HC)' $(TEST_HC_OPTS) $(HC_OPTS) --make -v0 TH_spliceE5_prof_ext.hs -prof -auto-all -fexternal-interpreter -o $@
+	./$@
+
 .PHONY: TH_Depends
 TH_Depends:
 	$(RM) TH_Depends_external.txt
diff --git a/testsuite/tests/th/TH_Roles2.stderr b/testsuite/tests/th/TH_Roles2.stderr
index e4a9a47858..c44e5b9d9d 100644
--- a/testsuite/tests/th/TH_Roles2.stderr
+++ b/testsuite/tests/th/TH_Roles2.stderr
@@ -5,9 +5,10 @@ TYPE CONSTRUCTORS
     Kind: forall k1. k1 -> *
 COERCION AXIOMS
 Dependent modules: []
-Dependent packages: [array-0.5.1.0, base-4.9.0.0, deepseq-1.4.2.0,
+Dependent packages: [array-0.5.1.0, base-4.9.0.0, binary-0.7.5.0,
+                     bytestring-0.10.7.0, containers-0.5.6.3, deepseq-1.4.2.0,
                      ghc-boot-0.0.0.0, ghc-prim-0.5.0.0, integer-gmp-1.0.0.0,
-                     pretty-1.1.2.0, template-haskell-2.11.0.0]
+                     pretty-1.1.3.2, template-haskell-2.11.0.0]
 
 ==================== Typechecker ====================
 TH_Roles2.$tcT
diff --git a/testsuite/tests/th/TH_finalizer.hs b/testsuite/tests/th/TH_finalizer.hs
new file mode 100644
index 0000000000..f59364edaf
--- /dev/null
+++ b/testsuite/tests/th/TH_finalizer.hs
@@ -0,0 +1,11 @@
+{-# LANGUAGE TemplateHaskell #-}
+
+module ShouldCompile where
+
+import Language.Haskell.TH
+import Language.Haskell.TH.Syntax
+
+$( do
+     addModFinalizer (do b <- getQ; reportWarning (show (b::Maybe Bool)))
+     return [] )
+$( putQ True >> return [] )
diff --git a/testsuite/tests/th/TH_finalizer.stderr b/testsuite/tests/th/TH_finalizer.stderr
new file mode 100644
index 0000000000..e89d434adb
--- /dev/null
+++ b/testsuite/tests/th/TH_finalizer.stderr
@@ -0,0 +1,2 @@
+
+TH_finalizer.hs:1:1: warning: Just True
diff --git a/testsuite/tests/th/TH_spliceE5_prof_ext.hs b/testsuite/tests/th/TH_spliceE5_prof_ext.hs
new file mode 100644
index 0000000000..255b1c5b8d
--- /dev/null
+++ b/testsuite/tests/th/TH_spliceE5_prof_ext.hs
@@ -0,0 +1,14 @@
+module Main where
+
+import TH_spliceE5_prof_ext_Lib
+
+v1 = "foo"
+
+main = putStrLn $(expandVars ["v1","v2"])
+-- The splice expands to refer to both v1 and v2,
+-- and the test checks that we don't dependency-analyse
+-- the program so that one or the other isn't in scope
+-- to the type checker
+
+
+v2 = "bar"
diff --git a/testsuite/tests/th/TH_spliceE5_prof_ext.stdout b/testsuite/tests/th/TH_spliceE5_prof_ext.stdout
new file mode 100644
index 0000000000..323fae03f4
--- /dev/null
+++ b/testsuite/tests/th/TH_spliceE5_prof_ext.stdout
@@ -0,0 +1 @@
+foobar
diff --git a/testsuite/tests/th/TH_spliceE5_prof_ext_Lib.hs b/testsuite/tests/th/TH_spliceE5_prof_ext_Lib.hs
new file mode 100644
index 0000000000..eb598c03d7
--- /dev/null
+++ b/testsuite/tests/th/TH_spliceE5_prof_ext_Lib.hs
@@ -0,0 +1,8 @@
+module TH_spliceE5_prof_ext_Lib where
+
+import Language.Haskell.TH
+
+expandVars :: [String] -> Q Exp
+expandVars s = [| concat $(return (ListE (map f s))) |]
+  where
+    f x = VarE (mkName x)
diff --git a/testsuite/tests/th/all.T b/testsuite/tests/th/all.T
index af8531c8b1..45ee2df13b 100644
--- a/testsuite/tests/th/all.T
+++ b/testsuite/tests/th/all.T
@@ -51,6 +51,14 @@ test('TH_spliceE5_prof',
      run_command,
      ['$MAKE -s --no-print-directory TH_spliceE5_prof'])
 
+test('TH_spliceE5_prof_ext',
+     [req_profiling,
+      omit_ways(['ghci']),
+      extra_clean(['TH_spliceE5_prof_ext_Lib.hi',
+                   'TH_spliceE5_prof_ext_Lib.o'])],
+     run_command,
+     ['$MAKE -s --no-print-directory TH_spliceE5_prof_ext'])
+
 test('TH_spliceD1',
      extra_clean(['TH_spliceD1_Lib.hi', 'TH_spliceD1_Lib.o']),
      multimod_compile_fail,
@@ -370,3 +378,5 @@ test('T10819',
      multimod_compile,
      ['T10819.hs', '-v0 ' + config.ghc_th_way_flags])
 test('T10820', normal, compile_and_run, ['-v0'])
+
+test('TH_finalizer', normal, compile, ['-v0'])