Remote GHCi, -fexternal-interpreter

Summary:
(Apologies for the size of this patch, I couldn't make a smaller one
that was validate-clean and also made sense independently)

(Some of this code is derived from GHCJS.)

This commit adds support for running interpreted code (for GHCi and
TemplateHaskell) in a separate process.  The functionality is
experimental, so for now it is off by default and enabled by the flag
-fexternal-interpreter.

Reaosns we want this:

* compiling Template Haskell code with -prof does not require
  building the code without -prof first

* when GHC itself is profiled, it can interpret unprofiled code, and
  the same applies to dynamic linking.  We would no longer need to
  force -dynamic-too with TemplateHaskell, and we can load ordinary
  objects into a dynamically-linked GHCi (and vice versa).

* An unprofiled GHCi can load and run profiled code, which means it
  can use the stack-trace functionality provided by profiling without
  taking the performance hit on the compiler that profiling would
  entail.

Amongst other things; see
https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details.

Notes on the implementation are in Note [Remote GHCi] in the new
module compiler/ghci/GHCi.hs.  It probably needs more documenting,
feel free to suggest things I could elaborate on.

Things that are not currently implemented for -fexternal-interpreter:

* The GHCi debugger
* :set prog, :set args in GHCi
* `recover` in Template Haskell
* Redirecting stdin/stdout for the external process

These are all doable, I just wanted to get to a working validate-clean
patch first.

I also haven't done any benchmarking yet.  I expect there to be slight hit
to link times for byte code and some penalty due to having to
serialize/deserialize TH syntax, but I don't expect it to be a serious
problem.  There's also lots of low-hanging fruit in the byte code
generator/linker that we could exploit to speed things up.

Test Plan:
* validate
* I've run parts of the test suite with
EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th.
There are a few failures due to the things not currently implemented
(see above).

Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1562

6 files changed, 22 insertions, 5 deletions

diff --git a/testsuite/tests/ghci.debugger/scripts/break006.stderr b/testsuite/tests/ghci.debugger/scripts/break006.stderr
index 16ba5ec004..58faa697c5 100644
--- a/testsuite/tests/ghci.debugger/scripts/break006.stderr
+++ b/testsuite/tests/ghci.debugger/scripts/break006.stderr
@@ -10,7 +10,7 @@
         instance Show Ordering -- Defined in ‘GHC.Show’
         instance Show Integer -- Defined in ‘GHC.Show’
         ...plus 23 others
-        ...plus 13 instance involving out-of-scope typess
+        ...plus 19 instance involving out-of-scope typess
         (use -fprint-potential-instances to see them all)
     • In a stmt of an interactive GHCi command: print it
 
@@ -25,6 +25,6 @@
         instance Show Ordering -- Defined in ‘GHC.Show’
         instance Show Integer -- Defined in ‘GHC.Show’
         ...plus 23 others
-        ...plus 13 instance involving out-of-scope typess
+        ...plus 19 instance involving out-of-scope typess
         (use -fprint-potential-instances to see them all)
     • In a stmt of an interactive GHCi command: print it
diff --git a/testsuite/tests/ghci.debugger/scripts/break011.script b/testsuite/tests/ghci.debugger/scripts/break011.script
index f9ca3fe393..e913a2f240 100644
--- a/testsuite/tests/ghci.debugger/scripts/break011.script
+++ b/testsuite/tests/ghci.debugger/scripts/break011.script
@@ -15,4 +15,7 @@ error "foo"
 :force _exception
 :show bindings
 :force _result
+-- the exception is caught by the withProg/withArgs wrappers, hence 3 :continues
+:continue
+:continue
 :continue
diff --git a/testsuite/tests/ghci.debugger/scripts/break011.stdout b/testsuite/tests/ghci.debugger/scripts/break011.stdout
index dee4d94360..ec0b3e9609 100644
--- a/testsuite/tests/ghci.debugger/scripts/break011.stdout
+++ b/testsuite/tests/ghci.debugger/scripts/break011.stdout
@@ -32,6 +32,18 @@ _exception :: SomeException = SomeException
 *** Exception: foo
 CallStack (from ImplicitParams):
   error, called at ../Test7.hs:2:18 in main:Main
+Stopped at <exception thrown>
+_exception :: e = SomeException
+                    (ErrorCallWithLocation
+                       "foo"
+                       "CallStack (from ImplicitParams):
+  error, called at ../Test7.hs:2:18 in main:Main")
+Stopped at <exception thrown>
+_exception :: e = SomeException
+                    (ErrorCallWithLocation
+                       "foo"
+                       "CallStack (from ImplicitParams):
+  error, called at ../Test7.hs:2:18 in main:Main")
 *** Exception: foo
 CallStack (from ImplicitParams):
   error, called at ../Test7.hs:2:18 in main:Main
diff --git a/testsuite/tests/ghci.debugger/scripts/break013.stdout b/testsuite/tests/ghci.debugger/scripts/break013.stdout
index 4c3d5f3d76..13d203f0b3 100644
--- a/testsuite/tests/ghci.debugger/scripts/break013.stdout
+++ b/testsuite/tests/ghci.debugger/scripts/break013.stdout
@@ -5,7 +5,7 @@ _result :: (Bool, Bool, ()) = _
 a :: Bool = _
 b :: Bool = _
 c :: () = _
-c :: () = _
 b :: Bool = _
+c :: () = _
 a :: Bool = _
 _result :: (Bool, Bool, ()) = _
diff --git a/testsuite/tests/ghci.debugger/scripts/break024.stdout b/testsuite/tests/ghci.debugger/scripts/break024.stdout
index dc3bd73425..548e7a4470 100644
--- a/testsuite/tests/ghci.debugger/scripts/break024.stdout
+++ b/testsuite/tests/ghci.debugger/scripts/break024.stdout
@@ -10,7 +10,9 @@ _exception :: e = _
 _exception = SomeException
                (GHC.IO.Exception.IOError
                   Nothing GHC.IO.Exception.UserError [] "error" Nothing Nothing)
-*** Exception: user error (error)
+Stopped at <exception thrown>
+_exception :: e = SomeException
+                    (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError ....)
 Stopped at <exception thrown>
 _exception :: e = _
 _exception = SomeException
diff --git a/testsuite/tests/ghci.debugger/scripts/print019.stderr b/testsuite/tests/ghci.debugger/scripts/print019.stderr
index fa53d94701..c1dc48bf21 100644
--- a/testsuite/tests/ghci.debugger/scripts/print019.stderr
+++ b/testsuite/tests/ghci.debugger/scripts/print019.stderr
@@ -9,6 +9,6 @@
         instance Show Ordering -- Defined in ‘GHC.Show’
         instance Show TyCon -- Defined in ‘GHC.Show’
         ...plus 30 others
-        ...plus two instance involving out-of-scope typess
+        ...plus 8 instance involving out-of-scope typess
         (use -fprint-potential-instances to see them all)
     • In a stmt of an interactive GHCi command: print it