Remote GHCi, -fexternal-interpreter

Summary:
(Apologies for the size of this patch, I couldn't make a smaller one
that was validate-clean and also made sense independently)

(Some of this code is derived from GHCJS.)

This commit adds support for running interpreted code (for GHCi and
TemplateHaskell) in a separate process.  The functionality is
experimental, so for now it is off by default and enabled by the flag
-fexternal-interpreter.

Reaosns we want this:

* compiling Template Haskell code with -prof does not require
  building the code without -prof first

* when GHC itself is profiled, it can interpret unprofiled code, and
  the same applies to dynamic linking.  We would no longer need to
  force -dynamic-too with TemplateHaskell, and we can load ordinary
  objects into a dynamically-linked GHCi (and vice versa).

* An unprofiled GHCi can load and run profiled code, which means it
  can use the stack-trace functionality provided by profiling without
  taking the performance hit on the compiler that profiling would
  entail.

Amongst other things; see
https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details.

Notes on the implementation are in Note [Remote GHCi] in the new
module compiler/ghci/GHCi.hs.  It probably needs more documenting,
feel free to suggest things I could elaborate on.

Things that are not currently implemented for -fexternal-interpreter:

* The GHCi debugger
* :set prog, :set args in GHCi
* `recover` in Template Haskell
* Redirecting stdin/stdout for the external process

These are all doable, I just wanted to get to a working validate-clean
patch first.

I also haven't done any benchmarking yet.  I expect there to be slight hit
to link times for byte code and some penalty due to having to
serialize/deserialize TH syntax, but I don't expect it to be a serious
problem.  There's also lots of low-hanging fruit in the byte code
generator/linker that we could exploit to speed things up.

Test Plan:
* validate
* I've run parts of the test suite with
EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th.
There are a few failures due to the things not currently implemented
(see above).

Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1562

4 files changed, 15 insertions, 6 deletions

diff --git a/testsuite/tests/ghci/prog001/prog001-ext.stdout b/testsuite/tests/ghci/prog001/prog001-ext.stdout
new file mode 100644
index 0000000000..eef24c2153
--- /dev/null
+++ b/testsuite/tests/ghci/prog001/prog001-ext.stdout
@@ -0,0 +1,4 @@
+"hello world84"
+"hello world84"
+"hello world84"
+44
diff --git a/testsuite/tests/ghci/prog001/prog001.T b/testsuite/tests/ghci/prog001/prog001.T
index 1ef3c0430c..af221e6fd9 100644
--- a/testsuite/tests/ghci/prog001/prog001.T
+++ b/testsuite/tests/ghci/prog001/prog001.T
@@ -1,5 +1,5 @@
 test('prog001',
      [extra_clean(['C.hs', 'D.hs', 'D.hi', 'D.o']),
-      cmd_prefix('ghciWayFlags=' + config.ghci_way_flags)],
+      cmd_prefix('ghciWayFlags=' + config.ghci_way_flags),
+      unless(opsys('mingw32'),extra_ways(['ghci-ext']))],
      ghci_script, ['prog001.script'])
-
diff --git a/testsuite/tests/ghci/scripts/T10110A.hs b/testsuite/tests/ghci/scripts/T10110A.hs
index 8482e7f673..ab64cfd016 100644
--- a/testsuite/tests/ghci/scripts/T10110A.hs
+++ b/testsuite/tests/ghci/scripts/T10110A.hs
@@ -1,4 +1,5 @@
 module T10110A (a) where
+import Debug.Trace
 {-# NOINLINE a #-}
 a :: Int
 a = 3
diff --git a/testsuite/tests/ghci/scripts/all.T b/testsuite/tests/ghci/scripts/all.T
index 1a664d128d..5c25cf8bb0 100755
--- a/testsuite/tests/ghci/scripts/all.T
+++ b/testsuite/tests/ghci/scripts/all.T
@@ -3,9 +3,13 @@
 test('ghci001', combined_output, ghci_script, ['ghci001.script'])
 test('ghci002', combined_output, ghci_script, ['ghci002.script'])
 test('ghci003', combined_output, ghci_script, ['ghci003.script'])
-test('ghci004', combined_output, ghci_script, ['ghci004.script'])
+test('ghci004', [ combined_output,
+                  unless(opsys('mingw32'),extra_ways(['ghci-ext'])) ],
+                ghci_script, ['ghci004.script'])
 test('ghci005', combined_output, ghci_script, ['ghci005.script'])
-test('ghci006', combined_output, ghci_script, ['ghci006.script'])
+test('ghci006', [ combined_output,
+                  unless(opsys('mingw32'),extra_ways(['ghci-ext'])) ],
+                ghci_script, ['ghci006.script'])
 test('ghci007', combined_output, ghci_script, ['ghci007.script'])
 test('ghci008', combined_output, ghci_script, ['ghci008.script'])
 test('ghci009', combined_output, ghci_script, ['ghci009.script'])
@@ -84,7 +88,7 @@ test('ghci056',
      ],
      ghci_script, ['ghci056.script'])
 
-test('ghci057', normal, ghci_script_without_flag('-fno-warn-tabs'), ['ghci057.script'])
+test('ghci057', extra_hc_opts('-fwarn-tabs'), ghci_script, ['ghci057.script'])
 
 test('T2452', normal, ghci_script, ['T2452.script'])
 test('T2766', normal, ghci_script, ['T2766.script'])
@@ -192,7 +196,7 @@ test('T9181', normal, ghci_script, ['T9181.script'])
 test('T9086b', normal, ghci_script, ['T9086b.script'])
 test('T9140', combined_output, ghci_script, ['T9140.script'])
 test('T9658', normal, ghci_script, ['T9658.script'])
-test('T9293', normal, ghci_script_without_flag('-fno-warn-tabs'), ['T9293.script'])
+test('T9293', extra_hc_opts('-fwarn-tabs'), ghci_script, ['T9293.script'])
 test('T9762',
  [ unless(have_dynamic(),skip)
  , pre_cmd('$MAKE -s --no-print-directory T9762_prep')