Remote GHCi, -fexternal-interpreter

Summary:
(Apologies for the size of this patch, I couldn't make a smaller one
that was validate-clean and also made sense independently)

(Some of this code is derived from GHCJS.)

This commit adds support for running interpreted code (for GHCi and
TemplateHaskell) in a separate process.  The functionality is
experimental, so for now it is off by default and enabled by the flag
-fexternal-interpreter.

Reaosns we want this:

* compiling Template Haskell code with -prof does not require
  building the code without -prof first

* when GHC itself is profiled, it can interpret unprofiled code, and
  the same applies to dynamic linking.  We would no longer need to
  force -dynamic-too with TemplateHaskell, and we can load ordinary
  objects into a dynamically-linked GHCi (and vice versa).

* An unprofiled GHCi can load and run profiled code, which means it
  can use the stack-trace functionality provided by profiling without
  taking the performance hit on the compiler that profiling would
  entail.

Amongst other things; see
https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details.

Notes on the implementation are in Note [Remote GHCi] in the new
module compiler/ghci/GHCi.hs.  It probably needs more documenting,
feel free to suggest things I could elaborate on.

Things that are not currently implemented for -fexternal-interpreter:

* The GHCi debugger
* :set prog, :set args in GHCi
* `recover` in Template Haskell
* Redirecting stdin/stdout for the external process

These are all doable, I just wanted to get to a working validate-clean
patch first.

I also haven't done any benchmarking yet.  I expect there to be slight hit
to link times for byte code and some penalty due to having to
serialize/deserialize TH syntax, but I don't expect it to be a serious
problem.  There's also lots of low-hanging fruit in the byte code
generator/linker that we could exploit to speed things up.

Test Plan:
* validate
* I've run parts of the test suite with
EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th.
There are a few failures due to the things not currently implemented
(see above).

Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1562

1 files changed, 5 insertions, 18 deletions

diff --git a/testsuite/driver/testlib.py b/testsuite/driver/testlib.py
index 60e5d465ac..c41bb8cb65 100644
--- a/testsuite/driver/testlib.py
+++ b/testsuite/driver/testlib.py
@@ -207,7 +207,7 @@ def _extra_ways( name, opts, ways ):
 
 def only_compiler_types( _compiler_types ):
    # Don't delete yet. The libraries unix, stm and hpc still call this function.
-   return lambda _name, _opts: None 
+   return lambda _name, _opts: None
 
 # -----
 
@@ -949,25 +949,12 @@ def run_command( name, way, cmd ):
 # -----------------------------------------------------------------------------
 # GHCi tests
 
-def ghci_script_without_flag(flag):
-    def apply(name, way, script):
-        overrides = [f for f in getTestOpts().compiler_always_flags if f != flag]
-        return ghci_script_override_default_flags(overrides)(name, way, script)
-
-    return apply
-
-def ghci_script_override_default_flags(overrides):
-    def apply(name, way, script):
-        return ghci_script(name, way, script, overrides)
-
-    return apply
-
 def ghci_script( name, way, script, override_flags = None ):
     # filter out -fforce-recomp from compiler_always_flags, because we're
     # actually testing the recompilation behaviour in the GHCi tests.
     flags = ' '.join(get_compiler_flags(override_flags, noforce=True))
 
-    way_flags = ' '.join(config.way_flags(name)['ghci'])
+    way_flags = ' '.join(config.way_flags(name)[way])
 
     # We pass HC and HC_OPTS as environment variables, so that the
     # script can invoke the correct compiler by using ':! $HC $HC_OPTS'
@@ -1093,7 +1080,7 @@ def compile_and_run__( name, way, top_mod, extra_mods, extra_hc_opts ):
        return result
     extra_hc_opts = result['hc_opts']
 
-    if way == 'ghci': # interpreted...
+    if way.startswith('ghci'): # interpreted...
         return interpreter_run( name, way, extra_hc_opts, 0, top_mod )
     else: # compiled...
         force = 0
@@ -1174,7 +1161,7 @@ def checkStats(name, way, stats_file, range_fields):
                 display('    Actual      ' + full_name + ' ' + field + ':', val, '')
                 if val != expected:
                     display('    Deviation   ' + full_name + ' ' + field + ':', deviation, '%')
-                
+
     return result
 
 # -----------------------------------------------------------------------------
@@ -2031,7 +2018,7 @@ def addTestFilesWrittenHelper(name, fn):
                 pass
             else:
                 framework_fail(name, 'strace', "Can't understand strace line: " + line)
- 
+
 def checkForFilesWrittenProblems(file):
     foundProblem = False