Remote GHCi, -fexternal-interpreter

Summary:
(Apologies for the size of this patch, I couldn't make a smaller one
that was validate-clean and also made sense independently)

(Some of this code is derived from GHCJS.)

This commit adds support for running interpreted code (for GHCi and
TemplateHaskell) in a separate process.  The functionality is
experimental, so for now it is off by default and enabled by the flag
-fexternal-interpreter.

Reaosns we want this:

* compiling Template Haskell code with -prof does not require
  building the code without -prof first

* when GHC itself is profiled, it can interpret unprofiled code, and
  the same applies to dynamic linking.  We would no longer need to
  force -dynamic-too with TemplateHaskell, and we can load ordinary
  objects into a dynamically-linked GHCi (and vice versa).

* An unprofiled GHCi can load and run profiled code, which means it
  can use the stack-trace functionality provided by profiling without
  taking the performance hit on the compiler that profiling would
  entail.

Amongst other things; see
https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details.

Notes on the implementation are in Note [Remote GHCi] in the new
module compiler/ghci/GHCi.hs.  It probably needs more documenting,
feel free to suggest things I could elaborate on.

Things that are not currently implemented for -fexternal-interpreter:

* The GHCi debugger
* :set prog, :set args in GHCi
* `recover` in Template Haskell
* Redirecting stdin/stdout for the external process

These are all doable, I just wanted to get to a working validate-clean
patch first.

I also haven't done any benchmarking yet.  I expect there to be slight hit
to link times for byte code and some penalty due to having to
serialize/deserialize TH syntax, but I don't expect it to be a serious
problem.  There's also lots of low-hanging fruit in the byte code
generator/linker that we could exploit to speed things up.

Test Plan:
* validate
* I've run parts of the test suite with
EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th.
There are a few failures due to the things not currently implemented
(see above).

Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1562

2 files changed, 14 insertions, 6 deletions

diff --git a/rules/build-prog.mk b/rules/build-prog.mk
index 1a497078c5..8352d88412 100644
--- a/rules/build-prog.mk
+++ b/rules/build-prog.mk
@@ -48,7 +48,6 @@ endef
 
 
 
-
 define build-prog-vars
 # $1 = dir
 # $2 = distdir
@@ -92,18 +91,27 @@ else
 ifneq "$$($$($1_$2_PROGNAME)_INPLACE)" ""
 $$(error $$($1_$2_PROGNAME)_INPLACE defined twice)
 endif
+#
+# Where do we install the wrapper and the binary?
+#   $$($1_$2_PROGNAME)_INPLACE  The thing we run (script or binary)
+#   $1_$2_INPLACE               The binary
+#
 ifeq "$$($1_$2_TOPDIR)" "YES"
-$$($1_$2_PROGNAME)_INPLACE = $$(INPLACE_TOPDIR)/$$($1_$2_PROG)
+$$($1_$2_PROGNAME)_INPLACE = $$(INPLACE_LIB)/bin/$$($1_$2_PROG)
+ifeq "$$($1_$2_WANT_INPLACE_WRAPPER)" "YES"
+$1_$2_INPLACE = $$(INPLACE_LIB)/bin/$$($1_$2_PROG).bin
 else
-$$($1_$2_PROGNAME)_INPLACE = $$(INPLACE_BIN)/$$($1_$2_PROG)
+$1_$2_INPLACE = $$(INPLACE_LIB)/bin/$$($1_$2_PROG)
 endif
-# Where do we install the inplace version?
+else
+$$($1_$2_PROGNAME)_INPLACE = $$(INPLACE_BIN)/$$($1_$2_PROG)
 ifeq "$$($1_$2_WANT_INPLACE_WRAPPER)" "YES"
 $1_$2_INPLACE = $$(INPLACE_LIB)/bin/$$($1_$2_PROG)
 else
 $1_$2_INPLACE = $$($$($1_$2_PROGNAME)_INPLACE)
 endif
 endif
+endif
 
 endef
 
@@ -297,7 +305,7 @@ endif
 ifeq "$$($1_$2_WANT_INSTALLED_WRAPPER)" "YES"
 INSTALL_LIBEXECS += $1/$2/build/tmp/$$($1_$2_PROG)
 else ifeq "$$($1_$2_TOPDIR)" "YES"
-INSTALL_TOPDIR_BINS += $1/$2/build/tmp/$$($1_$2_PROG)
+INSTALL_LIBEXECS += $1/$2/build/tmp/$$($1_$2_PROG)
 else
 INSTALL_BINS     += $1/$2/build/tmp/$$($1_$2_PROG)
 endif
diff --git a/rules/shell-wrapper.mk b/rules/shell-wrapper.mk
index 11eef67a83..f674c0dfc2 100644
--- a/rules/shell-wrapper.mk
+++ b/rules/shell-wrapper.mk
@@ -26,7 +26,7 @@ ifeq "$$($1_$2_WANT_INPLACE_WRAPPER)" "YES"
 $1_$2_INPLACE_SHELL_WRAPPER_NAME = $$($1_$2_PROG)
 
 ifeq "$$($1_$2_TOPDIR)" "YES"
-$1_$2_INPLACE_WRAPPER = $$(INPLACE_LIB)/$$($1_$2_INPLACE_SHELL_WRAPPER_NAME)
+$1_$2_INPLACE_WRAPPER = $$(INPLACE_LIB)/bin/$$($1_$2_INPLACE_SHELL_WRAPPER_NAME)
 else
 $1_$2_INPLACE_WRAPPER = $$(INPLACE_BIN)/$$($1_$2_INPLACE_SHELL_WRAPPER_NAME)
 endif