Remote GHCi, -fexternal-interpreter

Summary:
(Apologies for the size of this patch, I couldn't make a smaller one
that was validate-clean and also made sense independently)

(Some of this code is derived from GHCJS.)

This commit adds support for running interpreted code (for GHCi and
TemplateHaskell) in a separate process.  The functionality is
experimental, so for now it is off by default and enabled by the flag
-fexternal-interpreter.

Reaosns we want this:

* compiling Template Haskell code with -prof does not require
  building the code without -prof first

* when GHC itself is profiled, it can interpret unprofiled code, and
  the same applies to dynamic linking.  We would no longer need to
  force -dynamic-too with TemplateHaskell, and we can load ordinary
  objects into a dynamically-linked GHCi (and vice versa).

* An unprofiled GHCi can load and run profiled code, which means it
  can use the stack-trace functionality provided by profiling without
  taking the performance hit on the compiler that profiling would
  entail.

Amongst other things; see
https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi for more details.

Notes on the implementation are in Note [Remote GHCi] in the new
module compiler/ghci/GHCi.hs.  It probably needs more documenting,
feel free to suggest things I could elaborate on.

Things that are not currently implemented for -fexternal-interpreter:

* The GHCi debugger
* :set prog, :set args in GHCi
* `recover` in Template Haskell
* Redirecting stdin/stdout for the external process

These are all doable, I just wanted to get to a working validate-clean
patch first.

I also haven't done any benchmarking yet.  I expect there to be slight hit
to link times for byte code and some penalty due to having to
serialize/deserialize TH syntax, but I don't expect it to be a serious
problem.  There's also lots of low-hanging fruit in the byte code
generator/linker that we could exploit to speed things up.

Test Plan:
* validate
* I've run parts of the test suite with
EXTRA_HC_OPTS=-fexternal-interpreter, notably tests/ghci and tests/th.
There are a few failures due to the things not currently implemented
(see above).

Reviewers: simonpj, goldfire, ezyang, austin, alanz, hvr, niteria, bgamari, gibiansky, luite

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1562

1 files changed, 90 insertions, 0 deletions

diff --git a/compiler/ghci/ByteCodeTypes.hs b/compiler/ghci/ByteCodeTypes.hs
new file mode 100644
index 0000000000..0a8dd304b6
--- /dev/null
+++ b/compiler/ghci/ByteCodeTypes.hs
@@ -0,0 +1,90 @@
+{-# LANGUAGE MagicHash #-}
+--
+--  (c) The University of Glasgow 2002-2006
+--
+
+-- | Bytecode assembler types
+module ByteCodeTypes
+  ( CompiledByteCode(..), FFIInfo(..)
+  , UnlinkedBCO(..), BCOPtr(..), BCONPtr(..)
+  , ItblEnv, ItblPtr(..)
+  , BreakInfo(..)
+  ) where
+
+import FastString
+import Id
+import Module
+import Name
+import NameEnv
+import Outputable
+import PrimOp
+import SizedSeq
+import Type
+
+import Foreign
+import Data.Array.Base  ( UArray(..) )
+import Data.ByteString (ByteString)
+import GHC.Exts
+
+
+data CompiledByteCode
+  = ByteCode [UnlinkedBCO] -- Bunch of interpretable bindings
+             ItblEnv       -- A mapping from DataCons to their itbls
+             [FFIInfo]     -- ffi blocks we allocated
+                -- ToDo: we're not tracking strings that we malloc'd
+
+newtype FFIInfo = FFIInfo (Ptr ())
+  deriving Show
+
+instance Outputable CompiledByteCode where
+  ppr (ByteCode bcos _ _) = ppr bcos
+
+type ItblEnv = NameEnv (Name, ItblPtr)
+        -- We need the Name in the range so we know which
+        -- elements to filter out when unloading a module
+
+newtype ItblPtr = ItblPtr (Ptr ()) deriving Show
+
+data UnlinkedBCO
+   = UnlinkedBCO {
+        unlinkedBCOName   :: Name,
+        unlinkedBCOArity  :: Int,
+        unlinkedBCOInstrs :: UArray Int Word16,         -- insns
+        unlinkedBCOBitmap :: UArray Int Word,           -- bitmap
+        unlinkedBCOLits   :: (SizedSeq BCONPtr),        -- non-ptrs
+        unlinkedBCOPtrs   :: (SizedSeq BCOPtr)          -- ptrs
+   }
+
+data BCOPtr
+  = BCOPtrName   Name
+  | BCOPtrPrimOp PrimOp
+  | BCOPtrBCO    UnlinkedBCO
+  | BCOPtrBreakInfo  BreakInfo
+  | BCOPtrArray (MutableByteArray# RealWorld)
+
+data BCONPtr
+  = BCONPtrWord  Word
+  | BCONPtrLbl   FastString
+  | BCONPtrItbl  Name
+  | BCONPtrStr   ByteString
+
+data BreakInfo
+   = BreakInfo
+   { breakInfo_module :: Module
+   , breakInfo_number :: {-# UNPACK #-} !Int
+   , breakInfo_vars   :: [(Id,Word16)]
+   , breakInfo_resty  :: Type
+   }
+
+instance Outputable UnlinkedBCO where
+   ppr (UnlinkedBCO nm _arity _insns _bitmap lits ptrs)
+      = sep [text "BCO", ppr nm, text "with",
+             ppr (sizeSS lits), text "lits",
+             ppr (sizeSS ptrs), text "ptrs" ]
+
+instance Outputable BreakInfo where
+   ppr info = text "BreakInfo" <+>
+              parens (ppr (breakInfo_module info) <+>
+                      ppr (breakInfo_number info) <+>
+                      ppr (breakInfo_vars info) <+>
+                      ppr (breakInfo_resty info))