path: root/docs/backpack/algorithm.tex

\documentclass{article}

\usepackage{pifont}
\usepackage{graphicx} %[pdftex] OR [dvips]
\usepackage{fullpage}
\usepackage{wrapfig}
\usepackage{float}
\usepackage{titling}
\usepackage{hyperref}
\usepackage{tikz}
\usepackage{color}
\usepackage{footnote}
\usepackage{float}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{bigfoot}
\usepackage{amssymb}
\usepackage{framed}

% Alter some LaTeX defaults for better treatment of figures:
% See p.105 of "TeX Unbound" for suggested values.
% See pp. 199-200 of Lamport's "LaTeX" book for details.
%   General parameters, for ALL pages:
\renewcommand{\topfraction}{0.9}	% max fraction of floats at top
\renewcommand{\bottomfraction}{0.8}	% max fraction of floats at bottom
%   Parameters for TEXT pages (not float pages):
\setcounter{topnumber}{2}
\setcounter{bottomnumber}{2}
\setcounter{totalnumber}{4}     % 2 may work better
\setcounter{dbltopnumber}{2}    % for 2-column pages
\renewcommand{\dbltopfraction}{0.9}	% fit big float above 2-col. text
\renewcommand{\textfraction}{0.07}	% allow minimal text w. figs
%   Parameters for FLOAT pages (not text pages):
\renewcommand{\floatpagefraction}{0.7}	% require fuller float pages
% N.B.: floatpagefraction MUST be less than topfraction !!
\renewcommand{\dblfloatpagefraction}{0.7}	% require fuller float pages

% remember to use [htp] or [htpb] for placement

\newcommand{\I}[1]{\ensuremath{\mathit{#1}}}

\newcommand{\optionrule}{\noindent\rule{1.0\textwidth}{0.75pt}}

\newenvironment{aside}
  {\begin{figure}\def\FrameCommand{\hspace{2em}}
   \MakeFramed {\advance\hsize-\width}\optionrule\small}
{\par\vskip-\smallskipamount\optionrule\endMakeFramed\end{figure}}

\setlength{\droptitle}{-6em}

\newcommand{\Red}[1]{{\color{red} #1}}

\title{The Backpack algorithm}

\begin{document}

\maketitle

This document describes the Backpack shaping and typechecking
passes, as we intend to implement it.

\section{Changelog}

\paragraph{April 28, 2015}  A signature declaration no longer provides
a signature in the technical shaping sense; the motivation for this change
is explained in \textbf{In-scope signatures are not provisions}.  The simplest
consequence of this is that all requirements are importable (Derek has stated that he doesn't
think this will be too much of a problem in practice); it is also possible to
extend shape with a \verb|signatures| field, although some work has to be
done specifying coherence conditions between \verb|signatures| and \verb|requirements|.

\section{Front-end syntax}

\begin{figure}[htpb]
$$
\begin{array}{rcll}
p,q,r && \mbox{Package names} \\
m,n   && \mbox{Module names} \\[1em]
\multicolumn{3}{l}{\mbox{\bf Packages}} \\
  \I{pkg} & ::= & \verb|package|\; p\; [\I{provreq}]\; \verb|where {| d_1 \verb|;| \ldots \verb|;| d_n \verb|}| \\[1em]
\multicolumn{3}{l}{\mbox{\bf Declarations}} \\
  d & ::= & \verb|module|\;    m \; [exports]\; \verb|where|\; \I{body} \\
    & |   & \verb|signature|\; m \; [exports]\; \verb|where|\; \I{body} \\
    & |   & \verb|include|\; p \; [provreq] \\[1em]
\multicolumn{3}{l}{\mbox{\bf Provides/requires specification}} \\
\I{provreq} & ::= & \verb|(| \, \I{rns} \, \verb|)| \; 
        [ \verb|requires(|\, \I{rns} \, \verb|)| ] \\
\I{rns} & ::= & \I{rn}_0 \verb|,| \, \ldots \verb|,| \, \I{rn}_n [\verb|,|] & \mbox{Renamings} \\
\I{rn} & ::= & m\; \verb|as| \; n & \mbox{Renaming} \\[1em] 
\multicolumn{3}{l}{\mbox{\bf Haskell code}} \\
\I{exports} & & \mbox{A Haskell module export list} \\
\I{body}    & & \mbox{A Haskell module body} \\
\end{array}
$$
\caption{Syntax of Backpack} \label{fig:syntax}
\end{figure}

The syntax of Backpack is given in Figure~\ref{fig:syntax}.
See the ``Backpack manual'' for more explanation about the syntax.  It
is slightly simplified here by removing any constructs which are easily implemented as
syntactic sugar (e.g., a bare $m$ in a renaming is simply $m\; \verb|as|\; m$.)

\newpage
\section{Shaping}

\begin{figure}[htpb]
$$
\begin{array}{rcll}
\I{Shape} & ::= & \verb|provides:|\; m \; \verb|->|\; \I{Module}\; \verb|{|\, \I{Name} \verb|,|\, \ldots \, \verb|};| \ldots \\
      &     & \verb|requires:| \; m \; \verb|->|\; \textcolor{white}{\I{Module}}\; \verb|{| \, \I{Name} \verb|,| \, \ldots \, \verb|}| \verb|;| \ldots \\
\I{PkgKey} & ::= & p \verb|(| \, m \; \verb|->| \; \I{Module} \verb|,|\, \ldots\, \verb|)| \\
\I{Module} & ::= & \I{PkgKey} \verb|:| m \\
\I{Name}   & ::= & \I{Module} \verb|.| \I{OccName} \\
\I{OccName} & & \mbox{Unqualified name in a namespace}
\end{array}
$$
\caption{Semantic entities in Backpack} \label{fig:semantic}
\end{figure}

Shaping computes a \I{Shape}, whose form is described in Figure~\ref{fig:semantic}.
Initializing the shape context to the empty shape, we incrementally
build the context as follows:

\begin{enumerate}
    \item Calculate the shape of a declaration, with respect to the
        current shape context.  (e.g., by renaming a module/signature,
        or using the shape from an included package.)
    \item Merge this shape into the shape context.
\end{enumerate}

The final shape context is the shape of the package as a whole.
Optionally, we can also compute the renamed syntax trees of
modules and signatures.

%   (There is a slight
%   technical difficulty here, where to do shaping, we actually need an \texttt{AvailInfo},
%   so we can resolve \texttt{T(..)} style imports.)

%   One variation of shaping also computes the renamed version of a package,
%   i.e., where each identifier in the module and signature is replaced with
%   the original name (equivalently, we record the output of GHC's renaming
%   pass). This simplifies type checking because you no longer have to
%   recalculate the set of available names, which otherwise would be lost.
%   See more about this in the type checking section.

In the description below, we'll assume \verb|THIS| is the package key
of the package being processed.

\begin{aside}
\textbf{\textit{OccName} is implied by \textit{Name}.}
In Haskell, the following is not valid syntax:

\begin{verbatim}
    import A (foobar as baz)
\end{verbatim}
In particular, a \I{Name} which is in scope will always have the same
\I{OccName} (even if it may be qualified.)  You might imagine relaxing
this restriction so that declarations can be used under different \I{OccName}s;
in such a world, we need a different definition of shape:

\begin{verbatim}
    Shape ::=
        provided: ModName -> { OccName -> Name }
        required: ModName -> { OccName -> Name }
\end{verbatim}
Presently, however, such an \I{OccName} annotation would be redundant: it can be inferred from the \I{Name}.
\end{aside}

\begin{aside}
\textbf{Holes of a package are a mapping, not a set.} Why can't the \I{PkgKey} just record a
set of \I{Module}s, e.g. $\I{PkgKey} ::= \I{SrcPkgKey} \; \verb|{| \; \I{Module} \; \verb|}|$?  Consider:

\begin{verbatim}
    package p (A) requires (H1, H2) where
        signature H1(T) where
            data T
        signature H2(T) where
            data T
        module A(A(..)) where
            import qualified H1
            import qualified H2
            data A = A H1.T H2.T

    package q (A12, A21) where
        module I1(T) where
            data T = T Int
        module I2(T) where
            data T = T Bool
        include p (A as A12) requires (H1 as I1, H2 as I2)
        include p (A as A21) requires (H1 as I2, H2 as I1)
\end{verbatim}
With a mapping, the first instance of \verb|p| has key \verb|p(H1 -> q():I1, H2 -> q():I2)|
while the second instance has key \verb|p(H1 -> q():I2, H2 -> q():I1)|; with
a set, both would have the key \verb|p(q():I1, q():I2)|.
\end{aside}


\begin{aside}
\textbf{Signatures can require a specific entity.}
With requirements like \verb|A -> { HOLE:A.T, HOLE:A.foo }|,
why not specify it as \verb|A -> { T, foo }|,
e.g., \verb|required: { ModName -> { OccName } }|?  Consider:

\begin{verbatim}
    package p () requires (A, B) where
        signature A(T) where
            data T
        signature B(T) where
            import T
\end{verbatim}
The requirements of this package specify that \verb|A.T| $=$ \verb|B.T|; this
can be expressed with \I{Name}s as

\begin{verbatim}
    A -> { HOLE:A.T }
    B -> { HOLE:A.T }
\end{verbatim}
But, without \I{Name}s, the sharing constraint is impossible:  \verb|A -> { T }; B -> { T }|. (NB: \verb|A| and \verb|B| don't have to be implemented with the same module.)
\end{aside}

\begin{aside}
\textbf{The \textit{Name} of a value is used to avoid
ambiguous identifier errors.}  We state that two types
are equal when their \I{Name}s are the same; however,
for values, it is less clear why we care.  But consider this example:

\begin{verbatim}
    package p (A) requires (H1, H2) where
        signature H1(x) where
            x :: Int
        signature H2(x) where
            import H1(x)
        module A(y) where
            import H1
            import H2
            y = x
\end{verbatim}
The reference to \verb|x| in \verb|A| is unambiguous, because it is known
that \verb|x| from \verb|H1| and \verb|x| from \verb|H2| are the same (have
the same \I{Name}.)  If they were not the same, it would be ambiguous and
should cause an error.  Knowing the \I{Name} of a value distinguishes
between these two cases.
\end{aside}

\begin{aside}
\textbf{Holes are linear}
Requirements do not record what \I{Module} represents
the identity of a requirement, which means that it's not possible to assert
that hole \verb|A| and hole \verb|B| should be implemented with the same module,
as might occur with aliasing:

\begin{verbatim}
    signature A where
    signature B where
    alias A = B
\end{verbatim}
%
The benefit of this restriction is that when a requirement is filled,
it is obvious that this is the only requirement that is filled: you won't
magically cause some other requirements to be filled.  The downside is
it's not possible to write a package which looks for an interface it is
looking for in one of $n$ names, accepting any name as an acceptable linkage.
If aliasing was allowed, we'd need a separate physical shaping context,
to make sure multiple mentions of the same hole were consistent.

\end{aside}

%\newpage

\subsection{\texttt{module M}}

A module declaration provides a module \verb|THIS:M| at module name \verb|M|. It
has the shape:

\begin{verbatim}
    provides: M -> THIS:M { exports of renamed M }
    requires: (nothing)
\end{verbatim}
Example:

\begin{verbatim}
    module A(T) where
        data T = T

    -- provides: A -> THIS:A { THIS:A.T }
    -- requires: (nothing)
\end{verbatim}

\newpage
\subsection{\texttt{signature M}}

A signature declaration creates a requirement at module name \verb|M|.  It has the shape:

\begin{verbatim}
    provides: (nothing)
    requires: M -> { exports of renamed M }
\end{verbatim}

\noindent Example:

\begin{verbatim}
    signature H(T) where
        data T

    -- provides: H -> (nothing)
    -- requires: H -> { HOLE:H.T }
\end{verbatim}

\begin{aside}
\textbf{In-scope signatures are not provisions}.  We enforce the invariant that
a provision is always (syntactically) a \verb|module| and a requirement
is always a \verb|signature|.  This means that if you have a requirement
and a provision of the same name, the requirement can \emph{always} be filled
with the provision. Without this invariant, it's not clear if a provision
will actually fill a signature.  Consider this example, where
a signature is required and exposed:

\begin{verbatim}
    package a-sigs (A) requires (A) where -- ***
        signature A where
            data T

    package a-user (B) requires (A) where
        signature A where
            data T
            x :: T
        module B where
            ...

    package p where
        include a-sigs
        include a-user
\end{verbatim}
%
When we consider merging in the shape of \verb|a-user|, does the
\verb|A| provided by \verb|a-sigs| fill in the \verb|A| requirement
in \verb|a-user|?  It \emph{should not}, since \verb|a-sigs| does not
actually provide enough declarations to satisfy \verb|a-user|'s
requirement: the intended semantics \emph{merges} the requirements
of \verb|a-sigs| and \verb|a-user|.



\begin{verbatim}
    package a-sigs (M as A) requires (H as A) where
        signature H(T) where
            data T
        module M(T) where
            import H(T)
\end{verbatim}
%
We rightly should error, since the provision is a module. And in this situation:

\begin{verbatim}
    package a-sigs (H as A) requires (H) where
        signature H(T) where
            data T
\end{verbatim}
%
The requirements should be merged, but should the merged requirement
be under the name \verb|H| or \verb|A|?

It may still be possible to use the \verb|(A) requires (A)| syntax to
indicate exposed signatures, but this would be a mere syntactic
alternative to \verb|() requires (exposed A)|.
\end{aside}
%

\newpage
\subsection{\texttt{include pkg (X) requires (Y)}}

We merge with the transformed shape of package \verb|pkg|, where this
shape is transformed by:

\begin{itemize}
    \item Renaming and thinning the provisions according to \verb|(X)|
    \item Renaming requirements according to \verb|(Y)| (requirements cannot
          be thinned, so non-mentioned requirements are implicitly passed through.)
          For each renamed requirement from \verb|Y| to \verb|Y'|,
          substitute \verb|HOLE:Y| with \verb|HOLE:Y'| in the
          \I{Module}s and \I{Name}s of the provides and requires.
\end{itemize}
%
If there are no thinnings/renamings, you just merge the
shape unchanged! Here is an example:

\begin{verbatim}
    package p (M) requires (H) where
        signature H where
            data T
        module M where
            import H
            data S = S T

    package q (A) where
        module X where
            data T = T
        include p (M as A) requires (H as X)
\end{verbatim}
%
The shape of package \verb|p| is:

\begin{verbatim}
    requires: M -> { p(H -> HOLE:H):M.S }
    provides: H -> { HOLE:H.T }
\end{verbatim}
%
Thus, when we process the \verb|include| in package \verb|q|,
we make the following two changes: we rename the provisions,
and we rename the requirements, substituting \verb|HOLE|s.
The resulting shape to be merged in is:

\begin{verbatim}
    provides: A -> { p(H -> HOLE:X):M.S }
    requires: X -> { HOLE:X.T }
\end{verbatim}
%
After merging this in, the final shape of \verb|q| is:

\begin{verbatim}
    provides: X -> { q():X.T }              -- from shaping 'module X'
              A -> { p(H -> q():X):M.S }
    requires: (nothing)                     -- discharged by provided X
\end{verbatim}

\newpage

\subsection{Merging}

The shapes we've given for individual declarations have been quite
simple.  Merging combines two shapes, filling requirements with
implementations, unifying \I{Name}s, and unioning requirements; it is
the most complicated part of the shaping process.

The best way to think about merging is that we take two packages with
inputs (requirements) and outputs (provisions) and ``wiring'' them up so
that outputs feed into inputs.  In the absence
of mutual recursion, this wiring process is \emph{directed}: the provisions
of the first package feed into the requirements of the second package,
but never vice versa.  (With mutual recursion, things can go in the opposite
direction as well.)

Suppose we are merging shape $p$ with shape $q$ (e.g., $p; q$).  Merging
proceeds as follows:

\begin{enumerate}
    \item \emph{Fill every requirement of $q$ with provided modules from
        $p$.} For each requirement $M$ of $q$ that is provided by $p$ (in particular,
        all of its required \verb|Name|s are provided),
        substitute each \I{Module} occurrence of \verb|HOLE:M| with the
        provided $p(M)$, unify the names, and remove the requirement from $q$.
        If the names of the provision are not a superset of the required names, error.
    \item If mutual recursion is supported, \emph{fill every requirement of $p$ with provided modules from $q$.}
    \item \emph{Merge leftover requirements.}  For each requirement $M$ of $q$ that is not
        provided by $p$ but required by $p$, unify the names, and union them together to form the new requirement.  (It's not
        necessary to substitute \I{Module}s, since they are guaranteed to be the same.)
    \item \emph{Add provisions of $q$.} Union the provisions of $p$ and $q$, erroring
        if there is a duplicate that doesn't have the same identity.
\end{enumerate}
%
To unify two sets of names, find each pair of names with matching \I{OccName}s $n$ and $m$ and do the following:

\begin{enumerate}
    \item If both are from holes, pick a canonical representative $m$ and substitute $n$ with $m$.
    \item If one $n$ is from a hole, substitute $n$ with $m$.
    \item Otherwise, error if the names are not the same.
\end{enumerate}
%
It is important to note that substitutions on \I{Module}s and substitutions on
\I{Name}s are disjoint: a substitution from \verb|HOLE:A| to \verb|HOLE:B|
does \emph{not} substitute inside the name \verb|HOLE:A.T|.

Since merging is the most complicated step of shaping, here are a big
pile of examples of it in action.

\subsubsection{A simple example}

In the following set of packages:

\begin{verbatim}
    package p(M) requires (A) where
        signature A(T) where
            data T
        module M(T, S) where
            import A(T)
            data S = S T

    package q where
        module A where
            data T = T
        include p
\end{verbatim}

When we \verb|include p|, we need to merge the partial shape
of \verb|q| (with just provides \verb|A|) with the shape
of \verb|p|.  Here is each step of the merging process:

\begin{verbatim}
          shape 1                       shape 2
          --------------------------------------------------------------------------------
(initial shapes)
provides: A -> THIS:A { q():A.T }       M -> p(A -> HOLE:A) { HOLE:A.T, p(A -> HOLE:A).S }
requires: (nothing)                     A ->                { HOLE:A.T }

(after filling requirements)
provides: A -> THIS:A { q():A.T }       M -> p(A -> THIS:A) { q():A.T, p(A -> THIS:A).S }
requires: (nothing)                     (nothing)

(after adding provides)
provides: A -> THIS:A         { q():A.T }
          M -> p(A -> THIS:A) { q():A.T, p(A -> THIS:A).S }
requires: (nothing)
\end{verbatim}

Notice that we substituted \verb|HOLE:A| with \verb|THIS:A|, but \verb|HOLE:A.T| with \verb|q():A.T|.

\subsubsection{Requirements merging can affect provisions}

When a merge results in a substitution, we substitute over both
requirements and provisions:

\begin{verbatim}
    signature H(T) where
        data T
    module A(T) where
        import H(T)
    module B(T) where
        data T = T

    -- provides: A -> THIS:A { HOLE:H.T }
    --           B -> THIS:B { THIS:B.T }
    -- requires: H ->        { HOLE:H.T }

    signature H(T, f) where
        import B(T)
        f :: a -> a

    -- provides: A -> THIS:A { THIS:B.T }           -- UPDATED
    --           B -> THIS:B { THIS:B.T }
    -- requires: H ->        { THIS:B.T, HOLE:H.f } -- UPDATED
\end{verbatim}

\subsubsection{Sharing constraints}

Suppose you have two signature which both independently define a type,
and you would like to assert that these two types are the same.  In the
ML world, such a constraint is known as a sharing constraint.  Sharing
constraints can be encoded in Backpacks via clever use of reexports;
they are also an instructive example for signature merging.

\begin{verbatim}
    signature A(T) where
        data T
    signature B(T) where
        data T

    -- requires: A -> { HOLE:A.T }
                 B -> { HOLE:B.T }

    -- the sharing constraint!
    signature A(T) where
        import B(T)
    -- (shape to merge)
    -- requires: A -> { HOLE:B.T }

    -- (after merge)
    -- requires: A -> { HOLE:A.T }
    --           B -> { HOLE:A.T }
\end{verbatim}
%
\Red{I'm pretty sure any choice of \textit{Name} is OK, since the
subsequent substitution will make it alpha-equivalent.}

%   \subsubsection{Leaky requirements}

%   Both requirements and provisions can be imported, but requirements
%   are always available

%\Red{How to relax this so hs-boot works}

%\Red{Example of how loopy modules which rename requirements make it un-obvious whether or not
%a requirement is still required.  Same example works declaration level.}

%\Red{package p (A) requires (A); the input output ports should be the same}

% We figure out the requirements (because no loopy modules)
%
% package p (A, B) requires (B)
%   include base
%   sig B(T)
%       import Prelude(T)
%
% requirement example
%
% mental model: you start with an empty package, and you start accreting
% things on things, merging things together as you discover that this is
% the case.

%\newpage

\subsection{Export declarations}

If an explicit export declaration is given, the final shape is the
computed shape, minus any provisions not mentioned in the list, with the
appropriate renaming applied to provisions and requirements.  (Requirements
are implicitly passed through if they are not named.)
If no explicit export declaration is given, the final shape is
the computed shape, including only provisions which were defined
in the declarations of the package.

\begin{aside}
\textbf{Signature visibility, and defaulting}
The simplest formulation of requirements is to have them always be
visible.  Signature visibility could be controlled by associating
every requirement with a flag indicating if it is importable or
not: a signature declaration sets a requirement to be visible, and
an explicit export list can specify if a requirement is to be visible
or not.

When an export list is absent, we have to pick a default visibility
for a signature.  If we use the same behavior as with modules,
a strange situation can occur:

\begin{verbatim}
    package p where -- S is visible
        signature S where
            x :: True

    package q where -- use defaulting
        include p
        signature S where
            y :: True
        module M where
            import S
            z = x && y      -- OK

    package r where
        include q
        module N where
            import S
            z = y           -- OK
            z = x           -- ???
\end{verbatim}
%
Absent the second signature declaration in \verb|q|, \verb|S.x| clearly
should not be visible in \verb|N|.  However, what ought to occur when this signature
declaration is added?  One interpretation is to say that only some
(but not all) declarations are provided (\verb|S.x| remains invisible);
another interpretation is that adding \verb|S| is enough to treat
the signature as ``in-line'', and all declarations are now provided
(\verb|S.x| is visible).

The latter interpretation avoids having to keep track of providedness
per declarations, and means that you can always express defaulting
behavior by writing an explicit provides declaration on the package.
However, it has the odd behavior of making empty signatures semantically
meaningful:

\begin{verbatim}
package q where
    include p
    signature S where
\end{verbatim}
\end{aside}
%
%   SPJ: This would be too complicated (if there's yet a third way)

\subsection{Package key}

What is \verb|THIS|?  It is the package name, plus for every requirement \verb|M|,
a mapping \verb|M -> HOLE:M|.  Annoyingly, you don't know the full set of
requirements until the end of shaping, so you don't know the package key ahead of time;
however, it can be substituted at the end easily.

\clearpage
\newpage

\section{Type constructor exports}

In the previous section, we described the \I{Name}s of a
module as a flat namespace; but actually, there is one level of
hierarchy associated with type-constructors.  The type:

\begin{verbatim}
    data A = B { foo :: Int }
\end{verbatim}
%
brings three \I{OccName}s into scope, \verb|A|, \verb|B|
and \verb|foo|, but the constructors and record selectors are
considered \emph{children}
of \verb|A|: in an import list, they can be implicitly brought
into scope with \verb|A(..)|, or individually brought into
scope with \verb|foo| or \verb|pattern B| (using the new \verb|PatternSynonyms|
extension).  Symmetrically, a module may export only \emph{some}
of the constructors/selectors of a type; it may not even
export the type itself!

We \emph{absolutely} need this information to rename a module or
signature, which means that there is a little bit of extra information
we have to collect when shaping.  What is this information?  If we take
GHC's internal representation at face value, we have the more complex
semantic representation seen in Figure~\ref{fig:semantic2}:

\begin{figure}[htpb]
$$
\begin{array}{rcll}
\I{Shape} & ::= & \verb|provides:|\; m \; \verb|->|\; \I{Module}\; \verb|{|\, \I{AvailInfo} \verb|,|\, \ldots \, \verb|};| \ldots \\
      &     & \verb|requires:| \; m \; \verb|->|\; \textcolor{white}{\I{Module}}\; \verb|{| \, \I{AvailInfo} \verb|,| \, \ldots \, \verb|}| \verb|;| \ldots \\
\I{AvailInfo} & ::= & \I{Name} & \mbox{Plain identifiers} \\
          & |   & \I{Name} \, \verb|{| \, \I{Name}_0\verb|,| \, \ldots\verb|,| \, \I{Name}_n \, \verb|}| & \mbox{Type constructors} \\
\end{array}
$$
\caption{Enriched semantic entities in Backpack} \label{fig:semantic2}
\end{figure}
%
For type constructors, the outer \I{Name} identifies the \emph{parent}
identifier, which may not necessarily be in scope (define this to be the \texttt{availName}); the inner list consists
of the children identifiers that are actually in scope.  If a wildcard
is written, all of the child identifiers are brought into scope.
In the following examples, we've ensured that
types and constructors are unambiguous, although in Haskell proper they
live in separate namespaces; we've also elided the \verb|THIS| package
key from the identifiers.

\begin{verbatim}
    module M(A(..)) where
        data A = B { foo :: Int }
    -- M.A{ M.A, M.B, M.foo }

    module N(A) where
        data A = B { foo :: Int }
    -- N.A{ N.A }

    module O(foo) where
        data A = B { foo :: Int }
    -- O.A{ O.foo }

    module A where
        data T = S { bar :: Int }
    module B where
        data T = S { baz :: Bool }
    module C(bar, baz) where
        import A
        import B
    -- A.T{ A.bar }, B.T{ B.baz }
    -- NB: it would be illegal for the type constructors
    -- A.T and B.T to be both exported from C!
\end{verbatim}
%
Previously, we stated that we simply merged \I{Name}s based on their
\I{OccName}s.  We now must consider what it means to merge \I{AvailInfo}s.

\subsection{Algorithm}

Our merging algorithm takes two sets of \I{AvailInfo}s and merges them
into one set.  In the degenerate case where every \I{AvailInfo} is a
$Name$, this algorithm operates the same as the original algorithm.
Merging proceeds in two steps: unification and then simple union.

Unification proceeds as follows: for each pair of \I{Name}s with
matching \I{OccName}s, unify the names.  For each pair of $\I{Name}\, \verb|{|\,
\I{Name}_0\verb|,|\, \ldots\verb|,|\, \I{Name}_n\, \verb|}|$, where there
exists some pair of child names with matching \I{OccName}s, unify the
parent \I{Name}s.  (A single \I{AvailInfo} may participate in multiple such
pairs.)  A simple identifier and a type constructor \I{AvailInfo} with
overlapping in-scope names fails to unify.  After unification,
the simple union combines entries with matching \verb|availName|s (parent
name in the case of a type constructor), recursively unioning the child
names of type constructor \I{AvailInfo}s.

Unification of \I{Name}s results in a substitution, and a \I{Name} substitution
on \I{AvailInfo} is a little unconventional.  Specifically, substitution on $\I{Name}\, \verb|{|\,
\I{Name}_0\verb|,|\, \ldots\verb|,|\, \I{Name}_n\, \verb|}|$ proceeds specially:
a substitution from \I{Name} to $\I{Name}'$ induces a substitution from
\I{Module} to $Module'$ (as the \I{OccName}s of the \I{Name}s are guaranteed
to be equal), so for each child $\I{Name}_i$, perform the \I{Module}
substitution.  So for example, the substitution \verb|HOLE:A.T| to \verb|THIS:A.T|
takes the \I{AvailInfo} \verb|HOLE:A.T { HOLE:A.B, HOLE:A.foo }| to
\verb|THIS:A.T { THIS:A.B, THIS:A.foo }|.  In particular, substitution
on children \I{Name}s is \emph{only} carried out by substituting on the outer name;
we will never directly substitute children.

\subsection{Examples}

Unfortunately, there are a number of tricky scenarios:

\paragraph{Merging when type constructors are not in scope}

\begin{verbatim}
    signature A1(foo) where
        data A = A { foo :: Int, bar :: Bool }

    signature A2(bar) where
        data A = A { foo :: Int, bar :: Bool }
\end{verbatim}
%
If we merge \verb|A1| and \verb|A2|, are we supposed to conclude that
the types \verb|A1.A| and \verb|A2.A| (not in scope!) are the same?
The answer is no!  Consider these implementations:

\begin{verbatim}
    module A1(A(..)) where
        data A = A { foo :: Int, bar :: Bool }

    module A2(A(..)) where
        data A = A { foo :: Int, bar :: Bool }

    module A(foo, bar) where
        import A1
        import A2
\end{verbatim}

Here, \verb|module A1| implements \verb|signature A1|, \verb|module A2| implements \verb|signature A2|,
and \verb|module A| implements \verb|signature A1| and \verb|signature A2| individually
and should certainly implement their merge.  This is why we cannot simply
merge type constructors based on the \I{OccName} of their top-level type;
merging only occurs between in-scope identifiers.

\paragraph{Does merging a selector merge the type constructor?}

\begin{verbatim}
    signature A1(A(..)) where
        data A = A { foo :: Int, bar :: Bool }

    signature A2(A(..)) where
        data A = A { foo :: Int, bar :: Bool }

    signature A2(foo) where
        import A1(foo)
\end{verbatim}
%
Does the last signature, which is written in the style of a sharing constraint on \verb|foo|,
also cause \verb|bar| and the type and constructor \verb|A| to be unified?
Because a merge of a child name results in a substitution on the parent name,
the answer is yes.

\paragraph{Incomplete data declarations}

\begin{verbatim}
    signature A1(A(foo)) where
        data A = A { foo :: Int }

    signature A2(A(bar)) where
        data A = A { bar :: Bool }
\end{verbatim}
%
Should \verb|A1| and \verb|A2| merge?  If yes, this would imply
that data definitions in signatures could only be \emph{partial}
specifications of their true data types.  This seems complicated,
which suggests this should not be supported; however, in fact,
this sort of definition, while disallowed during type checking,
should be \emph{allowed} during shaping. The reason that the
shape we abscribe to the signatures \verb|A1| and \verb|A2| are
equivalent to the shapes for these which should merge:

\begin{verbatim}
    signature A1(A(foo)) where
        data A = A { foo :: Int, bar :: Bool }

    signature A2(A(bar)) where
        data A = A { foo :: Int, bar :: Bool }
\end{verbatim}

\subsection{Subtyping record selectors as functions}

\begin{verbatim}
    signature H(foo) where
        data A
        foo :: A -> Int

    module M(foo) where
        data A = A { foo :: Int, bar :: Bool }
\end{verbatim}
%
Does \verb|M| successfully fill \verb|H|?  If so, it means that anywhere
a signature requests a function \verb|foo|, we can instead validly
provide a record selector.  This capability seems quite attractive
but actually it is quite complicated, because we can no longer assume
that every child name is associated with a parent name.

As a workaround, \verb|H| can equivalently be written as:

\begin{verbatim}
    signature H(foo) where
        data A = A { foo :: Int, bar :: Bool }
\end{verbatim}
%
This is suboptimal, however, as the otherwise irrelevant \verb|bar| must be mentioned
in the definition.

So what if we actually want to write the original signature \verb|H|?
The technical difficulty is that we now need to unify a plain identifier
\I{AvailInfo} (from the signature) with a type constructor \I{AvailInfo}
(from a module.)  It is not clear what this should mean.
Consider this situation:

\begin{verbatim}
    package p where
        signature H(A, foo, bar) where
            data A
            foo :: A -> Int
            bar :: A -> Bool
        module X(A, foo) where
            import H
    package q where
        include p
        signature H(bar) where
            data A = A { foo :: Int, bar :: Bool }
        module Y where
            import X(A(..)) -- ???
\end{verbatim}

Should the wildcard import on \verb|X| be allowed?  Probably not?
How about this situation:

\begin{verbatim}
    package p where
        -- define without record selectors
        signature X1(A, foo) where
            data A
            foo :: A -> Int
        module M1(A, foo) where
            import X1

    package q where
        -- define with record selectors (X1s unify)
        signature X1(A(..)) where
            data A = A { foo :: Int, bar :: Bool }
        signature X2(A(..)) where
            data A = A { foo :: Int, bar :: Bool }

        -- export some record selectors
        signature Y1(bar) where
            import X1
        signature Y2(bar) where
            import X2

    package r where
        include p
        include q

        -- sharing constraint
        signature Y2(bar) where
            import Y1(bar)

        -- the payload
        module Test where
            import M1(foo)
            import X2(foo)
            ... foo ... -- conflict?
\end{verbatim}

Without the sharing constraint, the \verb|foo|s from \verb|M1| and \verb|X2|
should conflict.  With it, however, we should conclude that the \verb|foo|s
are the same, even though the \verb|foo| from \verb|M1| is \emph{not}
considered a child of \verb|A|, and even though in the sharing constraint
we \emph{only} unified \verb|bar| (and its parent \verb|A|).  To know that
\verb|foo| from \verb|M1| should also be unified, we have to know a bit
more about \verb|A| when the sharing constraint performs unification;
however, the \I{AvailInfo} will only tell us about what is in-scope, which
is \emph{not} enough information.

%\newpage

\section{Type checking}

\begin{figure}[htpb]
$$
\begin{array}{rcll}
\I{PkgType} & ::= & \I{ModIface}_0 \verb|;|\, \ldots\verb|;|\, \I{ModIface}_n \\[1em]
\multicolumn{3}{l}{\mbox{\bf Module interface}} \\
\I{ModIface} & ::= & \verb|module| \; \I{Module} \; \verb|(| \I{mi\_exports} \verb|)| \; \verb|where| \\
& & \qquad \I{mi\_decls} \\
& & \qquad \I{mi\_insts} \\
\I{mi\_exports} & ::= & \I{AvailInfo}_0 \verb|,|\, \ldots \verb|,|\, \I{AvailInfo}_n \\
\I{mi\_decls} & ::= & \I{IfaceDecl}_0 \verb|;|\, \ldots \verb|;|\, \I{IfaceDecl}_n \\
\I{mi\_insts} & ::= & \I{IfaceClsInst}_0 \verb|;|\, \ldots \verb|;|\, \I{IfaceClsInst}_n \\[1em]
\multicolumn{3}{l}{\mbox{\bf Interface declarations}} \\
\I{IfaceDecl} & ::= & \I{OccName} \; \verb|::| \; \I{IfaceId} \\
              & |   & \verb|data| \; \I{OccName} \; \verb|=| \;\ \I{IfaceData} \\
              & |   & \ldots \\
\I{IfaceClsInst} & & \mbox{A type-class instance} \\
\I{IfaceId} & & \mbox{Interface of top-level binder} \\
\I{IfaceData} & & \mbox{Interface of type constructor} \\
\end{array}
$$
\caption{Module interfaces in GHC} \label{fig:typecheck}
\end{figure}

Type checking an indefinite package (a package with holes) involves
calculating, for every module, a \I{ModIface} representing the
type/interface of the module in question (which is serialized
to disk).  The general form of these
interface files are described in Figure~\ref{fig:typecheck}; notably,
the interfaces \I{IfaceId}, \I{IfaceData}, etc. contain \I{Name} references,
which must be resolved by
looking up a \I{ModIface} corresponding to the \I{Module} associated
with the \I{Name}. (We will say more about this lookup process shortly.)
For example, given:

\begin{verbatim}
    package p where
        signature H where
            data T
        module A(S, T) where
            import H
            data S = S T
\end{verbatim}
%
the \I{PkgType} is:

\begin{verbatim}
    module HOLE:H (HOLE:H.T) where
        data T -- abstract type constructor
    module THIS:A (THIS:A.S, HOLE:H.T) where
        data S = S HOLE:H.T
    -- where THIS = p(H -> HOLE:H)
\end{verbatim}
%
However, a \I{PkgType} of \I{ModIface}s is not the whole story:
when a package has holes, a \I{PkgType} specified in this manner
defines a family of possible types, based on how the holes are
shaped and instantiated.  A package which writes \verb|include p requires (H as S)|
and has a sharing constraint of \verb|H.T| with \verb|q():B.T| may
end up with this ``type'':

\begin{verbatim}
    module HOLE:S (q():B.T) where
    module THIS:A (THIS:A.S, q():B.T) where
        data S = S q():B.T
    -- where THIS = p(H -> HOLE:S)
\end{verbatim}
%
Furthermore, for ease of implementation, GHC prefers to resolve all
indirect \I{Name} references (which are just strings) into a
\I{ModIface} into direct \I{TyThing} references (which are data
structures that have type information).  This resolution is done lazily
with some hackery!

Thus, given a shaping and a hole instantiation, a \I{ModIface} can be
converted into an in-memory \I{ModDetails}, described in
Figure~\ref{fig:typecheck-more}.  (Technically, the \I{Module} does not
have to be recorded as it is recorded in the \I{Name} associated with a
\I{TyThing}; so you can think of a \I{ModDetails} as a big bag of type-checkable
entities which GHC knows about; in the source code this
is referred to as the \emph{external package state (EPT)}).

\begin{figure}[htpb]
$$
\begin{array}{rcll}
\I{ModDetails} & ::= & \langle\I{md\_types} \verb|;|\; \I{md\_insts}\rangle \\
\I{md\_types}  & ::= & \I{TyThing}_0 \verb|,|\, \ldots\verb|,|\, \I{TyThing}_n \\
\I{md\_insts}  & ::= & \I{ClsInst}_0 \verb|,|\, \ldots\verb|,|\, \I{ClsInst}_n \\[1em]
\multicolumn{3}{l}{\mbox{\bf Type-checked declarations}} \\
\I{TyThing}    &     & \mbox{Type-checked thing with a \I{Name}} \\
\I{ClsInst}    &     & \mbox{Type-checked type class instance} \\
\end{array}
$$
\caption{Semantic objects in GHC} \label{fig:typecheck-more}
\end{figure}

Type checking, thus, is a delicate balancing act between module
interfaces and our semantic objects.  Given a shaping, here
are some important operations we have to do in type checking:

\paragraph{Imports}  When a module/signature imports a module name,
we must consult the exports of \I{ModIface}s associated with this
module name, modulo what we calculated into the shaping pass.
(In fact, we can get all of this information directly from the
shaping pass.)

\paragraph{Includes and name lookups}  When we include a package,
take all of the \I{ModIface}s it brings into scope, type-check
them with respect to the instantiation of holes and shaping, and add
them to the EPT.

Even better, this process should be done lazily on name lookup.
When we have a renamed identifier, e.g. a \I{Name},
we first check if we know about this object in EPT, and if not,
we must find the \I{ModIface}(s) (plural!) that would have brought the object into
scope, add them to EPT and try again.  The process of adding semantic
objects to the EPT may cause us to learn \emph{more} information
about an object than we knew previously.

\paragraph{Cross-package compilation}  When we begin type-checking a new
indefinite package, we must \emph{clear} all \I{ModDetails} which depend on
holes.  This is because shaping may cause the type-checked entities to refer
to different semantic objects.

\subsection{The basic plan}

Given a module or signature of a package, we can type check given these two assumptions:

\begin{itemize}
    \item We have a renamed syntax tree, whose identifiers have been
          resolved as according to the result of the shaping pass.
    \item For any \I{Name} in the renamed tree, the corresponding
          \I{ModDetails} for the \I{Module} has been loaded
          (or can be lazily loaded).
\end{itemize}

The result of type checking is a \I{ModDetails} which can then be
converted into a \I{ModIface}, because we assumed each signature
to serve as an uninstantiated hole (thus, the computed \I{ModDetails} is
in its most general form).
Arranging for these two assumptions to be true is the bulk of the
complexity of type checking.

\subsection{Loading modules from indefinite packages}

\paragraph{Everything is done modulo a shape}  Consider
this package:

\begin{verbatim}
package p where
    signature H(T) where
        data T = T
    module A(T) where
        data T = T
    signature H(T) where
        import A(T)

-- provides: A -> THIS:A { THIS:A.T }
--           H -> HOLE:H { THIS:A.T }
-- requires: H ->        { THIS:A.T }
\end{verbatim}

With this shaping information, when we are type-checking the first
signature for \verb|H|, it is completely wrong to try to create
a definition for \verb|HOLE:H.T|, since we know that it refers
to \verb|THIS:A.T| via the requirements of the shape.  This applies
even if \verb|H| is included from another package.  Thus, when
we are loading \I{ModDetails} into memory, it is always done
\emph{with respect to some shaping}.  Whenever you reshape,
you must clear the module environment.

\paragraph{Figuring out where to consult for shape information}

For this example, let's suppose we have already type-checked
this package \verb|p|:

\begin{verbatim}
package p (A) requires (S) where
    signature S where
        data S
        data T
    module A(A) where
        import S
        data A = A S T
\end{verbatim}

giving us the following \I{ModIface}s:

\begin{verbatim}
module HOLE:S.S where
    data S
    data T
module THIS:A where
    data A = A HOLE:S.S HOLE:S.T
-- where THIS = p(S -> HOLE:S)
\end{verbatim}

Next, we'd like to type check a package which includes \verb|p|:

\begin{verbatim}
package q (T, A, B) requires (H) where
    include p (A) requires (S as H)
    module T(T) where
        data T = T
    signature H(T) where
        import T(T)
    module B(B) where
        import A
        data B = B A
\end{verbatim}

%   package r where
%       include q
%       module H(S,T) where
%           import T(T)
%           data S = S
%       module C where
%           import A
%           import B
%           ...

Prior to typechecking, we compute its shape:

\begin{verbatim}
provides: (elided)
requires: H -> { HOLE:H.S, THIS:T.T }
-- where THIS = q(H -> HOLE:H)
\end{verbatim}

Our goal is to get the following type:

\begin{verbatim}
module THIS:T where
    data T = T
module THIS:B where
    data B = B p(S -> HOLE:H):A.A
        -- where data A = A HOLE:H.S THIS:T.T
-- where THIS = q(H -> HOLE:H)
\end{verbatim}

This type information does \emph{not} match the pre-existing
type information from \verb|p|: when we translate the \I{ModIface} for
\verb|A| in the context into a \I{ModDetails} from this typechecking,
we need to substitute \I{Name}s and \I{Module}s
as specified by shaping.  Specifically, when we load \verb|p(S -> HOLE:H):A|
to find out the type of \verb|p(S -> HOLE:H):A.A|,
we need to take \verb|HOLE:S.S| to \verb|HOLE:H.S| and \verb|HOLE:S.T| to \verb|THIS:T.T|.
In both cases, we can determine the right translation by looking at how \verb|S| is
instantiated in the package key for \verb|p| (it is instantiated with \verb|HOLE:H|),
and then consulting the shape in the requirements.

This process is done lazily, as we may not have typechecked the original
\I{Name} in question when doing this.  \verb|hs-boot| considerations apply
if things are loopy: we have to treat the type abstractly and re-typecheck it
to the right type later.


\subsection{Re-renaming}

Theoretically, the cleanest way to do shaping and typechecking is to have shaping
result in a fully renamed syntax tree, which we then typecheck: when done this way,
we don't have to worry about logical contexts (i.e., what is in scope) because
shaping will already have complained if things were not in scope.

However, for practical purposes, it's better if we don't try to keep
around renamed syntax trees, because this could result in very large
memory use; additionally, whenever a substitution occurs, we would have
to substitute over all of the renamed syntax trees.  Thus, while
type-checking, we'll also re-compute what is in scope (i.e.,  just the
\I{OccName} bits of \verb|provided|). Nota bene: we still use the
\I{Name}s from the shape as the destinations of these
\I{OccName}s!  Note that we can't just use the final shape, because
this may report more things in scope than we actually want.  (It's also
worth noting that if we could reduce the set of provided things in
scope in a single package, just the \I{Shape} would not be enough.)

\subsection{Merging \texttt{ModDetails}}

After type-checking a signature, we may turn to add it to our module
environment and discover there is already an entry for it!  In that case,
we simply merge it with the existing entry, erroring if there are incompatible
entries.

\end{document}