API Annotations documentation update, parsing issue, add example testwip/api-annot-tweaks-7.10

Summary:
Add a reference note to each AnnKeywordId haddock comment so GHC
developers will have an idea why they are there.

Add a new test to ghc-api/annotations to serve as a template for other
GHC developers when they need to update the parser. It provides output
which checks that each SrcSpan that an annotation is attached to
actually appears in the `ParsedSource`, and lists the individual
annotations. The idea is that a developer writes a version of this
which parses a sample file using whatever syntax is changed in
Parser.y, and can then check that all the annotations come through.

Depends on D538

Test Plan: ./validate

Reviewers: simonpj, hvr, austin

Reviewed By: austin

Subscribers: thomie, jstolarek

Differential Revision: https://phabricator.haskell.org/D620

(cherry picked from commit 851ed7211fb18fea938be84c99b6389f6762b30d)

3 files changed, 22 insertions, 13 deletions

diff --git a/compiler/parser/ApiAnnotation.hs b/compiler/parser/ApiAnnotation.hs
index 60f917222f..e8ad8ea879 100644
--- a/compiler/parser/ApiAnnotation.hs
+++ b/compiler/parser/ApiAnnotation.hs
@@ -17,9 +17,9 @@ import qualified Data.Map as Map
 import Data.Data
 
 
-{- Note [Api annotations]
-   ~~~~~~~~~~~~~~~~~~~~~~
-
+{-
+Note [Api annotations]
+~~~~~~~~~~~~~~~~~~~~~~
 In order to do source to source conversions using the GHC API, the
 locations of all elements of the original source needs to be tracked.
 The includes keywords such as 'let' / 'in' / 'do' etc as well as
@@ -66,8 +66,8 @@ This is done in the lexer / parser as follows.
 
 The PState variable in the lexer has the following variables added
 
->  annotations :: [(ApiAnnKey,SrcSpan)],
->  comment_q :: [Located Token],
+>  annotations :: [(ApiAnnKey,[SrcSpan])],
+>  comment_q :: [Located AnnotationComment],
 >  annotations_comments :: [(SrcSpan,[Located AnnotationComment])]
 
 The first and last store the values that end up in the ApiAnns value
@@ -115,6 +115,9 @@ This adds an AnnLet annotation for 'let', an AnnIn for 'in', as well
 as any annotations that may arise in the binds. This will include open
 and closing braces if they are used to delimit the let expressions.
 
+The wiki page describing this feature is
+https://ghc.haskell.org/trac/ghc/wiki/ApiAnnotations
+
 -}
 -- ---------------------------------------------------------------------
 
@@ -173,6 +176,9 @@ getAndRemoveAnnotationComments (anns,canns) span =
 -- Comments are only retained if @'Opt_KeepRawTokenStream'@ is set in
 -- @'DynFlags.DynFlags'@ before parsing.
 --
+-- The wiki page describing this feature is
+-- https://ghc.haskell.org/trac/ghc/wiki/ApiAnnotations
+--
 -- Note: in general the names of these are taken from the
 -- corresponding token, unless otherwise noted
 -- See note [Api annotations] above for details of the usage
diff --git a/compiler/parser/Lexer.x b/compiler/parser/Lexer.x
index 495605e70c..abb2477783 100644
--- a/compiler/parser/Lexer.x
+++ b/compiler/parser/Lexer.x
@@ -639,15 +639,15 @@ data Token
 
   | ITdupipvarid   FastString   -- GHC extension: implicit param: ?x
 
-  | ITchar     SourceText Char       -- Note [literal source text] in BasicTypes
-  | ITstring   SourceText FastString -- Note [literal source text] in BasicTypes
-  | ITinteger  SourceText Integer    -- Note [literal source text] in BasicTypes
+  | ITchar     SourceText Char       -- Note [Literal source text] in BasicTypes
+  | ITstring   SourceText FastString -- Note [Literal source text] in BasicTypes
+  | ITinteger  SourceText Integer    -- Note [Literal source text] in BasicTypes
   | ITrational FractionalLit
 
-  | ITprimchar   SourceText Char     -- Note [literal source text] in BasicTypes
-  | ITprimstring SourceText ByteString -- Note [literal source text] @BasicTypes
-  | ITprimint    SourceText Integer  -- Note [literal source text] in BasicTypes
-  | ITprimword   SourceText Integer  -- Note [literal source text] in BasicTypes
+  | ITprimchar   SourceText Char     -- Note [Literal source text] in BasicTypes
+  | ITprimstring SourceText ByteString -- Note [Literal source text] @BasicTypes
+  | ITprimint    SourceText Integer  -- Note [Literal source text] in BasicTypes
+  | ITprimword   SourceText Integer  -- Note [Literal source text] in BasicTypes
   | ITprimfloat  FractionalLit
   | ITprimdouble FractionalLit
 
diff --git a/compiler/parser/Parser.y b/compiler/parser/Parser.y
index 9e3d5ff14e..e3760906dd 100644
--- a/compiler/parser/Parser.y
+++ b/compiler/parser/Parser.y
@@ -253,7 +253,10 @@ structured editors.
 
 The helper functions are defined at the bottom of this file.
 
-See https://ghc.haskell.org/trac/ghc/wiki/GhcAstAnnotations for some background.
+See
+  https://ghc.haskell.org/trac/ghc/wiki/ApiAnnotations and
+  https://ghc.haskell.org/trac/ghc/wiki/GhcAstAnnotations
+for some background.
 
 -- -----------------------------------------------------------------------------