Accumulate Haddock comments in P (#17544, #17561, #8944)

Haddock comments are, first and foremost, comments. It's very annoying
to incorporate them into the grammar. We can take advantage of an
important property: adding a Haddock comment does not change the parse
tree in any way other than wrapping some nodes in HsDocTy and the like
(and if it does, that's a bug).

This patch implements the following:

* Accumulate Haddock comments with their locations in the P monad.
  This is handled in the lexer.

* After parsing, do a pass over the AST to associate Haddock comments
  with AST nodes using location info.

* Report the leftover comments to the user as a warning (-Winvalid-haddock).

1 files changed, 5 insertions, 4 deletions

diff --git a/testsuite/tests/ghc-api/T11579.hs b/testsuite/tests/ghc-api/T11579.hs
index 7ea08c9216..9f1cc41f92 100644
--- a/testsuite/tests/ghc-api/T11579.hs
+++ b/testsuite/tests/ghc-api/T11579.hs
@@ -5,6 +5,7 @@ import GHC
 import GHC.Data.StringBuffer
 import GHC.Parser.Lexer
 import GHC.Types.SrcLoc
+import Data.Foldable (toList)
 
 main :: IO ()
 main = do
@@ -13,14 +14,14 @@ main = do
     let stringBuffer = stringToStringBuffer "-- $bar some\n-- named chunk"
         loc = mkRealSrcLoc (mkFastString "Foo.hs") 1 1
 
-    token <- runGhc (Just libdir) $ do
+    hdk_comments <- runGhc (Just libdir) $ do
         dflags <- getSessionDynFlags
         let pstate = mkPState (dflags `gopt_set` Opt_Haddock) stringBuffer loc
         case unP (lexer False return) pstate of
-            POk _ token -> return (unLoc token)
-            _           -> error "No token"
+            POk s (L _ ITeof) -> return (map unLoc (toList (hdk_comments s)))
+            _                 -> error "No token"
 
     -- #11579
     -- Expected:                    "ITdocCommentNamed "bar some\n named chunk"
     -- Actual (with ghc-8.0.1-rc2): "ITdocCommentNamed "bar some"
-    print token
+    mapM_ print hdk_comments