[Syntax] Do not add a node for 'eof' into the tree

Summary:
While useful as a sentinel value when iterating over tokens, having
'eof' in the tree, seems to do more harm than good.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: javed.absar, kristof.beyls, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64576

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@368062 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 8 insertions, 4 deletions

diff --git a/lib/Tooling/Syntax/BuildTree.cpp b/lib/Tooling/Syntax/BuildTree.cpp
index 03c439c59e..a0b653df13 100644
--- a/lib/Tooling/Syntax/BuildTree.cpp
+++ b/lib/Tooling/Syntax/BuildTree.cpp
@@ -58,8 +58,11 @@ public:
   /// Finish building the tree and consume the root node.
   syntax::TranslationUnit *finalize() && {
     auto Tokens = Arena.tokenBuffer().expandedTokens();
+    assert(!Tokens.empty());
+    assert(Tokens.back().kind() == tok::eof);
+
     // Build the root of the tree, consuming all the children.
-    Pending.foldChildren(Tokens,
+    Pending.foldChildren(Tokens.drop_back(),
                          new (Arena.allocator()) syntax::TranslationUnit);
 
     return cast<syntax::TranslationUnit>(std::move(Pending).finalize());
@@ -96,10 +99,11 @@ private:
   /// Ensures that added nodes properly nest and cover the whole token stream.
   struct Forest {
     Forest(syntax::Arena &A) {
-      // FIXME: do not add 'eof' to the tree.
-
+      assert(!A.tokenBuffer().expandedTokens().empty());
+      assert(A.tokenBuffer().expandedTokens().back().kind() == tok::eof);
       // Create all leaf nodes.
-      for (auto &T : A.tokenBuffer().expandedTokens())
+      // Note that we do not have 'eof' in the tree.
+      for (auto &T : A.tokenBuffer().expandedTokens().drop_back())
         Trees.insert(Trees.end(),
                      {&T, NodeAndRole{new (A.allocator()) syntax::Leaf(&T)}});
     }