Fix lexing of nested heredoc strings in token_get_all()

This fixes bug #60097.

Before two global variables CG(heredoc) and CG(heredoc_len) were used to
track the current heredoc label. In order to support nested heredoc
strings the *previous* heredoc label was assigned as the token value of
T_START_HEREDOC and the language_parser.y assigned that to CG(heredoc).

This created a dependency of the lexer on the parser. Thus the
token_get_all() function, which accesses the lexer directly without
also running the parser, was not able to tokenize nested heredoc strings
(and leaked memory). Same applies for the source-code highlighting
functions.

The new approach is to maintain a heredoc_label_stack in the lexer, which
contains all active heredoc labels.

As it is no longer required, T_START_HEREDOC and T_END_HEREDOC now don't
carry a token value anymore.

In order to make the work with zend_ptr_stack in this context more
convenient I added a new function zend_ptr_stack_top(), which retrieves the
top element of the stack (similar to zend_stack_top()).

1 files changed, 121 insertions, 0 deletions

diff --git a/ext/tokenizer/tests/bug60097.phpt b/ext/tokenizer/tests/bug60097.phpt
new file mode 100644
index 0000000000..620b4fda4c
--- /dev/null
+++ b/ext/tokenizer/tests/bug60097.phpt
@@ -0,0 +1,121 @@
+--TEST--
+Bug 60097: token_get_all fails to lex nested heredoc
+--FILE--
+<?php
+
+var_dump(token_get_all('<?php
+<<<DOC1
+{$s(<<<DOC2
+DOC2
+)}
+DOC1;
+'));
+
+?>
+--EXPECT--
+array(14) {
+  [0]=>
+  array(3) {
+    [0]=>
+    int(372)
+    [1]=>
+    string(6) "<?php
+"
+    [2]=>
+    int(1)
+  }
+  [1]=>
+  array(3) {
+    [0]=>
+    int(376)
+    [1]=>
+    string(8) "<<<DOC1
+"
+    [2]=>
+    int(2)
+  }
+  [2]=>
+  array(3) {
+    [0]=>
+    int(379)
+    [1]=>
+    string(1) "{"
+    [2]=>
+    int(3)
+  }
+  [3]=>
+  array(3) {
+    [0]=>
+    int(309)
+    [1]=>
+    string(2) "$s"
+    [2]=>
+    int(3)
+  }
+  [4]=>
+  string(1) "("
+  [5]=>
+  array(3) {
+    [0]=>
+    int(376)
+    [1]=>
+    string(8) "<<<DOC2
+"
+    [2]=>
+    int(3)
+  }
+  [6]=>
+  array(3) {
+    [0]=>
+    int(377)
+    [1]=>
+    string(4) "DOC2"
+    [2]=>
+    int(4)
+  }
+  [7]=>
+  array(3) {
+    [0]=>
+    int(375)
+    [1]=>
+    string(1) "
+"
+    [2]=>
+    int(4)
+  }
+  [8]=>
+  string(1) ")"
+  [9]=>
+  string(1) "}"
+  [10]=>
+  array(3) {
+    [0]=>
+    int(314)
+    [1]=>
+    string(1) "
+"
+    [2]=>
+    int(5)
+  }
+  [11]=>
+  array(3) {
+    [0]=>
+    int(377)
+    [1]=>
+    string(4) "DOC1"
+    [2]=>
+    int(6)
+  }
+  [12]=>
+  string(1) ";"
+  [13]=>
+  array(3) {
+    [0]=>
+    int(375)
+    [1]=>
+    string(1) "
+"
+    [2]=>
+    int(6)
+  }
+}