Fix lexing of nested heredoc strings in token_get_all()

This fixes bug #60097.

Before two global variables CG(heredoc) and CG(heredoc_len) were used to
track the current heredoc label. In order to support nested heredoc
strings the *previous* heredoc label was assigned as the token value of
T_START_HEREDOC and the language_parser.y assigned that to CG(heredoc).

This created a dependency of the lexer on the parser. Thus the
token_get_all() function, which accesses the lexer directly without
also running the parser, was not able to tokenize nested heredoc strings
(and leaked memory). Same applies for the source-code highlighting
functions.

The new approach is to maintain a heredoc_label_stack in the lexer, which
contains all active heredoc labels.

As it is no longer required, T_START_HEREDOC and T_END_HEREDOC now don't
carry a token value anymore.

In order to make the work with zend_ptr_stack in this context more
convenient I added a new function zend_ptr_stack_top(), which retrieves the
top element of the stack (similar to zend_stack_top()).

1 files changed, 1 insertions, 4 deletions

diff --git a/ext/tokenizer/tokenizer.c b/ext/tokenizer/tokenizer.c
index c5cbf6c7d2..d22fd71243 100644
--- a/ext/tokenizer/tokenizer.c
+++ b/ext/tokenizer/tokenizer.c
@@ -138,11 +138,8 @@ static void tokenize(zval *return_value TSRMLS_DC)
 					token_line = ++CG(zend_lineno);
 					CG(increment_lineno) = 0;
 				}
-				add_next_index_stringl(keyword, Z_STRVAL(token), Z_STRLEN(token), 1);
-				efree(Z_STRVAL(token));
-			} else {
-				add_next_index_stringl(keyword, (char *)zendtext, zendleng, 1);
 			}
+			add_next_index_stringl(keyword, (char *)zendtext, zendleng, 1);
 			add_next_index_long(keyword, token_line);
 			add_next_index_zval(return_value, keyword);
 		} else {