Implement facility to plug in syntax triggered by keywords

Date: Tue, 27 Oct 2009 01:29:40 +0000
From: Zefram <zefram@fysh.org>
To: perl5-porters@perl.org
Subject: bareword sub lookups

Attached is a patch that changes how the tokeniser looks up subroutines,
when they're referenced by a bareword, for prototype and const-sub
purposes.  Formerly, it has looked up bareword subs directly in the
package, which is contrary to the way the generated op tree looks up
the sub, via an rv2cv op.  The patch makes the tokeniser generate the
rv2cv op earlier, and dig around in that.

The motivation for this is to allow modules to hook the rv2cv op
creation, to affect the name->subroutine lookup process.  Currently,
such hooking affects op execution as intended, but everything goes wrong
with a bareword ref where the tokeniser looks at some unrelated CV,
or a blank space, in the package.  With the patch in place, an rv2cv
hook correctly affects the tokeniser and therefore the prototype-based
aspects of parsing.

The patch also changes ck_subr (which applies the argument context and
checking parts of prototype behaviour) to handle subs referenced by an
RV const op inside the rv2cv, where formerly it would only handle a gv
op inside the rv2cv.  This is to support the most likely kind of modified
rv2cv op.

[This commit includes the Makefile.PL for XS-APITest-KeywordRPN missing
from the original patch, as well as updates to perldiag.pod and a
MANIFEST sort]

1 files changed, 65 insertions, 3 deletions

diff --git a/perlvars.h b/perlvars.h
index 49f4d5e31d..3639bd6ac0 100644
--- a/perlvars.h
+++ b/perlvars.h
@@ -8,9 +8,9 @@
  *
  */
 
-/****************/
-/* Truly global */
-/****************/
+/*
+=head1 Global Variables
+*/
 
 /* Don't forget to re-run embed.pl to propagate changes! */
 
@@ -186,3 +186,65 @@ PERLVARI(Gglobal_struct_size,	U16,	sizeof(struct perl_vars))
 PERLVARI(Ginterp_size_5_10_0, U16,
 	 PERL_INTERPRETER_SIZE_UPTO_MEMBER(PERL_LAST_5_10_0_INTERP_MEMBER))
 #endif
+
+/*
+=for apidoc AmUx|Perl_keyword_plugin_t|PL_keyword_plugin
+
+Function pointer, pointing at a function used to handle extended keywords.
+The function should be declared as
+
+	int keyword_plugin_function(pTHX_
+		char *keyword_ptr, STRLEN keyword_len,
+		OP **op_ptr)
+
+The function is called from the tokeniser, whenever a possible keyword
+is seen.  C<keyword_ptr> points at the word in the parser's input
+buffer, and C<keyword_len> gives its length; it is not null-terminated.
+The function is expected to examine the word, and possibly other state
+such as L<%^H|perlvar/%^H>, to decide whether it wants to handle it
+as an extended keyword.  If it does not, the function should return
+C<KEYWORD_PLUGIN_DECLINE>, and the normal parser process will continue.
+
+If the function wants to handle the keyword, it first must
+parse anything following the keyword that is part of the syntax
+introduced by the keyword.  The lexer interface is poorly documented.
+Broadly speaking, parsing needs to look at the buffer that extends
+from C<PL_parser-E<gt>bufptr> to C<PL_parser-E<gt>bufend>, and
+C<PL_parser-E<gt>bufptr> must be advanced across whatever text is
+consumed by the parsing process.  The buffer end is not necessarily the
+real end of the input text, but refilling the buffer is too complicated
+to discuss here.  See L<Devel::Declare> for some parsing experience,
+and hope for more core support in a future version of Perl.
+
+When a keyword is being handled, the plugin function must build
+a tree of C<OP> structures, representing the code that was parsed.
+The root of the tree must be stored in C<*op_ptr>.  The function then
+returns a contant indicating the syntactic role of the construct that
+it has parsed: C<KEYWORD_PLUGIN_STMT> if it is a complete statement, or
+C<KEYWORD_PLUGIN_EXPR> if it is an expression.  Note that a statement
+construct cannot be used inside an expression (except via C<do BLOCK>
+and similar), and an expression is not a complete statement (it requires
+at least a terminating semicolon).
+
+When a keyword is handled, the plugin function may also have
+(compile-time) side effects.  It may modify C<%^H>, define functions, and
+so on.  Typically, if side effects are the main purpose of a handler,
+it does not wish to generate any ops to be included in the normal
+compilation.  In this case it is still required to supply an op tree,
+but it suffices to generate a single null op.
+
+That's how the C<*PL_keyword_plugin> function needs to behave overall.
+Conventionally, however, one does not completely replace the existing
+handler function.  Instead, take a copy of C<PL_keyword_plugin> before
+assigning your own function pointer to it.  Your handler function should
+look for keywords that it is interested in and handle those.  Where it
+is not interested, it should call the saved plugin function, passing on
+the arguments it received.  Thus C<PL_keyword_plugin> actually points
+at a chain of handler functions, all of which have an opportunity to
+handle keywords, and only the last function in the chain (built into
+the Perl core) will normally return C<KEYWORD_PLUGIN_DECLINE>.
+
+=cut
+*/
+
+PERLVARI(Gkeyword_plugin, Perl_keyword_plugin_t, MEMBER_TO_FPTR(Perl_keyword_plugin_standard))