lexer API fixes

The attached patch contains these fixes for the lexer API work:

* fix MinGW-revealed problem in BOM logic (replacing Jan's patch)
* fix warnings from t/op/incfilter.t
* probably fix g++ failure due to goto bypassing initialisation
* perl5112delta update

-zefram

Signed-off-by: H.Merijn Brand <h.m.brand@xs4all.nl>

2 files changed, 17 insertions, 12 deletions

diff --git a/pod/perl5112delta.pod b/pod/perl5112delta.pod
index 82412b2616..03b142540d 100644
--- a/pod/perl5112delta.pod
+++ b/pod/perl5112delta.pod
@@ -58,15 +58,7 @@ Extension modules can now cleanly hook into the Perl parser to define new
 kinds of keyword-headed expression and compound statement.  The syntax
 following the keyword is defined entirely by the extension.  This allow
 a completely non-Perl sublanguage to be parsed inline, with the right
-ops cleanly generated.
-
-This feature is currently considered experimental, and using it to do
-anything interesting is difficult.  Many necessary supporting facilities,
-such as the lexer and the pad system, can only be accessed through
-unsupported internal interfaces.  It is intended that the Perl 5.13
-development cycle will see the addition of clean, supported interfaces
-for many of these functions.  In Perl 5.12 most uses of pluggable keywords
-will be via L<Devel::Declare>.
+ops cleanly generated.  This feature is currently considered experimental.
 
 See L<perlapi/PL_keyword_plugin> for the mechanism.  The Perl core source
 distribution also includes a new module L<XS::APItest::KeywordRPN>, which
@@ -74,6 +66,16 @@ implements reverse Polish notation arithmetic via pluggable keywords.
 This module is mainly used for test purposes, and is not normally
 installed, but also serves as an example of how to use the new mechanism.
 
+=head2 APIs for more internals
+
+The lowest layers of the lexer and parts of the pad system now have C
+APIs available to XS extensions.  These are necessary to support proper
+use of pluggable keywords, but have other uses too.  The new APIs are
+experimental, and only cover a small proportion of what would be necessary
+to take full advantage of the core's facilities in these areas.  It is
+intended that the Perl 5.13 development cycle will see the addition of
+a full range of clean, supported interfaces.
+
 =head2 Overridable function lookup
 
 Where an extension module hooks the creation of rv2cv ops, to modify
diff --git a/toke.c b/toke.c
index a17dc4cde8..a4e9471381 100644
--- a/toke.c
+++ b/toke.c
@@ -1232,6 +1232,8 @@ Perl_lex_next_chunk(pTHX_ U32 flags)
     } else if (filter_gets(linestr, old_bufend_pos)) {
 	got_some = 1;
     } else {
+	if (!SvPOK(linestr))   /* can get undefined by filter_gets */
+	    sv_setpvs(linestr, "");
 	eof:
 	/* End of real input.  Close filehandle (unless it was STDIN),
 	 * then add implicit termination.
@@ -3867,6 +3869,7 @@ Perl_yylex(pTHX)
     register char *d;
     STRLEN len;
     bool bof = FALSE;
+    U32 fake_eof = 0;
 
     /* orig_keyword, gvp, and gv are initialized here because
      * jump to the label just_a_word_zero can bypass their
@@ -4314,7 +4317,8 @@ Perl_yylex(pTHX)
 	    goto retry;
 	}
 	do {
-	    U32 fake_eof = 0;
+	    fake_eof = 0;
+	    bof = PL_rsfp ? TRUE : FALSE;
 	    if (0) {
 	      fake_eof:
 		fake_eof = LEX_FAKE_EOF;
@@ -4331,8 +4335,7 @@ Perl_yylex(pTHX)
 	    s = PL_bufptr;
 	    /* If it looks like the start of a BOM or raw UTF-16,
 	     * check if it in fact is. */
-	    bof = PL_rsfp ? TRUE : FALSE;
-	    if (bof &&
+	    if (bof && PL_rsfp &&
 		     (*s == 0 ||
 		      *(U8*)s == 0xEF ||
 		      *(U8*)s >= 0xFE ||