toke.c: Fix bugs where UTF-8 is turned on in mid chunk

Previous commits have tightened up the checking of UTF-8 for
well-formedness in the input program or string eval.  This is done in
lex_next_chunk and lex_start.  But it doesn't handle the case of

    use utf8; foo

because 'foo' is checked while UTF-8 is still off.  This solves that
problem by noticing when utf8 is turned on, and then rechecking at the
next opportunity.

See thread beginning at
http://nntp.perl.org/group/perl.perl5.porters/242916

This fixes [perl #130675].  A test will be added in a future commit

This catches some errors earlier than they used to be and aborts. so
some tests in the suite had to be split into multiple parts.

1 files changed, 2 insertions, 0 deletions

diff --git a/parser.h b/parser.h
index ad148c26eb..4187e0a93d 100644
--- a/parser.h
+++ b/parser.h
@@ -115,6 +115,8 @@ typedef struct yy_parser {
     IV          sig_optelems;   /* number of optional signature elems seen */
     char        sig_slurpy;     /* the sigil of the slurpy var (or null) */
 
+    bool        recheck_utf8_validity;
+
     PERL_BITFIELD16	in_pod:1;      /* lexer is within a =pod section */
     PERL_BITFIELD16	filtered:1;    /* source filters in evalbytes */
     PERL_BITFIELD16	saw_infix_sigil:1; /* saw & or * or % operator */