lexer API

Attached is a patch that adds a public API for the lowest layers of
lexing.  This is meant to provide a solid foundation for the parsing that
Devel::Declare and similar modules do, and it complements the pluggable
keyword mechanism.  The API consists of some existing variables combined
with some new functions, all marked as experimental (which making them
public certainly is).

1 files changed, 4 insertions, 0 deletions

diff --git a/parser.h b/parser.h
index 74d8ef2197..462dcfdef8 100644
--- a/parser.h
+++ b/parser.h
@@ -107,6 +107,10 @@ typedef struct yy_parser {
 
 } yy_parser;
 
+/* flags for lexer API */
+#define LEX_STUFF_UTF8		0x00000001
+#define LEX_KEEP_PREVIOUS	0x00000002
+
 /*
  * Local variables:
  * c-indentation-style: bsd