PerlIO documentation. Proofing and sub-editing requested.

p4raw-id: //depot/perlio@8581

1 files changed, 513 insertions, 0 deletions

diff --git a/pod/perliol.pod b/pod/perliol.pod
new file mode 100644
index 0000000000..6169d43949
--- /dev/null
+++ b/pod/perliol.pod
@@ -0,0 +1,513 @@
+
+=head1 NAME
+
+perliol - C API for Perl's implementation of IO in Layers.
+
+=head1 SYNOPSIS
+
+    /* Defining a layer ... */
+    #include <perliol.h>
+
+
+=head1 DESCRIPTION
+
+This document describes the behavior and implementation of the PerlIO abstraction
+described in L<perlapio> when C<USE_PERLIO> is defined (and C<USE_SFIO> is not).
+
+=head2 History and Background
+
+The PerlIO abstraction was introduced in perl5.003_02 but languished as just
+an abstraction until perl5.7.0. However during that time a number of perl extenstions
+switch to using it, so the API is mostly fixed to maintain (source) compatibility.
+
+The aim of the implementation is to provide the PerlIO API in a flexible and
+platform neutral manner. It is also a trial of an "Object Oriented C, with vtables"
+approach which may be applied to perl6.
+
+=head2 Layers vs Disciplines
+
+Initial discussion of the ability to modify IO streams behaviour used the term
+"discipline" for the entities which were added. This came (I believe) from the use
+of the term in "sfio", which in turn borowed it from "line disciplines" on Unix
+terminals. However, this document (and the C code) uses the term "layer".
+This is I hope a natural term given the implementation, and should avoid conotations
+that are inherent in earlier uses of "discipline" for things which are rather different.
+
+=head2 Data Structures
+
+The basic data structure is a PerlIOl:
+
+	typedef struct _PerlIO PerlIOl;
+	typedef struct _PerlIO_funcs PerlIO_funcs;
+	typedef PerlIOl *PerlIO;
+
+	struct _PerlIO
+	{
+	 PerlIOl *	next;       /* Lower layer */
+	 PerlIO_funcs *	tab;        /* Functions for this layer */
+	 IV		flags;      /* Various flags for state */
+	};
+
+A PerlIOl * is a pointer to to the struct, and the I<application> level PerlIO *
+is a pointer to a PerlIOl * - i.e. a pointer to a pointer to the struct.
+This allows the application level PerlIO * to remain constant while the actual
+PerlIOl * underneath changes. (Compare perl's SV * which remains constant
+while its sv_any field changes as the scalar's type changes.)
+An IO stream is then in general represented as a pointer to this linked-list
+of "layers".
+
+It should be noted that because of the double indirection in a PerlIO *,
+a &(perlio->next) "is" a PerlIO *, and so to some degree at least
+one layer can use the "standard" API on the next layer down.
+
+A "layer" is composed of two parts:
+
+=over 4
+
+=item 1. The functions and attributes of the "layer class".
+
+=item 2. The per-instance data for a particular handle.
+
+=back
+
+=head2 Functions and Attributes
+
+The functions and attributes are accessed via the "tab" (for table) member of
+PerlIOl. The functions (methods of the layer "class") are fixed, and are defined by the
+PerlIO_funcs type. They are broadly the same as the public PerlIO_xxxxx functions:
+
+	struct _PerlIO_funcs
+	{
+	 char *		name;
+	 Size_t		size;
+	 IV		kind;
+	 IV		(*Fileno)(PerlIO *f);
+	 PerlIO *	(*Fdopen)(PerlIO_funcs *tab, int fd, const char *mode);
+	 PerlIO *	(*Open)(PerlIO_funcs *tab, const char *path, const char *mode);
+	 int		(*Reopen)(const char *path, const char *mode, PerlIO *f);
+	 IV		(*Pushed)(PerlIO *f,const char *mode,const char *arg,STRLEN len);
+	 IV		(*Popped)(PerlIO *f);
+	 /* Unix-like functions - cf sfio line disciplines */
+	 SSize_t	(*Read)(PerlIO *f, void *vbuf, Size_t count);
+	 SSize_t	(*Unread)(PerlIO *f, const void *vbuf, Size_t count);
+	 SSize_t	(*Write)(PerlIO *f, const void *vbuf, Size_t count);
+	 IV		(*Seek)(PerlIO *f, Off_t offset, int whence);
+	 Off_t		(*Tell)(PerlIO *f);
+	 IV		(*Close)(PerlIO *f);
+	 /* Stdio-like buffered IO functions */
+	 IV		(*Flush)(PerlIO *f);
+	 IV		(*Fill)(PerlIO *f);
+	 IV		(*Eof)(PerlIO *f);
+	 IV		(*Error)(PerlIO *f);
+	 void		(*Clearerr)(PerlIO *f);
+	 void		(*Setlinebuf)(PerlIO *f);
+	 /* Perl's snooping functions */
+	 STDCHAR *	(*Get_base)(PerlIO *f);
+	 Size_t		(*Get_bufsiz)(PerlIO *f);
+	 STDCHAR *	(*Get_ptr)(PerlIO *f);
+	 SSize_t	(*Get_cnt)(PerlIO *f);
+	 void		(*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
+	};
+
+The first few members of the struct give a "name" for the layer, the size to C<malloc>
+for the per-instance data, and some flags which are attributes of the class as whole
+(such as whether it is a buffering layer), then follow the functions which fall into
+four basic groups:
+
+=over 4
+
+=item 1. Opening and setup functions
+
+=item 2. Basic IO operations
+
+=item 3. Stdio class buffering options.
+
+=item 4. Functions to support Perl's traditional "fast" access to the buffer.
+
+=back
+
+A layer does not have to implement all the functions, but the whole table has
+to be present. Unimplemented slots can be NULL (which will will result in an error
+when called) or can be filled in with stubs to "inherit" behaviour from
+a "base class". This "inheritance" is fixed for all instances of the layer,
+but as the layer chooses which stubs to populate the table, limited
+"multiple inheritance" is possible.
+
+=head2 Per-instance Data
+
+The per-instance data are held in memory beyond the basic PerlIOl struct,
+by making a PerlIOl the first member of the layer's struct thus:
+
+	typedef struct
+	{
+	 struct _PerlIO base;       /* Base "class" info */
+	 STDCHAR *	buf;        /* Start of buffer */
+	 STDCHAR *	end;        /* End of valid part of buffer */
+	 STDCHAR *	ptr;        /* Current position in buffer */
+	 Off_t		posn;       /* Offset of buf into the file */
+	 Size_t		bufsiz;     /* Real size of buffer */
+	 IV		oneword;    /* Emergency buffer */
+	} PerlIOBuf;
+
+In this way (as for perl's scalars) a pointer to a PerlIOBuf can be treated
+as a pointer to a PerlIOl.
+
+=head2 Layers in action.
+
+                table           perlio          unix
+            |           |
+            +-----------+    +----------+    +--------+
+   PerlIO ->|           |--->|  next    |--->|  NULL  |
+            +-----------+    +----------+    +--------+
+            |           |    |  buffer  |    |   fd   |
+            +-----------+    |          |    +--------+
+            |           |    +----------+
+
+
+The above attempts to show how the layer scheme works in a simple case.
+The applications PerlIO * points to an entry in the table(s) representing open
+(allocated) handles. For example the first three slots in the table correspond
+to C<stdin>,C<stdout> and C<stderr>. The table in turn points to the current
+"top" layer for the handle - in this case an instance of the generic buffering
+layer "perlio". That layer in turn points to the next layer down - in this
+case the lowlevel "unix" layer.
+
+The above is roughly equivalent to a "stdio" buffered stream, but with much more
+flexibility:
+
+=over 4
+
+=item *
+
+If Unix level read/write/lseek is not appropriate for (say) sockets then
+the "unix" layer can be replaced (at open time or even dynamically) with a
+"socket" layer.
+
+=item *
+
+Different handles can have different buffering schemes. The "top" layer
+could be the "mmap" layer if reading disk files was quicker using C<mmap>
+than C<read>. An "unbuffered" stream can be implemented simply by
+not having a buffer layer.
+
+=item *
+
+Extra layers can be inserted to process the data as it flows through.
+This was the driving need for including the scheme in perkl5.70+ - we needed a mechanism
+to allow data to be translated bewteen perl's internal encoding (conceptually
+at least Unicode as UTF-8), and the "native" format used by the system.
+This is provided by the ":encoding(xxxx)" layer which typically sits above
+the buffering layer.
+
+=item *
+
+A layer can be added that does "\n" to CRLF translation. This layer can be used
+on any platform, not just those that normally do such things.
+
+=back
+
+=head2 Per-instance flag bits
+
+The generic flag bits are a hybrid of O_XXXXX style flags deduced from
+the mode string passed to PerlIO_open() and state bits for typical buffer
+layers.
+
+=over4
+
+=item PERLIO_F_EOF
+
+End of file.
+
+=item PERLIO_F_CANWRITE
+
+Writes are permited i.e. opened as "w" or "r+" or "a". etc.
+
+=item  PERLIO_F_CANREAD
+
+Reads are permited i.e. opened "r" or "w+" (or even "a+" - ick).
+
+=item PERLIO_F_ERROR
+
+An error has occured (for PerlIO_error())
+
+=item PERLIO_F_TRUNCATE
+
+Truncate file suggested by open mode.
+
+=item PERLIO_F_APPEND
+
+All writes should be appends.
+
+=item PERLIO_F_CRLF
+
+Layer is performing Win32-like "\n" => CR,LF for output and CR,LF => "\n" for
+input. Normally the provided "crlf" layer is only layer than need bother about
+this. PerlIO_binmode() will mess with this flag rather than add/remove layers
+if the PERLIO_K_CANCRLF bit is set for the layers class.
+
+=item PERLIO_F_UTF8
+
+Data for this written to this layer should be UTF-8 encoded, data provided
+by this layer should be considered UTF-8 encoded. Can be set on any layer
+by ":utf8" dummy layer. Also set on ":encoding" layer.
+
+=item PERLIO_F_UNBUF
+
+Layer is unbuffered - i.e. write to next layer down should occur for
+each write to this layer.
+
+=item PERLIO_F_WRBUF
+
+The buffer for this layer currently holds data written to it but not sent
+to next layer.
+
+=item PERLIO_F_RDBUF
+
+The buffer for this layer currently holds unconsumed data read from
+layer below.
+
+=item PERLIO_F_LINEBUF
+
+Layer is line buffered. Write data should be passed to next layer down whenever a
+"\n" is seen. Any data beyond the "\n" should then be processed.
+
+=item PERLIO_F_TEMP
+
+File has been unlink()ed, or should be deleted on close().
+
+=item PERLIO_F_OPEN
+
+Handle is open.
+
+=item PERLIO_F_FASTGETS
+
+This instance of this layer supports the "fast gets" interface.
+Normally set based on PERLIO_K_FASTGETS for the class and by the
+existance of the function(s) in the table. However a class that
+normally provides that interface may need to avoid it on a
+particular instance. The "pending" layer needs to do this when
+it is pushed above an layer which does not support the interface.
+(Perls sv_gets() does not expect the steams fast gets behaviour
+to change during one "get".)
+
+=back
+
+=head2 Methods in Detail
+
+=over 4
+
+=item IV	(*Fileno)(PerlIO *f);
+
+Returns the Unix/Posix numeric file decriptor for the handle.
+Normally PerlIOBase_fileno() (which just asks next layer down) will suffice for this.
+
+=item  PerlIO *	(*Fdopen)(PerlIO_funcs *tab, int fd, const char *mode);
+
+Should (perhaps indirectly) call PerlIO_allocate() to allocate a slot
+in the table and associate it with the given numeric file descriptor,
+which will be open in an manner compatible with the supplied mode string.
+
+=item  PerlIO *	(*Open)(PerlIO_funcs *tab, const char *path, const char *mode);
+
+Should attempt to open the given path and if that succeeds then (perhaps indirectly)
+call PerlIO_allocate() to allocate a slot in the table and associate it with the
+layers information for the opened file.
+
+=item  int		(*Reopen)(const char *path, const char *mode, PerlIO *f);
+
+Re-open the supplied PerlIO * to connect it to C<path> in C<mode>. Returns as success flag.
+Perl does not use this and L<perlapio> marks it as subject to change.
+
+=item  IV		(*Pushed)(PerlIO *f,const char *mode,const char *arg,STRLEN len);
+
+Called when the layer is pushed onto the stack. The C<mode> argument may be NULL if this
+occurs post-open. The C<arg> and C<len> will be present if an argument string was
+passed. In most cases this should call PerlIOBase_pushed() to conver C<mode> into
+the appropriate PERLIO_F_XXXXX flags in addition to any actions the layer itself takes.
+
+=item  IV		(*Popped)(PerlIO *f);
+
+Called when the layer is popped from the stack. A layer will normally be popped after
+Close() is called. But a layer can be popped without being closed if the program
+is dynamically managing layers on the stream. In such cases Popped() should free
+any resources (buffers, translation tables, ...) not held directly in the layer's
+struct.
+
+=item  SSize_t	(*Read)(PerlIO *f, void *vbuf, Size_t count);
+
+Basic read operation. Returns actual bytes read, or -1 on an error.
+Typically will call Fill and manipulate pointers (possibly via the API).
+PerlIOBuf_read() may be suitable for derived classes which provide "fast gets" methods.
+
+=item  SSize_t	(*Unread)(PerlIO *f, const void *vbuf, Size_t count);
+
+A superset of stdio's ungetc(). Should arrange for future reads to see the bytes in C<vbuf>.
+If there is no obviously better implementation then PerlIOBase_unread() provides
+the function by pushing a "fake" "pending" layer above the calling layer.
+
+=item  SSize_t	(*Write)(PerlIO *f, const void *vbuf, Size_t count);
+
+Basic write operation. Returns bytes written or -1 on an error.
+
+=item  IV		(*Seek)(PerlIO *f, Off_t offset, int whence);
+
+Position the file pointer. Should normally call its own Flush method and
+then the Seek method of next layer down.
+
+=item  Off_t		(*Tell)(PerlIO *f);
+
+Return the file pointer. May be based on layers cached concept of position to
+avoid overhead.
+
+=item  IV		(*Close)(PerlIO *f);
+
+Close the stream. Should normally call PerlIOBase_close() to flush itself
+and Close layers below and then deallocate any data structures (buffers, translation
+tables, ...) not  held directly in the data structure.
+
+=item  IV		(*Flush)(PerlIO *f);
+
+Should make streams state consistent with layers below. That is any
+buffered write data should be written, and file position of lower layer
+adjusted for data read fron below but not actually consumed.
+
+=item  IV		(*Fill)(PerlIO *f);
+
+The buffer for this layer should be filled (for read) from layer below.
+
+=item  IV		(*Eof)(PerlIO *f);
+
+Return end-of-file indicator. PerlIOBase_eof() is normally sufficient.
+
+=item  IV		(*Error)(PerlIO *f);
+
+Return error indicator. PerlIOBase_error() is normally sufficient.
+
+=item  void		(*Clearerr)(PerlIO *f);
+
+Clear end-of-file and error indicators. Should call PerlIOBase_clearerr()
+to set the PERLIO_F_XXXXX flags, which may suffice.
+
+=item  void		(*Setlinebuf)(PerlIO *f);
+
+Mark the stream as line buffered.
+
+=item  STDCHAR *	(*Get_base)(PerlIO *f);
+
+Allocate (if not already done so) the read buffer for this layer and
+return pointer to it.
+
+=item  Size_t		(*Get_bufsiz)(PerlIO *f);
+
+Return the number of bytes that last Fill() put in the buffer.
+
+=item  STDCHAR *	(*Get_ptr)(PerlIO *f);
+
+Return the current read pointer relative to this layers buffer.
+
+=item  SSize_t	(*Get_cnt)(PerlIO *f);
+
+Return the number of bytes left to be read in the current buffer.
+
+=item  void		(*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
+
+Adjust the read pointer and count of bytes to match C<ptr> and/or C<cnt>.
+The application (or layer above) must ensure they are consistent.
+(Checking is allowed by the paranoid.)
+
+=back
+
+
+=head2 Core Layers
+
+The file C<perlio.c> provides the following layers:
+
+=over 4
+
+=item "unix"
+
+A basic non-buffered layer which calls Unix/POSIX read(), write(), lseek(), close().
+No buffering. Even on platforms that distinguish between O_TEXT and O_BINARY
+this layer is always O_BINARY.
+
+=item "perlio"
+
+A very complete generic buffering layer which provides the whole of PerlIO API.
+It is also intended to be used as a "base class" for other layers. (For example
+its Read() method is implemented in terms of the Get_cnt()/Get_ptr()/Set_ptrcnt()
+methods).
+
+"perlio" over "unix" provides a complete replacement for stdio as seen via PerlIO API.
+This is the default for USE_PERLIO when system's stdio does not permit perl's
+"fast gets" access, and which do not distinguish between O_TEXT and O_BINARY.
+
+=item "stdio"
+
+A layer which provides the PerlIO API via the layer scheme, but implements it by calling
+system's stdio. This is (currently) the default if system's stdio provides sufficient
+access to allow perl's "fast gets" access and which do not distinguish between O_TEXT and
+O_BINARY.
+
+=item "crlf"
+
+A layer derived using "perlio" as a base class. It provides Win32-like "\n" to CR,LF
+translation. Can either be applied above "perlio" or serve as the buffer layer itself.
+"crlf" over "unix" is the default if system distinguishes between O_TEXT and O_BINARY
+opens. (At some point "unix" will be replaced by a "native" Win32 IO layer on that
+platform, as Win32's read/write layer has various drawbacks.)
+The "crlf" layer is a reasonable model for a layer which transforms data in some way.
+
+=item "mmap"
+
+If Configure detects C<mmap()> functions this layer is provided (with "perlio" as a
+"base") which does "read" operations by mmap()ing the file. Performance improvement
+is marginal on modern systems, so it is mainly there as a proof of concept.
+It is likely to be unbundled from the core at some point.
+The "mmap" layer is a reasonable model for a minimalist "derived" layer.
+
+=item "pending"
+
+An "internal" derivative of "perlio" which can be used to provide Unread() function
+for layers which have no buffer or cannot be bothered.
+(Basically this layer's Fill() pops itself off the stack and so resumes reading
+from layer below.)
+
+=item "raw"
+
+A dummy layer which never exists on the layer stack. Instead when "pushed" it
+actually pops the stack!, removing itself, and any other layers until it reaches
+a layer with the class PERLIO_K_RAW bit set.
+
+=item "utf8"
+
+Another dummy layer. When pushed it pops itself and sets the PERLIO_F_UTF8 flag
+on the layer which was (and now is once more) the top of the stack.
+
+=back
+
+In addition C<perlio.c> also provides a number of PerlIOBase_xxxx() functions
+which are intended to be used in the table slots of classes which do not need
+to do anything special for a particular method.
+
+=head2 Extension Layers
+
+Layers can made available by extension modules.
+
+=over 4
+
+=item "encoding"
+
+   use Encoding;
+
+makes this layer available. It is an example of a layer which takes an argument.
+as it is called as:
+
+   open($fh,"<:encoding(iso-8859-7)",$pathname)
+
+=back
+
+
+=cut
+
+
+