Merge pull request #94 from erikrose/blessed-integration-docs-further-reading-and-growing-pains

(5) Blessed integration docs further reading and growing pains

3 files changed, 488 insertions, 0 deletions

diff --git a/docs/further.rst b/docs/further.rst
new file mode 100644
index 0000000..4847814
--- /dev/null
+++ b/docs/further.rst
@@ -0,0 +1,106 @@
+Further Reading
+===============
+
+As a developer's API, blessings is often bundled with frameworks and toolsets
+that dive deeper into Terminal I/O programming than :class:`~.Terminal` offers.
+Here are some recommended readings to help you along:
+
+- `terminfo(5)
+  <http://www.openbsd.org/cgi-bin/man.cgi?query=terminfo&apropos=0&sektion=5>`_
+  manpage of your preferred posix-like operating system. The capabilities
+  available as attributes of :class:`~.Terminal` are directly mapped to those
+  listed in the column **Cap-name**.
+
+- `termios(4)
+  <http://www.openbsd.org/cgi-bin/man.cgi?query=termios&apropos=0&sektion=4>`_
+  of your preferred posix-like operating system.
+
+- `The TTY demystified
+  <http://www.linusakesson.net/programming/tty/index.php>`_ by
+  Linus Åkesson.
+
+- `A Brief Introduction to Termios
+  <https://blog.nelhage.com/2009/12/a-brief-introduction-to-termios/>`_ by
+  Nelson Elhage.
+
+- Richard Steven's `Advance Unix Programming
+  <http://www.amazon.com/exec/obidos/ISBN=0201563177/wrichardstevensA/>`_
+  ("AUP") provides two very good chapters, "Terminal I/O" and
+  "Pseudo Terminals".
+
+- GNU's `The Termcap Manual
+  <https://www.gnu.org/software/termutils/manual/termcap-1.3/html_mono/termcap.html>`_
+  by Richard M. Stallman.
+
+- `Chapter 4 <http://compsci.hunter.cuny.edu/~sweiss/course_materials/unix_lecture_notes/chapter_04.pdf>`_
+  of CUNY's course material for *Introduction to System Programming*, by
+  `Stewart Weiss <http://compsci.hunter.cuny.edu/~sweiss/>`_
+
+- `Chapter 11
+  <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap11.html>`_
+  of the IEEE Open Group Base Specifications Issue 7, "General Terminal
+  Interface"
+
+- The GNU C Library documentation, section `Low-Level Terminal Interface
+  <http://www.gnu.org/software/libc/manual/html_mono/libc.html#toc-Low_002dLevel-Terminal-Interface-1>`_
+
+- The source code of many popular terminal emulators.  If there is ever any
+  question of "the meaning of a terminal capability", or whether or not your
+  preferred terminal emulator actually handles them, read the source!
+
+  These are often written in the C language, and directly map the
+  "Control Sequence Inducers" (CSI, literally ``\x1b[`` for most modern
+  terminal types) emitted by most terminal capabilities to an action in a
+  series of ``case`` switch statements.
+
+  - Many modern libraries are now based `https://github.com/GNOME/vte
+    <libvte>`_ (or just 'vte'): Gnome Terminal, sakura, Terminator, Lilyterm,
+    ROXTerm, evilvte, Termit, Termite, Tilda, tinyterm, lxterminal.
+  - `Thomas E. Dickey <http://invisible-island.net/>` has been maintaining
+    `xterm <http://invisible-island.net/xterm/xterm.html>`_, as well as a
+    primary maintainer of many related packages such as `ncurses 
+    <http://invisible-island.net/ncurses/ncurses.html>`_ for quite a long
+    while.  There is often speculation and misinformation among developers of
+    terminal emulators and programs that interact with them.  Thomas Dickey's
+    analysis is always thorough and complete.
+  - xterm, urxvt, SyncTerm, and EtherTerm.
+  - There are far too many to name, Chose one you like! Thomas Dickey
+
+
+- The source code of the tty(4), pty(4), and the given "console driver" for
+  any posix-like operating system.  If you search thoroughly enough, you will
+  eventually discover a terminal sequence decoder, usually a ``case`` switch
+  that translates ``\x1b[0m`` into a "reset color" action towards the video
+  driver.  Though ``tty.c`` is linked here (the only kernel file common among
+  them), it is probably not the most interesting, but it can get you started:
+
+     - `FreeBSD <https://github.com/freebsd/freebsd/blob/master/sys/kern/tty.c>`_
+     - `OpenBSD <http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/kern/tty.c?content-type=text/plain>`_
+     - `Illumos (Solaris) <https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/io/tty_common.c>`
+     - `Minix <https://github.com/minix3/minix/blob/master/minix/drivers/tty/tty/tty.c>`_
+     - `Linux <https://github.com/torvalds/linux/blob/master/drivers/tty/n_tty.c>`_
+
+  The TTY driver is a great introduction to Kernel and Systems programming,
+  because familiar components may be discovered and experimented with.  It is
+  available on all operating systems (except windows), and because of its
+  critical nature, examples of efficient file I/O, character buffers (often
+  implemented as "ring buffers") and even fine-grained kernel locking can be
+  found.
+
+- `termcap & terminfo (O'Reilly Nutshell)
+  <http://www.amazon.com/termcap-terminfo-OReilly-Nutshell-Linda/dp/0937175226>`_
+  by Linda Mui, Tim O'Reilly, and John Strang.
+
+- Note that System-V systems, also known as `Unix98
+  <http://en.wikipedia.org/wiki/Single_UNIX_Specification>`_ (SunOS, HP-UX,
+  AIX and others) use a `Streams <http://en.wikipedia.org/wiki/STREAMS>`_
+  interface.  On these systems, the `ioctl(2)
+  <http://pubs.opengroup.org/onlinepubs/009695399/functions/ioctl.html>`_
+  interface provides the ``PUSH`` and ``POP`` parameters to communicate with
+  a Streams device driver, which differs significantly from Linux and BSD.
+
+  Many of these systems provide compatible interfaces for Linux, but they may
+  not always be as complete as the counterpart they emulate, most especially
+  in regards to managing pseudo-terminals.
+
+
diff --git a/docs/index.rst b/docs/index.rst
index eabb7d7..986f483 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -11,6 +11,8 @@ Contents:
    intro
    overview
    api
+   further
+   pains
    history
 
 =======
diff --git a/docs/pains.rst b/docs/pains.rst
new file mode 100644
index 0000000..5d95d1e
--- /dev/null
+++ b/docs/pains.rst
@@ -0,0 +1,380 @@
+Growing Pains
+=============
+
+When constructing rich console applications, there are a surprisingly great
+number of portability issues.  Though Blessings provides an abstraction for
+the full terminal capability database to ensure compatibility across many
+types of terminals, there are still many portability issues that the
+terminal :attr:`~.kind` alone cannot abstract for you.
+
+This section intends to raise awareness and guide you through these kinds of
+issues.
+
+
+8 and 16 colors
+---------------
+
+Where 8 and 16 colors are used, they should be assumed to be the
+`CGA Color Palette`_.  Though there is no terminal standard that proclaims
+that the CGA colors are used, their values are the best approximations
+across all common hardware terminals and terminal emulators.
+
+A recent phenomenon of users is to customize their base 16 colors to provide
+(often, more "washed out") color schemes.  Furthermore, we are only recently
+getting LCD displays of colorspaces that achieve close approximation to the
+original video terminals.  Some find these values uncomfortably intense: in
+their original CRT form, their contrast and brightness was lowered by hardware
+dials, whereas today's LCD's typically display well only near full intensity.
+
+Though one can not *trust*, much less *query* through Terminal I/O routines
+the colorspace of the remote terminal, **we can**:
+
+- Trust that a close approximation of the `CGA Color Palette`_ for the base
+  16 colors will be displayed for **most** users.
+
+- Trust that users who have made the choice to adjust their palette have made
+  the choice to do so, and are able to re-adjust such palettes as necessary
+  to accommodate different programs (such as through the use of "Themes").
+
+.. note::
+
+   It has become popular to use dynamic system-wide color palette adjustments
+   in software such as `f.lux`_, which adjust the system-wide "Color Profile"
+   of the entire graphics display depending on the time of day.  One might
+   assume that ``term.blue("text")`` may be **completely** invisible to such
+   users during the night!
+
+
+Where is brown, purple, or grey?
+--------------------------------
+
+There are **only 8 color names** on even a 16-color terminal:  The second
+eight colors are "high intensity" versions of the first in direct series.  The
+colors *brown*, *purple*, and *grey* are not named in the first series, though
+they are available!
+
+- **brown**: *yellow is brown*, only high-intensity yellow
+  (``bright_yellow``) is yellow!
+
+- **purple**: *magenta is purple*.  In earlier, 4-bit color spaces, there
+  were only black, cyan, magenta, and white of low and high intensity, such
+  as found on a common home computer of the time such as the `ZX Spectrum`_.
+
+  Additional "colors" were only possible through dithering.  The color names
+  cyan and magenta on later graphics adapters are carried over from its
+  predecessors.  Although the color cyan remained true in RGB value on
+  16-color to its predecessor, magenta shifted farther towards blue from red
+  becoming purple (as true red was introduced as one of the base 8 colors).
+
+- **grey**: there are actually **three shades of grey** (or American spelling,
+  'gray'), though the color attribute named 'grey' does not exist!
+
+  In ascending order of intensity, the shades of grey are:
+
+  - ``bold_black``: in lieu of the uselessness of an "intense black", this is
+    color is instead mapped to "dark grey".
+  - ``white``: white is actually mild compared to the true color 'white': this
+    is more officially mapped to "common grey", and is often the default
+    foreground color.
+  - ``bright_white``: is pure white (``#ffffff``).
+
+
+white-on-black
+~~~~~~~~~~~~~~
+
+The default foreground and background should be assumed as *white-on-black*.
+
+For quite some time, the families of terminals produced by DEC, IBM, and
+Tektronix dominated the computing world with the default color scheme of
+*green-on-black* and less commonly *amber-on-black* monochrome displays:
+The inverse was a non-default configuration.  The IBM 3270 clients exclusively
+used *green-on-black* in both hardware and software emulators, and is likely
+a driving factor of the default *white-on-black* appearance of the first IBM
+Personal Computer.
+
+The less common *black-on-white* "ink paper" style of emulators is a valid
+concern for those designing terminal interfaces.  The color scheme of
+*black-on-white* directly conflicts with the intention of `bold is bright`_,
+where ``term.bright_red('ATTENTION!')`` may become difficult to read,
+as it appears as *pink on white*!
+
+
+History of ink-paper inspired black-on-white
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Early home computers with color video adapters, such as the Commodore 64
+provided *white-on-blue* as their basic video terminal configuration.  One can
+only assume such appearances were provided to demonstrate their color
+capabilities over competitors (such as the Apple ][).
+
+More common, X11's xterm and the software HyperTerm bundle with MS Windows
+provided an "ink on paper" *black-on-white* appearance as their default
+configuration.  Two popular emulators continue to supply *black-on-white* by
+default to this day: Xorg's xterm and Apple's Terminal.app.
+
+.. note:: Windows no longer supplies a terminal emulator: the "command prompt"
+   as we know it now uses the MSVCRT API routines to interact and does not
+   make use of terminal sequences, even ignoring those sequences that MS-DOS
+   family of systems previously interpreted through the ANSI.SYS driver,
+   though it continues to default to *white-on-black*.
+
+
+Bold is bright
+--------------
+
+**Where Bold is used, it should be assumed to be *Bright***.
+
+Due to the influence of early graphics adapters providing a set of 8
+"low-intensity" and 8 "high intensity" versions of the first, the term
+"bold" for terminals sequences is synonymous with "high intensity" in
+almost all circumstances.
+
+
+History of bold as "wide stroke"
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In typography, the true translation of "bold" is that a font should be
+displayed *with emphasis*.  In classical terms, this would be achieved by
+pen be re-writing over the same letters.  On a teletype or printer, this was
+similarly achieved by writing a character, backspacing, then repeating the
+same character in a form called **overstriking**.
+
+To bold a character, ``C``, one would emit the sequence ``C^HC`` where
+``^H`` is backspace (0x08).  To underline ``C``, one would would emit
+``C^H_``.
+
+**Video terminals do not support overstriking**.  Though the mdoc format for
+manual pages continue to emit overstriking sequences for bold and underline,
+translators such as mandoc will instead emit an appropriate terminal sequence.
+
+Many characters previously displayable by combining using overstriking of
+ASCII characters on teletypes, such as: ±, ≠, or ⩝ were delegated to a
+`code page`_ or lost entirely until the introduction of multibyte encodings.
+
+Much like the "ink paper" introduction in windowing systems for terminal
+emulators, "wide stroke" bold was introduced only much later when combined
+with operating systems that provided font routines such as TrueType.
+
+
+Enforcing white-on-black
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+In conclusion, *white-on-black* should be considered the default.  If there is
+a need to **enforce** *white-on-black* for terminal clients suspected to be
+defaulted as *black-on-white*, one would want to trust that a combination of
+``term.home + term.white_on_black + term.clear`` should repaint the entire
+emulator's window with the desired effect.
+
+However, this cannot be trusted to **all** terminal emulators to perform
+correctly!  Depending on your audience, you may instead ensure that the
+entire screen (including whitespace) is painted using the ``on_black``
+mnemonic.
+
+Beware of customized color schemes
+----------------------------------
+
+A recent phenomenon is for users to customize these first 16 colors of their
+preferred emulator to colors of their own liking.  Though this has always been
+possible with ``~/.XResources``, the introduction of PuTTy and iTerm2 to
+interactively adjustment these colors have made this much more common.
+
+This may cause your audience to see your intended interface in a wildly
+different form.  Your intended presentation may appear mildly unreadable.
+
+Users are certainly free to customize their colors however they like, but it
+should be known that displaying ``term.black_on_red("DANGER!")`` may appear
+as "grey on pastel red" to your audience, reducing the intended effect of
+intensity.
+
+
+256 colors can avoid customization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first instinct of a user who aliases ls(1) to ``ls -G`` or ``colorls``,
+when faced with the particularly low intensity of the default ``blue``
+attribute is **to adjust their terminal emulator's color scheme of the base
+16 colors**.
+
+This is not necessary: the environment variable ``LSCOLORS`` may be redefined
+to map an alternative color for blue, or to use ``bright_blue`` in its place.
+
+Furthermore, all common terminal text editors such as emacs or vim may be
+configured with "colorschemes" to make use of the 256-color support found in
+most modern emulators.  Many readable shades of blue are available, and many
+programs that emit such colors can be configured to emit a higher or lower
+intensity variant from the full 256 color space through program configuration.
+
+
+Monochrome and reverse
+----------------------
+
+Note that ``reverse`` takes the current foreground and background colors and
+reverses them.  In contrast, the compound formatter ``black_on_red`` would
+fail to set the background *or* foreground color on a monochrome display,
+resulting in the same stylization as ``normal`` -- it would not appear any
+different!
+
+If your userbase consists of monochrome terminals, you may wish to provide
+"lightbars" and other such effects using the compound formatter
+``red_reverse``.  In the literal sense of "set foreground color to red, then
+swap foreground and background", this produces a similar effect on
+**both** color and monochrome displays.
+
+For text, very few ``{color}_on_{color}`` formatters are visible with the
+base 16 colors, so you should generally wish for ``black_on_{color}``
+anyway.  By using ``{color}_reverse`` you may be portable with monochrome
+displays as well.
+
+
+Multibyte Encodings and Code pages
+----------------------------------
+
+A terminal that supports both multibyte encodings (UTF-8) and legacy 8-bit
+code pages (ISO 2022) may instruct the terminal to switch between both
+modes using the following sequences:
+
+  - ``\x1b%@``: Select default character set, that is ISO 8859-1 (ISO 2022).
+  - ``\x1b%G``: Select UTF-8 character set (ISO 2022).
+
+When a terminal is in ISO 2022 mode, you may use a sequence
+to request a terminal to change its `code page`_.  It begins by ``\x1b(``,
+followed by an ASCII character representing a code page selection.  For
+example ``\x1b(U`` on the VGA Linux console switches to the `IBM CP437`_
+`code page`_, allowing MS-DOS artwork to be displayed in its natural
+8-bit byte encoding.  A list of standard codes and the expected code page
+may be found on Thomas E. Dickey's xterm control sequences section on
+sequences following the `Control-Sequence-Inducer`_.
+
+Detecting multibyte
+~~~~~~~~~~~~~~~~~~~
+
+One can be assured that the connecting client is capable of representing
+UTF-8 and other multibyte character encodings by the Environment variable
+``LANG``.  If this is not possible, there is an alternative method:
+
+  - Emit Report Cursor Position (CPR), ``\x1b[6n`` and store response.
+  - Emit a multibyte UTF-8 character, such as ⦰ (``\x29\xb0``).
+  - Emit Report Cursor Position (CPR), ``\x1b[6n`` and store response.
+  - Determine the difference of the *(y, x)* location of the response.
+    If it is *1*, then the client decoded the two UTF-8 bytes as a
+    single character, and can be considered capable.  If it is *2*,
+    the client is using a `code page`_ and is incapable of decoding
+    a UTF-8 bytestream.
+
+Note that both SSH and Telnet protocols provide means for forwarding
+the ``LANG`` environment variable.  However, some transports such as
+a link by serial cable is incapable of forwarding Environment variables.
+
+Detecting screen size
+~~~~~~~~~~~~~~~~~~~~~
+
+While we're on the subject, there are times when :attr:`height` and
+:attr:`width` are not accurate -- when a transport does not provide the means
+to propagate the COLUMNS and ROWS Environment values, or propagate the
+SIGWINCH signals, such as through a serial link.
+
+The same means described above for multibyte encoding detection may be used to
+detect the remote client's window size:
+
+  - Move cursor to row 999, 999.
+  - Emit Report Cursor Position (CPR), ``\x1b[6n`` and store response.
+  - The return value is the window dimensions of the client.
+
+This is the method used by the program ``resize`` provided in the Xorg
+distribution, and its source may be viewed as file `resize.c`_.
+
+Alt or meta sends Escape
+------------------------
+
+Programs using GNU readline such as bash continue to provide default mappings
+such as *ALT+u/l* to uppercase/lowercase word after cursor. This is achieved
+by the configuration option, altSendsEscape or `metaSendsEscape
+<http://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Alt-and-Meta-Keys>`_
+
+The default for most terminals, however, is that the meta key is bound by
+the operating system (such as *META+f* for find), and that alt is used for
+inserting international keys, where the combination *ALT+u, a* is used
+to insert the character ``ä``.
+
+The ability to detect alt or meta key combinations is achieved by two means:
+
+  - The alt or meta key sets the 8th bit "high", so that *ALT+z* becomes
+    the value of ``'z'`` + 128: ``ú``
+  - The alt or meta key prefaces the combining character with escape, so
+    that *ALT+z* becomes *Escape + z*: ``\x1bz``
+
+It is therefore a recommendation to **avoid alt or meta keys entirely** in
+applications, and instead prefer the ctrl-key combinations, so as to avoid
+instructing your users to configure their terminal emulators to communicate
+such sequences.
+
+
+Backspace sends delete
+----------------------
+
+Typically, backspace is ``^H`` (8, or 0x08) and delete is ^? (127, or 0x7f).
+
+On some systems however, the key for backspace is actually labeled and
+transmitted as "delete", though its function in the operating system behaves
+just as backspace.  It is highly recommend to accept **both** ``KEY_DELETE``
+and ``KEY_BACKSPACE`` as having the same meaning except when implementing full
+screen editors.
+
+And only then, to provide the choice to map delete as true delete as a
+non-default configuration option.
+
+
+The misnomer of ANSI
+--------------------
+
+When people say 'ANSI Sequence', they are discussing:
+
+- Standard `ECMA-48`_: Control Functions for Coded Character Sets
+
+- `ANSI X3.64 <http://sydney.edu.au/engineering/it/~tapted/ansi.html>`_ from
+  1981, when the `American National Standards Institute
+  <http://www.ansi.org/>`_ adopted the `ECMA-48`_ as standard, which was later
+  withdrawn in 1997 (so in this sense it is *not* an ANSI standard).
+
+- The `ANSI.SYS`_ driver provided in MS-DOS and
+  clones.  The popularity of the IBM Personal Computer and MS-DOS of the era,
+  and its ability to display colored text further populated the idea that such
+  text "is ANSI".
+
+- The `IBM CP437`_ `code page`_ (which provided "block art" characters) paired
+  with `ECMA-48`_ sequences supported by the MS-DOS ANSI.SYS driver to create
+  artwork, known as `ANSI art <http://sixteencolors.net/>`_.
+
+  This is purely an American misnomer, because early IBM PC and clones in the
+  European nations did not ship with the `IBM CP437`_ `code page`_ by default.
+
+  Many people now mistake the difference between "ASCII art" and "ANSI art" to
+  be whether or not block art and other characters from the `IBM CP437`_
+  code page are used.  Where even such "ASCII art" may contain `ECMA-48`_
+  color codes!
+
+- The ANSI terminal database entry and its many descendants in the
+  `terminfo database
+  <http://invisible-island.net/ncurses/terminfo.src.html>`_.  This is mostly
+  due to terminals compatible with SCO UNIX, which was the successor of
+  Microsoft's Xenix, which brought some semblance of the Microsoft DOS
+  `ANSI.SYS`_ driver capabilities.
+
+- `Select Graphics Rendition (SGR) <http://vt100.net/docs/vt510-rm/SGR>`_
+  on vt100 clones, which include many of the common sequences in `ECMA-48`_.
+
+- Any sequence started by the `Control-Sequence-Inducer`_ is often
+  mistakenly termed as an "ANSI Escape Sequence" though not appearing in
+  `ECMA-48`_ or interpreted by the `ANSI.SYS`_ driver. The adjoining phrase
+  "Escape Sequence" is so termed because it follows the ASCII character
+  for the escape key (ESC, ``\x1b``).
+
+.. _code page: http://en.wikipedia.org/wiki/Code_page
+.. _IBM CP437: http://en.wikipedia.org/wiki/Code_page_437
+.. _CGA Color Palette: http://en.wikipedia.org/wiki/Color_Graphics_Adapter#With_an_RGBI_monitor
+.. _f.lux: https://justgetflux.com/
+.. _ZX Spectrum: http://en.wikipedia.org/wiki/List_of_8-bit_computer_hardware_palettes#ZX_Spectrum
+.. _Control-Sequence-Inducer: http://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Controls-beginning-with-ESC
+.. _resize.c: http://www.opensource.apple.com/source/X11apps/X11apps-13/xterm/xterm-207/resize.c
+.. _ANSI.SYS: http://www.kegel.com/nansi/
+.. _ECMA-48: http://www.ecma-international.org/publications/standards/Ecma-048.htm