@c -*-texinfo-*-
@c This is part of the GNU Guile Reference Manual.
@c Copyright (C) 2013, 2020 Free Software Foundation, Inc.
@c See the file guile.texi for copying conditions.

@c Note: Don't use "Texinfo" as the node name here because this leads to
@c a clash in the HTML output between texinfo.html (from the "texinfo"
@c node) and Texinfo.html on case-insensitive file systems such as
@c HFS+ (MacOS X).
@node Texinfo Processing
@section Texinfo Processing

@menu
* texinfo::              Parse texinfo files or fragments into @code{stexi}, a scheme representation
* texinfo docbook::      Transform a subset of docbook into @code{stexi}
* texinfo html::         Transform @code{stexi} into HTML
* texinfo indexing::     Extract an index from a piece of @code{stexi}
* texinfo string-utils::  String utility functions used by the texinfo processor
* texinfo plain-text::   Render @code{stexi} as plain text
* texinfo serialize::    Render @code{stexi} as texinfo
* texinfo reflection::   Enable texinfo across Guile's help system
@end menu

@node texinfo
@subsection (texinfo)
@subsubsection Overview
@subheading Texinfo processing in scheme
This module parses texinfo into SXML. TeX will always be the processor
of choice for print output, of course. However, although @code{makeinfo}
works well for info, its output in other formats is not very
customizable, and the program is not extensible as a whole. This module
aims to provide an extensible framework for texinfo processing that
integrates texinfo into the constellation of SXML processing tools.

@subheading Notes on the SXML vocabulary
Consider the following texinfo fragment:

@example 
 @@deffn Primitive set-car! pair value
 This function...
 @@end deffn
@end example

Logically, the category (Primitive), name (set-car!), and arguments
(pair value) are ``attributes'' of the deffn, with the description as
the content. However, texinfo allows for @@-commands within the
arguments to an environment, like @code{@@deffn}, which means that
texinfo ``attributes'' are PCDATA. XML attributes, on the other hand,
are CDATA. For this reason, ``attributes'' of texinfo @@-commands are
called ``arguments'', and are grouped under the special element, `%'.

Because `%' is not a valid NCName, stexinfo is a superset of SXML. In
the interests of interoperability, this module provides a conversion
function to replace the `%' with `texinfo-arguments'.

@subsubsection Usage
@anchor{texinfo call-with-file-and-dir}@defun call-with-file-and-dir filename proc
Call the one-argument procedure @var{proc} with an input port that reads
from @var{filename}. During the dynamic extent of @var{proc}'s
execution, the current directory will be @code{(dirname
@var{filename})}. This is useful for parsing documents that can include
files by relative path name.

@end defun

@anchor{texinfo texi-command-specs}@defvar texi-command-specs
@end defvar

@anchor{texinfo texi-command-depth}@defun texi-command-depth command max-depth
Given the texinfo command @var{command}, return its nesting level, or
@code{#f} if it nests too deep for @var{max-depth}.

Examples:

@example 
 (texi-command-depth 'chapter 4)        @result{} 1
 (texi-command-depth 'top 4)            @result{} 0
 (texi-command-depth 'subsection 4)     @result{} 3
 (texi-command-depth 'appendixsubsec 4) @result{} 3
 (texi-command-depth 'subsection 2)     @result{} #f
@end example

@end defun

@anchor{texinfo texi-fragment->stexi}@defun texi-fragment->stexi string-or-port
Parse the texinfo commands in @var{string-or-port}, and return the
resultant stexi tree. The head of the tree will be the special command,
@code{*fragment*}.

@end defun

@anchor{texinfo texi->stexi}@defun texi->stexi port
Read a full texinfo document from @var{port} and return the parsed stexi
tree. The parsing will start at the @code{@@settitle} and end at
@code{@@bye} or EOF.

@end defun

@anchor{texinfo stexi->sxml}@defun stexi->sxml tree
Transform the stexi tree @var{tree} into sxml. This involves replacing
the @code{%} element that keeps the texinfo arguments with an element
for each argument.

FIXME: right now it just changes % to @code{texinfo-arguments} -- that
doesn't hang with the idea of making a dtd at some point

@end defun

@node texinfo docbook
@subsection (texinfo docbook)
@subsubsection Overview
@c 
This module exports procedures for transforming a limited subset of the
SXML representation of docbook into stexi. It is not complete by any
means. The intention is to gather a number of routines and stylesheets
so that external modules can parse specific subsets of docbook, for
example that set generated by certain tools.

@subsubsection Usage
@anchor{texinfo docbook *sdocbook->stexi-rules*}@defvar *sdocbook->stexi-rules*
@end defvar

@anchor{texinfo docbook *sdocbook-block-commands*}@defvar *sdocbook-block-commands*
@end defvar

@anchor{texinfo docbook sdocbook-flatten}@defun sdocbook-flatten sdocbook
"Flatten" a fragment of sdocbook so that block elements do not nest
inside each other.

Docbook is a nested format, where e.g. a @code{refsect2} normally
appears inside a @code{refsect1}. Logical divisions in the document are
represented via the tree topology; a @code{refsect2} element
@emph{contains} all of the elements in its section.

On the contrary, texinfo is a flat format, in which sections are marked
off by standalone section headers like @code{@@subsection}, and block
elements do not nest inside each other.

This function takes a nested sdocbook fragment @var{sdocbook} and
flattens all of the sections, such that e.g.

@example 
 (refsect1 (refsect2 (para "Hello")))
@end example

becomes

@example 
 ((refsect1) (refsect2) (para "Hello"))
@end example

Oftentimes (always?) sectioning elements have @code{<title>} as their
first element child; users interested in processing the @code{refsect*}
elements into proper sectioning elements like @code{chapter} might be
interested in @code{replace-titles} and @code{filter-empty-elements}.
@xref{texinfo docbook replace-titles,,replace-titles}, and @ref{texinfo
docbook filter-empty-elements,,filter-empty-elements}.

Returns a nodeset; that is to say, an untagged list of stexi elements.
@xref{SXPath}, for the definition of a nodeset.

@end defun

@anchor{texinfo docbook filter-empty-elements}@defun filter-empty-elements sdocbook
Filters out empty elements in an sdocbook nodeset. Mostly useful after
running @code{sdocbook-flatten}.

@end defun

@anchor{texinfo docbook replace-titles}@defun replace-titles sdocbook-fragment
Iterate over the sdocbook nodeset @var{sdocbook-fragment}, transforming
contiguous @code{refsect} and @code{title} elements into the appropriate
texinfo sectioning command. Most useful after having run
@code{sdocbook-flatten}.

For example:

@example 
 (replace-titles '((refsect1) (title "Foo") (para "Bar.")))
    @result{} '((chapter "Foo") (para "Bar."))
@end example

@end defun

@node texinfo html
@subsection (texinfo html)
@subsubsection Overview
This module implements transformation from @code{stexi} to HTML. Note
that the output of @code{stexi->shtml} is actually SXML with the HTML
vocabulary. This means that the output can be further processed, and
that it must eventually be serialized by @code{sxml->xml}.
@xref{Reading and Writing XML}.

References (i.e., the @code{@@ref} family of commands) are resolved by a
@dfn{ref-resolver}.  @xref{texinfo html
add-ref-resolver!,add-ref-resolver!}.

@subsubsection Usage
@anchor{texinfo html add-ref-resolver!}@defun add-ref-resolver! proc
Add @var{proc} to the head of the list of ref-resolvers. @var{proc} will
be expected to take the name of a node and the name of a manual and
return the URL of the referent, or @code{#f} to pass control to the next
ref-resolver in the list.

The default ref-resolver will return the concatenation of the manual
name, @code{#}, and the node name.

@end defun

@anchor{texinfo html stexi->shtml}@defun stexi->shtml tree
Transform the stexi @var{tree} into shtml, resolving references via
ref-resolvers. See the module commentary for more details.

@end defun

@anchor{texinfo html urlify}@defun urlify str
@end defun

@node texinfo indexing
@subsection (texinfo indexing)
@subsubsection Overview
@c texinfo formatting
Given a piece of stexi, return an index of a specified variety.

Note that currently, @code{stexi-extract-index} doesn't differentiate
between different kinds of index entries. That's a bug ;)

@subsubsection Usage
@anchor{texinfo indexing stexi-extract-index}@defun stexi-extract-index tree manual-name kind
Given an stexi tree @var{tree}, index all of the entries of type
@var{kind}. @var{kind} can be one of the predefined texinfo indices
(@code{concept}, @code{variable}, @code{function}, @code{key},
@code{program}, @code{type}) or one of the special symbols @code{auto}
or @code{all}. @code{auto} will scan the stext for a @code{(printindex)}
statement, and @code{all} will generate an index from all entries,
regardless of type.

The returned index is a list of pairs, the @sc{car} of which is the
entry (a string) and the @sc{cdr} of which is a node name (a string).

@end defun

@node texinfo string-utils
@subsection (texinfo string-utils)
@subsubsection Overview
Module @samp{(texinfo string-utils)} provides various string-related
functions useful to Guile's texinfo support.

@subsubsection Usage
@anchor{texinfo string-utils escape-special-chars}@defun escape-special-chars str special-chars escape-char
Returns a copy of @var{str} with all given special characters preceded
by the given @var{escape-char}.

@var{special-chars} can either be a single character, or a string
consisting of all the special characters.

@lisp 
;; make a string regexp-safe...
 (escape-special-chars "***(Example String)***"  
                      "[]()/*." 
                      #\\)
=> "\\*\\*\\*\\(Example String\\)\\*\\*\\*"

;; also can escape a singe char...
 (escape-special-chars "richardt@@vzavenue.net"
                      #\@@
                      #\@@)
=> "richardt@@@@vzavenue.net"
@end lisp

@end defun

@anchor{texinfo string-utils transform-string}@defun transform-string str match? replace [start] [end]
Uses @var{match?} against each character in @var{str}, and performs a
replacement on each character for which matches are found.

@var{match?} may either be a function, a character, a string, or
@code{#t}. If @var{match?} is a function, then it takes a single
character as input, and should return @samp{#t} for matches.
@var{match?} is a character, it is compared to each string character
using @code{char=?}. If @var{match?} is a string, then any character in
that string will be considered a match. @code{#t} will cause every
character to be a match.

If @var{replace} is a function, it is called with the matched character
as an argument, and the returned value is sent to the output string via
@samp{display}. If @var{replace} is anything else, it is sent through
the output string via @samp{display}.

Note that the replacement for the matched characters does not need to be
a single character. That is what differentiates this function from
@samp{string-map}, and what makes it useful for applications such as
converting @samp{#\&} to @samp{"&amp;"} in web page text. Some other
functions in this module are just wrappers around common uses of
@samp{transform-string}. Transformations not possible with this function
should probably be done with regular expressions.

If @var{start} and @var{end} are given, they control which portion of
the string undergoes transformation. The entire input string is still
output, though. So, if @var{start} is @samp{5}, then the first five
characters of @var{str} will still appear in the returned string.

@lisp 
; these two are equivalent...
 (transform-string str #\space #\-) ; change all spaces to -'s
 (transform-string str (lambda (c) (char=? #\space c)) #\-)
@end lisp

@end defun

@anchor{texinfo string-utils expand-tabs}@defun expand-tabs str [tab-size]
Returns a copy of @var{str} with all tabs expanded to spaces.
@var{tab-size} defaults to 8.

Assuming tab size of 8, this is equivalent to:

@lisp 
 (transform-string str #\tab "        ")
@end lisp

@end defun

@anchor{texinfo string-utils center-string}@defun center-string str [width] [chr] [rchr]
Returns a copy of @var{str} centered in a field of @var{width}
characters. Any needed padding is done by character @var{chr}, which
defaults to @samp{#\space}. If @var{rchr} is provided, then the padding
to the right will use it instead. See the examples below. left and
@var{rchr} on the right. The default @var{width} is 80. The default
@var{chr} and @var{rchr} is @samp{#\space}. The string is never
truncated.

@lisp 
 (center-string "Richard Todd" 24)
=> "      Richard Todd      "

 (center-string " Richard Todd " 24 #\=)
=> "===== Richard Todd ====="

 (center-string " Richard Todd " 24 #\< #\>)
=> "<<<<< Richard Todd >>>>>"
@end lisp

@end defun

@anchor{texinfo string-utils left-justify-string}@defun left-justify-string str [width] [chr]
@code{left-justify-string str [width chr]}. Returns a copy of @var{str}
padded with @var{chr} such that it is left justified in a field of
@var{width} characters. The default @var{width} is 80. Unlike
@samp{string-pad} from srfi-13, the string is never truncated.

@end defun

@anchor{texinfo string-utils right-justify-string}@defun right-justify-string str [width] [chr]
Returns a copy of @var{str} padded with @var{chr} such that it is right
justified in a field of @var{width} characters. The default @var{width}
is 80. The default @var{chr} is @samp{#\space}. Unlike @samp{string-pad}
from srfi-13, the string is never truncated.

@end defun

@anchor{texinfo string-utils collapse-repeated-chars}@defun collapse-repeated-chars str [chr] [num]
Returns a copy of @var{str} with all repeated instances of @var{chr}
collapsed down to at most @var{num} instances. The default value for
@var{chr} is @samp{#\space}, and the default value for @var{num} is 1.

@lisp 
 (collapse-repeated-chars "H  e  l  l  o")
=> "H e l l o"
 (collapse-repeated-chars "H--e--l--l--o" #\-)
=> "H-e-l-l-o"
 (collapse-repeated-chars "H-e--l---l----o" #\- 2)
=> "H-e--l--l--o"
@end lisp

@end defun

@anchor{texinfo string-utils make-text-wrapper}@defun make-text-wrapper [#:line-width] [#:expand-tabs?] [#:tab-width] [#:collapse-whitespace?] [#:subsequent-indent] [#:initial-indent] [#:break-long-words?]
Returns a procedure that will split a string into lines according to the
given parameters.

@table @code
@item #:line-width
This is the target length used when deciding where to wrap lines.
Default is 80.

@item #:expand-tabs?
Boolean describing whether tabs in the input should be expanded. Default
is #t.

@item #:tab-width
If tabs are expanded, this will be the number of spaces to which they
expand. Default is 8.

@item #:collapse-whitespace?
Boolean describing whether the whitespace inside the existing text
should be removed or not. Default is #t.

If text is already well-formatted, and is just being wrapped to fit in a
different width, then set this to @samp{#f}. This way, many common text
conventions (such as two spaces between sentences) can be preserved if
in the original text. If the input text spacing cannot be trusted, then
leave this setting at the default, and all repeated whitespace will be
collapsed down to a single space.

@item #:initial-indent
Defines a string that will be put in front of the first line of wrapped
text. Default is the empty string, ``''.

@item #:subsequent-indent
Defines a string that will be put in front of all lines of wrapped text,
except the first one. Default is the empty string, ``''.

@item #:break-long-words?
If a single word is too big to fit on a line, this setting tells the
wrapper what to do. Defaults to #t, which will break up long words. When
set to #f, the line will be allowed, even though it is longer than the
defined @code{#:line-width}.

@end table

The return value is a procedure of one argument, the input string, which
returns a list of strings, where each element of the list is one line.

@end defun

@anchor{texinfo string-utils fill-string}@defun fill-string str . kwargs
Wraps the text given in string @var{str} according to the parameters
provided in @var{kwargs}, or the default setting if they are not given.
Returns a single string with the wrapped text. Valid keyword arguments
are discussed in @code{make-text-wrapper}.

@end defun

@anchor{texinfo string-utils string->wrapped-lines}@defun string->wrapped-lines str . kwargs
@code{string->wrapped-lines str keywds ...}. Wraps the text given in
string @var{str} according to the parameters provided in @var{keywds},
or the default setting if they are not given. Returns a list of strings
representing the formatted lines. Valid keyword arguments are discussed
in @code{make-text-wrapper}.

@end defun

@node texinfo plain-text
@subsection (texinfo plain-text)
@subsubsection Overview
Transformation from stexi to plain-text. Strives to re-create the output
from @code{info}; comes pretty damn close.

@subsubsection Usage
@anchor{texinfo plain-text stexi->plain-text}@defun stexi->plain-text tree
Transform @var{tree} into plain text. Returns a string.

@end defun

@defvr {Scheme Variable} *line-width*
This fluid (@pxref{Fluids and Dynamic States}) specifies the length of
line for the purposes of line wrapping in the @code{stexi->plain-text}
conversion.
@end defvr

@node texinfo serialize
@subsection (texinfo serialize)
@subsubsection Overview
Serialization of @code{stexi} to plain texinfo.

@subsubsection Usage
@anchor{texinfo serialize stexi->texi}@defun stexi->texi tree
Serialize the stexi @var{tree} into plain texinfo.

@end defun

@node texinfo reflection
@subsection (texinfo reflection)
@subsubsection Overview
Routines to generare @code{stexi} documentation for objects and modules.

Note that in this context, an @dfn{object} is just a value associated
with a location. It has nothing to do with GOOPS.

@subsubsection Usage
@anchor{texinfo reflection module-stexi-documentation}@defun module-stexi-documentation sym-name [%docs-resolver] [#:docs-resolver]
Return documentation for the module named @var{sym-name}. The
documentation will be formatted as @code{stexi}
(@pxref{texinfo,texinfo}).

@end defun

@anchor{texinfo reflection script-stexi-documentation}@defun script-stexi-documentation scriptpath
Return documentation for given script. The documentation will be taken
from the script's commentary, and will be returned in the @code{stexi}
format (@pxref{texinfo,texinfo}).

@end defun

@anchor{texinfo reflection object-stexi-documentation}@defun object-stexi-documentation _ [_] [#:force]
@end defun

@anchor{texinfo reflection package-stexi-standard-copying}@defun package-stexi-standard-copying name version updated years copyright-holder permissions
Create a standard texinfo @code{copying} section.

@var{years} is a list of years (as integers) in which the modules being
documented were released. All other arguments are strings.

@end defun

@anchor{texinfo reflection package-stexi-standard-titlepage}@defun package-stexi-standard-titlepage name version updated authors
Create a standard GNU title page.

@var{authors} is a list of @code{(@var{name} . @var{email})} pairs. All
other arguments are strings.

Here is an example of the usage of this procedure:

@smallexample 
 (package-stexi-standard-titlepage
  "Foolib"
  "3.2"
  "26 September 2006"
  '(("Alyssa P Hacker" . "alyssa@@example.com"))
  '(2004 2005 2006)
  "Free Software Foundation, Inc."
  "Standard GPL permissions blurb goes here")
@end smallexample

@end defun

@anchor{texinfo reflection package-stexi-generic-menu}@defun package-stexi-generic-menu name entries
Create a menu from a generic alist of entries, the car of which should
be the node name, and the cdr the description. As an exception, an entry
of @code{#f} will produce a separator.

@end defun

@anchor{texinfo reflection package-stexi-standard-menu}@defun package-stexi-standard-menu name modules module-descriptions extra-entries
Create a standard top node and menu, suitable for processing by
makeinfo.

@end defun

@anchor{texinfo reflection package-stexi-extended-menu}@defun package-stexi-extended-menu name module-pairs script-pairs extra-entries
Create an "extended" menu, like the standard menu but with a section for
scripts.

@end defun

@anchor{texinfo reflection package-stexi-standard-prologue}@defun package-stexi-standard-prologue name filename category description copying titlepage menu
Create a standard prologue, suitable for later serialization to texinfo
and .info creation with makeinfo.

Returns a list of stexinfo forms suitable for passing to
@code{package-stexi-documentation} as the prologue. @xref{texinfo
reflection package-stexi-documentation}, @ref{texinfo reflection
package-stexi-standard-titlepage,package-stexi-standard-titlepage},
@ref{texinfo reflection
package-stexi-standard-copying,package-stexi-standard-copying}, and
@ref{texinfo reflection
package-stexi-standard-menu,package-stexi-standard-menu}.

@end defun

@anchor{texinfo reflection package-stexi-documentation}@defun package-stexi-documentation modules name filename prologue epilogue [#:module-stexi-documentation-args] [#:scripts]
Create stexi documentation for a @dfn{package}, where a package is a set
of modules that is released together.

@var{modules} is expected to be a list of module names, where a module
name is a list of symbols. The stexi that is returned will be titled
@var{name} and a texinfo filename of @var{filename}.

@var{prologue} and @var{epilogue} are lists of stexi forms that will be
spliced into the output document before and after the generated modules
documentation, respectively. @xref{texinfo reflection
package-stexi-standard-prologue}, to create a conventional GNU texinfo
prologue.

@var{module-stexi-documentation-args} is an optional argument that, if
given, will be added to the argument list when
@code{module-texi-documentation} is called. For example, it might be
useful to define a @code{#:docs-resolver} argument.

@end defun

@anchor{texinfo reflection package-stexi-documentation-for-include}@defun package-stexi-documentation-for-include modules module-descriptions [#:module-stexi-documentation-args]
Create stexi documentation for a @dfn{package}, where a package is a set
of modules that is released together.

@var{modules} is expected to be a list of module names, where a module
name is a list of symbols. Returns an stexinfo fragment.

Unlike @code{package-stexi-documentation}, this function simply produces
a menu and the module documentations instead of producing a full texinfo
document. This can be useful if you write part of your manual by hand,
and just use @code{@@include} to pull in the automatically generated
parts.

@var{module-stexi-documentation-args} is an optional argument that, if
given, will be added to the argument list when
@code{module-texi-documentation} is called. For example, it might be
useful to define a @code{#:docs-resolver} argument.

@end defun