diff options
| author | Arnold D. Robbins <arnold@skeeve.com> | 2016-10-26 21:52:49 +0300 |
|---|---|---|
| committer | Arnold D. Robbins <arnold@skeeve.com> | 2016-10-26 21:52:49 +0300 |
| commit | 627c1b8f9913547703c7c53b0716b913f327a402 (patch) | |
| tree | f34e72451f571b4be5640e18ed9df054d4c3fff3 | |
| parent | e404706d5e2ea41229fe5be9b0725202f49bf308 (diff) | |
| parent | e5abd6a16d42fc0f42277919a2d0a2c28476788c (diff) | |
| download | gawk-627c1b8f9913547703c7c53b0716b913f327a402.tar.gz | |
Merge branch 'master' into feature/typed-regex
| -rw-r--r-- | .gitignore | 2 | ||||
| -rw-r--r-- | doc/gawk.info | 35781 | ||||
| -rw-r--r-- | doc/gawkinet.info | 4406 |
3 files changed, 40187 insertions, 2 deletions
@@ -16,5 +16,3 @@ gawk stamp-h1 test/fmtspcl.ok - -doc/*.info diff --git a/doc/gawk.info b/doc/gawk.info new file mode 100644 index 00000000..b8ab365a --- /dev/null +++ b/doc/gawk.info @@ -0,0 +1,35781 @@ +This is gawk.info, produced by makeinfo version 6.1 from gawk.texi. + +Copyright (C) 1989, 1991, 1992, 1993, 1996-2005, 2007, 2009-2016 +Free Software Foundation, Inc. + + + This is Edition 4.1 of 'GAWK: Effective AWK Programming: A User's +Guide for GNU Awk', for the 4.1.4 (or later) version of the GNU +implementation of AWK. + + Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with the +Invariant Sections being "GNU General Public License", with the +Front-Cover Texts being "A GNU Manual", and with the Back-Cover Texts as +in (a) below. A copy of the license is included in the section entitled +"GNU Free Documentation License". + + a. The FSF's Back-Cover Text is: "You have the freedom to copy and + modify this GNU manual." +INFO-DIR-SECTION Text creation and manipulation +START-INFO-DIR-ENTRY +* Gawk: (gawk). A text scanning and processing language. +END-INFO-DIR-ENTRY + +INFO-DIR-SECTION Individual utilities +START-INFO-DIR-ENTRY +* awk: (gawk)Invoking gawk. Text scanning and processing. +END-INFO-DIR-ENTRY + + +File: gawk.info, Node: Top, Next: Foreword3, Up: (dir) + +General Introduction +******************** + +This file documents 'awk', a program that you can use to select +particular records in a file and perform operations upon them. + + Copyright (C) 1989, 1991, 1992, 1993, 1996-2005, 2007, 2009-2016 +Free Software Foundation, Inc. + + + This is Edition 4.1 of 'GAWK: Effective AWK Programming: A User's +Guide for GNU Awk', for the 4.1.4 (or later) version of the GNU +implementation of AWK. + + Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with the +Invariant Sections being "GNU General Public License", with the +Front-Cover Texts being "A GNU Manual", and with the Back-Cover Texts as +in (a) below. A copy of the license is included in the section entitled +"GNU Free Documentation License". + + a. The FSF's Back-Cover Text is: "You have the freedom to copy and + modify this GNU manual." + +* Menu: + +* Foreword3:: Some nice words about this + Info file. +* Foreword4:: More nice words. +* Preface:: What this Info file is about; brief + history and acknowledgments. +* Getting Started:: A basic introduction to using + 'awk'. How to run an 'awk' + program. Command-line syntax. +* Invoking Gawk:: How to run 'gawk'. +* Regexp:: All about matching things using regular + expressions. +* Reading Files:: How to read files and manipulate fields. +* Printing:: How to print using 'awk'. Describes + the 'print' and 'printf' + statements. Also describes redirection of + output. +* Expressions:: Expressions are the basic building blocks + of statements. +* Patterns and Actions:: Overviews of patterns and actions. +* Arrays:: The description and use of arrays. Also + includes array-oriented control statements. +* Functions:: Built-in and user-defined functions. +* Library Functions:: A Library of 'awk' Functions. +* Sample Programs:: Many 'awk' programs with complete + explanations. +* Advanced Features:: Stuff for advanced users, specific to + 'gawk'. +* Internationalization:: Getting 'gawk' to speak your + language. +* Debugger:: The 'gawk' debugger. +* Arbitrary Precision Arithmetic:: Arbitrary precision arithmetic with + 'gawk'. +* Dynamic Extensions:: Adding new built-in functions to + 'gawk'. +* Language History:: The evolution of the 'awk' + language. +* Installation:: Installing 'gawk' under various + operating systems. +* Notes:: Notes about adding things to 'gawk' + and possible future work. +* Basic Concepts:: A very quick introduction to programming + concepts. +* Glossary:: An explanation of some unfamiliar terms. +* Copying:: Your right to copy and distribute + 'gawk'. +* GNU Free Documentation License:: The license for this Info file. +* Index:: Concept and Variable Index. + +* History:: The history of 'gawk' and + 'awk'. +* Names:: What name to use to find + 'awk'. +* This Manual:: Using this Info file. Includes + sample input files that you can use. +* Conventions:: Typographical Conventions. +* Manual History:: Brief history of the GNU project and + this Info file. +* How To Contribute:: Helping to save the world. +* Acknowledgments:: Acknowledgments. +* Running gawk:: How to run 'gawk' programs; + includes command-line syntax. +* One-shot:: Running a short throwaway + 'awk' program. +* Read Terminal:: Using no input files (input from the + keyboard instead). +* Long:: Putting permanent 'awk' + programs in files. +* Executable Scripts:: Making self-contained 'awk' + programs. +* Comments:: Adding documentation to 'gawk' + programs. +* Quoting:: More discussion of shell quoting + issues. +* DOS Quoting:: Quoting in Windows Batch Files. +* Sample Data Files:: Sample data files for use in the + 'awk' programs illustrated in + this Info file. +* Very Simple:: A very simple example. +* Two Rules:: A less simple one-line example using + two rules. +* More Complex:: A more complex example. +* Statements/Lines:: Subdividing or combining statements + into lines. +* Other Features:: Other Features of 'awk'. +* When:: When to use 'gawk' and when to + use other things. +* Intro Summary:: Summary of the introduction. +* Command Line:: How to run 'awk'. +* Options:: Command-line options and their + meanings. +* Other Arguments:: Input file names and variable + assignments. +* Naming Standard Input:: How to specify standard input with + other files. +* Environment Variables:: The environment variables + 'gawk' uses. +* AWKPATH Variable:: Searching directories for + 'awk' programs. +* AWKLIBPATH Variable:: Searching directories for + 'awk' shared libraries. +* Other Environment Variables:: The environment variables. +* Exit Status:: 'gawk''s exit status. +* Include Files:: Including other files into your + program. +* Loading Shared Libraries:: Loading shared libraries into your + program. +* Obsolete:: Obsolete Options and/or features. +* Undocumented:: Undocumented Options and Features. +* Invoking Summary:: Invocation summary. +* Regexp Usage:: How to Use Regular Expressions. +* Escape Sequences:: How to write nonprinting characters. +* Regexp Operators:: Regular Expression Operators. +* Bracket Expressions:: What can go between '[...]'. +* Leftmost Longest:: How much text matches. +* Computed Regexps:: Using Dynamic Regexps. +* GNU Regexp Operators:: Operators specific to GNU software. +* Case-sensitivity:: How to do case-insensitive matching. +* Strong Regexp Constants:: Strongly typed regexp constants. +* Regexp Summary:: Regular expressions summary. +* Records:: Controlling how data is split into + records. +* awk split records:: How standard 'awk' splits + records. +* gawk split records:: How 'gawk' splits records. +* Fields:: An introduction to fields. +* Nonconstant Fields:: Nonconstant Field Numbers. +* Changing Fields:: Changing the Contents of a Field. +* Field Separators:: The field separator and how to change + it. +* Default Field Splitting:: How fields are normally separated. +* Regexp Field Splitting:: Using regexps as the field separator. +* Single Character Fields:: Making each character a separate + field. +* Command Line Field Separator:: Setting 'FS' from the command + line. +* Full Line Fields:: Making the full line be a single + field. +* Field Splitting Summary:: Some final points and a summary table. +* Constant Size:: Reading constant width data. +* Splitting By Content:: Defining Fields By Content +* Multiple Line:: Reading multiline records. +* Getline:: Reading files under explicit program + control using the 'getline' + function. +* Plain Getline:: Using 'getline' with no + arguments. +* Getline/Variable:: Using 'getline' into a variable. +* Getline/File:: Using 'getline' from a file. +* Getline/Variable/File:: Using 'getline' into a variable + from a file. +* Getline/Pipe:: Using 'getline' from a pipe. +* Getline/Variable/Pipe:: Using 'getline' into a variable + from a pipe. +* Getline/Coprocess:: Using 'getline' from a coprocess. +* Getline/Variable/Coprocess:: Using 'getline' into a variable + from a coprocess. +* Getline Notes:: Important things to know about + 'getline'. +* Getline Summary:: Summary of 'getline' Variants. +* Read Timeout:: Reading input with a timeout. +* Retrying Input:: Retrying input after certain errors. +* Command-line directories:: What happens if you put a directory on + the command line. +* Input Summary:: Input summary. +* Input Exercises:: Exercises. +* Print:: The 'print' statement. +* Print Examples:: Simple examples of 'print' + statements. +* Output Separators:: The output separators and how to + change them. +* OFMT:: Controlling Numeric Output With + 'print'. +* Printf:: The 'printf' statement. +* Basic Printf:: Syntax of the 'printf' statement. +* Control Letters:: Format-control letters. +* Format Modifiers:: Format-specification modifiers. +* Printf Examples:: Several examples. +* Redirection:: How to redirect output to multiple + files and pipes. +* Special FD:: Special files for I/O. +* Special Files:: File name interpretation in + 'gawk'. 'gawk' allows + access to inherited file descriptors. +* Other Inherited Files:: Accessing other open files with + 'gawk'. +* Special Network:: Special files for network + communications. +* Special Caveats:: Things to watch out for. +* Close Files And Pipes:: Closing Input and Output Files and + Pipes. +* Nonfatal:: Enabling Nonfatal Output. +* Output Summary:: Output summary. +* Output Exercises:: Exercises. +* Values:: Constants, Variables, and Regular + Expressions. +* Constants:: String, numeric and regexp constants. +* Scalar Constants:: Numeric and string constants. +* Nondecimal-numbers:: What are octal and hex numbers. +* Regexp Constants:: Regular Expression constants. +* Using Constant Regexps:: When and how to use a regexp constant. +* Variables:: Variables give names to values for + later use. +* Using Variables:: Using variables in your programs. +* Assignment Options:: Setting variables on the command line + and a summary of command-line syntax. + This is an advanced method of input. +* Conversion:: The conversion of strings to numbers + and vice versa. +* Strings And Numbers:: How 'awk' Converts Between + Strings And Numbers. +* Locale influences conversions:: How the locale may affect conversions. +* All Operators:: 'gawk''s operators. +* Arithmetic Ops:: Arithmetic operations ('+', + '-', etc.) +* Concatenation:: Concatenating strings. +* Assignment Ops:: Changing the value of a variable or a + field. +* Increment Ops:: Incrementing the numeric value of a + variable. +* Truth Values and Conditions:: Testing for true and false. +* Truth Values:: What is "true" and what is + "false". +* Typing and Comparison:: How variables acquire types and how + this affects comparison of numbers and + strings with '<', etc. +* Variable Typing:: String type versus numeric type. +* Comparison Operators:: The comparison operators. +* POSIX String Comparison:: String comparison with POSIX rules. +* Boolean Ops:: Combining comparison expressions using + boolean operators '||' ("or"), + '&&' ("and") and '!' + ("not"). +* Conditional Exp:: Conditional expressions select between + two subexpressions under control of a + third subexpression. +* Function Calls:: A function call is an expression. +* Precedence:: How various operators nest. +* Locales:: How the locale affects things. +* Expressions Summary:: Expressions summary. +* Pattern Overview:: What goes into a pattern. +* Regexp Patterns:: Using regexps as patterns. +* Expression Patterns:: Any expression can be used as a + pattern. +* Ranges:: Pairs of patterns specify record + ranges. +* BEGIN/END:: Specifying initialization and cleanup + rules. +* Using BEGIN/END:: How and why to use BEGIN/END rules. +* I/O And BEGIN/END:: I/O issues in BEGIN/END rules. +* BEGINFILE/ENDFILE:: Two special patterns for advanced + control. +* Empty:: The empty pattern, which matches every + record. +* Using Shell Variables:: How to use shell variables with + 'awk'. +* Action Overview:: What goes into an action. +* Statements:: Describes the various control + statements in detail. +* If Statement:: Conditionally execute some + 'awk' statements. +* While Statement:: Loop until some condition is + satisfied. +* Do Statement:: Do specified action while looping + until some condition is satisfied. +* For Statement:: Another looping statement, that + provides initialization and increment + clauses. +* Switch Statement:: Switch/case evaluation for conditional + execution of statements based on a + value. +* Break Statement:: Immediately exit the innermost + enclosing loop. +* Continue Statement:: Skip to the end of the innermost + enclosing loop. +* Next Statement:: Stop processing the current input + record. +* Nextfile Statement:: Stop processing the current file. +* Exit Statement:: Stop execution of 'awk'. +* Built-in Variables:: Summarizes the predefined variables. +* User-modified:: Built-in variables that you change to + control 'awk'. +* Auto-set:: Built-in variables where 'awk' + gives you information. +* ARGC and ARGV:: Ways to use 'ARGC' and + 'ARGV'. +* Pattern Action Summary:: Patterns and Actions summary. +* Array Basics:: The basics of arrays. +* Array Intro:: Introduction to Arrays +* Reference to Elements:: How to examine one element of an + array. +* Assigning Elements:: How to change an element of an array. +* Array Example:: Basic Example of an Array +* Scanning an Array:: A variation of the 'for' + statement. It loops through the + indices of an array's existing + elements. +* Controlling Scanning:: Controlling the order in which arrays + are scanned. +* Numeric Array Subscripts:: How to use numbers as subscripts in + 'awk'. +* Uninitialized Subscripts:: Using Uninitialized variables as + subscripts. +* Delete:: The 'delete' statement removes an + element from an array. +* Multidimensional:: Emulating multidimensional arrays in + 'awk'. +* Multiscanning:: Scanning multidimensional arrays. +* Arrays of Arrays:: True multidimensional arrays. +* Arrays Summary:: Summary of arrays. +* Built-in:: Summarizes the built-in functions. +* Calling Built-in:: How to call built-in functions. +* Numeric Functions:: Functions that work with numbers, + including 'int()', 'sin()' + and 'rand()'. +* String Functions:: Functions for string manipulation, + such as 'split()', 'match()' + and 'sprintf()'. +* Gory Details:: More than you want to know about + '\' and '&' with + 'sub()', 'gsub()', and + 'gensub()'. +* I/O Functions:: Functions for files and shell + commands. +* Time Functions:: Functions for dealing with timestamps. +* Bitwise Functions:: Functions for bitwise operations. +* Type Functions:: Functions for type information. +* I18N Functions:: Functions for string translation. +* User-defined:: Describes User-defined functions in + detail. +* Definition Syntax:: How to write definitions and what they + mean. +* Function Example:: An example function definition and + what it does. +* Function Caveats:: Things to watch out for. +* Calling A Function:: Don't use spaces. +* Variable Scope:: Controlling variable scope. +* Pass By Value/Reference:: Passing parameters. +* Return Statement:: Specifying the value a function + returns. +* Dynamic Typing:: How variable types can change at + runtime. +* Indirect Calls:: Choosing the function to call at + runtime. +* Functions Summary:: Summary of functions. +* Library Names:: How to best name private global + variables in library functions. +* General Functions:: Functions that are of general use. +* Strtonum Function:: A replacement for the built-in + 'strtonum()' function. +* Assert Function:: A function for assertions in + 'awk' programs. +* Round Function:: A function for rounding if + 'sprintf()' does not do it + correctly. +* Cliff Random Function:: The Cliff Random Number Generator. +* Ordinal Functions:: Functions for using characters as + numbers and vice versa. +* Join Function:: A function to join an array into a + string. +* Getlocaltime Function:: A function to get formatted times. +* Readfile Function:: A function to read an entire file at + once. +* Shell Quoting:: A function to quote strings for the + shell. +* Data File Management:: Functions for managing command-line + data files. +* Filetrans Function:: A function for handling data file + transitions. +* Rewind Function:: A function for rereading the current + file. +* File Checking:: Checking that data files are readable. +* Empty Files:: Checking for zero-length files. +* Ignoring Assigns:: Treating assignments as file names. +* Getopt Function:: A function for processing command-line + arguments. +* Passwd Functions:: Functions for getting user + information. +* Group Functions:: Functions for getting group + information. +* Walking Arrays:: A function to walk arrays of arrays. +* Library Functions Summary:: Summary of library functions. +* Library Exercises:: Exercises. +* Running Examples:: How to run these examples. +* Clones:: Clones of common utilities. +* Cut Program:: The 'cut' utility. +* Egrep Program:: The 'egrep' utility. +* Id Program:: The 'id' utility. +* Split Program:: The 'split' utility. +* Tee Program:: The 'tee' utility. +* Uniq Program:: The 'uniq' utility. +* Wc Program:: The 'wc' utility. +* Miscellaneous Programs:: Some interesting 'awk' + programs. +* Dupword Program:: Finding duplicated words in a + document. +* Alarm Program:: An alarm clock. +* Translate Program:: A program similar to the 'tr' + utility. +* Labels Program:: Printing mailing labels. +* Word Sorting:: A program to produce a word usage + count. +* History Sorting:: Eliminating duplicate entries from a + history file. +* Extract Program:: Pulling out programs from Texinfo + source files. +* Simple Sed:: A Simple Stream Editor. +* Igawk Program:: A wrapper for 'awk' that + includes files. +* Anagram Program:: Finding anagrams from a dictionary. +* Signature Program:: People do amazing things with too much + time on their hands. +* Programs Summary:: Summary of programs. +* Programs Exercises:: Exercises. +* Nondecimal Data:: Allowing nondecimal input data. +* Array Sorting:: Facilities for controlling array + traversal and sorting arrays. +* Controlling Array Traversal:: How to use PROCINFO["sorted_in"]. +* Array Sorting Functions:: How to use 'asort()' and + 'asorti()'. +* Two-way I/O:: Two-way communications with another + process. +* TCP/IP Networking:: Using 'gawk' for network + programming. +* Profiling:: Profiling your 'awk' programs. +* Advanced Features Summary:: Summary of advanced features. +* I18N and L10N:: Internationalization and Localization. +* Explaining gettext:: How GNU 'gettext' works. +* Programmer i18n:: Features for the programmer. +* Translator i18n:: Features for the translator. +* String Extraction:: Extracting marked strings. +* Printf Ordering:: Rearranging 'printf' arguments. +* I18N Portability:: 'awk'-level portability + issues. +* I18N Example:: A simple i18n example. +* Gawk I18N:: 'gawk' is also + internationalized. +* I18N Summary:: Summary of I18N stuff. +* Debugging:: Introduction to 'gawk' + debugger. +* Debugging Concepts:: Debugging in General. +* Debugging Terms:: Additional Debugging Concepts. +* Awk Debugging:: Awk Debugging. +* Sample Debugging Session:: Sample debugging session. +* Debugger Invocation:: How to Start the Debugger. +* Finding The Bug:: Finding the Bug. +* List of Debugger Commands:: Main debugger commands. +* Breakpoint Control:: Control of Breakpoints. +* Debugger Execution Control:: Control of Execution. +* Viewing And Changing Data:: Viewing and Changing Data. +* Execution Stack:: Dealing with the Stack. +* Debugger Info:: Obtaining Information about the + Program and the Debugger State. +* Miscellaneous Debugger Commands:: Miscellaneous Commands. +* Readline Support:: Readline support. +* Limitations:: Limitations and future plans. +* Debugging Summary:: Debugging summary. +* Computer Arithmetic:: A quick intro to computer math. +* Math Definitions:: Defining terms used. +* MPFR features:: The MPFR features in 'gawk'. +* FP Math Caution:: Things to know. +* Inexactness of computations:: Floating point math is not exact. +* Inexact representation:: Numbers are not exactly represented. +* Comparing FP Values:: How to compare floating point values. +* Errors accumulate:: Errors get bigger as they go. +* Getting Accuracy:: Getting more accuracy takes some work. +* Try To Round:: Add digits and round. +* Setting precision:: How to set the precision. +* Setting the rounding mode:: How to set the rounding mode. +* Arbitrary Precision Integers:: Arbitrary Precision Integer Arithmetic + with 'gawk'. +* POSIX Floating Point Problems:: Standards Versus Existing Practice. +* Floating point summary:: Summary of floating point discussion. +* Extension Intro:: What is an extension. +* Plugin License:: A note about licensing. +* Extension Mechanism Outline:: An outline of how it works. +* Extension API Description:: A full description of the API. +* Extension API Functions Introduction:: Introduction to the API functions. +* General Data Types:: The data types. +* Memory Allocation Functions:: Functions for allocating memory. +* Constructor Functions:: Functions for creating values. +* Registration Functions:: Functions to register things with + 'gawk'. +* Extension Functions:: Registering extension functions. +* Exit Callback Functions:: Registering an exit callback. +* Extension Version String:: Registering a version string. +* Input Parsers:: Registering an input parser. +* Output Wrappers:: Registering an output wrapper. +* Two-way processors:: Registering a two-way processor. +* Printing Messages:: Functions for printing messages. +* Updating ERRNO:: Functions for updating 'ERRNO'. +* Requesting Values:: How to get a value. +* Accessing Parameters:: Functions for accessing parameters. +* Symbol Table Access:: Functions for accessing global + variables. +* Symbol table by name:: Accessing variables by name. +* Symbol table by cookie:: Accessing variables by "cookie". +* Cached values:: Creating and using cached values. +* Array Manipulation:: Functions for working with arrays. +* Array Data Types:: Data types for working with arrays. +* Array Functions:: Functions for working with arrays. +* Flattening Arrays:: How to flatten arrays. +* Creating Arrays:: How to create and populate arrays. +* Redirection API:: How to access and manipulate redirections. +* Extension API Variables:: Variables provided by the API. +* Extension Versioning:: API Version information. +* Extension API Informational Variables:: Variables providing information about + 'gawk''s invocation. +* Extension API Boilerplate:: Boilerplate code for using the API. +* Finding Extensions:: How 'gawk' finds compiled + extensions. +* Extension Example:: Example C code for an extension. +* Internal File Description:: What the new functions will do. +* Internal File Ops:: The code for internal file operations. +* Using Internal File Ops:: How to use an external extension. +* Extension Samples:: The sample extensions that ship with + 'gawk'. +* Extension Sample File Functions:: The file functions sample. +* Extension Sample Fnmatch:: An interface to 'fnmatch()'. +* Extension Sample Fork:: An interface to 'fork()' and + other process functions. +* Extension Sample Inplace:: Enabling in-place file editing. +* Extension Sample Ord:: Character to value to character + conversions. +* Extension Sample Readdir:: An interface to 'readdir()'. +* Extension Sample Revout:: Reversing output sample output + wrapper. +* Extension Sample Rev2way:: Reversing data sample two-way + processor. +* Extension Sample Read write array:: Serializing an array to a file. +* Extension Sample Readfile:: Reading an entire file into a string. +* Extension Sample Time:: An interface to 'gettimeofday()' + and 'sleep()'. +* Extension Sample API Tests:: Tests for the API. +* gawkextlib:: The 'gawkextlib' project. +* Extension summary:: Extension summary. +* Extension Exercises:: Exercises. +* V7/SVR3.1:: The major changes between V7 and + System V Release 3.1. +* SVR4:: Minor changes between System V + Releases 3.1 and 4. +* POSIX:: New features from the POSIX standard. +* BTL:: New features from Brian Kernighan's + version of 'awk'. +* POSIX/GNU:: The extensions in 'gawk' not + in POSIX 'awk'. +* Feature History:: The history of the features in + 'gawk'. +* Common Extensions:: Common Extensions Summary. +* Ranges and Locales:: How locales used to affect regexp + ranges. +* Contributors:: The major contributors to + 'gawk'. +* History summary:: History summary. +* Gawk Distribution:: What is in the 'gawk' + distribution. +* Getting:: How to get the distribution. +* Extracting:: How to extract the distribution. +* Distribution contents:: What is in the distribution. +* Unix Installation:: Installing 'gawk' under + various versions of Unix. +* Quick Installation:: Compiling 'gawk' under Unix. +* Shell Startup Files:: Shell convenience functions. +* Additional Configuration Options:: Other compile-time options. +* Configuration Philosophy:: How it's all supposed to work. +* Non-Unix Installation:: Installation on Other Operating + Systems. +* PC Installation:: Installing and Compiling 'gawk' on + Microsoft Windows. +* PC Binary Installation:: Installing a prepared distribution. +* PC Compiling:: Compiling 'gawk' for Windows32. +* PC Using:: Running 'gawk' on Windows32. +* Cygwin:: Building and running 'gawk' + for Cygwin. +* MSYS:: Using 'gawk' In The MSYS + Environment. +* VMS Installation:: Installing 'gawk' on VMS. +* VMS Compilation:: How to compile 'gawk' under + VMS. +* VMS Dynamic Extensions:: Compiling 'gawk' dynamic + extensions on VMS. +* VMS Installation Details:: How to install 'gawk' under + VMS. +* VMS Running:: How to run 'gawk' under VMS. +* VMS GNV:: The VMS GNV Project. +* VMS Old Gawk:: An old version comes with some VMS + systems. +* Bugs:: Reporting Problems and Bugs. +* Bug address:: Where to send reports to. +* Usenet:: Where not to send reports to. +* Maintainers:: Maintainers of non-*nix ports. +* Other Versions:: Other freely available 'awk' + implementations. +* Installation summary:: Summary of installation. +* Compatibility Mode:: How to disable certain 'gawk' + extensions. +* Additions:: Making Additions To 'gawk'. +* Accessing The Source:: Accessing the Git repository. +* Adding Code:: Adding code to the main body of + 'gawk'. +* New Ports:: Porting 'gawk' to a new + operating system. +* Derived Files:: Why derived files are kept in the Git + repository. +* Future Extensions:: New features that may be implemented + one day. +* Implementation Limitations:: Some limitations of the + implementation. +* Extension Design:: Design notes about the extension API. +* Old Extension Problems:: Problems with the old mechanism. +* Extension New Mechanism Goals:: Goals for the new mechanism. +* Extension Other Design Decisions:: Some other design decisions. +* Extension Future Growth:: Some room for future growth. +* Old Extension Mechanism:: Some compatibility for old extensions. +* Notes summary:: Summary of implementation notes. +* Basic High Level:: The high level view. +* Basic Data Typing:: A very quick intro to data types. + + To my parents, for their love, and for the wonderful example they set +for me. + + To my wife Miriam, for making me complete. Thank you for building +your life together with me. + + To our children Chana, Rivka, Nachum and Malka, for enrichening our +lives in innumerable ways. + + +File: gawk.info, Node: Foreword3, Next: Foreword4, Prev: Top, Up: Top + +Foreword to the Third Edition +***************************** + +Arnold Robbins and I are good friends. We were introduced in 1990 by +circumstances--and our favorite programming language, AWK. The +circumstances started a couple of years earlier. I was working at a new +job and noticed an unplugged Unix computer sitting in the corner. No +one knew how to use it, and neither did I. However, a couple of days +later, it was running, and I was 'root' and the one-and-only user. That +day, I began the transition from statistician to Unix programmer. + + On one of many trips to the library or bookstore in search of books +on Unix, I found the gray AWK book, a.k.a. Alfred V. Aho, Brian W. +Kernighan, and Peter J. Weinberger's 'The AWK Programming Language' +(Addison-Wesley, 1988). 'awk''s simple programming paradigm--find a +pattern in the input and then perform an action--often reduced complex +or tedious data manipulations to a few lines of code. I was excited to +try my hand at programming in AWK. + + Alas, the 'awk' on my computer was a limited version of the language +described in the gray book. I discovered that my computer had "old +'awk'" and the book described "new 'awk'." I learned that this was +typical; the old version refused to step aside or relinquish its name. +If a system had a new 'awk', it was invariably called 'nawk', and few +systems had it. The best way to get a new 'awk' was to 'ftp' the source +code for 'gawk' from 'prep.ai.mit.edu'. 'gawk' was a version of new +'awk' written by David Trueman and Arnold, and available under the GNU +General Public License. + + (Incidentally, it's no longer difficult to find a new 'awk'. 'gawk' +ships with GNU/Linux, and you can download binaries or source code for +almost any system; my wife uses 'gawk' on her VMS box.) + + My Unix system started out unplugged from the wall; it certainly was +not plugged into a network. So, oblivious to the existence of 'gawk' +and the Unix community in general, and desiring a new 'awk', I wrote my +own, called 'mawk'. Before I was finished, I knew about 'gawk', but it +was too late to stop, so I eventually posted to a 'comp.sources' +newsgroup. + + A few days after my posting, I got a friendly email from Arnold +introducing himself. He suggested we share design and algorithms and +attached a draft of the POSIX standard so that I could update 'mawk' to +support language extensions added after publication of 'The AWK +Programming Language'. + + Frankly, if our roles had been reversed, I would not have been so +open and we probably would have never met. I'm glad we did meet. He is +an AWK expert's AWK expert and a genuinely nice person. Arnold +contributes significant amounts of his expertise and time to the Free +Software Foundation. + + This book is the 'gawk' reference manual, but at its core it is a +book about AWK programming that will appeal to a wide audience. It is a +definitive reference to the AWK language as defined by the 1987 Bell +Laboratories release and codified in the 1992 POSIX Utilities standard. + + On the other hand, the novice AWK programmer can study a wealth of +practical programs that emphasize the power of AWK's basic idioms: +data-driven control flow, pattern matching with regular expressions, and +associative arrays. Those looking for something new can try out +'gawk''s interface to network protocols via special '/inet' files. + + The programs in this book make clear that an AWK program is typically +much smaller and faster to develop than a counterpart written in C. +Consequently, there is often a payoff to prototyping an algorithm or +design in AWK to get it running quickly and expose problems early. +Often, the interpreted performance is adequate and the AWK prototype +becomes the product. + + The new 'pgawk' (profiling 'gawk'), produces program execution +counts. I recently experimented with an algorithm that for n lines of +input, exhibited ~ C n^2 performance, while theory predicted ~ C n log n +behavior. A few minutes poring over the 'awkprof.out' profile +pinpointed the problem to a single line of code. 'pgawk' is a welcome +addition to my programmer's toolbox. + + Arnold has distilled over a decade of experience writing and using +AWK programs, and developing 'gawk', into this book. If you use AWK or +want to learn how, then read this book. + + Michael Brennan + Author of 'mawk' + March 2001 + + +File: gawk.info, Node: Foreword4, Next: Preface, Prev: Foreword3, Up: Top + +Foreword to the Fourth Edition +****************************** + +Some things don't change. Thirteen years ago I wrote: "If you use AWK +or want to learn how, then read this book." True then, and still true +today. + + Learning to use a programming language is about more than mastering +the syntax. One needs to acquire an understanding of how to use the +features of the language to solve practical programming problems. A +focus of this book is many examples that show how to use AWK. + + Some things do change. Our computers are much faster and have more +memory. Consequently, speed and storage inefficiencies of a high-level +language matter less. Prototyping in AWK and then rewriting in C for +performance reasons happens less, because more often the prototype is +fast enough. + + Of course, there are computing operations that are best done in C or +C++. With 'gawk' 4.1 and later, you do not have to choose between +writing your program in AWK or in C/C++. You can write most of your +program in AWK and the aspects that require C/C++ capabilities can be +written in C/C++, and then the pieces glued together when the 'gawk' +module loads the C/C++ module as a dynamic plug-in. *note Dynamic +Extensions::, has all the details, and, as expected, many examples to +help you learn the ins and outs. + + I enjoy programming in AWK and had fun (re)reading this book. I +think you will too. + + Michael Brennan + Author of 'mawk' + October 2014 + + +File: gawk.info, Node: Preface, Next: Getting Started, Prev: Foreword4, Up: Top + +Preface +******* + +Several kinds of tasks occur repeatedly when working with text files. +You might want to extract certain lines and discard the rest. Or you +may need to make changes wherever certain patterns appear, but leave the +rest of the file alone. Such jobs are often easy with 'awk'. The 'awk' +utility interprets a special-purpose programming language that makes it +easy to handle simple data-reformatting jobs. + + The GNU implementation of 'awk' is called 'gawk'; if you invoke it +with the proper options or environment variables, it is fully compatible +with the POSIX(1) specification of the 'awk' language and with the Unix +version of 'awk' maintained by Brian Kernighan. This means that all +properly written 'awk' programs should work with 'gawk'. So most of the +time, we don't distinguish between 'gawk' and other 'awk' +implementations. + + Using 'awk' you can: + + * Manage small, personal databases + + * Generate reports + + * Validate data + + * Produce indexes and perform other document-preparation tasks + + * Experiment with algorithms that you can adapt later to other + computer languages + + In addition, 'gawk' provides facilities that make it easy to: + + * Extract bits and pieces of data for processing + + * Sort data + + * Perform simple network communications + + * Profile and debug 'awk' programs + + * Extend the language with functions written in C or C++ + + This Info file teaches you about the 'awk' language and how you can +use it effectively. You should already be familiar with basic system +commands, such as 'cat' and 'ls',(2) as well as basic shell facilities, +such as input/output (I/O) redirection and pipes. + + Implementations of the 'awk' language are available for many +different computing environments. This Info file, while describing the +'awk' language in general, also describes the particular implementation +of 'awk' called 'gawk' (which stands for "GNU 'awk'"). 'gawk' runs on a +broad range of Unix systems, ranging from Intel-architecture PC-based +computers up through large-scale systems. 'gawk' has also been ported +to Mac OS X, Microsoft Windows (all versions), and OpenVMS.(3) + +* Menu: + +* History:: The history of 'gawk' and + 'awk'. +* Names:: What name to use to find 'awk'. +* This Manual:: Using this Info file. Includes sample + input files that you can use. +* Conventions:: Typographical Conventions. +* Manual History:: Brief history of the GNU project and this + Info file. +* How To Contribute:: Helping to save the world. +* Acknowledgments:: Acknowledgments. + + ---------- Footnotes ---------- + + (1) The 2008 POSIX standard is accessible online at +<http://www.opengroup.org/onlinepubs/9699919799/>. + + (2) These utilities are available on POSIX-compliant systems, as well +as on traditional Unix-based systems. If you are using some other +operating system, you still need to be familiar with the ideas of I/O +redirection and pipes. + + (3) Some other, obsolete systems to which 'gawk' was once ported are +no longer supported and the code for those systems has been removed. + + +File: gawk.info, Node: History, Next: Names, Up: Preface + +History of 'awk' and 'gawk' +=========================== + + Recipe for a Programming Language + + 1 part 'egrep' 1 part 'snobol' + 2 parts 'ed' 3 parts C + + Blend all parts well using 'lex' and 'yacc'. Document minimally and +release. + + After eight years, add another part 'egrep' and two more parts C. +Document very well and release. + + The name 'awk' comes from the initials of its designers: Alfred V. +Aho, Peter J. Weinberger, and Brian W. Kernighan. The original version +of 'awk' was written in 1977 at AT&T Bell Laboratories. In 1985, a new +version made the programming language more powerful, introducing +user-defined functions, multiple input streams, and computed regular +expressions. This new version became widely available with Unix System +V Release 3.1 (1987). The version in System V Release 4 (1989) added +some new features and cleaned up the behavior in some of the "dark +corners" of the language. The specification for 'awk' in the POSIX +Command Language and Utilities standard further clarified the language. +Both the 'gawk' designers and the original 'awk' designers at Bell +Laboratories provided feedback for the POSIX specification. + + Paul Rubin wrote 'gawk' in 1986. Jay Fenlason completed it, with +advice from Richard Stallman. John Woods contributed parts of the code +as well. In 1988 and 1989, David Trueman, with help from me, thoroughly +reworked 'gawk' for compatibility with the newer 'awk'. Circa 1994, I +became the primary maintainer. Current development focuses on bug +fixes, performance improvements, standards compliance, and, +occasionally, new features. + + In May 1997, Ju"rgen Kahrs felt the need for network access from +'awk', and with a little help from me, set about adding features to do +this for 'gawk'. At that time, he also wrote the bulk of 'TCP/IP +Internetworking with 'gawk'' (a separate document, available as part of +the 'gawk' distribution). His code finally became part of the main +'gawk' distribution with 'gawk' version 3.1. + + John Haque rewrote the 'gawk' internals, in the process providing an +'awk'-level debugger. This version became available as 'gawk' version +4.0 in 2011. + + *Note Contributors:: for a full list of those who have made important +contributions to 'gawk'. + + +File: gawk.info, Node: Names, Next: This Manual, Prev: History, Up: Preface + +A Rose by Any Other Name +======================== + +The 'awk' language has evolved over the years. Full details are +provided in *note Language History::. The language described in this +Info file is often referred to as "new 'awk'." By analogy, the original +version of 'awk' is referred to as "old 'awk'." + + On most current systems, when you run the 'awk' utility you get some +version of new 'awk'.(1) If your system's standard 'awk' is the old +one, you will see something like this if you try the test program: + + $ awk 1 /dev/null + error-> awk: syntax error near line 1 + error-> awk: bailing out near line 1 + +In this case, you should find a version of new 'awk', or just install +'gawk'! + + Throughout this Info file, whenever we refer to a language feature +that should be available in any complete implementation of POSIX 'awk', +we simply use the term 'awk'. When referring to a feature that is +specific to the GNU implementation, we use the term 'gawk'. + + ---------- Footnotes ---------- + + (1) Only Solaris systems still use an old 'awk' for the default 'awk' +utility. A more modern 'awk' lives in '/usr/xpg6/bin' on these systems. + + +File: gawk.info, Node: This Manual, Next: Conventions, Prev: Names, Up: Preface + +Using This Book +=============== + +The term 'awk' refers to a particular program as well as to the language +you use to tell this program what to do. When we need to be careful, we +call the language "the 'awk' language," and the program "the 'awk' +utility." This Info file explains both how to write programs in the +'awk' language and how to run the 'awk' utility. The term "'awk' +program" refers to a program written by you in the 'awk' programming +language. + + Primarily, this Info file explains the features of 'awk' as defined +in the POSIX standard. It does so in the context of the 'gawk' +implementation. While doing so, it also attempts to describe important +differences between 'gawk' and other 'awk' implementations.(1) Finally, +it notes any 'gawk' features that are not in the POSIX standard for +'awk'. + + There are sidebars scattered throughout the Info file. They add a +more complete explanation of points that are relevant, but not likely to +be of interest on first reading. All appear in the index, under the +heading "sidebar." + + Most of the time, the examples use complete 'awk' programs. Some of +the more advanced minor nodes show only the part of the 'awk' program +that illustrates the concept being described. + + Although this Info file is aimed principally at people who have not +been exposed to 'awk', there is a lot of information here that even the +'awk' expert should find useful. In particular, the description of +POSIX 'awk' and the example programs in *note Library Functions::, and +in *note Sample Programs::, should be of interest. + + This Info file is split into several parts, as follows: + + * Part I describes the 'awk' language and the 'gawk' program in + detail. It starts with the basics, and continues through all of + the features of 'awk'. It contains the following chapters: + + - *note Getting Started::, provides the essentials you need to + know to begin using 'awk'. + + - *note Invoking Gawk::, describes how to run 'gawk', the + meaning of its command-line options, and how it finds 'awk' + program source files. + + - *note Regexp::, introduces regular expressions in general, and + in particular the flavors supported by POSIX 'awk' and 'gawk'. + + - *note Reading Files::, describes how 'awk' reads your data. + It introduces the concepts of records and fields, as well as + the 'getline' command. I/O redirection is first described + here. Network I/O is also briefly introduced here. + + - *note Printing::, describes how 'awk' programs can produce + output with 'print' and 'printf'. + + - *note Expressions::, describes expressions, which are the + basic building blocks for getting most things done in a + program. + + - *note Patterns and Actions::, describes how to write patterns + for matching records, actions for doing something when a + record is matched, and the predefined variables 'awk' and + 'gawk' use. + + - *note Arrays::, covers 'awk''s one-and-only data structure: + the associative array. Deleting array elements and whole + arrays is described, as well as sorting arrays in 'gawk'. The + major node also describes how 'gawk' provides arrays of + arrays. + + - *note Functions::, describes the built-in functions 'awk' and + 'gawk' provide, as well as how to define your own functions. + It also discusses how 'gawk' lets you call functions + indirectly. + + * Part II shows how to use 'awk' and 'gawk' for problem solving. + There is lots of code here for you to read and learn from. This + part contains the following chapters: + + - *note Library Functions::, provides a number of functions + meant to be used from main 'awk' programs. + + - *note Sample Programs::, provides many sample 'awk' programs. + + Reading these two chapters allows you to see 'awk' solving real + problems. + + * Part III focuses on features specific to 'gawk'. It contains the + following chapters: + + - *note Advanced Features::, describes a number of advanced + features. Of particular note are the abilities to control the + order of array traversal, have two-way communications with + another process, perform TCP/IP networking, and profile your + 'awk' programs. + + - *note Internationalization::, describes special features for + translating program messages into different languages at + runtime. + + - *note Debugger::, describes the 'gawk' debugger. + + - *note Arbitrary Precision Arithmetic::, describes advanced + arithmetic facilities. + + - *note Dynamic Extensions::, describes how to add new variables + and functions to 'gawk' by writing extensions in C or C++. + + * Part IV provides the appendices, the Glossary, and two licenses + that cover the 'gawk' source code and this Info file, respectively. + It contains the following appendices: + + - *note Language History::, describes how the 'awk' language has + evolved since its first release to the present. It also + describes how 'gawk' has acquired features over time. + + - *note Installation::, describes how to get 'gawk', how to + compile it on POSIX-compatible systems, and how to compile and + use it on different non-POSIX systems. It also describes how + to report bugs in 'gawk' and where to get other freely + available 'awk' implementations. + + - *note Notes::, describes how to disable 'gawk''s extensions, + as well as how to contribute new code to 'gawk', and some + possible future directions for 'gawk' development. + + - *note Basic Concepts::, provides some very cursory background + material for those who are completely unfamiliar with computer + programming. + + The *note Glossary::, defines most, if not all, of the + significant terms used throughout the Info file. If you find + terms that you aren't familiar with, try looking them up here. + + - *note Copying::, and *note GNU Free Documentation License::, + present the licenses that cover the 'gawk' source code and + this Info file, respectively. + + ---------- Footnotes ---------- + + (1) All such differences appear in the index under the entry +"differences in 'awk' and 'gawk'." + + +File: gawk.info, Node: Conventions, Next: Manual History, Prev: This Manual, Up: Preface + +Typographical Conventions +========================= + +This Info file is written in Texinfo +(http://www.gnu.org/software/texinfo/), the GNU documentation formatting +language. A single Texinfo source file is used to produce both the +printed and online versions of the documentation. This minor node +briefly documents the typographical conventions used in Texinfo. + + Examples you would type at the command line are preceded by the +common shell primary and secondary prompts, '$' and '>'. Input that you +type is shown 'like this'. Output from the command is preceded by the +glyph "-|". This typically represents the command's standard output. +Error messages and other output on the command's standard error are +preceded by the glyph "error->". For example: + + $ echo hi on stdout + -| hi on stdout + $ echo hello on stderr 1>&2 + error-> hello on stderr + + Characters that you type at the keyboard look 'like this'. In +particular, there are special characters called "control characters." +These are characters that you type by holding down both the 'CONTROL' +key and another key, at the same time. For example, a 'Ctrl-d' is typed +by first pressing and holding the 'CONTROL' key, next pressing the 'd' +key, and finally releasing both keys. + + For the sake of brevity, throughout this Info file, we refer to Brian +Kernighan's version of 'awk' as "BWK 'awk'." (*Note Other Versions:: +for information on his and other versions.) + +Dark Corners +------------ + + Dark corners are basically fractal--no matter how much you + illuminate, there's always a smaller but darker one. + -- _Brian Kernighan_ + + Until the POSIX standard (and 'GAWK: Effective AWK Programming'), +many features of 'awk' were either poorly documented or not documented +at all. Descriptions of such features (often called "dark corners") are +noted in this Info file with "(d.c.)." They also appear in the index +under the heading "dark corner." + + But, as noted by the opening quote, any coverage of dark corners is +by definition incomplete. + + Extensions to the standard 'awk' language that are supported by more +than one 'awk' implementation are marked "(c.e.)," and listed in the +index under "common extensions" and "extensions, common." + + +File: gawk.info, Node: Manual History, Next: How To Contribute, Prev: Conventions, Up: Preface + +The GNU Project and This Book +============================= + +The Free Software Foundation (FSF) is a nonprofit organization dedicated +to the production and distribution of freely distributable software. It +was founded by Richard M. Stallman, the author of the original Emacs +editor. GNU Emacs is the most widely used version of Emacs today. + + The GNU(1) Project is an ongoing effort on the part of the Free +Software Foundation to create a complete, freely distributable, +POSIX-compliant computing environment. The FSF uses the GNU General +Public License (GPL) to ensure that its software's source code is always +available to the end user. A copy of the GPL is included for your +reference (*note Copying::). The GPL applies to the C language source +code for 'gawk'. To find out more about the FSF and the GNU Project +online, see the GNU Project's home page (http://www.gnu.org). This Info +file may also be read from GNU's website +(http://www.gnu.org/software/gawk/manual/). + + A shell, an editor (Emacs), highly portable optimizing C, C++, and +Objective-C compilers, a symbolic debugger and dozens of large and small +utilities (such as 'gawk'), have all been completed and are freely +available. The GNU operating system kernel (the HURD), has been +released but remains in an early stage of development. + + Until the GNU operating system is more fully developed, you should +consider using GNU/Linux, a freely distributable, Unix-like operating +system for Intel, Power Architecture, Sun SPARC, IBM S/390, and other +systems.(2) Many GNU/Linux distributions are available for download +from the Internet. + + The Info file itself has gone through multiple previous editions. +Paul Rubin wrote the very first draft of 'The GAWK Manual'; it was +around 40 pages long. Diane Close and Richard Stallman improved it, +yielding a version that was around 90 pages and barely described the +original, "old" version of 'awk'. + + I started working with that version in the fall of 1988. As work on +it progressed, the FSF published several preliminary versions (numbered +0.X). In 1996, edition 1.0 was released with 'gawk' 3.0.0. The FSF +published the first two editions under the title 'The GNU Awk User's +Guide'. + + This edition maintains the basic structure of the previous editions. +For FSF edition 4.0, the content was thoroughly reviewed and updated. +All references to 'gawk' versions prior to 4.0 were removed. Of +significant note for that edition was the addition of *note Debugger::. + + For FSF edition 4.1, the content has been reorganized into parts, and +the major new additions are *note Arbitrary Precision Arithmetic::, and +*note Dynamic Extensions::. + + This Info file will undoubtedly continue to evolve. If you find an +error in the Info file, please report it! *Note Bugs:: for information +on submitting problem reports electronically. + + ---------- Footnotes ---------- + + (1) GNU stands for "GNU's Not Unix." + + (2) The terminology "GNU/Linux" is explained in the *note Glossary::. + + +File: gawk.info, Node: How To Contribute, Next: Acknowledgments, Prev: Manual History, Up: Preface + +How to Contribute +================= + +As the maintainer of GNU 'awk', I once thought that I would be able to +manage a collection of publicly available 'awk' programs and I even +solicited contributions. Making things available on the Internet helps +keep the 'gawk' distribution down to manageable size. + + The initial collection of material, such as it is, is still available +at <ftp://ftp.freefriends.org/arnold/Awkstuff>. In the hopes of doing +something more broad, I acquired the 'awk.info' domain. + + However, I found that I could not dedicate enough time to managing +contributed code: the archive did not grow and the domain went unused +for several years. + + Late in 2008, a volunteer took on the task of setting up an +'awk'-related website--<http://awk.info>--and did a very nice job. + + If you have written an interesting 'awk' program, or have written a +'gawk' extension that you would like to share with the rest of the +world, please see <http://awk.info/?contribute> for how to contribute it +to the website. + + +File: gawk.info, Node: Acknowledgments, Prev: How To Contribute, Up: Preface + +Acknowledgments +=============== + +The initial draft of 'The GAWK Manual' had the following +acknowledgments: + + Many people need to be thanked for their assistance in producing + this manual. Jay Fenlason contributed many ideas and sample + programs. Richard Mlynarik and Robert Chassell gave helpful + comments on drafts of this manual. The paper 'A Supplemental + Document for AWK' by John W. Pierce of the Chemistry Department at + UC San Diego, pinpointed several issues relevant both to 'awk' + implementation and to this manual, that would otherwise have + escaped us. + + I would like to acknowledge Richard M. Stallman, for his vision of a +better world and for his courage in founding the FSF and starting the +GNU Project. + + Earlier editions of this Info file had the following +acknowledgements: + + The following people (in alphabetical order) provided helpful + comments on various versions of this book: Rick Adams, Dr. Nelson + H.F. Beebe, Karl Berry, Dr. Michael Brennan, Rich Burridge, Claire + Cloutier, Diane Close, Scott Deifik, Christopher ("Topher") Eliot, + Jeffrey Friedl, Dr. Darrel Hankerson, Michal Jaegermann, Dr. + Richard J. LeBlanc, Michael Lijewski, Pat Rankin, Miriam Robbins, + Mary Sheehan, and Chuck Toporek. + + Robert J. Chassell provided much valuable advice on the use of + Texinfo. He also deserves special thanks for convincing me _not_ + to title this Info file 'How to Gawk Politely'. Karl Berry helped + significantly with the TeX part of Texinfo. + + I would like to thank Marshall and Elaine Hartholz of Seattle and + Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet + vacation time in their homes, which allowed me to make significant + progress on this Info file and on 'gawk' itself. + + Phil Hughes of SSC contributed in a very important way by loaning + me his laptop GNU/Linux system, not once, but twice, which allowed + me to do a lot of work while away from home. + + David Trueman deserves special credit; he has done a yeoman job of + evolving 'gawk' so that it performs well and without bugs. + Although he is no longer involved with 'gawk', working with him on + this project was a significant pleasure. + + The intrepid members of the GNITS mailing list, and most notably + Ulrich Drepper, provided invaluable help and feedback for the + design of the internationalization features. + + Chuck Toporek, Mary Sheehan, and Claire Cloutier of O'Reilly & + Associates contributed significant editorial help for this Info + file for the 3.1 release of 'gawk'. + + Dr. Nelson Beebe, Andreas Buening, Dr. Manuel Collado, Antonio +Colombo, Stephen Davies, Scott Deifik, Akim Demaille, Daniel Richard G., +Darrel Hankerson, Michal Jaegermann, Ju"rgen Kahrs, Stepan Kasal, John +Malmberg, Dave Pitts, Chet Ramey, Pat Rankin, Andrew Schorr, Corinna +Vinschen, and Eli Zaretskii (in alphabetical order) make up the current +'gawk' "crack portability team." Without their hard work and help, +'gawk' would not be nearly the robust, portable program it is today. It +has been and continues to be a pleasure working with this team of fine +people. + + Notable code and documentation contributions were made by a number of +people. *Note Contributors:: for the full list. + + Thanks to Michael Brennan for the Forewords. + + Thanks to Patrice Dumas for the new 'makeinfo' program. Thanks to +Karl Berry, who continues to work to keep the Texinfo markup language +sane. + + Robert P.J. Day, Michael Brennan, and Brian Kernighan kindly acted as +reviewers for the 2015 edition of this Info file. Their feedback helped +improve the final work. + + I would also like to thank Brian Kernighan for his invaluable +assistance during the testing and debugging of 'gawk', and for his +ongoing help and advice in clarifying numerous points about the +language. We could not have done nearly as good a job on either 'gawk' +or its documentation without his help. + + Brian is in a class by himself as a programmer and technical author. +I have to thank him (yet again) for his ongoing friendship and for being +a role model to me for close to 30 years! Having him as a reviewer is +an exciting privilege. It has also been extremely humbling... + + I must thank my wonderful wife, Miriam, for her patience through the +many versions of this project, for her proofreading, and for sharing me +with the computer. I would like to thank my parents for their love, and +for the grace with which they raised and educated me. Finally, I also +must acknowledge my gratitude to G-d, for the many opportunities He has +sent my way, as well as for the gifts He has given me with which to take +advantage of those opportunities. + + +Arnold Robbins +Nof Ayalon +Israel +February 2015 + + +File: gawk.info, Node: Getting Started, Next: Invoking Gawk, Prev: Preface, Up: Top + +1 Getting Started with 'awk' +**************************** + +The basic function of 'awk' is to search files for lines (or other units +of text) that contain certain patterns. When a line matches one of the +patterns, 'awk' performs specified actions on that line. 'awk' +continues to process input lines in this way until it reaches the end of +the input files. + + Programs in 'awk' are different from programs in most other +languages, because 'awk' programs are "data driven" (i.e., you describe +the data you want to work with and then what to do when you find it). +Most other languages are "procedural"; you have to describe, in great +detail, every step the program should take. When working with +procedural languages, it is usually much harder to clearly describe the +data your program will process. For this reason, 'awk' programs are +often refreshingly easy to read and write. + + When you run 'awk', you specify an 'awk' "program" that tells 'awk' +what to do. The program consists of a series of "rules" (it may also +contain "function definitions", an advanced feature that we will ignore +for now; *note User-defined::). Each rule specifies one pattern to +search for and one action to perform upon finding the pattern. + + Syntactically, a rule consists of a "pattern" followed by an +"action". The action is enclosed in braces to separate it from the +pattern. Newlines usually separate rules. Therefore, an 'awk' program +looks like this: + + PATTERN { ACTION } + PATTERN { ACTION } + ... + +* Menu: + +* Running gawk:: How to run 'gawk' programs; includes + command-line syntax. +* Sample Data Files:: Sample data files for use in the 'awk' + programs illustrated in this Info file. +* Very Simple:: A very simple example. +* Two Rules:: A less simple one-line example using two + rules. +* More Complex:: A more complex example. +* Statements/Lines:: Subdividing or combining statements into + lines. +* Other Features:: Other Features of 'awk'. +* When:: When to use 'gawk' and when to use + other things. +* Intro Summary:: Summary of the introduction. + + +File: gawk.info, Node: Running gawk, Next: Sample Data Files, Up: Getting Started + +1.1 How to Run 'awk' Programs +============================= + +There are several ways to run an 'awk' program. If the program is +short, it is easiest to include it in the command that runs 'awk', like +this: + + awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ... + + When the program is long, it is usually more convenient to put it in +a file and run it with a command like this: + + awk -f PROGRAM-FILE INPUT-FILE1 INPUT-FILE2 ... + + This minor node discusses both mechanisms, along with several +variations of each. + +* Menu: + +* One-shot:: Running a short throwaway 'awk' + program. +* Read Terminal:: Using no input files (input from the keyboard + instead). +* Long:: Putting permanent 'awk' programs in + files. +* Executable Scripts:: Making self-contained 'awk' programs. +* Comments:: Adding documentation to 'gawk' + programs. +* Quoting:: More discussion of shell quoting issues. + + +File: gawk.info, Node: One-shot, Next: Read Terminal, Up: Running gawk + +1.1.1 One-Shot Throwaway 'awk' Programs +--------------------------------------- + +Once you are familiar with 'awk', you will often type in simple programs +the moment you want to use them. Then you can write the program as the +first argument of the 'awk' command, like this: + + awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ... + +where PROGRAM consists of a series of patterns and actions, as described +earlier. + + This command format instructs the "shell", or command interpreter, to +start 'awk' and use the PROGRAM to process records in the input file(s). +There are single quotes around PROGRAM so the shell won't interpret any +'awk' characters as special shell characters. The quotes also cause the +shell to treat all of PROGRAM as a single argument for 'awk', and allow +PROGRAM to be more than one line long. + + This format is also useful for running short or medium-sized 'awk' +programs from shell scripts, because it avoids the need for a separate +file for the 'awk' program. A self-contained shell script is more +reliable because there are no other files to misplace. + + Later in this chapter, in *note Very Simple::, we'll see examples of +several short, self-contained programs. + + +File: gawk.info, Node: Read Terminal, Next: Long, Prev: One-shot, Up: Running gawk + +1.1.2 Running 'awk' Without Input Files +--------------------------------------- + +You can also run 'awk' without any input files. If you type the +following command line: + + awk 'PROGRAM' + +'awk' applies the PROGRAM to the "standard input", which usually means +whatever you type on the keyboard. This continues until you indicate +end-of-file by typing 'Ctrl-d'. (On non-POSIX operating systems, the +end-of-file character may be different.) + + As an example, the following program prints a friendly piece of +advice (from Douglas Adams's 'The Hitchhiker's Guide to the Galaxy'), to +keep you from worrying about the complexities of computer programming: + + $ awk 'BEGIN { print "Don\47t Panic!" }' + -| Don't Panic! + + 'awk' executes statements associated with 'BEGIN' before reading any +input. If there are no other statements in your program, as is the case +here, 'awk' just stops, instead of trying to read input it doesn't know +how to process. The '\47' is a magic way (explained later) of getting a +single quote into the program, without having to engage in ugly shell +quoting tricks. + + NOTE: If you use Bash as your shell, you should execute the command + 'set +H' before running this program interactively, to disable the + C shell-style command history, which treats '!' as a special + character. We recommend putting this command into your personal + startup file. + + This next simple 'awk' program emulates the 'cat' utility; it copies +whatever you type on the keyboard to its standard output (why this works +is explained shortly): + + $ awk '{ print }' + Now is the time for all good men + -| Now is the time for all good men + to come to the aid of their country. + -| to come to the aid of their country. + Four score and seven years ago, ... + -| Four score and seven years ago, ... + What, me worry? + -| What, me worry? + Ctrl-d + + +File: gawk.info, Node: Long, Next: Executable Scripts, Prev: Read Terminal, Up: Running gawk + +1.1.3 Running Long Programs +--------------------------- + +Sometimes 'awk' programs are very long. In these cases, it is more +convenient to put the program into a separate file. In order to tell +'awk' to use that file for its program, you type: + + awk -f SOURCE-FILE INPUT-FILE1 INPUT-FILE2 ... + + The '-f' instructs the 'awk' utility to get the 'awk' program from +the file SOURCE-FILE (*note Options::). Any file name can be used for +SOURCE-FILE. For example, you could put the program: + + BEGIN { print "Don't Panic!" } + +into the file 'advice'. Then this command: + + awk -f advice + +does the same thing as this one: + + awk 'BEGIN { print "Don\47t Panic!" }' + +This was explained earlier (*note Read Terminal::). Note that you don't +usually need single quotes around the file name that you specify with +'-f', because most file names don't contain any of the shell's special +characters. Notice that in 'advice', the 'awk' program did not have +single quotes around it. The quotes are only needed for programs that +are provided on the 'awk' command line. (Also, placing the program in a +file allows us to use a literal single quote in the program text, +instead of the magic '\47'.) + + If you want to clearly identify an 'awk' program file as such, you +can add the extension '.awk' to the file name. This doesn't affect the +execution of the 'awk' program but it does make "housekeeping" easier. + + +File: gawk.info, Node: Executable Scripts, Next: Comments, Prev: Long, Up: Running gawk + +1.1.4 Executable 'awk' Programs +------------------------------- + +Once you have learned 'awk', you may want to write self-contained 'awk' +scripts, using the '#!' script mechanism. You can do this on many +systems.(1) For example, you could update the file 'advice' to look +like this: + + #! /bin/awk -f + + BEGIN { print "Don't Panic!" } + +After making this file executable (with the 'chmod' utility), simply +type 'advice' at the shell and the system arranges to run 'awk' as if +you had typed 'awk -f advice': + + $ chmod +x advice + $ advice + -| Don't Panic! + +(We assume you have the current directory in your shell's search path +variable [typically '$PATH']. If not, you may need to type './advice' +at the shell.) + + Self-contained 'awk' scripts are useful when you want to write a +program that users can invoke without their having to know that the +program is written in 'awk'. + + Understanding '#!' + + 'awk' is an "interpreted" language. This means that the 'awk' +utility reads your program and then processes your data according to the +instructions in your program. (This is different from a "compiled" +language such as C, where your program is first compiled into machine +code that is executed directly by your system's processor.) The 'awk' +utility is thus termed an "interpreter". Many modern languages are +interpreted. + + The line beginning with '#!' lists the full file name of an +interpreter to run and a single optional initial command-line argument +to pass to that interpreter. The operating system then runs the +interpreter with the given argument and the full argument list of the +executed program. The first argument in the list is the full file name +of the 'awk' program. The rest of the argument list contains either +options to 'awk', or data files, or both. (Note that on many systems +'awk' may be found in '/usr/bin' instead of in '/bin'.) + + Some systems limit the length of the interpreter name to 32 +characters. Often, this can be dealt with by using a symbolic link. + + You should not put more than one argument on the '#!' line after the +path to 'awk'. It does not work. The operating system treats the rest +of the line as a single argument and passes it to 'awk'. Doing this +leads to confusing behavior--most likely a usage diagnostic of some sort +from 'awk'. + + Finally, the value of 'ARGV[0]' (*note Built-in Variables::) varies +depending upon your operating system. Some systems put 'awk' there, +some put the full pathname of 'awk' (such as '/bin/awk'), and some put +the name of your script ('advice'). (d.c.) Don't rely on the value of +'ARGV[0]' to provide your script name. + + ---------- Footnotes ---------- + + (1) The '#!' mechanism works on GNU/Linux systems, BSD-based systems, +and commercial Unix systems. + + +File: gawk.info, Node: Comments, Next: Quoting, Prev: Executable Scripts, Up: Running gawk + +1.1.5 Comments in 'awk' Programs +-------------------------------- + +A "comment" is some text that is included in a program for the sake of +human readers; it is not really an executable part of the program. +Comments can explain what the program does and how it works. Nearly all +programming languages have provisions for comments, as programs are +typically hard to understand without them. + + In the 'awk' language, a comment starts with the number sign +character ('#') and continues to the end of the line. The '#' does not +have to be the first character on the line. The 'awk' language ignores +the rest of a line following a number sign. For example, we could have +put the following into 'advice': + + # This program prints a nice, friendly message. It helps + # keep novice users from being afraid of the computer. + BEGIN { print "Don't Panic!" } + + You can put comment lines into keyboard-composed throwaway 'awk' +programs, but this usually isn't very useful; the purpose of a comment +is to help you or another person understand the program when reading it +at a later time. + + CAUTION: As mentioned in *note One-shot::, you can enclose short to + medium-sized programs in single quotes, in order to keep your shell + scripts self-contained. When doing so, _don't_ put an apostrophe + (i.e., a single quote) into a comment (or anywhere else in your + program). The shell interprets the quote as the closing quote for + the entire program. As a result, usually the shell prints a + message about mismatched quotes, and if 'awk' actually runs, it + will probably print strange messages about syntax errors. For + example, look at the following: + + $ awk 'BEGIN { print "hello" } # let's be cute' + > + + The shell sees that the first two quotes match, and that a new + quoted object begins at the end of the command line. It therefore + prompts with the secondary prompt, waiting for more input. With + Unix 'awk', closing the quoted string produces this result: + + $ awk '{ print "hello" } # let's be cute' + > ' + error-> awk: can't open file be + error-> source line number 1 + + Putting a backslash before the single quote in 'let's' wouldn't + help, because backslashes are not special inside single quotes. + The next node describes the shell's quoting rules. + + +File: gawk.info, Node: Quoting, Prev: Comments, Up: Running gawk + +1.1.6 Shell Quoting Issues +-------------------------- + +* Menu: + +* DOS Quoting:: Quoting in Windows Batch Files. + +For short to medium-length 'awk' programs, it is most convenient to +enter the program on the 'awk' command line. This is best done by +enclosing the entire program in single quotes. This is true whether you +are entering the program interactively at the shell prompt, or writing +it as part of a larger shell script: + + awk 'PROGRAM TEXT' INPUT-FILE1 INPUT-FILE2 ... + + Once you are working with the shell, it is helpful to have a basic +knowledge of shell quoting rules. The following rules apply only to +POSIX-compliant, Bourne-style shells (such as Bash, the GNU Bourne-Again +Shell). If you use the C shell, you're on your own. + + Before diving into the rules, we introduce a concept that appears +throughout this Info file, which is that of the "null", or empty, +string. + + The null string is character data that has no value. In other words, +it is empty. It is written in 'awk' programs like this: '""'. In the +shell, it can be written using single or double quotes: '""' or ''''. +Although the null string has no characters in it, it does exist. For +example, consider this command: + + $ echo "" + +Here, the 'echo' utility receives a single argument, even though that +argument has no characters in it. In the rest of this Info file, we use +the terms "null string" and "empty string" interchangeably. Now, on to +the quoting rules: + + * Quoted items can be concatenated with nonquoted items as well as + with other quoted items. The shell turns everything into one + argument for the command. + + * Preceding any single character with a backslash ('\') quotes that + character. The shell removes the backslash and passes the quoted + character on to the command. + + * Single quotes protect everything between the opening and closing + quotes. The shell does no interpretation of the quoted text, + passing it on verbatim to the command. It is _impossible_ to embed + a single quote inside single-quoted text. Refer back to *note + Comments:: for an example of what happens if you try. + + * Double quotes protect most things between the opening and closing + quotes. The shell does at least variable and command substitution + on the quoted text. Different shells may do additional kinds of + processing on double-quoted text. + + Because certain characters within double-quoted text are processed + by the shell, they must be "escaped" within the text. Of note are + the characters '$', '`', '\', and '"', all of which must be + preceded by a backslash within double-quoted text if they are to be + passed on literally to the program. (The leading backslash is + stripped first.) Thus, the example seen in *note Read Terminal::: + + awk 'BEGIN { print "Don\47t Panic!" }' + + could instead be written this way: + + $ awk "BEGIN { print \"Don't Panic!\" }" + -| Don't Panic! + + Note that the single quote is not special within double quotes. + + * Null strings are removed when they occur as part of a non-null + command-line argument, while explicit null objects are kept. For + example, to specify that the field separator 'FS' should be set to + the null string, use: + + awk -F "" 'PROGRAM' FILES # correct + + Don't use this: + + awk -F"" 'PROGRAM' FILES # wrong! + + In the second case, 'awk' attempts to use the text of the program + as the value of 'FS', and the first file name as the text of the + program! This results in syntax errors at best, and confusing + behavior at worst. + + Mixing single and double quotes is difficult. You have to resort to +shell quoting tricks, like this: + + $ awk 'BEGIN { print "Here is a single quote <'"'"'>" }' + -| Here is a single quote <'> + +This program consists of three concatenated quoted strings. The first +and the third are single-quoted, and the second is double-quoted. + + This can be "simplified" to: + + $ awk 'BEGIN { print "Here is a single quote <'\''>" }' + -| Here is a single quote <'> + +Judge for yourself which of these two is the more readable. + + Another option is to use double quotes, escaping the embedded, +'awk'-level double quotes: + + $ awk "BEGIN { print \"Here is a single quote <'>\" }" + -| Here is a single quote <'> + +This option is also painful, because double quotes, backslashes, and +dollar signs are very common in more advanced 'awk' programs. + + A third option is to use the octal escape sequence equivalents (*note +Escape Sequences::) for the single- and double-quote characters, like +so: + + $ awk 'BEGIN { print "Here is a single quote <\47>" }' + -| Here is a single quote <'> + $ awk 'BEGIN { print "Here is a double quote <\42>" }' + -| Here is a double quote <"> + +This works nicely, but you should comment clearly what the escapes mean. + + A fourth option is to use command-line variable assignment, like +this: + + $ awk -v sq="'" 'BEGIN { print "Here is a single quote <" sq ">" }' + -| Here is a single quote <'> + + (Here, the two string constants and the value of 'sq' are +concatenated into a single string that is printed by 'print'.) + + If you really need both single and double quotes in your 'awk' +program, it is probably best to move it into a separate file, where the +shell won't be part of the picture and you can say what you mean. + + +File: gawk.info, Node: DOS Quoting, Up: Quoting + +1.1.6.1 Quoting in MS-Windows Batch Files +......................................... + +Although this Info file generally only worries about POSIX systems and +the POSIX shell, the following issue arises often enough for many users +that it is worth addressing. + + The "shells" on Microsoft Windows systems use the double-quote +character for quoting, and make it difficult or impossible to include an +escaped double-quote character in a command-line script. The following +example, courtesy of Jeroen Brink, shows how to print all lines in a +file surrounded by double quotes: + + gawk "{ print \"\042\" $0 \"\042\" }" FILE + + +File: gawk.info, Node: Sample Data Files, Next: Very Simple, Prev: Running gawk, Up: Getting Started + +1.2 Data files for the Examples +=============================== + +Many of the examples in this Info file take their input from two sample +data files. The first, 'mail-list', represents a list of peoples' names +together with their email addresses and information about those people. +The second data file, called 'inventory-shipped', contains information +about monthly shipments. In both files, each line is considered to be +one "record". + + In 'mail-list', each record contains the name of a person, his/her +phone number, his/her email address, and a code for his/her relationship +with the author of the list. The columns are aligned using spaces. An +'A' in the last column means that the person is an acquaintance. An 'F' +in the last column means that the person is a friend. An 'R' means that +the person is a relative: + + Amelia 555-5553 amelia.zodiacusque@gmail.com F + Anthony 555-3412 anthony.asserturo@hotmail.com A + Becky 555-7685 becky.algebrarum@gmail.com A + Bill 555-1675 bill.drowning@hotmail.com A + Broderick 555-0542 broderick.aliquotiens@yahoo.com R + Camilla 555-2912 camilla.infusarum@skynet.be R + Fabius 555-1234 fabius.undevicesimus@ucb.edu F + Julie 555-6699 julie.perscrutabor@skeeve.com F + Martin 555-6480 martin.codicibus@hotmail.com A + Samuel 555-3430 samuel.lanceolis@shu.edu A + Jean-Paul 555-2127 jeanpaul.campanorum@nyu.edu R + + The data file 'inventory-shipped' represents information about +shipments during the year. Each record contains the month, the number +of green crates shipped, the number of red boxes shipped, the number of +orange bags shipped, and the number of blue packages shipped, +respectively. There are 16 entries, covering the 12 months of last year +and the first four months of the current year. An empty line separates +the data for the two years: + + Jan 13 25 15 115 + Feb 15 32 24 226 + Mar 15 24 34 228 + Apr 31 52 63 420 + May 16 34 29 208 + Jun 31 42 75 492 + Jul 24 34 67 436 + Aug 15 34 47 316 + Sep 13 55 37 277 + Oct 29 54 68 525 + Nov 20 87 82 577 + Dec 17 35 61 401 + + Jan 21 36 64 620 + Feb 26 58 80 652 + Mar 24 75 70 495 + Apr 21 70 74 514 + + The sample files are included in the 'gawk' distribution, in the +directory 'awklib/eg/data'. + + +File: gawk.info, Node: Very Simple, Next: Two Rules, Prev: Sample Data Files, Up: Getting Started + +1.3 Some Simple Examples +======================== + +The following command runs a simple 'awk' program that searches the +input file 'mail-list' for the character string 'li' (a grouping of +characters is usually called a "string"; the term "string" is based on +similar usage in English, such as "a string of pearls" or "a string of +cars in a train"): + + awk '/li/ { print $0 }' mail-list + +When lines containing 'li' are found, they are printed because +'print $0' means print the current line. (Just 'print' by itself means +the same thing, so we could have written that instead.) + + You will notice that slashes ('/') surround the string 'li' in the +'awk' program. The slashes indicate that 'li' is the pattern to search +for. This type of pattern is called a "regular expression", which is +covered in more detail later (*note Regexp::). The pattern is allowed +to match parts of words. There are single quotes around the 'awk' +program so that the shell won't interpret any of it as special shell +characters. + + Here is what this program prints: + + $ awk '/li/ { print $0 }' mail-list + -| Amelia 555-5553 amelia.zodiacusque@gmail.com F + -| Broderick 555-0542 broderick.aliquotiens@yahoo.com R + -| Julie 555-6699 julie.perscrutabor@skeeve.com F + -| Samuel 555-3430 samuel.lanceolis@shu.edu A + + In an 'awk' rule, either the pattern or the action can be omitted, +but not both. If the pattern is omitted, then the action is performed +for _every_ input line. If the action is omitted, the default action is +to print all lines that match the pattern. + + Thus, we could leave out the action (the 'print' statement and the +braces) in the previous example and the result would be the same: 'awk' +prints all lines matching the pattern 'li'. By comparison, omitting the +'print' statement but retaining the braces makes an empty action that +does nothing (i.e., no lines are printed). + + Many practical 'awk' programs are just a line or two long. Following +is a collection of useful, short programs to get you started. Some of +these programs contain constructs that haven't been covered yet. (The +description of the program will give you a good idea of what is going +on, but you'll need to read the rest of the Info file to become an 'awk' +expert!) Most of the examples use a data file named 'data'. This is +just a placeholder; if you use these programs yourself, substitute your +own file names for 'data'. For future reference, note that there is +often more than one way to do things in 'awk'. At some point, you may +want to look back at these examples and see if you can come up with +different ways to do the same things shown here: + + * Print every line that is longer than 80 characters: + + awk 'length($0) > 80' data + + The sole rule has a relational expression as its pattern and has no + action--so it uses the default action, printing the record. + + * Print the length of the longest input line: + + awk '{ if (length($0) > max) max = length($0) } + END { print max }' data + + The code associated with 'END' executes after all input has been + read; it's the other side of the coin to 'BEGIN'. + + * Print the length of the longest line in 'data': + + expand data | awk '{ if (x < length($0)) x = length($0) } + END { print "maximum line length is " x }' + + This example differs slightly from the previous one: the input is + processed by the 'expand' utility to change TABs into spaces, so + the widths compared are actually the right-margin columns, as + opposed to the number of input characters on each line. + + * Print every line that has at least one field: + + awk 'NF > 0' data + + This is an easy way to delete blank lines from a file (or rather, + to create a new file similar to the old file but from which the + blank lines have been removed). + + * Print seven random numbers from 0 to 100, inclusive: + + awk 'BEGIN { for (i = 1; i <= 7; i++) + print int(101 * rand()) }' + + * Print the total number of bytes used by FILES: + + ls -l FILES | awk '{ x += $5 } + END { print "total bytes: " x }' + + * Print the total number of kilobytes used by FILES: + + ls -l FILES | awk '{ x += $5 } + END { print "total K-bytes:", x / 1024 }' + + * Print a sorted list of the login names of all users: + + awk -F: '{ print $1 }' /etc/passwd | sort + + * Count the lines in a file: + + awk 'END { print NR }' data + + * Print the even-numbered lines in the data file: + + awk 'NR % 2 == 0' data + + If you used the expression 'NR % 2 == 1' instead, the program would + print the odd-numbered lines. + + +File: gawk.info, Node: Two Rules, Next: More Complex, Prev: Very Simple, Up: Getting Started + +1.4 An Example with Two Rules +============================= + +The 'awk' utility reads the input files one line at a time. For each +line, 'awk' tries the patterns of each rule. If several patterns match, +then several actions execute in the order in which they appear in the +'awk' program. If no patterns match, then no actions run. + + After processing all the rules that match the line (and perhaps there +are none), 'awk' reads the next line. (However, *note Next Statement:: +and also *note Nextfile Statement::.) This continues until the program +reaches the end of the file. For example, the following 'awk' program +contains two rules: + + /12/ { print $0 } + /21/ { print $0 } + +The first rule has the string '12' as the pattern and 'print $0' as the +action. The second rule has the string '21' as the pattern and also has +'print $0' as the action. Each rule's action is enclosed in its own +pair of braces. + + This program prints every line that contains the string '12' _or_ the +string '21'. If a line contains both strings, it is printed twice, once +by each rule. + + This is what happens if we run this program on our two sample data +files, 'mail-list' and 'inventory-shipped': + + $ awk '/12/ { print $0 } + > /21/ { print $0 }' mail-list inventory-shipped + -| Anthony 555-3412 anthony.asserturo@hotmail.com A + -| Camilla 555-2912 camilla.infusarum@skynet.be R + -| Fabius 555-1234 fabius.undevicesimus@ucb.edu F + -| Jean-Paul 555-2127 jeanpaul.campanorum@nyu.edu R + -| Jean-Paul 555-2127 jeanpaul.campanorum@nyu.edu R + -| Jan 21 36 64 620 + -| Apr 21 70 74 514 + +Note how the line beginning with 'Jean-Paul' in 'mail-list' was printed +twice, once for each rule. + + +File: gawk.info, Node: More Complex, Next: Statements/Lines, Prev: Two Rules, Up: Getting Started + +1.5 A More Complex Example +========================== + +Now that we've mastered some simple tasks, let's look at what typical +'awk' programs do. This example shows how 'awk' can be used to +summarize, select, and rearrange the output of another utility. It uses +features that haven't been covered yet, so don't worry if you don't +understand all the details: + + ls -l | awk '$6 == "Nov" { sum += $5 } + END { print sum }' + + This command prints the total number of bytes in all the files in the +current directory that were last modified in November (of any year). +The 'ls -l' part of this example is a system command that gives you a +listing of the files in a directory, including each file's size and the +date the file was last modified. Its output looks like this: + + -rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile + -rw-r--r-- 1 arnold user 10809 Nov 7 13:03 awk.h + -rw-r--r-- 1 arnold user 983 Apr 13 12:14 awk.tab.h + -rw-r--r-- 1 arnold user 31869 Jun 15 12:20 awkgram.y + -rw-r--r-- 1 arnold user 22414 Nov 7 13:03 awk1.c + -rw-r--r-- 1 arnold user 37455 Nov 7 13:03 awk2.c + -rw-r--r-- 1 arnold user 27511 Dec 9 13:07 awk3.c + -rw-r--r-- 1 arnold user 7989 Nov 7 13:03 awk4.c + +The first field contains read-write permissions, the second field +contains the number of links to the file, and the third field identifies +the file's owner. The fourth field identifies the file's group. The +fifth field contains the file's size in bytes. The sixth, seventh, and +eighth fields contain the month, day, and time, respectively, that the +file was last modified. Finally, the ninth field contains the file +name. + + The '$6 == "Nov"' in our 'awk' program is an expression that tests +whether the sixth field of the output from 'ls -l' matches the string +'Nov'. Each time a line has the string 'Nov' for its sixth field, 'awk' +performs the action 'sum += $5'. This adds the fifth field (the file's +size) to the variable 'sum'. As a result, when 'awk' has finished +reading all the input lines, 'sum' is the total of the sizes of the +files whose lines matched the pattern. (This works because 'awk' +variables are automatically initialized to zero.) + + After the last line of output from 'ls' has been processed, the 'END' +rule executes and prints the value of 'sum'. In this example, the value +of 'sum' is 80600. + + These more advanced 'awk' techniques are covered in later minor nodes +(*note Action Overview::). Before you can move on to more advanced +'awk' programming, you have to know how 'awk' interprets your input and +displays your output. By manipulating fields and using 'print' +statements, you can produce some very useful and impressive-looking +reports. + + +File: gawk.info, Node: Statements/Lines, Next: Other Features, Prev: More Complex, Up: Getting Started + +1.6 'awk' Statements Versus Lines +================================= + +Most often, each line in an 'awk' program is a separate statement or +separate rule, like this: + + awk '/12/ { print $0 } + /21/ { print $0 }' mail-list inventory-shipped + + However, 'gawk' ignores newlines after any of the following symbols +and keywords: + + , { ? : || && do else + +A newline at any other point is considered the end of the statement.(1) + + If you would like to split a single statement into two lines at a +point where a newline would terminate it, you can "continue" it by +ending the first line with a backslash character ('\'). The backslash +must be the final character on the line in order to be recognized as a +continuation character. A backslash is allowed anywhere in the +statement, even in the middle of a string or regular expression. For +example: + + awk '/This regular expression is too long, so continue it\ + on the next line/ { print $1 }' + +We have generally not used backslash continuation in our sample +programs. 'gawk' places no limit on the length of a line, so backslash +continuation is never strictly necessary; it just makes programs more +readable. For this same reason, as well as for clarity, we have kept +most statements short in the programs presented throughout the Info +file. Backslash continuation is most useful when your 'awk' program is +in a separate source file instead of entered from the command line. You +should also note that many 'awk' implementations are more particular +about where you may use backslash continuation. For example, they may +not allow you to split a string constant using backslash continuation. +Thus, for maximum portability of your 'awk' programs, it is best not to +split your lines in the middle of a regular expression or a string. + + CAUTION: _Backslash continuation does not work as described with + the C shell._ It works for 'awk' programs in files and for + one-shot programs, _provided_ you are using a POSIX-compliant + shell, such as the Unix Bourne shell or Bash. But the C shell + behaves differently! There you must use two backslashes in a row, + followed by a newline. Note also that when using the C shell, + _every_ newline in your 'awk' program must be escaped with a + backslash. To illustrate: + + % awk 'BEGIN { \ + ? print \\ + ? "hello, world" \ + ? }' + -| hello, world + + Here, the '%' and '?' are the C shell's primary and secondary + prompts, analogous to the standard shell's '$' and '>'. + + Compare the previous example to how it is done with a + POSIX-compliant shell: + + $ awk 'BEGIN { + > print \ + > "hello, world" + > }' + -| hello, world + + 'awk' is a line-oriented language. Each rule's action has to begin +on the same line as the pattern. To have the pattern and action on +separate lines, you _must_ use backslash continuation; there is no other +option. + + Another thing to keep in mind is that backslash continuation and +comments do not mix. As soon as 'awk' sees the '#' that starts a +comment, it ignores _everything_ on the rest of the line. For example: + + $ gawk 'BEGIN { print "dont panic" # a friendly \ + > BEGIN rule + > }' + error-> gawk: cmd. line:2: BEGIN rule + error-> gawk: cmd. line:2: ^ syntax error + +In this case, it looks like the backslash would continue the comment +onto the next line. However, the backslash-newline combination is never +even noticed because it is "hidden" inside the comment. Thus, the +'BEGIN' is noted as a syntax error. + + When 'awk' statements within one rule are short, you might want to +put more than one of them on a line. This is accomplished by separating +the statements with a semicolon (';'). This also applies to the rules +themselves. Thus, the program shown at the start of this minor node +could also be written this way: + + /12/ { print $0 } ; /21/ { print $0 } + + NOTE: The requirement that states that rules on the same line must + be separated with a semicolon was not in the original 'awk' + language; it was added for consistency with the treatment of + statements within an action. + + ---------- Footnotes ---------- + + (1) The '?' and ':' referred to here is the three-operand conditional +expression described in *note Conditional Exp::. Splitting lines after +'?' and ':' is a minor 'gawk' extension; if '--posix' is specified +(*note Options::), then this extension is disabled. + + +File: gawk.info, Node: Other Features, Next: When, Prev: Statements/Lines, Up: Getting Started + +1.7 Other Features of 'awk' +=========================== + +The 'awk' language provides a number of predefined, or "built-in", +variables that your programs can use to get information from 'awk'. +There are other variables your program can set as well to control how +'awk' processes your data. + + In addition, 'awk' provides a number of built-in functions for doing +common computational and string-related operations. 'gawk' provides +built-in functions for working with timestamps, performing bit +manipulation, for runtime string translation (internationalization), +determining the type of a variable, and array sorting. + + As we develop our presentation of the 'awk' language, we will +introduce most of the variables and many of the functions. They are +described systematically in *note Built-in Variables:: and in *note +Built-in::. + + +File: gawk.info, Node: When, Next: Intro Summary, Prev: Other Features, Up: Getting Started + +1.8 When to Use 'awk' +===================== + +Now that you've seen some of what 'awk' can do, you might wonder how +'awk' could be useful for you. By using utility programs, advanced +patterns, field separators, arithmetic statements, and other selection +criteria, you can produce much more complex output. The 'awk' language +is very useful for producing reports from large amounts of raw data, +such as summarizing information from the output of other utility +programs like 'ls'. (*Note More Complex::.) + + Programs written with 'awk' are usually much smaller than they would +be in other languages. This makes 'awk' programs easy to compose and +use. Often, 'awk' programs can be quickly composed at your keyboard, +used once, and thrown away. Because 'awk' programs are interpreted, you +can avoid the (usually lengthy) compilation part of the typical +edit-compile-test-debug cycle of software development. + + Complex programs have been written in 'awk', including a complete +retargetable assembler for eight-bit microprocessors (*note Glossary::, +for more information), and a microcode assembler for a special-purpose +Prolog computer. The original 'awk''s capabilities were strained by +tasks of such complexity, but modern versions are more capable. + + If you find yourself writing 'awk' scripts of more than, say, a few +hundred lines, you might consider using a different programming +language. The shell is good at string and pattern matching; in +addition, it allows powerful use of the system utilities. Python offers +a nice balance between high-level ease of programming and access to +system facilities.(1) + + ---------- Footnotes ---------- + + (1) Other popular scripting languages include Ruby and Perl. + + +File: gawk.info, Node: Intro Summary, Prev: When, Up: Getting Started + +1.9 Summary +=========== + + * Programs in 'awk' consist of PATTERN-ACTION pairs. + + * An ACTION without a PATTERN always runs. The default ACTION for a + pattern without one is '{ print $0 }'. + + * Use either 'awk 'PROGRAM' FILES' or 'awk -f PROGRAM-FILE FILES' to + run 'awk'. + + * You may use the special '#!' header line to create 'awk' programs + that are directly executable. + + * Comments in 'awk' programs start with '#' and continue to the end + of the same line. + + * Be aware of quoting issues when writing 'awk' programs as part of a + larger shell script (or MS-Windows batch file). + + * You may use backslash continuation to continue a source line. + Lines are automatically continued after a comma, open brace, + question mark, colon, '||', '&&', 'do', and 'else'. + + +File: gawk.info, Node: Invoking Gawk, Next: Regexp, Prev: Getting Started, Up: Top + +2 Running 'awk' and 'gawk' +************************** + +This major node covers how to run 'awk', both POSIX-standard and +'gawk'-specific command-line options, and what 'awk' and 'gawk' do with +nonoption arguments. It then proceeds to cover how 'gawk' searches for +source files, reading standard input along with other files, 'gawk''s +environment variables, 'gawk''s exit status, using include files, and +obsolete and undocumented options and/or features. + + Many of the options and features described here are discussed in more +detail later in the Info file; feel free to skip over things in this +major node that don't interest you right now. + +* Menu: + +* Command Line:: How to run 'awk'. +* Options:: Command-line options and their meanings. +* Other Arguments:: Input file names and variable assignments. +* Naming Standard Input:: How to specify standard input with other + files. +* Environment Variables:: The environment variables 'gawk' uses. +* Exit Status:: 'gawk''s exit status. +* Include Files:: Including other files into your program. +* Loading Shared Libraries:: Loading shared libraries into your program. +* Obsolete:: Obsolete Options and/or features. +* Undocumented:: Undocumented Options and Features. +* Invoking Summary:: Invocation summary. + + +File: gawk.info, Node: Command Line, Next: Options, Up: Invoking Gawk + +2.1 Invoking 'awk' +================== + +There are two ways to run 'awk'--with an explicit program or with one or +more program files. Here are templates for both of them; items enclosed +in [...] in these templates are optional: + + 'awk' [OPTIONS] '-f' PROGFILE ['--'] FILE ... + 'awk' [OPTIONS] ['--'] ''PROGRAM'' FILE ... + + In addition to traditional one-letter POSIX-style options, 'gawk' +also supports GNU long options. + + It is possible to invoke 'awk' with an empty program: + + awk '' datafile1 datafile2 + +Doing so makes little sense, though; 'awk' exits silently when given an +empty program. (d.c.) If '--lint' has been specified on the command +line, 'gawk' issues a warning that the program is empty. + + +File: gawk.info, Node: Options, Next: Other Arguments, Prev: Command Line, Up: Invoking Gawk + +2.2 Command-Line Options +======================== + +Options begin with a dash and consist of a single character. GNU-style +long options consist of two dashes and a keyword. The keyword can be +abbreviated, as long as the abbreviation allows the option to be +uniquely identified. If the option takes an argument, either the +keyword is immediately followed by an equals sign ('=') and the +argument's value, or the keyword and the argument's value are separated +by whitespace. If a particular option with a value is given more than +once, it is the last value that counts. + + Each long option for 'gawk' has a corresponding POSIX-style short +option. The long and short options are interchangeable in all contexts. +The following list describes options mandated by the POSIX standard: + +'-F FS' +'--field-separator FS' + Set the 'FS' variable to FS (*note Field Separators::). + +'-f SOURCE-FILE' +'--file SOURCE-FILE' + Read the 'awk' program source from SOURCE-FILE instead of in the + first nonoption argument. This option may be given multiple times; + the 'awk' program consists of the concatenation of the contents of + each specified SOURCE-FILE. + +'-v VAR=VAL' +'--assign VAR=VAL' + Set the variable VAR to the value VAL _before_ execution of the + program begins. Such variable values are available inside the + 'BEGIN' rule (*note Other Arguments::). + + The '-v' option can only set one variable, but it can be used more + than once, setting another variable each time, like this: 'awk + -v foo=1 -v bar=2 ...'. + + CAUTION: Using '-v' to set the values of the built-in + variables may lead to surprising results. 'awk' will reset + the values of those variables as it needs to, possibly + ignoring any initial value you may have given. + +'-W GAWK-OPT' + Provide an implementation-specific option. This is the POSIX + convention for providing implementation-specific options. These + options also have corresponding GNU-style long options. Note that + the long options may be abbreviated, as long as the abbreviations + remain unique. The full list of 'gawk'-specific options is + provided next. + +'--' + Signal the end of the command-line options. The following + arguments are not treated as options even if they begin with '-'. + This interpretation of '--' follows the POSIX argument parsing + conventions. + + This is useful if you have file names that start with '-', or in + shell scripts, if you have file names that will be specified by the + user that could start with '-'. It is also useful for passing + options on to the 'awk' program; see *note Getopt Function::. + + The following list describes 'gawk'-specific options: + +'-b' +'--characters-as-bytes' + Cause 'gawk' to treat all input data as single-byte characters. In + addition, all output written with 'print' or 'printf' is treated as + single-byte characters. + + Normally, 'gawk' follows the POSIX standard and attempts to process + its input data according to the current locale (*note Locales::). + This can often involve converting multibyte characters into wide + characters (internally), and can lead to problems or confusion if + the input data does not contain valid multibyte characters. This + option is an easy way to tell 'gawk', "Hands off my data!" + +'-c' +'--traditional' + Specify "compatibility mode", in which the GNU extensions to the + 'awk' language are disabled, so that 'gawk' behaves just like BWK + 'awk'. *Note POSIX/GNU::, which summarizes the extensions. Also + see *note Compatibility Mode::. + +'-C' +'--copyright' + Print the short version of the General Public License and then + exit. + +'-d'[FILE] +'--dump-variables'['='FILE] + Print a sorted list of global variables, their types, and final + values to FILE. If no FILE is provided, print this list to a file + named 'awkvars.out' in the current directory. No space is allowed + between the '-d' and FILE, if FILE is supplied. + + Having a list of all global variables is a good way to look for + typographical errors in your programs. You would also use this + option if you have a large program with a lot of functions, and you + want to be sure that your functions don't inadvertently use global + variables that you meant to be local. (This is a particularly easy + mistake to make with simple variable names like 'i', 'j', etc.) + +'-D'[FILE] +'--debug'['='FILE] + Enable debugging of 'awk' programs (*note Debugging::). By + default, the debugger reads commands interactively from the + keyboard (standard input). The optional FILE argument allows you + to specify a file with a list of commands for the debugger to + execute noninteractively. No space is allowed between the '-D' and + FILE, if FILE is supplied. + +'-e' PROGRAM-TEXT +'--source' PROGRAM-TEXT + Provide program source code in the PROGRAM-TEXT. This option + allows you to mix source code in files with source code that you + enter on the command line. This is particularly useful when you + have library functions that you want to use from your command-line + programs (*note AWKPATH Variable::). + +'-E' FILE +'--exec' FILE + Similar to '-f', read 'awk' program text from FILE. There are two + differences from '-f': + + * This option terminates option processing; anything else on the + command line is passed on directly to the 'awk' program. + + * Command-line variable assignments of the form 'VAR=VALUE' are + disallowed. + + This option is particularly necessary for World Wide Web CGI + applications that pass arguments through the URL; using this option + prevents a malicious (or other) user from passing in options, + assignments, or 'awk' source code (via '-e') to the CGI + application.(1) This option should be used with '#!' scripts + (*note Executable Scripts::), like so: + + #! /usr/local/bin/gawk -E + + AWK PROGRAM HERE ... + +'-g' +'--gen-pot' + Analyze the source program and generate a GNU 'gettext' portable + object template file on standard output for all string constants + that have been marked for translation. *Note + Internationalization::, for information about this option. + +'-h' +'--help' + Print a "usage" message summarizing the short- and long-style + options that 'gawk' accepts and then exit. + +'-i' SOURCE-FILE +'--include' SOURCE-FILE + Read an 'awk' source library from SOURCE-FILE. This option is + completely equivalent to using the '@include' directive inside your + program. It is very similar to the '-f' option, but there are two + important differences. First, when '-i' is used, the program + source is not loaded if it has been previously loaded, whereas with + '-f', 'gawk' always loads the file. Second, because this option is + intended to be used with code libraries, 'gawk' does not recognize + such files as constituting main program input. Thus, after + processing an '-i' argument, 'gawk' still expects to find the main + source code via the '-f' option or on the command line. + +'-l' EXT +'--load' EXT + Load a dynamic extension named EXT. Extensions are stored as + system shared libraries. This option searches for the library + using the 'AWKLIBPATH' environment variable. The correct library + suffix for your platform will be supplied by default, so it need + not be specified in the extension name. The extension + initialization routine should be named 'dl_load()'. An alternative + is to use the '@load' keyword inside the program to load a shared + library. This advanced feature is described in detail in *note + Dynamic Extensions::. + +'-L'[VALUE] +'--lint'['='VALUE] + Warn about constructs that are dubious or nonportable to other + 'awk' implementations. No space is allowed between the '-L' and + VALUE, if VALUE is supplied. Some warnings are issued when 'gawk' + first reads your program. Others are issued at runtime, as your + program executes. With an optional argument of 'fatal', lint + warnings become fatal errors. This may be drastic, but its use + will certainly encourage the development of cleaner 'awk' programs. + With an optional argument of 'invalid', only warnings about things + that are actually invalid are issued. (This is not fully + implemented yet.) + + Some warnings are only printed once, even if the dubious constructs + they warn about occur multiple times in your 'awk' program. Thus, + when eliminating problems pointed out by '--lint', you should take + care to search for all occurrences of each inappropriate construct. + As 'awk' programs are usually short, doing so is not burdensome. + +'-M' +'--bignum' + Select arbitrary-precision arithmetic on numbers. This option has + no effect if 'gawk' is not compiled to use the GNU MPFR and MP + libraries (*note Arbitrary Precision Arithmetic::). + +'-n' +'--non-decimal-data' + Enable automatic interpretation of octal and hexadecimal values in + input data (*note Nondecimal Data::). + + CAUTION: This option can severely break old programs. Use + with care. Also note that this option may disappear in a + future version of 'gawk'. + +'-N' +'--use-lc-numeric' + Force the use of the locale's decimal point character when parsing + numeric input data (*note Locales::). + +'-o'[FILE] +'--pretty-print'['='FILE] + Enable pretty-printing of 'awk' programs. Implies '--no-optimize'. + By default, the output program is created in a file named + 'awkprof.out' (*note Profiling::). The optional FILE argument + allows you to specify a different file name for the output. No + space is allowed between the '-o' and FILE, if FILE is supplied. + + NOTE: In the past, this option would also execute your + program. This is no longer the case. + +'-O' +'--optimize' + Enable 'gawk''s default optimizations on the internal + representation of the program. At the moment, this includes simple + constant folding and tail recursion elimination in function calls. + + These optimizations are enabled by default. This option remains + primarily for backwards compatibility. However, it may be used to + cancel the effect of an earlier '-s' option (see later in this + list). + +'-p'[FILE] +'--profile'['='FILE] + Enable profiling of 'awk' programs (*note Profiling::). Implies + '--no-optimize'. By default, profiles are created in a file named + 'awkprof.out'. The optional FILE argument allows you to specify a + different file name for the profile file. No space is allowed + between the '-p' and FILE, if FILE is supplied. + + The profile contains execution counts for each statement in the + program in the left margin, and function call counts for each + function. + +'-P' +'--posix' + Operate in strict POSIX mode. This disables all 'gawk' extensions + (just like '--traditional') and disables all extensions not allowed + by POSIX. *Note Common Extensions:: for a summary of the extensions + in 'gawk' that are disabled by this option. Also, the following + additional restrictions apply: + + * Newlines are not allowed after '?' or ':' (*note Conditional + Exp::). + + * Specifying '-Ft' on the command line does not set the value of + 'FS' to be a single TAB character (*note Field Separators::). + + * The locale's decimal point character is used for parsing input + data (*note Locales::). + + If you supply both '--traditional' and '--posix' on the command + line, '--posix' takes precedence. 'gawk' issues a warning if both + options are supplied. + +'-r' +'--re-interval' + Allow interval expressions (*note Regexp Operators::) in regexps. + This is now 'gawk''s default behavior. Nevertheless, this option + remains (both for backward compatibility and for use in combination + with '--traditional'). + +'-s' +'--no-optimize' + Disable 'gawk''s default optimizations on the internal + representation of the program. + +'-S' +'--sandbox' + Disable the 'system()' function, input redirections with 'getline', + output redirections with 'print' and 'printf', and dynamic + extensions. This is particularly useful when you want to run 'awk' + scripts from questionable sources and need to make sure the scripts + can't access your system (other than the specified input data + file). + +'-t' +'--lint-old' + Warn about constructs that are not available in the original + version of 'awk' from Version 7 Unix (*note V7/SVR3.1::). + +'-V' +'--version' + Print version information for this particular copy of 'gawk'. This + allows you to determine if your copy of 'gawk' is up to date with + respect to whatever the Free Software Foundation is currently + distributing. It is also useful for bug reports (*note Bugs::). + + As long as program text has been supplied, any other options are +flagged as invalid with a warning message but are otherwise ignored. + + In compatibility mode, as a special case, if the value of FS supplied +to the '-F' option is 't', then 'FS' is set to the TAB character +('"\t"'). This is true only for '--traditional' and not for '--posix' +(*note Field Separators::). + + The '-f' option may be used more than once on the command line. If +it is, 'awk' reads its program source from all of the named files, as if +they had been concatenated together into one big file. This is useful +for creating libraries of 'awk' functions. These functions can be +written once and then retrieved from a standard place, instead of having +to be included in each individual program. The '-i' option is similar +in this regard. (As mentioned in *note Definition Syntax::, function +names must be unique.) + + With standard 'awk', library functions can still be used, even if the +program is entered at the keyboard, by specifying '-f /dev/tty'. After +typing your program, type 'Ctrl-d' (the end-of-file character) to +terminate it. (You may also use '-f -' to read program source from the +standard input, but then you will not be able to also use the standard +input as a source of data.) + + Because it is clumsy using the standard 'awk' mechanisms to mix +source file and command-line 'awk' programs, 'gawk' provides the '-e' +option. This does not require you to preempt the standard input for +your source code; it allows you to easily mix command-line and library +source code (*note AWKPATH Variable::). As with '-f', the '-e' and '-i' +options may also be used multiple times on the command line. + + If no '-f' or '-e' option is specified, then 'gawk' uses the first +nonoption command-line argument as the text of the program source code. + + If the environment variable 'POSIXLY_CORRECT' exists, then 'gawk' +behaves in strict POSIX mode, exactly as if you had supplied '--posix'. +Many GNU programs look for this environment variable to suppress +extensions that conflict with POSIX, but 'gawk' behaves differently: it +suppresses all extensions, even those that do not conflict with POSIX, +and behaves in strict POSIX mode. If '--lint' is supplied on the +command line and 'gawk' turns on POSIX mode because of +'POSIXLY_CORRECT', then it issues a warning message indicating that +POSIX mode is in effect. You would typically set this variable in your +shell's startup file. For a Bourne-compatible shell (such as Bash), you +would add these lines to the '.profile' file in your home directory: + + POSIXLY_CORRECT=true + export POSIXLY_CORRECT + + For a C shell-compatible shell,(2) you would add this line to the +'.login' file in your home directory: + + setenv POSIXLY_CORRECT true + + Having 'POSIXLY_CORRECT' set is not recommended for daily use, but it +is good for testing the portability of your programs to other +environments. + + ---------- Footnotes ---------- + + (1) For more detail, please see Section 4.4 of RFC 3875 +(http://www.ietf.org/rfc/rfc3875). Also see the explanatory note sent +to the 'gawk' bug mailing list +(http://lists.gnu.org/archive/html/bug-gawk/2014-11/msg00022.html). + + (2) Not recommended. + + +File: gawk.info, Node: Other Arguments, Next: Naming Standard Input, Prev: Options, Up: Invoking Gawk + +2.3 Other Command-Line Arguments +================================ + +Any additional arguments on the command line are normally treated as +input files to be processed in the order specified. However, an +argument that has the form 'VAR=VALUE', assigns the value VALUE to the +variable VAR--it does not specify a file at all. (See *note Assignment +Options::.) In the following example, COUNT=1 is a variable assignment, +not a file name: + + awk -f program.awk file1 count=1 file2 + + All the command-line arguments are made available to your 'awk' +program in the 'ARGV' array (*note Built-in Variables::). Command-line +options and the program text (if present) are omitted from 'ARGV'. All +other arguments, including variable assignments, are included. As each +element of 'ARGV' is processed, 'gawk' sets 'ARGIND' to the index in +'ARGV' of the current element. + + Changing 'ARGC' and 'ARGV' in your 'awk' program lets you control how +'awk' processes the input files; this is described in more detail in +*note ARGC and ARGV::. + + The distinction between file name arguments and variable-assignment +arguments is made when 'awk' is about to open the next input file. At +that point in execution, it checks the file name to see whether it is +really a variable assignment; if so, 'awk' sets the variable instead of +reading a file. + + Therefore, the variables actually receive the given values after all +previously specified files have been read. In particular, the values of +variables assigned in this fashion are _not_ available inside a 'BEGIN' +rule (*note BEGIN/END::), because such rules are run before 'awk' begins +scanning the argument list. + + The variable values given on the command line are processed for +escape sequences (*note Escape Sequences::). (d.c.) + + In some very early implementations of 'awk', when a variable +assignment occurred before any file names, the assignment would happen +_before_ the 'BEGIN' rule was executed. 'awk''s behavior was thus +inconsistent; some command-line assignments were available inside the +'BEGIN' rule, while others were not. Unfortunately, some applications +came to depend upon this "feature." When 'awk' was changed to be more +consistent, the '-v' option was added to accommodate applications that +depended upon the old behavior. + + The variable assignment feature is most useful for assigning to +variables such as 'RS', 'OFS', and 'ORS', which control input and output +formats, before scanning the data files. It is also useful for +controlling state if multiple passes are needed over a data file. For +example: + + awk 'pass == 1 { PASS 1 STUFF } + pass == 2 { PASS 2 STUFF }' pass=1 mydata pass=2 mydata + + Given the variable assignment feature, the '-F' option for setting +the value of 'FS' is not strictly necessary. It remains for historical +compatibility. + + +File: gawk.info, Node: Naming Standard Input, Next: Environment Variables, Prev: Other Arguments, Up: Invoking Gawk + +2.4 Naming Standard Input +========================= + +Often, you may wish to read standard input together with other files. +For example, you may wish to read one file, read standard input coming +from a pipe, and then read another file. + + The way to name the standard input, with all versions of 'awk', is to +use a single, standalone minus sign or dash, '-'. For example: + + SOME_COMMAND | awk -f myprog.awk file1 - file2 + +Here, 'awk' first reads 'file1', then it reads the output of +SOME_COMMAND, and finally it reads 'file2'. + + You may also use '"-"' to name standard input when reading files with +'getline' (*note Getline/File::). + + In addition, 'gawk' allows you to specify the special file name +'/dev/stdin', both on the command line and with 'getline'. Some other +versions of 'awk' also support this, but it is not standard. (Some +operating systems provide a '/dev/stdin' file in the filesystem; +however, 'gawk' always processes this file name itself.) + + +File: gawk.info, Node: Environment Variables, Next: Exit Status, Prev: Naming Standard Input, Up: Invoking Gawk + +2.5 The Environment Variables 'gawk' Uses +========================================= + +A number of environment variables influence how 'gawk' behaves. + +* Menu: + +* AWKPATH Variable:: Searching directories for 'awk' + programs. +* AWKLIBPATH Variable:: Searching directories for 'awk' shared + libraries. +* Other Environment Variables:: The environment variables. + + +File: gawk.info, Node: AWKPATH Variable, Next: AWKLIBPATH Variable, Up: Environment Variables + +2.5.1 The 'AWKPATH' Environment Variable +---------------------------------------- + +The previous minor node described how 'awk' program files can be named +on the command line with the '-f' option. In most 'awk' +implementations, you must supply a precise pathname for each program +file, unless the file is in the current directory. But with 'gawk', if +the file name supplied to the '-f' or '-i' options does not contain a +directory separator '/', then 'gawk' searches a list of directories +(called the "search path") one by one, looking for a file with the +specified name. + + The search path is a string consisting of directory names separated +by colons.(1) 'gawk' gets its search path from the 'AWKPATH' +environment variable. If that variable does not exist, or if it has an +empty value, 'gawk' uses a default path (described shortly). + + The search path feature is particularly helpful for building +libraries of useful 'awk' functions. The library files can be placed in +a standard directory in the default path and then specified on the +command line with a short file name. Otherwise, you would have to type +the full file name for each file. + + By using the '-i' or '-f' options, your command-line 'awk' programs +can use facilities in 'awk' library files (*note Library Functions::). +Path searching is not done if 'gawk' is in compatibility mode. This is +true for both '--traditional' and '--posix'. *Note Options::. + + If the source code file is not found after the initial search, the +path is searched again after adding the suffix '.awk' to the file name. + + 'gawk''s path search mechanism is similar to the shell's. (See 'The +Bourne-Again SHell manual' (http://www.gnu.org/software/bash/manual/).) +It treats a null entry in the path as indicating the current directory. +(A null entry is indicated by starting or ending the path with a colon +or by placing two colons next to each other ['::'].) + + NOTE: To include the current directory in the path, either place + '.' as an entry in the path or write a null entry in the path. + + Different past versions of 'gawk' would also look explicitly in the + current directory, either before or after the path search. As of + version 4.1.2, this no longer happens; if you wish to look in the + current directory, you must include '.' either as a separate entry + or as a null entry in the search path. + + The default value for 'AWKPATH' is '.:/usr/local/share/awk'.(2) +Since '.' is included at the beginning, 'gawk' searches first in the +current directory and then in '/usr/local/share/awk'. In practice, this +means that you will rarely need to change the value of 'AWKPATH'. + + *Note Shell Startup Files::, for information on functions that help +to manipulate the 'AWKPATH' variable. + + 'gawk' places the value of the search path that it used into +'ENVIRON["AWKPATH"]'. This provides access to the actual search path +value from within an 'awk' program. + + Although you can change 'ENVIRON["AWKPATH"]' within your 'awk' +program, this has no effect on the running program's behavior. This +makes sense: the 'AWKPATH' environment variable is used to find the +program source files. Once your program is running, all the files have +been found, and 'gawk' no longer needs to use 'AWKPATH'. + + ---------- Footnotes ---------- + + (1) Semicolons on MS-Windows. + + (2) Your version of 'gawk' may use a different directory; it will +depend upon how 'gawk' was built and installed. The actual directory is +the value of '$(datadir)' generated when 'gawk' was configured. You +probably don't need to worry about this, though. + + +File: gawk.info, Node: AWKLIBPATH Variable, Next: Other Environment Variables, Prev: AWKPATH Variable, Up: Environment Variables + +2.5.2 The 'AWKLIBPATH' Environment Variable +------------------------------------------- + +The 'AWKLIBPATH' environment variable is similar to the 'AWKPATH' +variable, but it is used to search for loadable extensions (stored as +system shared libraries) specified with the '-l' option rather than for +source files. If the extension is not found, the path is searched again +after adding the appropriate shared library suffix for the platform. +For example, on GNU/Linux systems, the suffix '.so' is used. The search +path specified is also used for extensions loaded via the '@load' +keyword (*note Loading Shared Libraries::). + + If 'AWKLIBPATH' does not exist in the environment, or if it has an +empty value, 'gawk' uses a default path; this is typically +'/usr/local/lib/gawk', although it can vary depending upon how 'gawk' +was built. + + *Note Shell Startup Files::, for information on functions that help +to manipulate the 'AWKLIBPATH' variable. + + 'gawk' places the value of the search path that it used into +'ENVIRON["AWKLIBPATH"]'. This provides access to the actual search path +value from within an 'awk' program. + + +File: gawk.info, Node: Other Environment Variables, Prev: AWKLIBPATH Variable, Up: Environment Variables + +2.5.3 Other Environment Variables +--------------------------------- + +A number of other environment variables affect 'gawk''s behavior, but +they are more specialized. Those in the following list are meant to be +used by regular users: + +'GAWK_MSEC_SLEEP' + Specifies the interval between connection retries, in milliseconds. + On systems that do not support the 'usleep()' system call, the + value is rounded up to an integral number of seconds. + +'GAWK_READ_TIMEOUT' + Specifies the time, in milliseconds, for 'gawk' to wait for input + before returning with an error. *Note Read Timeout::. + +'GAWK_SOCK_RETRIES' + Controls the number of times 'gawk' attempts to retry a two-way + TCP/IP (socket) connection before giving up. *Note TCP/IP + Networking::. Note that when nonfatal I/O is enabled (*note + Nonfatal::), 'gawk' only tries to open a TCP/IP socket once. + +'POSIXLY_CORRECT' + Causes 'gawk' to switch to POSIX-compatibility mode, disabling all + traditional and GNU extensions. *Note Options::. + + The environment variables in the following list are meant for use by +the 'gawk' developers for testing and tuning. They are subject to +change. The variables are: + +'AWKBUFSIZE' + This variable only affects 'gawk' on POSIX-compliant systems. With + a value of 'exact', 'gawk' uses the size of each input file as the + size of the memory buffer to allocate for I/O. Otherwise, the value + should be a number, and 'gawk' uses that number as the size of the + buffer to allocate. (When this variable is not set, 'gawk' uses + the smaller of the file's size and the "default" blocksize, which + is usually the filesystem's I/O blocksize.) + +'AWK_HASH' + If this variable exists with a value of 'gst', 'gawk' switches to + using the hash function from GNU Smalltalk for managing arrays. + This function may be marginally faster than the standard function. + +'AWKREADFUNC' + If this variable exists, 'gawk' switches to reading source files + one line at a time, instead of reading in blocks. This exists for + debugging problems on filesystems on non-POSIX operating systems + where I/O is performed in records, not in blocks. + +'GAWK_MSG_SRC' + If this variable exists, 'gawk' includes the file name and line + number within the 'gawk' source code from which warning and/or + fatal messages are generated. Its purpose is to help isolate the + source of a message, as there are multiple places that produce the + same warning or error message. + +'GAWK_LOCALE_DIR' + Specifies the location of compiled message object files for 'gawk' + itself. This is passed to the 'bindtextdomain()' function when + 'gawk' starts up. + +'GAWK_NO_DFA' + If this variable exists, 'gawk' does not use the DFA regexp matcher + for "does it match" kinds of tests. This can cause 'gawk' to be + slower. Its purpose is to help isolate differences between the two + regexp matchers that 'gawk' uses internally. (There aren't + supposed to be differences, but occasionally theory and practice + don't coordinate with each other.) + +'GAWK_STACKSIZE' + This specifies the amount by which 'gawk' should grow its internal + evaluation stack, when needed. + +'INT_CHAIN_MAX' + This specifies intended maximum number of items 'gawk' will + maintain on a hash chain for managing arrays indexed by integers. + +'STR_CHAIN_MAX' + This specifies intended maximum number of items 'gawk' will + maintain on a hash chain for managing arrays indexed by strings. + +'TIDYMEM' + If this variable exists, 'gawk' uses the 'mtrace()' library calls + from the GNU C library to help track down possible memory leaks. + + +File: gawk.info, Node: Exit Status, Next: Include Files, Prev: Environment Variables, Up: Invoking Gawk + +2.6 'gawk''s Exit Status +======================== + +If the 'exit' statement is used with a value (*note Exit Statement::), +then 'gawk' exits with the numeric value given to it. + + Otherwise, if there were no problems during execution, 'gawk' exits +with the value of the C constant 'EXIT_SUCCESS'. This is usually zero. + + If an error occurs, 'gawk' exits with the value of the C constant +'EXIT_FAILURE'. This is usually one. + + If 'gawk' exits because of a fatal error, the exit status is two. On +non-POSIX systems, this value may be mapped to 'EXIT_FAILURE'. + + +File: gawk.info, Node: Include Files, Next: Loading Shared Libraries, Prev: Exit Status, Up: Invoking Gawk + +2.7 Including Other Files into Your Program +=========================================== + +This minor node describes a feature that is specific to 'gawk'. + + The '@include' keyword can be used to read external 'awk' source +files. This gives you the ability to split large 'awk' source files +into smaller, more manageable pieces, and also lets you reuse common +'awk' code from various 'awk' scripts. In other words, you can group +together 'awk' functions used to carry out specific tasks into external +files. These files can be used just like function libraries, using the +'@include' keyword in conjunction with the 'AWKPATH' environment +variable. Note that source files may also be included using the '-i' +option. + + Let's see an example. We'll start with two (trivial) 'awk' scripts, +namely 'test1' and 'test2'. Here is the 'test1' script: + + BEGIN { + print "This is script test1." + } + +and here is 'test2': + + @include "test1" + BEGIN { + print "This is script test2." + } + + Running 'gawk' with 'test2' produces the following result: + + $ gawk -f test2 + -| This is script test1. + -| This is script test2. + + 'gawk' runs the 'test2' script, which includes 'test1' using the +'@include' keyword. So, to include external 'awk' source files, you +just use '@include' followed by the name of the file to be included, +enclosed in double quotes. + + NOTE: Keep in mind that this is a language construct and the file + name cannot be a string variable, but rather just a literal string + constant in double quotes. + + The files to be included may be nested; e.g., given a third script, +namely 'test3': + + @include "test2" + BEGIN { + print "This is script test3." + } + +Running 'gawk' with the 'test3' script produces the following results: + + $ gawk -f test3 + -| This is script test1. + -| This is script test2. + -| This is script test3. + + The file name can, of course, be a pathname. For example: + + @include "../io_funcs" + +and: + + @include "/usr/awklib/network" + +are both valid. The 'AWKPATH' environment variable can be of great +value when using '@include'. The same rules for the use of the +'AWKPATH' variable in command-line file searches (*note AWKPATH +Variable::) apply to '@include' also. + + This is very helpful in constructing 'gawk' function libraries. If +you have a large script with useful, general-purpose 'awk' functions, +you can break it down into library files and put those files in a +special directory. You can then include those "libraries," either by +using the full pathnames of the files, or by setting the 'AWKPATH' +environment variable accordingly and then using '@include' with just the +file part of the full pathname. Of course, you can keep library files +in more than one directory; the more complex the working environment is, +the more directories you may need to organize the files to be included. + + Given the ability to specify multiple '-f' options, the '@include' +mechanism is not strictly necessary. However, the '@include' keyword +can help you in constructing self-contained 'gawk' programs, thus +reducing the need for writing complex and tedious command lines. In +particular, '@include' is very useful for writing CGI scripts to be run +from web pages. + + As mentioned in *note AWKPATH Variable::, the current directory is +always searched first for source files, before searching in 'AWKPATH'; +this also applies to files named with '@include'. + + +File: gawk.info, Node: Loading Shared Libraries, Next: Obsolete, Prev: Include Files, Up: Invoking Gawk + +2.8 Loading Dynamic Extensions into Your Program +================================================ + +This minor node describes a feature that is specific to 'gawk'. + + The '@load' keyword can be used to read external 'awk' extensions +(stored as system shared libraries). This allows you to link in +compiled code that may offer superior performance and/or give you access +to extended capabilities not supported by the 'awk' language. The +'AWKLIBPATH' variable is used to search for the extension. Using +'@load' is completely equivalent to using the '-l' command-line option. + + If the extension is not initially found in 'AWKLIBPATH', another +search is conducted after appending the platform's default shared +library suffix to the file name. For example, on GNU/Linux systems, the +suffix '.so' is used: + + $ gawk '@load "ordchr"; BEGIN {print chr(65)}' + -| A + +This is equivalent to the following example: + + $ gawk -lordchr 'BEGIN {print chr(65)}' + -| A + +For command-line usage, the '-l' option is more convenient, but '@load' +is useful for embedding inside an 'awk' source file that requires access +to an extension. + + *note Dynamic Extensions::, describes how to write extensions (in C +or C++) that can be loaded with either '@load' or the '-l' option. It +also describes the 'ordchr' extension. + + +File: gawk.info, Node: Obsolete, Next: Undocumented, Prev: Loading Shared Libraries, Up: Invoking Gawk + +2.9 Obsolete Options and/or Features +==================================== + +This minor node describes features and/or command-line options from +previous releases of 'gawk' that either are not available in the current +version or are still supported but deprecated (meaning that they will +_not_ be in the next release). + + The process-related special files '/dev/pid', '/dev/ppid', +'/dev/pgrpid', and '/dev/user' were deprecated in 'gawk' 3.1, but still +worked. As of version 4.0, they are no longer interpreted specially by +'gawk'. (Use 'PROCINFO' instead; see *note Auto-set::.) + + +File: gawk.info, Node: Undocumented, Next: Invoking Summary, Prev: Obsolete, Up: Invoking Gawk + +2.10 Undocumented Options and Features +====================================== + + Use the Source, Luke! + -- _Obi-Wan_ + + This minor node intentionally left blank. + + +File: gawk.info, Node: Invoking Summary, Prev: Undocumented, Up: Invoking Gawk + +2.11 Summary +============ + + * Use either 'awk 'PROGRAM' FILES' or 'awk -f PROGRAM-FILE FILES' to + run 'awk'. + + * The three standard options for all versions of 'awk' are '-f', + '-F', and '-v'. 'gawk' supplies these and many others, as well as + corresponding GNU-style long options. + + * Nonoption command-line arguments are usually treated as file names, + unless they have the form 'VAR=VALUE', in which case they are taken + as variable assignments to be performed at that point in processing + the input. + + * All nonoption command-line arguments, excluding the program text, + are placed in the 'ARGV' array. Adjusting 'ARGC' and 'ARGV' + affects how 'awk' processes input. + + * You can use a single minus sign ('-') to refer to standard input on + the command line. 'gawk' also lets you use the special file name + '/dev/stdin'. + + * 'gawk' pays attention to a number of environment variables. + 'AWKPATH', 'AWKLIBPATH', and 'POSIXLY_CORRECT' are the most + important ones. + + * 'gawk''s exit status conveys information to the program that + invoked it. Use the 'exit' statement from within an 'awk' program + to set the exit status. + + * 'gawk' allows you to include other 'awk' source files into your + program using the '@include' statement and/or the '-i' and '-f' + command-line options. + + * 'gawk' allows you to load additional functions written in C or C++ + using the '@load' statement and/or the '-l' option. (This advanced + feature is described later, in *note Dynamic Extensions::.) + + +File: gawk.info, Node: Regexp, Next: Reading Files, Prev: Invoking Gawk, Up: Top + +3 Regular Expressions +********************* + +A "regular expression", or "regexp", is a way of describing a set of +strings. Because regular expressions are such a fundamental part of +'awk' programming, their format and use deserve a separate major node. + + A regular expression enclosed in slashes ('/') is an 'awk' pattern +that matches every input record whose text belongs to that set. The +simplest regular expression is a sequence of letters, numbers, or both. +Such a regexp matches any string that contains that sequence. Thus, the +regexp 'foo' matches any string containing 'foo'. Thus, the pattern +'/foo/' matches any input record containing the three adjacent +characters 'foo' _anywhere_ in the record. Other kinds of regexps let +you specify more complicated classes of strings. + +* Menu: + +* Regexp Usage:: How to Use Regular Expressions. +* Escape Sequences:: How to write nonprinting characters. +* Regexp Operators:: Regular Expression Operators. +* Bracket Expressions:: What can go between '[...]'. +* Leftmost Longest:: How much text matches. +* Computed Regexps:: Using Dynamic Regexps. +* GNU Regexp Operators:: Operators specific to GNU software. +* Case-sensitivity:: How to do case-insensitive matching. +* Strong Regexp Constants:: Strongly typed regexp constants. +* Regexp Summary:: Regular expressions summary. + + +File: gawk.info, Node: Regexp Usage, Next: Escape Sequences, Up: Regexp + +3.1 How to Use Regular Expressions +================================== + +A regular expression can be used as a pattern by enclosing it in +slashes. Then the regular expression is tested against the entire text +of each record. (Normally, it only needs to match some part of the text +in order to succeed.) For example, the following prints the second +field of each record where the string 'li' appears anywhere in the +record: + + $ awk '/li/ { print $2 }' mail-list + -| 555-5553 + -| 555-0542 + -| 555-6699 + -| 555-3430 + + Regular expressions can also be used in matching expressions. These +expressions allow you to specify the string to match against; it need +not be the entire current input record. The two operators '~' and '!~' +perform regular expression comparisons. Expressions using these +operators can be used as patterns, or in 'if', 'while', 'for', and 'do' +statements. (*Note Statements::.) For example, the following is true +if the expression EXP (taken as a string) matches REGEXP: + + EXP ~ /REGEXP/ + +This example matches, or selects, all input records with the uppercase +letter 'J' somewhere in the first field: + + $ awk '$1 ~ /J/' inventory-shipped + -| Jan 13 25 15 115 + -| Jun 31 42 75 492 + -| Jul 24 34 67 436 + -| Jan 21 36 64 620 + + So does this: + + awk '{ if ($1 ~ /J/) print }' inventory-shipped + + This next example is true if the expression EXP (taken as a character +string) does _not_ match REGEXP: + + EXP !~ /REGEXP/ + + The following example matches, or selects, all input records whose +first field _does not_ contain the uppercase letter 'J': + + $ awk '$1 !~ /J/' inventory-shipped + -| Feb 15 32 24 226 + -| Mar 15 24 34 228 + -| Apr 31 52 63 420 + -| May 16 34 29 208 + ... + + When a regexp is enclosed in slashes, such as '/foo/', we call it a +"regexp constant", much like '5.27' is a numeric constant and '"foo"' is +a string constant. + + +File: gawk.info, Node: Escape Sequences, Next: Regexp Operators, Prev: Regexp Usage, Up: Regexp + +3.2 Escape Sequences +==================== + +Some characters cannot be included literally in string constants +('"foo"') or regexp constants ('/foo/'). Instead, they should be +represented with "escape sequences", which are character sequences +beginning with a backslash ('\'). One use of an escape sequence is to +include a double-quote character in a string constant. Because a plain +double quote ends the string, you must use '\"' to represent an actual +double-quote character as a part of the string. For example: + + $ awk 'BEGIN { print "He said \"hi!\" to her." }' + -| He said "hi!" to her. + + The backslash character itself is another character that cannot be +included normally; you must write '\\' to put one backslash in the +string or regexp. Thus, the string whose contents are the two +characters '"' and '\' must be written '"\"\\"'. + + Other escape sequences represent unprintable characters such as TAB +or newline. There is nothing to stop you from entering most unprintable +characters directly in a string constant or regexp constant, but they +may look ugly. + + The following list presents all the escape sequences used in 'awk' +and what they represent. Unless noted otherwise, all these escape +sequences apply to both string constants and regexp constants: + +'\\' + A literal backslash, '\'. + +'\a' + The "alert" character, 'Ctrl-g', ASCII code 7 (BEL). (This often + makes some sort of audible noise.) + +'\b' + Backspace, 'Ctrl-h', ASCII code 8 (BS). + +'\f' + Formfeed, 'Ctrl-l', ASCII code 12 (FF). + +'\n' + Newline, 'Ctrl-j', ASCII code 10 (LF). + +'\r' + Carriage return, 'Ctrl-m', ASCII code 13 (CR). + +'\t' + Horizontal TAB, 'Ctrl-i', ASCII code 9 (HT). + +'\v' + Vertical TAB, 'Ctrl-k', ASCII code 11 (VT). + +'\NNN' + The octal value NNN, where NNN stands for 1 to 3 digits between '0' + and '7'. For example, the code for the ASCII ESC (escape) + character is '\033'. + +'\xHH...' + The hexadecimal value HH, where HH stands for a sequence of + hexadecimal digits ('0'-'9', and either 'A'-'F' or 'a'-'f'). A + maximum of two digts are allowed after the '\x'. Any further + hexadecimal digits are treated as simple letters or numbers. + (c.e.) (The '\x' escape sequence is not allowed in POSIX awk.) + + CAUTION: In ISO C, the escape sequence continues until the + first nonhexadecimal digit is seen. For many years, 'gawk' + would continue incorporating hexadecimal digits into the value + until a non-hexadecimal digit or the end of the string was + encountered. However, using more than two hexadecimal digits + produced undefined results. As of version 4.2, only two + digits are processed. + +'\/' + A literal slash (necessary for regexp constants only). This + sequence is used when you want to write a regexp constant that + contains a slash (such as '/.*:\/home\/[[:alnum:]]+:.*/'; the + '[[:alnum:]]' notation is discussed in *note Bracket + Expressions::). Because the regexp is delimited by slashes, you + need to escape any slash that is part of the pattern, in order to + tell 'awk' to keep processing the rest of the regexp. + +'\"' + A literal double quote (necessary for string constants only). This + sequence is used when you want to write a string constant that + contains a double quote (such as '"He said \"hi!\" to her."'). + Because the string is delimited by double quotes, you need to + escape any quote that is part of the string, in order to tell 'awk' + to keep processing the rest of the string. + + In 'gawk', a number of additional two-character sequences that begin +with a backslash have special meaning in regexps. *Note GNU Regexp +Operators::. + + In a regexp, a backslash before any character that is not in the +previous list and not listed in *note GNU Regexp Operators:: means that +the next character should be taken literally, even if it would normally +be a regexp operator. For example, '/a\+b/' matches the three +characters 'a+b'. + + For complete portability, do not use a backslash before any character +not shown in the previous list or that is not an operator. + + Backslash Before Regular Characters + + If you place a backslash in a string constant before something that +is not one of the characters previously listed, POSIX 'awk' purposely +leaves what happens as undefined. There are two choices: + +Strip the backslash out + This is what BWK 'awk' and 'gawk' both do. For example, '"a\qc"' + is the same as '"aqc"'. (Because this is such an easy bug both to + introduce and to miss, 'gawk' warns you about it.) Consider 'FS = + "[ \t]+\|[ \t]+"' to use vertical bars surrounded by whitespace as + the field separator. There should be two backslashes in the + string: 'FS = "[ \t]+\\|[ \t]+"'.) + +Leave the backslash alone + Some other 'awk' implementations do this. In such implementations, + typing '"a\qc"' is the same as typing '"a\\qc"'. + + To summarize: + + * The escape sequences in the preceding list are always processed + first, for both string constants and regexp constants. This + happens very early, as soon as 'awk' reads your program. + + * 'gawk' processes both regexp constants and dynamic regexps (*note + Computed Regexps::), for the special operators listed in *note GNU + Regexp Operators::. + + * A backslash before any other character means to treat that + character literally. + + Escape Sequences for Metacharacters + + Suppose you use an octal or hexadecimal escape to represent a regexp +metacharacter. (See *note Regexp Operators::.) Does 'awk' treat the +character as a literal character or as a regexp operator? + + Historically, such characters were taken literally. (d.c.) However, +the POSIX standard indicates that they should be treated as real +metacharacters, which is what 'gawk' does. In compatibility mode (*note +Options::), 'gawk' treats the characters represented by octal and +hexadecimal escape sequences literally when used in regexp constants. +Thus, '/a\52b/' is equivalent to '/a\*b/'. + + +File: gawk.info, Node: Regexp Operators, Next: Bracket Expressions, Prev: Escape Sequences, Up: Regexp + +3.3 Regular Expression Operators +================================ + +You can combine regular expressions with special characters, called +"regular expression operators" or "metacharacters", to increase the +power and versatility of regular expressions. + + The escape sequences described in *note Escape Sequences:: are valid +inside a regexp. They are introduced by a '\' and are recognized and +converted into corresponding real characters as the very first step in +processing regexps. + + Here is a list of metacharacters. All characters that are not escape +sequences and that are not listed here stand for themselves: + +'\' + This suppresses the special meaning of a character when matching. + For example, '\$' matches the character '$'. + +'^' + This matches the beginning of a string. '^@chapter' matches + '@chapter' at the beginning of a string, for example, and can be + used to identify chapter beginnings in Texinfo source files. The + '^' is known as an "anchor", because it anchors the pattern to + match only at the beginning of the string. + + It is important to realize that '^' does not match the beginning of + a line (the point right after a '\n' newline character) embedded in + a string. The condition is not true in the following example: + + if ("line1\nLINE 2" ~ /^L/) ... + +'$' + This is similar to '^', but it matches only at the end of a string. + For example, 'p$' matches a record that ends with a 'p'. The '$' + is an anchor and does not match the end of a line (the point right + before a '\n' newline character) embedded in a string. The + condition in the following example is not true: + + if ("line1\nLINE 2" ~ /1$/) ... + +'.' (period) + This matches any single character, _including_ the newline + character. For example, '.P' matches any single character followed + by a 'P' in a string. Using concatenation, we can make a regular + expression such as 'U.A', which matches any three-character + sequence that begins with 'U' and ends with 'A'. + + In strict POSIX mode (*note Options::), '.' does not match the NUL + character, which is a character with all bits equal to zero. + Otherwise, NUL is just another character. Other versions of 'awk' + may not be able to match the NUL character. + +'['...']' + This is called a "bracket expression".(1) It matches any _one_ of + the characters that are enclosed in the square brackets. For + example, '[MVX]' matches any one of the characters 'M', 'V', or 'X' + in a string. A full discussion of what can be inside the square + brackets of a bracket expression is given in *note Bracket + Expressions::. + +'[^'...']' + This is a "complemented bracket expression". The first character + after the '[' _must_ be a '^'. It matches any characters _except_ + those in the square brackets. For example, '[^awk]' matches any + character that is not an 'a', 'w', or 'k'. + +'|' + This is the "alternation operator" and it is used to specify + alternatives. The '|' has the lowest precedence of all the regular + expression operators. For example, '^P|[aeiouy]' matches any + string that matches either '^P' or '[aeiouy]'. This means it + matches any string that starts with 'P' or contains (anywhere + within it) a lowercase English vowel. + + The alternation applies to the largest possible regexps on either + side. + +'('...')' + Parentheses are used for grouping in regular expressions, as in + arithmetic. They can be used to concatenate regular expressions + containing the alternation operator, '|'. For example, + '@(samp|code)\{[^}]+\}' matches both '@code{foo}' and '@samp{bar}'. + (These are Texinfo formatting control sequences. The '+' is + explained further on in this list.) + +'*' + This symbol means that the preceding regular expression should be + repeated as many times as necessary to find a match. For example, + 'ph*' applies the '*' symbol to the preceding 'h' and looks for + matches of one 'p' followed by any number of 'h's. This also + matches just 'p' if no 'h's are present. + + There are two subtle points to understand about how '*' works. + First, the '*' applies only to the single preceding regular + expression component (e.g., in 'ph*', it applies just to the 'h'). + To cause '*' to apply to a larger subexpression, use parentheses: + '(ph)*' matches 'ph', 'phph', 'phphph', and so on. + + Second, '*' finds as many repetitions as possible. If the text to + be matched is 'phhhhhhhhhhhhhhooey', 'ph*' matches all of the 'h's. + +'+' + This symbol is similar to '*', except that the preceding expression + must be matched at least once. This means that 'wh+y' would match + 'why' and 'whhy', but not 'wy', whereas 'wh*y' would match all + three. + +'?' + This symbol is similar to '*', except that the preceding expression + can be matched either once or not at all. For example, 'fe?d' + matches 'fed' and 'fd', but nothing else. + +'{'N'}' +'{'N',}' +'{'N','M'}' + One or two numbers inside braces denote an "interval expression". + If there is one number in the braces, the preceding regexp is + repeated N times. If there are two numbers separated by a comma, + the preceding regexp is repeated N to M times. If there is one + number followed by a comma, then the preceding regexp is repeated + at least N times: + + 'wh{3}y' + Matches 'whhhy', but not 'why' or 'whhhhy'. + + 'wh{3,5}y' + Matches 'whhhy', 'whhhhy', or 'whhhhhy' only. + + 'wh{2,}y' + Matches 'whhy', 'whhhy', and so on. + + Interval expressions were not traditionally available in 'awk'. + They were added as part of the POSIX standard to make 'awk' and + 'egrep' consistent with each other. + + Initially, because old programs may use '{' and '}' in regexp + constants, 'gawk' did _not_ match interval expressions in regexps. + + However, beginning with version 4.0, 'gawk' does match interval + expressions by default. This is because compatibility with POSIX + has become more important to most 'gawk' users than compatibility + with old programs. + + For programs that use '{' and '}' in regexp constants, it is good + practice to always escape them with a backslash. Then the regexp + constants are valid and work the way you want them to, using any + version of 'awk'.(2) + + Finally, when '{' and '}' appear in regexp constants in a way that + cannot be interpreted as an interval expression (such as '/q{a}/'), + then they stand for themselves. + + In regular expressions, the '*', '+', and '?' operators, as well as +the braces '{' and '}', have the highest precedence, followed by +concatenation, and finally by '|'. As in arithmetic, parentheses can +change how operators are grouped. + + In POSIX 'awk' and 'gawk', the '*', '+', and '?' operators stand for +themselves when there is nothing in the regexp that precedes them. For +example, '/+/' matches a literal plus sign. However, many other +versions of 'awk' treat such a usage as a syntax error. + + If 'gawk' is in compatibility mode (*note Options::), interval +expressions are not available in regular expressions. + + ---------- Footnotes ---------- + + (1) In other literature, you may see a bracket expression referred to +as either a "character set", a "character class", or a "character list". + + (2) Use two backslashes if you're using a string constant with a +regexp operator or function. + + +File: gawk.info, Node: Bracket Expressions, Next: Leftmost Longest, Prev: Regexp Operators, Up: Regexp + +3.4 Using Bracket Expressions +============================= + +As mentioned earlier, a bracket expression matches any character among +those listed between the opening and closing square brackets. + + Within a bracket expression, a "range expression" consists of two +characters separated by a hyphen. It matches any single character that +sorts between the two characters, based upon the system's native +character set. For example, '[0-9]' is equivalent to '[0123456789]'. +(See *note Ranges and Locales:: for an explanation of how the POSIX +standard and 'gawk' have changed over time. This is mainly of +historical interest.) + + With the increasing popularity of the Unicode character standard +(http://www.unicode.org), there is an additional wrinkle to consider. +Octal and hexadecimal escape sequences inside bracket expressions are +taken to represent only single-byte characters (characters whose values +fit within the range 0-256). To match a range of characters where the +endpoints of the range are larger than 256, enter the multibyte +encodings of the characters directly. + + To include one of the characters '\', ']', '-', or '^' in a bracket +expression, put a '\' in front of it. For example: + + [d\]] + +matches either 'd' or ']'. Additionally, if you place ']' right after +the opening '[', the closing bracket is treated as one of the characters +to be matched. + + The treatment of '\' in bracket expressions is compatible with other +'awk' implementations and is also mandated by POSIX. The regular +expressions in 'awk' are a superset of the POSIX specification for +Extended Regular Expressions (EREs). POSIX EREs are based on the +regular expressions accepted by the traditional 'egrep' utility. + + "Character classes" are a feature introduced in the POSIX standard. +A character class is a special notation for describing lists of +characters that have a specific attribute, but the actual characters can +vary from country to country and/or from character set to character set. +For example, the notion of what is an alphabetic character differs +between the United States and France. + + A character class is only valid in a regexp _inside_ the brackets of +a bracket expression. Character classes consist of '[:', a keyword +denoting the class, and ':]'. *note Table 3.1: table-char-classes. +lists the character classes defined by the POSIX standard. + +Class Meaning +-------------------------------------------------------------------------- +'[:alnum:]' Alphanumeric characters +'[:alpha:]' Alphabetic characters +'[:blank:]' Space and TAB characters +'[:cntrl:]' Control characters +'[:digit:]' Numeric characters +'[:graph:]' Characters that are both printable and visible (a space is + printable but not visible, whereas an 'a' is both) +'[:lower:]' Lowercase alphabetic characters +'[:print:]' Printable characters (characters that are not control + characters) +'[:punct:]' Punctuation characters (characters that are not letters, + digits, control characters, or space characters) +'[:space:]' Space characters (such as space, TAB, and formfeed, to name + a few) +'[:upper:]' Uppercase alphabetic characters +'[:xdigit:]'Characters that are hexadecimal digits + +Table 3.1: POSIX character classes + + For example, before the POSIX standard, you had to write +'/[A-Za-z0-9]/' to match alphanumeric characters. If your character set +had other alphabetic characters in it, this would not match them. With +the POSIX character classes, you can write '/[[:alnum:]]/' to match the +alphabetic and numeric characters in your character set. + + Some utilities that match regular expressions provide a nonstandard +'[:ascii:]' character class; 'awk' does not. However, you can simulate +such a construct using '[\x00-\x7F]'. This matches all values +numerically between zero and 127, which is the defined range of the +ASCII character set. Use a complemented character list ('[^\x00-\x7F]') +to match any single-byte characters that are not in the ASCII range. + + Two additional special sequences can appear in bracket expressions. +These apply to non-ASCII character sets, which can have single symbols +(called "collating elements") that are represented with more than one +character. They can also have several characters that are equivalent +for "collating", or sorting, purposes. (For example, in French, a plain +"e" and a grave-accented "e`" are equivalent.) These sequences are: + +Collating symbols + Multicharacter collating elements enclosed between '[.' and '.]'. + For example, if 'ch' is a collating element, then '[[.ch.]]' is a + regexp that matches this collating element, whereas '[ch]' is a + regexp that matches either 'c' or 'h'. + +Equivalence classes + Locale-specific names for a list of characters that are equal. The + name is enclosed between '[=' and '=]'. For example, the name 'e' + might be used to represent all of "e," "e^," "e`," and "e'." In + this case, '[[=e=]]' is a regexp that matches any of 'e', 'e^', + 'e'', or 'e`'. + + These features are very valuable in non-English-speaking locales. + + CAUTION: The library functions that 'gawk' uses for regular + expression matching currently recognize only POSIX character + classes; they do not recognize collating symbols or equivalence + classes. + + Inside a bracket expression, an opening bracket ('[') that does not +start a character class, collating element or equivalence class is taken +literally. This is also true of '.' and '*'. + + +File: gawk.info, Node: Leftmost Longest, Next: Computed Regexps, Prev: Bracket Expressions, Up: Regexp + +3.5 How Much Text Matches? +========================== + +Consider the following: + + echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }' + + This example uses the 'sub()' function to make a change to the input +record. ('sub()' replaces the first instance of any text matched by the +first argument with the string provided as the second argument; *note +String Functions::.) Here, the regexp '/a+/' indicates "one or more 'a' +characters," and the replacement text is '<A>'. + + The input contains four 'a' characters. 'awk' (and POSIX) regular +expressions always match the leftmost, _longest_ sequence of input +characters that can match. Thus, all four 'a' characters are replaced +with '<A>' in this example: + + $ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }' + -| <A>bcd + + For simple match/no-match tests, this is not so important. But when +doing text matching and substitutions with the 'match()', 'sub()', +'gsub()', and 'gensub()' functions, it is very important. *Note String +Functions::, for more information on these functions. Understanding +this principle is also important for regexp-based record and field +splitting (*note Records::, and also *note Field Separators::). + + +File: gawk.info, Node: Computed Regexps, Next: GNU Regexp Operators, Prev: Leftmost Longest, Up: Regexp + +3.6 Using Dynamic Regexps +========================= + +The righthand side of a '~' or '!~' operator need not be a regexp +constant (i.e., a string of characters between slashes). It may be any +expression. The expression is evaluated and converted to a string if +necessary; the contents of the string are then used as the regexp. A +regexp computed in this way is called a "dynamic regexp" or a "computed +regexp": + + BEGIN { digits_regexp = "[[:digit:]]+" } + $0 ~ digits_regexp { print } + +This sets 'digits_regexp' to a regexp that describes one or more digits, +and tests whether the input record matches this regexp. + + NOTE: When using the '~' and '!~' operators, be aware that there is + a difference between a regexp constant enclosed in slashes and a + string constant enclosed in double quotes. If you are going to use + a string constant, you have to understand that the string is, in + essence, scanned _twice_: the first time when 'awk' reads your + program, and the second time when it goes to match the string on + the lefthand side of the operator with the pattern on the right. + This is true of any string-valued expression (such as + 'digits_regexp', shown in the previous example), not just string + constants. + + What difference does it make if the string is scanned twice? The +answer has to do with escape sequences, and particularly with +backslashes. To get a backslash into a regular expression inside a +string, you have to type two backslashes. + + For example, '/\*/' is a regexp constant for a literal '*'. Only one +backslash is needed. To do the same thing with a string, you have to +type '"\\*"'. The first backslash escapes the second one so that the +string actually contains the two characters '\' and '*'. + + Given that you can use both regexp and string constants to describe +regular expressions, which should you use? The answer is "regexp +constants," for several reasons: + + * String constants are more complicated to write and more difficult + to read. Using regexp constants makes your programs less + error-prone. Not understanding the difference between the two + kinds of constants is a common source of errors. + + * It is more efficient to use regexp constants. 'awk' can note that + you have supplied a regexp and store it internally in a form that + makes pattern matching more efficient. When using a string + constant, 'awk' must first convert the string into this internal + form and then perform the pattern matching. + + * Using regexp constants is better form; it shows clearly that you + intend a regexp match. + + Using '\n' in Bracket Expressions of Dynamic Regexps + + Some older versions of 'awk' do not allow the newline character to be +used inside a bracket expression for a dynamic regexp: + + $ awk '$0 ~ "[ \t\n]"' + error-> awk: newline in character class [ + error-> ]... + error-> source line number 1 + error-> context is + error-> $0 ~ "[ >>> \t\n]" <<< + + But a newline in a regexp constant works with no problem: + + $ awk '$0 ~ /[ \t\n]/' + here is a sample line + -| here is a sample line + Ctrl-d + + 'gawk' does not have this problem, and it isn't likely to occur often +in practice, but it's worth noting for future reference. + + +File: gawk.info, Node: GNU Regexp Operators, Next: Case-sensitivity, Prev: Computed Regexps, Up: Regexp + +3.7 'gawk'-Specific Regexp Operators +==================================== + +GNU software that deals with regular expressions provides a number of +additional regexp operators. These operators are described in this +minor node and are specific to 'gawk'; they are not available in other +'awk' implementations. Most of the additional operators deal with word +matching. For our purposes, a "word" is a sequence of one or more +letters, digits, or underscores ('_'): + +'\s' + Matches any whitespace character. Think of it as shorthand for + '[[:space:]]'. + +'\S' + Matches any character that is not whitespace. Think of it as + shorthand for '[^[:space:]]'. + +'\w' + Matches any word-constituent character--that is, it matches any + letter, digit, or underscore. Think of it as shorthand for + '[[:alnum:]_]'. + +'\W' + Matches any character that is not word-constituent. Think of it as + shorthand for '[^[:alnum:]_]'. + +'\<' + Matches the empty string at the beginning of a word. For example, + '/\<away/' matches 'away' but not 'stowaway'. + +'\>' + Matches the empty string at the end of a word. For example, + '/stow\>/' matches 'stow' but not 'stowaway'. + +'\y' + Matches the empty string at either the beginning or the end of a + word (i.e., the word boundar*y*). For example, '\yballs?\y' + matches either 'ball' or 'balls', as a separate word. + +'\B' + Matches the empty string that occurs between two word-constituent + characters. For example, '/\Brat\B/' matches 'crate', but it does + not match 'dirty rat'. '\B' is essentially the opposite of '\y'. + + There are two other operators that work on buffers. In Emacs, a +"buffer" is, naturally, an Emacs buffer. Other GNU programs, including +'gawk', consider the entire string to match as the buffer. The +operators are: + +'\`' + Matches the empty string at the beginning of a buffer (string) + +'\'' + Matches the empty string at the end of a buffer (string) + + Because '^' and '$' always work in terms of the beginning and end of +strings, these operators don't add any new capabilities for 'awk'. They +are provided for compatibility with other GNU software. + + In other GNU software, the word-boundary operator is '\b'. However, +that conflicts with the 'awk' language's definition of '\b' as +backspace, so 'gawk' uses a different letter. An alternative method +would have been to require two backslashes in the GNU operators, but +this was deemed too confusing. The current method of using '\y' for the +GNU '\b' appears to be the lesser of two evils. + + The various command-line options (*note Options::) control how 'gawk' +interprets characters in regexps: + +No options + In the default case, 'gawk' provides all the facilities of POSIX + regexps and the GNU regexp operators described in *note Regexp + Operators::. + +'--posix' + Match only POSIX regexps; the GNU operators are not special (e.g., + '\w' matches a literal 'w'). Interval expressions are allowed. + +'--traditional' + Match traditional Unix 'awk' regexps. The GNU operators are not + special, and interval expressions are not available. Because BWK + 'awk' supports them, the POSIX character classes ('[[:alnum:]]', + etc.) are available. Characters described by octal and + hexadecimal escape sequences are treated literally, even if they + represent regexp metacharacters. + +'--re-interval' + Allow interval expressions in regexps, if '--traditional' has been + provided. Otherwise, interval expressions are available by + default. + + +File: gawk.info, Node: Case-sensitivity, Next: Strong Regexp Constants, Prev: GNU Regexp Operators, Up: Regexp + +3.8 Case Sensitivity in Matching +================================ + +Case is normally significant in regular expressions, both when matching +ordinary characters (i.e., not metacharacters) and inside bracket +expressions. Thus, a 'w' in a regular expression matches only a +lowercase 'w' and not an uppercase 'W'. + + The simplest way to do a case-independent match is to use a bracket +expression--for example, '[Ww]'. However, this can be cumbersome if you +need to use it often, and it can make the regular expressions harder to +read. There are two alternatives that you might prefer. + + One way to perform a case-insensitive match at a particular point in +the program is to convert the data to a single case, using the +'tolower()' or 'toupper()' built-in string functions (which we haven't +discussed yet; *note String Functions::). For example: + + tolower($1) ~ /foo/ { ... } + +converts the first field to lowercase before matching against it. This +works in any POSIX-compliant 'awk'. + + Another method, specific to 'gawk', is to set the variable +'IGNORECASE' to a nonzero value (*note Built-in Variables::). When +'IGNORECASE' is not zero, _all_ regexp and string operations ignore +case. + + Changing the value of 'IGNORECASE' dynamically controls the case +sensitivity of the program as it runs. Case is significant by default +because 'IGNORECASE' (like most variables) is initialized to zero: + + x = "aB" + if (x ~ /ab/) ... # this test will fail + + IGNORECASE = 1 + if (x ~ /ab/) ... # now it will succeed + + In general, you cannot use 'IGNORECASE' to make certain rules case +insensitive and other rules case sensitive, as there is no +straightforward way to set 'IGNORECASE' just for the pattern of a +particular rule.(1) To do this, use either bracket expressions or +'tolower()'. However, one thing you can do with 'IGNORECASE' only is +dynamically turn case sensitivity on or off for all the rules at once. + + 'IGNORECASE' can be set on the command line or in a 'BEGIN' rule +(*note Other Arguments::; also *note Using BEGIN/END::). Setting +'IGNORECASE' from the command line is a way to make a program case +insensitive without having to edit it. + + In multibyte locales, the equivalences between upper- and lowercase +characters are tested based on the wide-character values of the locale's +character set. Otherwise, the characters are tested based on the +ISO-8859-1 (ISO Latin-1) character set. This character set is a +superset of the traditional 128 ASCII characters, which also provides a +number of characters suitable for use with European languages.(2) + + The value of 'IGNORECASE' has no effect if 'gawk' is in compatibility +mode (*note Options::). Case is always significant in compatibility +mode. + + ---------- Footnotes ---------- + + (1) Experienced C and C++ programmers will note that it is possible, +using something like 'IGNORECASE = 1 && /foObAr/ { ... }' and +'IGNORECASE = 0 || /foobar/ { ... }'. However, this is somewhat obscure +and we don't recommend it. + + (2) If you don't understand this, don't worry about it; it just means +that 'gawk' does the right thing. + + +File: gawk.info, Node: Strong Regexp Constants, Next: Regexp Summary, Prev: Case-sensitivity, Up: Regexp + +3.9 Strongly Typed Regexp Constants +=================================== + +This minor node describes a 'gawk'-specific feature. + + Regexp constants ('/.../') hold a strange position in the 'awk' +language. In most contexts, they act like an expression: '$0 ~ /.../'. +In other contexts, they denote only a regexp to be matched. In no case +are they really a "first class citizen" of the language. That is, you +cannot define a scalar variable whose type is "regexp" in the same sense +that you can define a variable to be a number or a string: + + num = 42 Numeric variable + str = "hi" String variable + re = /foo/ Wrong! re is the result of $0 ~ /foo/ + + +File: gawk.info, Node: Regexp Summary, Prev: Strong Regexp Constants, Up: Regexp + +3.10 Summary +============ + + * Regular expressions describe sets of strings to be matched. In + 'awk', regular expression constants are written enclosed between + slashes: '/'...'/'. + + * Regexp constants may be used standalone in patterns and in + conditional expressions, or as part of matching expressions using + the '~' and '!~' operators. + + * Escape sequences let you represent nonprintable characters and also + let you represent regexp metacharacters as literal characters to be + matched. + + * Regexp operators provide grouping, alternation, and repetition. + + * Bracket expressions give you a shorthand for specifying sets of + characters that can match at a particular point in a regexp. + Within bracket expressions, POSIX character classes let you specify + certain groups of characters in a locale-independent fashion. + + * Regular expressions match the leftmost longest text in the string + being matched. This matters for cases where you need to know the + extent of the match, such as for text substitution and when the + record separator is a regexp. + + * Matching expressions may use dynamic regexps (i.e., string values + treated as regular expressions). + + * 'gawk''s 'IGNORECASE' variable lets you control the case + sensitivity of regexp matching. In other 'awk' versions, use + 'tolower()' or 'toupper()'. + + +File: gawk.info, Node: Reading Files, Next: Printing, Prev: Regexp, Up: Top + +4 Reading Input Files +********************* + +In the typical 'awk' program, 'awk' reads all input either from the +standard input (by default, this is the keyboard, but often it is a pipe +from another command) or from files whose names you specify on the 'awk' +command line. If you specify input files, 'awk' reads them in order, +processing all the data from one before going on to the next. The name +of the current input file can be found in the predefined variable +'FILENAME' (*note Built-in Variables::). + + The input is read in units called "records", and is processed by the +rules of your program one record at a time. By default, each record is +one line. Each record is automatically split into chunks called +"fields". This makes it more convenient for programs to work on the +parts of a record. + + On rare occasions, you may need to use the 'getline' command. The +'getline' command is valuable both because it can do explicit input from +any number of files, and because the files used with it do not have to +be named on the 'awk' command line (*note Getline::). + +* Menu: + +* Records:: Controlling how data is split into records. +* Fields:: An introduction to fields. +* Nonconstant Fields:: Nonconstant Field Numbers. +* Changing Fields:: Changing the Contents of a Field. +* Field Separators:: The field separator and how to change it. +* Constant Size:: Reading constant width data. +* Splitting By Content:: Defining Fields By Content +* Multiple Line:: Reading multiline records. +* Getline:: Reading files under explicit program control + using the 'getline' function. +* Read Timeout:: Reading input with a timeout. +* Retrying Input:: Retrying input after certain errors. +* Command-line directories:: What happens if you put a directory on the + command line. +* Input Summary:: Input summary. +* Input Exercises:: Exercises. + + +File: gawk.info, Node: Records, Next: Fields, Up: Reading Files + +4.1 How Input Is Split into Records +=================================== + +'awk' divides the input for your program into records and fields. It +keeps track of the number of records that have been read so far from the +current input file. This value is stored in a predefined variable +called 'FNR', which is reset to zero every time a new file is started. +Another predefined variable, 'NR', records the total number of input +records read so far from all data files. It starts at zero, but is +never automatically reset to zero. + +* Menu: + +* awk split records:: How standard 'awk' splits records. +* gawk split records:: How 'gawk' splits records. + + +File: gawk.info, Node: awk split records, Next: gawk split records, Up: Records + +4.1.1 Record Splitting with Standard 'awk' +------------------------------------------ + +Records are separated by a character called the "record separator". By +default, the record separator is the newline character. This is why +records are, by default, single lines. To use a different character for +the record separator, simply assign that character to the predefined +variable 'RS'. + + Like any other variable, the value of 'RS' can be changed in the +'awk' program with the assignment operator, '=' (*note Assignment +Ops::). The new record-separator character should be enclosed in +quotation marks, which indicate a string constant. Often, the right +time to do this is at the beginning of execution, before any input is +processed, so that the very first record is read with the proper +separator. To do this, use the special 'BEGIN' pattern (*note +BEGIN/END::). For example: + + awk 'BEGIN { RS = "u" } + { print $0 }' mail-list + +changes the value of 'RS' to 'u', before reading any input. The new +value is a string whose first character is the letter "u"; as a result, +records are separated by the letter "u". Then the input file is read, +and the second rule in the 'awk' program (the action with no pattern) +prints each record. Because each 'print' statement adds a newline at +the end of its output, this 'awk' program copies the input with each 'u' +changed to a newline. Here are the results of running the program on +'mail-list': + + $ awk 'BEGIN { RS = "u" } + > { print $0 }' mail-list + -| Amelia 555-5553 amelia.zodiac + -| sq + -| e@gmail.com F + -| Anthony 555-3412 anthony.assert + -| ro@hotmail.com A + -| Becky 555-7685 becky.algebrar + -| m@gmail.com A + -| Bill 555-1675 bill.drowning@hotmail.com A + -| Broderick 555-0542 broderick.aliq + -| otiens@yahoo.com R + -| Camilla 555-2912 camilla.inf + -| sar + -| m@skynet.be R + -| Fabi + -| s 555-1234 fabi + -| s. + -| ndevicesim + -| s@ + -| cb.ed + -| F + -| J + -| lie 555-6699 j + -| lie.perscr + -| tabor@skeeve.com F + -| Martin 555-6480 martin.codicib + -| s@hotmail.com A + -| Sam + -| el 555-3430 sam + -| el.lanceolis@sh + -| .ed + -| A + -| Jean-Pa + -| l 555-2127 jeanpa + -| l.campanor + -| m@ny + -| .ed + -| R + -| + +Note that the entry for the name 'Bill' is not split. In the original +data file (*note Sample Data Files::), the line looks like this: + + Bill 555-1675 bill.drowning@hotmail.com A + +It contains no 'u', so there is no reason to split the record, unlike +the others, which each have one or more occurrences of the 'u'. In +fact, this record is treated as part of the previous record; the newline +separating them in the output is the original newline in the data file, +not the one added by 'awk' when it printed the record! + + Another way to change the record separator is on the command line, +using the variable-assignment feature (*note Other Arguments::): + + awk '{ print $0 }' RS="u" mail-list + +This sets 'RS' to 'u' before processing 'mail-list'. + + Using an alphabetic character such as 'u' for the record separator is +highly likely to produce strange results. Using an unusual character +such as '/' is more likely to produce correct behavior in the majority +of cases, but there are no guarantees. The moral is: Know Your Data. + + When using regular characters as the record separator, there is one +unusual case that occurs when 'gawk' is being fully POSIX-compliant +(*note Options::). Then, the following (extreme) pipeline prints a +surprising '1': + + $ echo | gawk --posix 'BEGIN { RS = "a" } ; { print NF }' + -| 1 + + There is one field, consisting of a newline. The value of the +built-in variable 'NF' is the number of fields in the current record. +(In the normal case, 'gawk' treats the newline as whitespace, printing +'0' as the result. Most other versions of 'awk' also act this way.) + + Reaching the end of an input file terminates the current input +record, even if the last character in the file is not the character in +'RS'. (d.c.) + + The empty string '""' (a string without any characters) has a special +meaning as the value of 'RS'. It means that records are separated by +one or more blank lines and nothing else. *Note Multiple Line:: for +more details. + + If you change the value of 'RS' in the middle of an 'awk' run, the +new value is used to delimit subsequent records, but the record +currently being processed, as well as records already processed, are not +affected. + + After the end of the record has been determined, 'gawk' sets the +variable 'RT' to the text in the input that matched 'RS'. + + +File: gawk.info, Node: gawk split records, Prev: awk split records, Up: Records + +4.1.2 Record Splitting with 'gawk' +---------------------------------- + +When using 'gawk', the value of 'RS' is not limited to a one-character +string. It can be any regular expression (*note Regexp::). (c.e.) In +general, each record ends at the next string that matches the regular +expression; the next record starts at the end of the matching string. +This general rule is actually at work in the usual case, where 'RS' +contains just a newline: a record ends at the beginning of the next +matching string (the next newline in the input), and the following +record starts just after the end of this string (at the first character +of the following line). The newline, because it matches 'RS', is not +part of either record. + + When 'RS' is a single character, 'RT' contains the same single +character. However, when 'RS' is a regular expression, 'RT' contains +the actual input text that matched the regular expression. + + If the input file ends without any text matching 'RS', 'gawk' sets +'RT' to the null string. + + The following example illustrates both of these features. It sets +'RS' equal to a regular expression that matches either a newline or a +series of one or more uppercase letters with optional leading and/or +trailing whitespace: + + $ echo record 1 AAAA record 2 BBBB record 3 | + > gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" } + > { print "Record =", $0,"and RT = [" RT "]" }' + -| Record = record 1 and RT = [ AAAA ] + -| Record = record 2 and RT = [ BBBB ] + -| Record = record 3 and RT = [ + -| ] + +The square brackets delineate the contents of 'RT', letting you see the +leading and trailing whitespace. The final value of 'RT' is a newline. +*Note Simple Sed:: for a more useful example of 'RS' as a regexp and +'RT'. + + If you set 'RS' to a regular expression that allows optional trailing +text, such as 'RS = "abc(XYZ)?"', it is possible, due to implementation +constraints, that 'gawk' may match the leading part of the regular +expression, but not the trailing part, particularly if the input text +that could match the trailing part is fairly long. 'gawk' attempts to +avoid this problem, but currently, there's no guarantee that this will +never happen. + + NOTE: Remember that in 'awk', the '^' and '$' anchor metacharacters + match the beginning and end of a _string_, and not the beginning + and end of a _line_. As a result, something like 'RS = + "^[[:upper:]]"' can only match at the beginning of a file. This is + because 'gawk' views the input file as one long string that happens + to contain newline characters. It is thus best to avoid anchor + metacharacters in the value of 'RS'. + + The use of 'RS' as a regular expression and the 'RT' variable are +'gawk' extensions; they are not available in compatibility mode (*note +Options::). In compatibility mode, only the first character of the +value of 'RS' determines the end of the record. + + 'RS = "\0"' Is Not Portable + + There are times when you might want to treat an entire data file as a +single record. The only way to make this happen is to give 'RS' a value +that you know doesn't occur in the input file. This is hard to do in a +general way, such that a program always works for arbitrary input files. + + You might think that for text files, the NUL character, which +consists of a character with all bits equal to zero, is a good value to +use for 'RS' in this case: + + BEGIN { RS = "\0" } # whole file becomes one record? + + 'gawk' in fact accepts this, and uses the NUL character for the +record separator. This works for certain special files, such as +'/proc/environ' on GNU/Linux systems, where the NUL character is in fact +the record separator. However, this usage is _not_ portable to most +other 'awk' implementations. + + Almost all other 'awk' implementations(1) store strings internally as +C-style strings. C strings use the NUL character as the string +terminator. In effect, this means that 'RS = "\0"' is the same as 'RS = +""'. (d.c.) + + It happens that recent versions of 'mawk' can use the NUL character +as a record separator. However, this is a special case: 'mawk' does not +allow embedded NUL characters in strings. (This may change in a future +version of 'mawk'.) + + *Note Readfile Function:: for an interesting way to read whole files. +If you are using 'gawk', see *note Extension Sample Readfile:: for +another option. + + ---------- Footnotes ---------- + + (1) At least that we know about. + + +File: gawk.info, Node: Fields, Next: Nonconstant Fields, Prev: Records, Up: Reading Files + +4.2 Examining Fields +==================== + +When 'awk' reads an input record, the record is automatically "parsed" +or separated by the 'awk' utility into chunks called "fields". By +default, fields are separated by "whitespace", like words in a line. +Whitespace in 'awk' means any string of one or more spaces, TABs, or +newlines; other characters that are considered whitespace by other +languages (such as formfeed, vertical tab, etc.) are _not_ considered +whitespace by 'awk'. + + The purpose of fields is to make it more convenient for you to refer +to these pieces of the record. You don't have to use them--you can +operate on the whole record if you want--but fields are what make simple +'awk' programs so powerful. + + You use a dollar sign ('$') to refer to a field in an 'awk' program, +followed by the number of the field you want. Thus, '$1' refers to the +first field, '$2' to the second, and so on. (Unlike in the Unix shells, +the field numbers are not limited to single digits. '$127' is the 127th +field in the record.) For example, suppose the following is a line of +input: + + This seems like a pretty nice example. + +Here the first field, or '$1', is 'This', the second field, or '$2', is +'seems', and so on. Note that the last field, '$7', is 'example.'. +Because there is no space between the 'e' and the '.', the period is +considered part of the seventh field. + + 'NF' is a predefined variable whose value is the number of fields in +the current record. 'awk' automatically updates the value of 'NF' each +time it reads a record. No matter how many fields there are, the last +field in a record can be represented by '$NF'. So, '$NF' is the same as +'$7', which is 'example.'. If you try to reference a field beyond the +last one (such as '$8' when the record has only seven fields), you get +the empty string. (If used in a numeric operation, you get zero.) + + The use of '$0', which looks like a reference to the "zeroth" field, +is a special case: it represents the whole input record. Use it when +you are not interested in specific fields. Here are some more examples: + + $ awk '$1 ~ /li/ { print $0 }' mail-list + -| Amelia 555-5553 amelia.zodiacusque@gmail.com F + -| Julie 555-6699 julie.perscrutabor@skeeve.com F + +This example prints each record in the file 'mail-list' whose first +field contains the string 'li'. + + By contrast, the following example looks for 'li' in _the entire +record_ and prints the first and last fields for each matching input +record: + + $ awk '/li/ { print $1, $NF }' mail-list + -| Amelia F + -| Broderick R + -| Julie F + -| Samuel A + + +File: gawk.info, Node: Nonconstant Fields, Next: Changing Fields, Prev: Fields, Up: Reading Files + +4.3 Nonconstant Field Numbers +============================= + +A field number need not be a constant. Any expression in the 'awk' +language can be used after a '$' to refer to a field. The value of the +expression specifies the field number. If the value is a string, rather +than a number, it is converted to a number. Consider this example: + + awk '{ print $NR }' + +Recall that 'NR' is the number of records read so far: one in the first +record, two in the second, and so on. So this example prints the first +field of the first record, the second field of the second record, and so +on. For the twentieth record, field number 20 is printed; most likely, +the record has fewer than 20 fields, so this prints a blank line. Here +is another example of using expressions as field numbers: + + awk '{ print $(2*2) }' mail-list + + 'awk' evaluates the expression '(2*2)' and uses its value as the +number of the field to print. The '*' represents multiplication, so the +expression '2*2' evaluates to four. The parentheses are used so that +the multiplication is done before the '$' operation; they are necessary +whenever there is a binary operator(1) in the field-number expression. +This example, then, prints the type of relationship (the fourth field) +for every line of the file 'mail-list'. (All of the 'awk' operators are +listed, in order of decreasing precedence, in *note Precedence::.) + + If the field number you compute is zero, you get the entire record. +Thus, '$(2-2)' has the same value as '$0'. Negative field numbers are +not allowed; trying to reference one usually terminates the program. +(The POSIX standard does not define what happens when you reference a +negative field number. 'gawk' notices this and terminates your program. +Other 'awk' implementations may behave differently.) + + As mentioned in *note Fields::, 'awk' stores the current record's +number of fields in the built-in variable 'NF' (also *note Built-in +Variables::). Thus, the expression '$NF' is not a special feature--it +is the direct consequence of evaluating 'NF' and using its value as a +field number. + + ---------- Footnotes ---------- + + (1) A "binary operator", such as '*' for multiplication, is one that +takes two operands. The distinction is required because 'awk' also has +unary (one-operand) and ternary (three-operand) operators. + + +File: gawk.info, Node: Changing Fields, Next: Field Separators, Prev: Nonconstant Fields, Up: Reading Files + +4.4 Changing the Contents of a Field +==================================== + +The contents of a field, as seen by 'awk', can be changed within an +'awk' program; this changes what 'awk' perceives as the current input +record. (The actual input is untouched; 'awk' _never_ modifies the +input file.) Consider the following example and its output: + + $ awk '{ nboxes = $3 ; $3 = $3 - 10 + > print nboxes, $3 }' inventory-shipped + -| 25 15 + -| 32 22 + -| 24 14 + ... + +The program first saves the original value of field three in the +variable 'nboxes'. The '-' sign represents subtraction, so this program +reassigns field three, '$3', as the original value of field three minus +ten: '$3 - 10'. (*Note Arithmetic Ops::.) Then it prints the original +and new values for field three. (Someone in the warehouse made a +consistent mistake while inventorying the red boxes.) + + For this to work, the text in '$3' must make sense as a number; the +string of characters must be converted to a number for the computer to +do arithmetic on it. The number resulting from the subtraction is +converted back to a string of characters that then becomes field three. +*Note Conversion::. + + When the value of a field is changed (as perceived by 'awk'), the +text of the input record is recalculated to contain the new field where +the old one was. In other words, '$0' changes to reflect the altered +field. Thus, this program prints a copy of the input file, with 10 +subtracted from the second field of each line: + + $ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped + -| Jan 3 25 15 115 + -| Feb 5 32 24 226 + -| Mar 5 24 34 228 + ... + + It is also possible to assign contents to fields that are out of +range. For example: + + $ awk '{ $6 = ($5 + $4 + $3 + $2) + > print $6 }' inventory-shipped + -| 168 + -| 297 + -| 301 + ... + +We've just created '$6', whose value is the sum of fields '$2', '$3', +'$4', and '$5'. The '+' sign represents addition. For the file +'inventory-shipped', '$6' represents the total number of parcels shipped +for a particular month. + + Creating a new field changes 'awk''s internal copy of the current +input record, which is the value of '$0'. Thus, if you do 'print $0' +after adding a field, the record printed includes the new field, with +the appropriate number of field separators between it and the previously +existing fields. + + This recomputation affects and is affected by 'NF' (the number of +fields; *note Fields::). For example, the value of 'NF' is set to the +number of the highest field you create. The exact format of '$0' is +also affected by a feature that has not been discussed yet: the "output +field separator", 'OFS', used to separate the fields (*note Output +Separators::). + + Note, however, that merely _referencing_ an out-of-range field does +_not_ change the value of either '$0' or 'NF'. Referencing an +out-of-range field only produces an empty string. For example: + + if ($(NF+1) != "") + print "can't happen" + else + print "everything is normal" + +should print 'everything is normal', because 'NF+1' is certain to be out +of range. (*Note If Statement:: for more information about 'awk''s +'if-else' statements. *Note Typing and Comparison:: for more +information about the '!=' operator.) + + It is important to note that making an assignment to an existing +field changes the value of '$0' but does not change the value of 'NF', +even when you assign the empty string to a field. For example: + + $ echo a b c d | awk '{ OFS = ":"; $2 = "" + > print $0; print NF }' + -| a::c:d + -| 4 + +The field is still there; it just has an empty value, delimited by the +two colons between 'a' and 'c'. This example shows what happens if you +create a new field: + + $ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new" + > print $0; print NF }' + -| a::c:d::new + -| 6 + +The intervening field, '$5', is created with an empty value (indicated +by the second pair of adjacent colons), and 'NF' is updated with the +value six. + + Decrementing 'NF' throws away the values of the fields after the new +value of 'NF' and recomputes '$0'. (d.c.) Here is an example: + + $ echo a b c d e f | awk '{ print "NF =", NF; + > NF = 3; print $0 }' + -| NF = 6 + -| a b c + + CAUTION: Some versions of 'awk' don't rebuild '$0' when 'NF' is + decremented. + + Finally, there are times when it is convenient to force 'awk' to +rebuild the entire record, using the current values of the fields and +'OFS'. To do this, use the seemingly innocuous assignment: + + $1 = $1 # force record to be reconstituted + print $0 # or whatever else with $0 + +This forces 'awk' to rebuild the record. It does help to add a comment, +as we've shown here. + + There is a flip side to the relationship between '$0' and the fields. +Any assignment to '$0' causes the record to be reparsed into fields +using the _current_ value of 'FS'. This also applies to any built-in +function that updates '$0', such as 'sub()' and 'gsub()' (*note String +Functions::). + + Understanding '$0' + + It is important to remember that '$0' is the _full_ record, exactly +as it was read from the input. This includes any leading or trailing +whitespace, and the exact whitespace (or other characters) that +separates the fields. + + It is a common error to try to change the field separators in a +record simply by setting 'FS' and 'OFS', and then expecting a plain +'print' or 'print $0' to print the modified record. + + But this does not work, because nothing was done to change the record +itself. Instead, you must force the record to be rebuilt, typically +with a statement such as '$1 = $1', as described earlier. + + +File: gawk.info, Node: Field Separators, Next: Constant Size, Prev: Changing Fields, Up: Reading Files + +4.5 Specifying How Fields Are Separated +======================================= + +* Menu: + +* Default Field Splitting:: How fields are normally separated. +* Regexp Field Splitting:: Using regexps as the field separator. +* Single Character Fields:: Making each character a separate field. +* Command Line Field Separator:: Setting 'FS' from the command line. +* Full Line Fields:: Making the full line be a single field. +* Field Splitting Summary:: Some final points and a summary table. + +The "field separator", which is either a single character or a regular +expression, controls the way 'awk' splits an input record into fields. +'awk' scans the input record for character sequences that match the +separator; the fields themselves are the text between the matches. + + In the examples that follow, we use the bullet symbol (*) to +represent spaces in the output. If the field separator is 'oo', then +the following line: + + moo goo gai pan + +is split into three fields: 'm', '*g', and '*gai*pan'. Note the leading +spaces in the values of the second and third fields. + + The field separator is represented by the predefined variable 'FS'. +Shell programmers take note: 'awk' does _not_ use the name 'IFS' that is +used by the POSIX-compliant shells (such as the Unix Bourne shell, 'sh', +or Bash). + + The value of 'FS' can be changed in the 'awk' program with the +assignment operator, '=' (*note Assignment Ops::). Often, the right +time to do this is at the beginning of execution before any input has +been processed, so that the very first record is read with the proper +separator. To do this, use the special 'BEGIN' pattern (*note +BEGIN/END::). For example, here we set the value of 'FS' to the string +'","': + + awk 'BEGIN { FS = "," } ; { print $2 }' + +Given the input line: + + John Q. Smith, 29 Oak St., Walamazoo, MI 42139 + +this 'awk' program extracts and prints the string '*29*Oak*St.'. + + Sometimes the input data contains separator characters that don't +separate fields the way you thought they would. For instance, the +person's name in the example we just used might have a title or suffix +attached, such as: + + John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139 + +The same program would extract '*LXIX' instead of '*29*Oak*St.'. If you +were expecting the program to print the address, you would be surprised. +The moral is to choose your data layout and separator characters +carefully to prevent such problems. (If the data is not in a form that +is easy to process, perhaps you can massage it first with a separate +'awk' program.) + + +File: gawk.info, Node: Default Field Splitting, Next: Regexp Field Splitting, Up: Field Separators + +4.5.1 Whitespace Normally Separates Fields +------------------------------------------ + +Fields are normally separated by whitespace sequences (spaces, TABs, and +newlines), not by single spaces. Two spaces in a row do not delimit an +empty field. The default value of the field separator 'FS' is a string +containing a single space, '" "'. If 'awk' interpreted this value in +the usual way, each space character would separate fields, so two spaces +in a row would make an empty field between them. The reason this does +not happen is that a single space as the value of 'FS' is a special +case--it is taken to specify the default manner of delimiting fields. + + If 'FS' is any other single character, such as '","', then each +occurrence of that character separates two fields. Two consecutive +occurrences delimit an empty field. If the character occurs at the +beginning or the end of the line, that too delimits an empty field. The +space character is the only single character that does not follow these +rules. + + +File: gawk.info, Node: Regexp Field Splitting, Next: Single Character Fields, Prev: Default Field Splitting, Up: Field Separators + +4.5.2 Using Regular Expressions to Separate Fields +-------------------------------------------------- + +The previous node discussed the use of single characters or simple +strings as the value of 'FS'. More generally, the value of 'FS' may be +a string containing any regular expression. In this case, each match in +the record for the regular expression separates fields. For example, +the assignment: + + FS = ", \t" + +makes every area of an input line that consists of a comma followed by a +space and a TAB into a field separator. ('\t' is an "escape sequence" +that stands for a TAB; *note Escape Sequences::, for the complete list +of similar escape sequences.) + + For a less trivial example of a regular expression, try using single +spaces to separate fields the way single commas are used. 'FS' can be +set to '"[ ]"' (left bracket, space, right bracket). This regular +expression matches a single space and nothing else (*note Regexp::). + + There is an important difference between the two cases of 'FS = " "' +(a single space) and 'FS = "[ \t\n]+"' (a regular expression matching +one or more spaces, TABs, or newlines). For both values of 'FS', fields +are separated by "runs" (multiple adjacent occurrences) of spaces, TABs, +and/or newlines. However, when the value of 'FS' is '" "', 'awk' first +strips leading and trailing whitespace from the record and then decides +where the fields are. For example, the following pipeline prints 'b': + + $ echo ' a b c d ' | awk '{ print $2 }' + -| b + +However, this pipeline prints 'a' (note the extra spaces around each +letter): + + $ echo ' a b c d ' | awk 'BEGIN { FS = "[ \t\n]+" } + > { print $2 }' + -| a + +In this case, the first field is null, or empty. + + The stripping of leading and trailing whitespace also comes into play +whenever '$0' is recomputed. For instance, study this pipeline: + + $ echo ' a b c d' | awk '{ print; $2 = $2; print }' + -| a b c d + -| a b c d + +The first 'print' statement prints the record as it was read, with +leading whitespace intact. The assignment to '$2' rebuilds '$0' by +concatenating '$1' through '$NF' together, separated by the value of +'OFS' (which is a space by default). Because the leading whitespace was +ignored when finding '$1', it is not part of the new '$0'. Finally, the +last 'print' statement prints the new '$0'. + + There is an additional subtlety to be aware of when using regular +expressions for field splitting. It is not well specified in the POSIX +standard, or anywhere else, what '^' means when splitting fields. Does +the '^' match only at the beginning of the entire record? Or is each +field separator a new string? It turns out that different 'awk' +versions answer this question differently, and you should not rely on +any specific behavior in your programs. (d.c.) + + As a point of information, BWK 'awk' allows '^' to match only at the +beginning of the record. 'gawk' also works this way. For example: + + $ echo 'xxAA xxBxx C' | + > gawk -F '(^x+)|( +)' '{ for (i = 1; i <= NF; i++) + > printf "-->%s<--\n", $i }' + -| --><-- + -| -->AA<-- + -| -->xxBxx<-- + -| -->C<-- + + +File: gawk.info, Node: Single Character Fields, Next: Command Line Field Separator, Prev: Regexp Field Splitting, Up: Field Separators + +4.5.3 Making Each Character a Separate Field +-------------------------------------------- + +There are times when you may want to examine each character of a record +separately. This can be done in 'gawk' by simply assigning the null +string ('""') to 'FS'. (c.e.) In this case, each individual character +in the record becomes a separate field. For example: + + $ echo a b | gawk 'BEGIN { FS = "" } + > { + > for (i = 1; i <= NF; i = i + 1) + > print "Field", i, "is", $i + > }' + -| Field 1 is a + -| Field 2 is + -| Field 3 is b + + Traditionally, the behavior of 'FS' equal to '""' was not defined. +In this case, most versions of Unix 'awk' simply treat the entire record +as only having one field. (d.c.) In compatibility mode (*note +Options::), if 'FS' is the null string, then 'gawk' also behaves this +way. + + +File: gawk.info, Node: Command Line Field Separator, Next: Full Line Fields, Prev: Single Character Fields, Up: Field Separators + +4.5.4 Setting 'FS' from the Command Line +---------------------------------------- + +'FS' can be set on the command line. Use the '-F' option to do so. For +example: + + awk -F, 'PROGRAM' INPUT-FILES + +sets 'FS' to the ',' character. Notice that the option uses an +uppercase 'F' instead of a lowercase 'f'. The latter option ('-f') +specifies a file containing an 'awk' program. + + The value used for the argument to '-F' is processed in exactly the +same way as assignments to the predefined variable 'FS'. Any special +characters in the field separator must be escaped appropriately. For +example, to use a '\' as the field separator on the command line, you +would have to type: + + # same as FS = "\\" + awk -F\\\\ '...' files ... + +Because '\' is used for quoting in the shell, 'awk' sees '-F\\'. Then +'awk' processes the '\\' for escape characters (*note Escape +Sequences::), finally yielding a single '\' to use for the field +separator. + + As a special case, in compatibility mode (*note Options::), if the +argument to '-F' is 't', then 'FS' is set to the TAB character. If you +type '-F\t' at the shell, without any quotes, the '\' gets deleted, so +'awk' figures that you really want your fields to be separated with TABs +and not 't's. Use '-v FS="t"' or '-F"[t]"' on the command line if you +really do want to separate your fields with 't's. Use '-F '\t'' when +not in compatibility mode to specify that TABs separate fields. + + As an example, let's use an 'awk' program file called 'edu.awk' that +contains the pattern '/edu/' and the action 'print $1': + + /edu/ { print $1 } + + Let's also set 'FS' to be the '-' character and run the program on +the file 'mail-list'. The following command prints a list of the names +of the people that work at or attend a university, and the first three +digits of their phone numbers: + + $ awk -F- -f edu.awk mail-list + -| Fabius 555 + -| Samuel 555 + -| Jean + +Note the third line of output. The third line in the original file +looked like this: + + Jean-Paul 555-2127 jeanpaul.campanorum@nyu.edu R + + The '-' as part of the person's name was used as the field separator, +instead of the '-' in the phone number that was originally intended. +This demonstrates why you have to be careful in choosing your field and +record separators. + + Perhaps the most common use of a single character as the field +separator occurs when processing the Unix system password file. On many +Unix systems, each user has a separate entry in the system password +file, with one line per user. The information in these lines is +separated by colons. The first field is the user's login name and the +second is the user's encrypted or shadow password. (A shadow password +is indicated by the presence of a single 'x' in the second field.) A +password file entry might look like this: + + arnold:x:2076:10:Arnold Robbins:/home/arnold:/bin/bash + + The following program searches the system password file and prints +the entries for users whose full name is not indicated: + + awk -F: '$5 == ""' /etc/passwd + + +File: gawk.info, Node: Full Line Fields, Next: Field Splitting Summary, Prev: Command Line Field Separator, Up: Field Separators + +4.5.5 Making the Full Line Be a Single Field +-------------------------------------------- + +Occasionally, it's useful to treat the whole input line as a single +field. This can be done easily and portably simply by setting 'FS' to +'"\n"' (a newline):(1) + + awk -F'\n' 'PROGRAM' FILES ... + +When you do this, '$1' is the same as '$0'. + + Changing 'FS' Does Not Affect the Fields + + According to the POSIX standard, 'awk' is supposed to behave as if +each record is split into fields at the time it is read. In particular, +this means that if you change the value of 'FS' after a record is read, +the values of the fields (i.e., how they were split) should reflect the +old value of 'FS', not the new one. + + However, many older implementations of 'awk' do not work this way. +Instead, they defer splitting the fields until a field is actually +referenced. The fields are split using the _current_ value of 'FS'! +(d.c.) This behavior can be difficult to diagnose. The following +example illustrates the difference between the two methods: + + sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }' + +which usually prints: + + root + +on an incorrect implementation of 'awk', while 'gawk' prints the full +first line of the file, something like: + + root:x:0:0:Root:/: + + (The 'sed'(2) command prints just the first line of '/etc/passwd'.) + + ---------- Footnotes ---------- + + (1) Thanks to Andrew Schorr for this tip. + + (2) The 'sed' utility is a "stream editor." Its behavior is also +defined by the POSIX standard. + + +File: gawk.info, Node: Field Splitting Summary, Prev: Full Line Fields, Up: Field Separators + +4.5.6 Field-Splitting Summary +----------------------------- + +It is important to remember that when you assign a string constant as +the value of 'FS', it undergoes normal 'awk' string processing. For +example, with Unix 'awk' and 'gawk', the assignment 'FS = "\.."' assigns +the character string '".."' to 'FS' (the backslash is stripped). This +creates a regexp meaning "fields are separated by occurrences of any two +characters." If instead you want fields to be separated by a literal +period followed by any single character, use 'FS = "\\.."'. + + The following list summarizes how fields are split, based on the +value of 'FS' ('==' means "is equal to"): + +'FS == " "' + Fields are separated by runs of whitespace. Leading and trailing + whitespace are ignored. This is the default. + +'FS == ANY OTHER SINGLE CHARACTER' + Fields are separated by each occurrence of the character. Multiple + successive occurrences delimit empty fields, as do leading and + trailing occurrences. The character can even be a regexp + metacharacter; it does not need to be escaped. + +'FS == REGEXP' + Fields are separated by occurrences of characters that match + REGEXP. Leading and trailing matches of REGEXP delimit empty + fields. + +'FS == ""' + Each individual character in the record becomes a separate field. + (This is a common extension; it is not specified by the POSIX + standard.) + + 'FS' and 'IGNORECASE' + + The 'IGNORECASE' variable (*note User-modified::) affects field +splitting _only_ when the value of 'FS' is a regexp. It has no effect +when 'FS' is a single character, even if that character is a letter. +Thus, in the following code: + + FS = "c" + IGNORECASE = 1 + $0 = "aCa" + print $1 + +The output is 'aCa'. If you really want to split fields on an +alphabetic character while ignoring case, use a regexp that will do it +for you (e.g., 'FS = "[c]"'). In this case, 'IGNORECASE' will take +effect. + + +File: gawk.info, Node: Constant Size, Next: Splitting By Content, Prev: Field Separators, Up: Reading Files + +4.6 Reading Fixed-Width Data +============================ + +This minor node discusses an advanced feature of 'gawk'. If you are a +novice 'awk' user, you might want to skip it on the first reading. + + 'gawk' provides a facility for dealing with fixed-width fields with +no distinctive field separator. For example, data of this nature arises +in the input for old Fortran programs where numbers are run together, or +in the output of programs that did not anticipate the use of their +output as input for other programs. + + An example of the latter is a table where all the columns are lined +up by the use of a variable number of spaces and _empty fields are just +spaces_. Clearly, 'awk''s normal field splitting based on 'FS' does not +work well in this case. Although a portable 'awk' program can use a +series of 'substr()' calls on '$0' (*note String Functions::), this is +awkward and inefficient for a large number of fields. + + The splitting of an input record into fixed-width fields is specified +by assigning a string containing space-separated numbers to the built-in +variable 'FIELDWIDTHS'. Each number specifies the width of the field, +_including_ columns between fields. If you want to ignore the columns +between fields, you can specify the width as a separate field that is +subsequently ignored. It is a fatal error to supply a field width that +has a negative value. The following data is the output of the Unix 'w' +utility. It is useful to illustrate the use of 'FIELDWIDTHS': + + 10:06pm up 21 days, 14:04, 23 users + User tty login idle JCPU PCPU what + hzuo ttyV0 8:58pm 9 5 vi p24.tex + hzang ttyV3 6:37pm 50 -csh + eklye ttyV5 9:53pm 7 1 em thes.tex + dportein ttyV6 8:17pm 1:47 -csh + gierd ttyD3 10:00pm 1 elm + dave ttyD4 9:47pm 4 4 w + brent ttyp0 26Jun91 4:46 26:46 4:41 bash + dave ttyq4 26Jun9115days 46 46 wnewmail + + The following program takes this input, converts the idle time to +number of seconds, and prints out the first two fields and the +calculated idle time: + + BEGIN { FIELDWIDTHS = "9 6 10 6 7 7 35" } + NR > 2 { + idle = $4 + sub(/^ +/, "", idle) # strip leading spaces + if (idle == "") + idle = 0 + if (idle ~ /:/) { + split(idle, t, ":") + idle = t[1] * 60 + t[2] + } + if (idle ~ /days/) + idle *= 24 * 60 * 60 + + print $1, $2, idle + } + + NOTE: The preceding program uses a number of 'awk' features that + haven't been introduced yet. + + Running the program on the data produces the following results: + + hzuo ttyV0 0 + hzang ttyV3 50 + eklye ttyV5 0 + dportein ttyV6 107 + gierd ttyD3 1 + dave ttyD4 0 + brent ttyp0 286 + dave ttyq4 1296000 + + Another (possibly more practical) example of fixed-width input data +is the input from a deck of balloting cards. In some parts of the +United States, voters mark their choices by punching holes in computer +cards. These cards are then processed to count the votes for any +particular candidate or on any particular issue. Because a voter may +choose not to vote on some issue, any column on the card may be empty. +An 'awk' program for processing such data could use the 'FIELDWIDTHS' +feature to simplify reading the data. (Of course, getting 'gawk' to run +on a system with card readers is another story!) + + Assigning a value to 'FS' causes 'gawk' to use 'FS' for field +splitting again. Use 'FS = FS' to make this happen, without having to +know the current value of 'FS'. In order to tell which kind of field +splitting is in effect, use 'PROCINFO["FS"]' (*note Auto-set::). The +value is '"FS"' if regular field splitting is being used, or +'"FIELDWIDTHS"' if fixed-width field splitting is being used: + + if (PROCINFO["FS"] == "FS") + REGULAR FIELD SPLITTING ... + else if (PROCINFO["FS"] == "FIELDWIDTHS") + FIXED-WIDTH FIELD SPLITTING ... + else + CONTENT-BASED FIELD SPLITTING ... (see next minor node) + + This information is useful when writing a function that needs to +temporarily change 'FS' or 'FIELDWIDTHS', read some records, and then +restore the original settings (*note Passwd Functions:: for an example +of such a function). + + +File: gawk.info, Node: Splitting By Content, Next: Multiple Line, Prev: Constant Size, Up: Reading Files + +4.7 Defining Fields by Content +============================== + +This minor node discusses an advanced feature of 'gawk'. If you are a +novice 'awk' user, you might want to skip it on the first reading. + + Normally, when using 'FS', 'gawk' defines the fields as the parts of +the record that occur in between each field separator. In other words, +'FS' defines what a field _is not_, instead of what a field _is_. +However, there are times when you really want to define the fields by +what they are, and not by what they are not. + + The most notorious such case is so-called "comma-separated values" +(CSV) data. Many spreadsheet programs, for example, can export their +data into text files, where each record is terminated with a newline, +and fields are separated by commas. If commas only separated the data, +there wouldn't be an issue. The problem comes when one of the fields +contains an _embedded_ comma. In such cases, most programs embed the +field in double quotes.(1) So, we might have data like this: + + Robbins,Arnold,"1234 A Pretty Street, NE",MyTown,MyState,12345-6789,USA + + The 'FPAT' variable offers a solution for cases like this. The value +of 'FPAT' should be a string that provides a regular expression. This +regular expression describes the contents of each field. + + In the case of CSV data as presented here, each field is either +"anything that is not a comma," or "a double quote, anything that is not +a double quote, and a closing double quote." If written as a regular +expression constant (*note Regexp::), we would have +'/([^,]+)|("[^"]+")/'. Writing this as a string requires us to escape +the double quotes, leading to: + + FPAT = "([^,]+)|(\"[^\"]+\")" + + Putting this to use, here is a simple program to parse the data: + + BEGIN { + FPAT = "([^,]+)|(\"[^\"]+\")" + } + + { + print "NF = ", NF + for (i = 1; i <= NF; i++) { + printf("$%d = <%s>\n", i, $i) + } + } + + When run, we get the following: + + $ gawk -f simple-csv.awk addresses.csv + NF = 7 + $1 = <Robbins> + $2 = <Arnold> + $3 = <"1234 A Pretty Street, NE"> + $4 = <MyTown> + $5 = <MyState> + $6 = <12345-6789> + $7 = <USA> + + Note the embedded comma in the value of '$3'. + + A straightforward improvement when processing CSV data of this sort +would be to remove the quotes when they occur, with something like this: + + if (substr($i, 1, 1) == "\"") { + len = length($i) + $i = substr($i, 2, len - 2) # Get text within the two quotes + } + + As with 'FS', the 'IGNORECASE' variable (*note User-modified::) +affects field splitting with 'FPAT'. + + Assigning a value to 'FPAT' overrides field splitting with 'FS' and +with 'FIELDWIDTHS'. Similar to 'FIELDWIDTHS', the value of +'PROCINFO["FS"]' will be '"FPAT"' if content-based field splitting is +being used. + + NOTE: Some programs export CSV data that contains embedded newlines + between the double quotes. 'gawk' provides no way to deal with + this. Even though a formal specification for CSV data exists, + there isn't much more to be done; the 'FPAT' mechanism provides an + elegant solution for the majority of cases, and the 'gawk' + developers are satisfied with that. + + As written, the regexp used for 'FPAT' requires that each field +contain at least one character. A straightforward modification +(changing the first '+' to '*') allows fields to be empty: + + FPAT = "([^,]*)|(\"[^\"]+\")" + + Finally, the 'patsplit()' function makes the same functionality +available for splitting regular strings (*note String Functions::). + + To recap, 'gawk' provides three independent methods to split input +records into fields. The mechanism used is based on which of the three +variables--'FS', 'FIELDWIDTHS', or 'FPAT'--was last assigned to. + + ---------- Footnotes ---------- + + (1) The CSV format lacked a formal standard definition for many +years. RFC 4180 (http://www.ietf.org/rfc/rfc4180.txt) standardizes the +most common practices. + + +File: gawk.info, Node: Multiple Line, Next: Getline, Prev: Splitting By Content, Up: Reading Files + +4.8 Multiple-Line Records +========================= + +In some databases, a single line cannot conveniently hold all the +information in one entry. In such cases, you can use multiline records. +The first step in doing this is to choose your data format. + + One technique is to use an unusual character or string to separate +records. For example, you could use the formfeed character (written +'\f' in 'awk', as in C) to separate them, making each record a page of +the file. To do this, just set the variable 'RS' to '"\f"' (a string +containing the formfeed character). Any other character could equally +well be used, as long as it won't be part of the data in a record. + + Another technique is to have blank lines separate records. By a +special dispensation, an empty string as the value of 'RS' indicates +that records are separated by one or more blank lines. When 'RS' is set +to the empty string, each record always ends at the first blank line +encountered. The next record doesn't start until the first nonblank +line that follows. No matter how many blank lines appear in a row, they +all act as one record separator. (Blank lines must be completely empty; +lines that contain only whitespace do not count.) + + You can achieve the same effect as 'RS = ""' by assigning the string +'"\n\n+"' to 'RS'. This regexp matches the newline at the end of the +record and one or more blank lines after the record. In addition, a +regular expression always matches the longest possible sequence when +there is a choice (*note Leftmost Longest::). So, the next record +doesn't start until the first nonblank line that follows--no matter how +many blank lines appear in a row, they are considered one record +separator. + + However, there is an important difference between 'RS = ""' and 'RS = +"\n\n+"'. In the first case, leading newlines in the input data file +are ignored, and if a file ends without extra blank lines after the last +record, the final newline is removed from the record. In the second +case, this special processing is not done. (d.c.) + + Now that the input is separated into records, the second step is to +separate the fields in the records. One way to do this is to divide +each of the lines into fields in the normal manner. This happens by +default as the result of a special feature. When 'RS' is set to the +empty string _and_ 'FS' is set to a single character, the newline +character _always_ acts as a field separator. This is in addition to +whatever field separations result from 'FS'.(1) + + The original motivation for this special exception was probably to +provide useful behavior in the default case (i.e., 'FS' is equal to +'" "'). This feature can be a problem if you really don't want the +newline character to separate fields, because there is no way to prevent +it. However, you can work around this by using the 'split()' function +to break up the record manually (*note String Functions::). If you have +a single-character field separator, you can work around the special +feature in a different way, by making 'FS' into a regexp for that single +character. For example, if the field separator is a percent character, +instead of 'FS = "%"', use 'FS = "[%]"'. + + Another way to separate fields is to put each field on a separate +line: to do this, just set the variable 'FS' to the string '"\n"'. +(This single-character separator matches a single newline.) A practical +example of a data file organized this way might be a mailing list, where +blank lines separate the entries. Consider a mailing list in a file +named 'addresses', which looks like this: + + Jane Doe + 123 Main Street + Anywhere, SE 12345-6789 + + John Smith + 456 Tree-lined Avenue + Smallville, MW 98765-4321 + ... + +A simple program to process this file is as follows: + + # addrs.awk --- simple mailing list program + + # Records are separated by blank lines. + # Each line is one field. + BEGIN { RS = "" ; FS = "\n" } + + { + print "Name is:", $1 + print "Address is:", $2 + print "City and State are:", $3 + print "" + } + + Running the program produces the following output: + + $ awk -f addrs.awk addresses + -| Name is: Jane Doe + -| Address is: 123 Main Street + -| City and State are: Anywhere, SE 12345-6789 + -| + -| Name is: John Smith + -| Address is: 456 Tree-lined Avenue + -| City and State are: Smallville, MW 98765-4321 + -| + ... + + *Note Labels Program:: for a more realistic program dealing with +address lists. The following list summarizes how records are split, +based on the value of 'RS'. ('==' means "is equal to.") + +'RS == "\n"' + Records are separated by the newline character ('\n'). In effect, + every line in the data file is a separate record, including blank + lines. This is the default. + +'RS == ANY SINGLE CHARACTER' + Records are separated by each occurrence of the character. + Multiple successive occurrences delimit empty records. + +'RS == ""' + Records are separated by runs of blank lines. When 'FS' is a + single character, then the newline character always serves as a + field separator, in addition to whatever value 'FS' may have. + Leading and trailing newlines in a file are ignored. + +'RS == REGEXP' + Records are separated by occurrences of characters that match + REGEXP. Leading and trailing matches of REGEXP delimit empty + records. (This is a 'gawk' extension; it is not specified by the + POSIX standard.) + + If not in compatibility mode (*note Options::), 'gawk' sets 'RT' to +the input text that matched the value specified by 'RS'. But if the +input file ended without any text that matches 'RS', then 'gawk' sets +'RT' to the null string. + + ---------- Footnotes ---------- + + (1) When 'FS' is the null string ('""') or a regexp, this special +feature of 'RS' does not apply. It does apply to the default field +separator of a single space: 'FS = " "'. + + +File: gawk.info, Node: Getline, Next: Read Timeout, Prev: Multiple Line, Up: Reading Files + +4.9 Explicit Input with 'getline' +================================= + +So far we have been getting our input data from 'awk''s main input +stream--either the standard input (usually your keyboard, sometimes the +output from another program) or the files specified on the command line. +The 'awk' language has a special built-in command called 'getline' that +can be used to read input under your explicit control. + + The 'getline' command is used in several different ways and should +_not_ be used by beginners. The examples that follow the explanation of +the 'getline' command include material that has not been covered yet. +Therefore, come back and study the 'getline' command _after_ you have +reviewed the rest of this Info file and have a good knowledge of how +'awk' works. + + The 'getline' command returns 1 if it finds a record and 0 if it +encounters the end of the file. If there is some error in getting a +record, such as a file that cannot be opened, then 'getline' returns -1. +In this case, 'gawk' sets the variable 'ERRNO' to a string describing +the error that occurred. + + If 'ERRNO' indicates that the I/O operation may be retried, and +'PROCINFO["INPUT", "RETRY"]' is set, then 'getline' returns -2 instead +of -1, and further calls to 'getline' may be attempted. *Note Retrying +Input:: for further information about this feature. + + In the following examples, COMMAND stands for a string value that +represents a shell command. + + NOTE: When '--sandbox' is specified (*note Options::), reading + lines from files, pipes, and coprocesses is disabled. + +* Menu: + +* Plain Getline:: Using 'getline' with no arguments. +* Getline/Variable:: Using 'getline' into a variable. +* Getline/File:: Using 'getline' from a file. +* Getline/Variable/File:: Using 'getline' into a variable from a + file. +* Getline/Pipe:: Using 'getline' from a pipe. +* Getline/Variable/Pipe:: Using 'getline' into a variable from a + pipe. +* Getline/Coprocess:: Using 'getline' from a coprocess. +* Getline/Variable/Coprocess:: Using 'getline' into a variable from a + coprocess. +* Getline Notes:: Important things to know about 'getline'. +* Getline Summary:: Summary of 'getline' Variants. + + +File: gawk.info, Node: Plain Getline, Next: Getline/Variable, Up: Getline + +4.9.1 Using 'getline' with No Arguments +--------------------------------------- + +The 'getline' command can be used without arguments to read input from +the current input file. All it does in this case is read the next input +record and split it up into fields. This is useful if you've finished +processing the current record, but want to do some special processing on +the next record _right now_. For example: + + # Remove text between /* and */, inclusive + { + if ((i = index($0, "/*")) != 0) { + out = substr($0, 1, i - 1) # leading part of the string + rest = substr($0, i + 2) # ... */ ... + j = index(rest, "*/") # is */ in trailing part? + if (j > 0) { + rest = substr(rest, j + 2) # remove comment + } else { + while (j == 0) { + # get more text + if (getline <= 0) { + print("unexpected EOF or error:", ERRNO) > "/dev/stderr" + exit + } + # build up the line using string concatenation + rest = rest $0 + j = index(rest, "*/") # is */ in trailing part? + if (j != 0) { + rest = substr(rest, j + 2) + break + } + } + } + # build up the output line using string concatenation + $0 = out rest + } + print $0 + } + + This 'awk' program deletes C-style comments ('/* ... */') from the +input. It uses a number of features we haven't covered yet, including +string concatenation (*note Concatenation::) and the 'index()' and +'substr()' built-in functions (*note String Functions::). By replacing +the 'print $0' with other statements, you could perform more complicated +processing on the decommented input, such as searching for matches of a +regular expression. (This program has a subtle problem--it does not +work if one comment ends and another begins on the same line.) + + This form of the 'getline' command sets 'NF', 'NR', 'FNR', 'RT', and +the value of '$0'. + + NOTE: The new value of '$0' is used to test the patterns of any + subsequent rules. The original value of '$0' that triggered the + rule that executed 'getline' is lost. By contrast, the 'next' + statement reads a new record but immediately begins processing it + normally, starting with the first rule in the program. *Note Next + Statement::. + + +File: gawk.info, Node: Getline/Variable, Next: Getline/File, Prev: Plain Getline, Up: Getline + +4.9.2 Using 'getline' into a Variable +------------------------------------- + +You can use 'getline VAR' to read the next record from 'awk''s input +into the variable VAR. No other processing is done. For example, +suppose the next line is a comment or a special string, and you want to +read it without triggering any rules. This form of 'getline' allows you +to read that line and store it in a variable so that the main +read-a-line-and-check-each-rule loop of 'awk' never sees it. The +following example swaps every two lines of input: + + { + if ((getline tmp) > 0) { + print tmp + print $0 + } else + print $0 + } + +It takes the following list: + + wan + tew + free + phore + +and produces these results: + + tew + wan + phore + free + + The 'getline' command used in this way sets only the variables 'NR', +'FNR', and 'RT' (and, of course, VAR). The record is not split into +fields, so the values of the fields (including '$0') and the value of +'NF' do not change. + + +File: gawk.info, Node: Getline/File, Next: Getline/Variable/File, Prev: Getline/Variable, Up: Getline + +4.9.3 Using 'getline' from a File +--------------------------------- + +Use 'getline < FILE' to read the next record from FILE. Here, FILE is a +string-valued expression that specifies the file name. '< FILE' is +called a "redirection" because it directs input to come from a different +place. For example, the following program reads its input record from +the file 'secondary.input' when it encounters a first field with a value +equal to 10 in the current input file: + + { + if ($1 == 10) { + getline < "secondary.input" + print + } else + print + } + + Because the main input stream is not used, the values of 'NR' and +'FNR' are not changed. However, the record it reads is split into +fields in the normal manner, so the values of '$0' and the other fields +are changed, resulting in a new value of 'NF'. 'RT' is also set. + + According to POSIX, 'getline < EXPRESSION' is ambiguous if EXPRESSION +contains unparenthesized operators other than '$'; for example, 'getline +< dir "/" file' is ambiguous because the concatenation operator (not +discussed yet; *note Concatenation::) is not parenthesized. You should +write it as 'getline < (dir "/" file)' if you want your program to be +portable to all 'awk' implementations. + + +File: gawk.info, Node: Getline/Variable/File, Next: Getline/Pipe, Prev: Getline/File, Up: Getline + +4.9.4 Using 'getline' into a Variable from a File +------------------------------------------------- + +Use 'getline VAR < FILE' to read input from the file FILE, and put it in +the variable VAR. As earlier, FILE is a string-valued expression that +specifies the file from which to read. + + In this version of 'getline', none of the predefined variables are +changed and the record is not split into fields. The only variable +changed is VAR.(1) For example, the following program copies all the +input files to the output, except for records that say +'@include FILENAME'. Such a record is replaced by the contents of the +file FILENAME: + + { + if (NF == 2 && $1 == "@include") { + while ((getline line < $2) > 0) + print line + close($2) + } else + print + } + + Note here how the name of the extra input file is not built into the +program; it is taken directly from the data, specifically from the +second field on the '@include' line. + + The 'close()' function is called to ensure that if two identical +'@include' lines appear in the input, the entire specified file is +included twice. *Note Close Files And Pipes::. + + One deficiency of this program is that it does not process nested +'@include' statements (i.e., '@include' statements in included files) +the way a true macro preprocessor would. *Note Igawk Program:: for a +program that does handle nested '@include' statements. + + ---------- Footnotes ---------- + + (1) This is not quite true. 'RT' could be changed if 'RS' is a +regular expression. + + +File: gawk.info, Node: Getline/Pipe, Next: Getline/Variable/Pipe, Prev: Getline/Variable/File, Up: Getline + +4.9.5 Using 'getline' from a Pipe +--------------------------------- + + Omniscience has much to recommend it. Failing that, attention to + details would be useful. + -- _Brian Kernighan_ + + The output of a command can also be piped into 'getline', using +'COMMAND | getline'. In this case, the string COMMAND is run as a shell +command and its output is piped into 'awk' to be used as input. This +form of 'getline' reads one record at a time from the pipe. For +example, the following program copies its input to its output, except +for lines that begin with '@execute', which are replaced by the output +produced by running the rest of the line as a shell command: + + { + if ($1 == "@execute") { + tmp = substr($0, 10) # Remove "@execute" + while ((tmp | getline) > 0) + print + close(tmp) + } else + print + } + +The 'close()' function is called to ensure that if two identical +'@execute' lines appear in the input, the command is run for each one. +*Note Close Files And Pipes::. Given the input: + + foo + bar + baz + @execute who + bletch + +the program might produce: + + foo + bar + baz + arnold ttyv0 Jul 13 14:22 + miriam ttyp0 Jul 13 14:23 (murphy:0) + bill ttyp1 Jul 13 14:23 (murphy:0) + bletch + +Notice that this program ran the command 'who' and printed the result. +(If you try this program yourself, you will of course get different +results, depending upon who is logged in on your system.) + + This variation of 'getline' splits the record into fields, sets the +value of 'NF', and recomputes the value of '$0'. The values of 'NR' and +'FNR' are not changed. 'RT' is set. + + According to POSIX, 'EXPRESSION | getline' is ambiguous if EXPRESSION +contains unparenthesized operators other than '$'--for example, '"echo " +"date" | getline' is ambiguous because the concatenation operator is not +parenthesized. You should write it as '("echo " "date") | getline' if +you want your program to be portable to all 'awk' implementations. + + NOTE: Unfortunately, 'gawk' has not been consistent in its + treatment of a construct like '"echo " "date" | getline'. Most + versions, including the current version, treat it at as '("echo " + "date") | getline'. (This is also how BWK 'awk' behaves.) Some + versions instead treat it as '"echo " ("date" | getline)'. (This + is how 'mawk' behaves.) In short, _always_ use explicit + parentheses, and then you won't have to worry. + + +File: gawk.info, Node: Getline/Variable/Pipe, Next: Getline/Coprocess, Prev: Getline/Pipe, Up: Getline + +4.9.6 Using 'getline' into a Variable from a Pipe +------------------------------------------------- + +When you use 'COMMAND | getline VAR', the output of COMMAND is sent +through a pipe to 'getline' and into the variable VAR. For example, the +following program reads the current date and time into the variable +'current_time', using the 'date' utility, and then prints it: + + BEGIN { + "date" | getline current_time + close("date") + print "Report printed on " current_time + } + + In this version of 'getline', none of the predefined variables are +changed and the record is not split into fields. However, 'RT' is set. + + According to POSIX, 'EXPRESSION | getline VAR' is ambiguous if +EXPRESSION contains unparenthesized operators other than '$'; for +example, '"echo " "date" | getline VAR' is ambiguous because the +concatenation operator is not parenthesized. You should write it as +'("echo " "date") | getline VAR' if you want your program to be portable +to other 'awk' implementations. + + +File: gawk.info, Node: Getline/Coprocess, Next: Getline/Variable/Coprocess, Prev: Getline/Variable/Pipe, Up: Getline + +4.9.7 Using 'getline' from a Coprocess +-------------------------------------- + +Reading input into 'getline' from a pipe is a one-way operation. The +command that is started with 'COMMAND | getline' only sends data _to_ +your 'awk' program. + + On occasion, you might want to send data to another program for +processing and then read the results back. 'gawk' allows you to start a +"coprocess", with which two-way communications are possible. This is +done with the '|&' operator. Typically, you write data to the coprocess +first and then read the results back, as shown in the following: + + print "SOME QUERY" |& "db_server" + "db_server" |& getline + +which sends a query to 'db_server' and then reads the results. + + The values of 'NR' and 'FNR' are not changed, because the main input +stream is not used. However, the record is split into fields in the +normal manner, thus changing the values of '$0', of the other fields, +and of 'NF' and 'RT'. + + Coprocesses are an advanced feature. They are discussed here only +because this is the minor node on 'getline'. *Note Two-way I/O::, where +coprocesses are discussed in more detail. + + +File: gawk.info, Node: Getline/Variable/Coprocess, Next: Getline Notes, Prev: Getline/Coprocess, Up: Getline + +4.9.8 Using 'getline' into a Variable from a Coprocess +------------------------------------------------------ + +When you use 'COMMAND |& getline VAR', the output from the coprocess +COMMAND is sent through a two-way pipe to 'getline' and into the +variable VAR. + + In this version of 'getline', none of the predefined variables are +changed and the record is not split into fields. The only variable +changed is VAR. However, 'RT' is set. + + Coprocesses are an advanced feature. They are discussed here only +because this is the minor node on 'getline'. *Note Two-way I/O::, where +coprocesses are discussed in more detail. + + +File: gawk.info, Node: Getline Notes, Next: Getline Summary, Prev: Getline/Variable/Coprocess, Up: Getline + +4.9.9 Points to Remember About 'getline' +---------------------------------------- + +Here are some miscellaneous points about 'getline' that you should bear +in mind: + + * When 'getline' changes the value of '$0' and 'NF', 'awk' does _not_ + automatically jump to the start of the program and start testing + the new record against every pattern. However, the new record is + tested against any subsequent rules. + + * Some very old 'awk' implementations limit the number of pipelines + that an 'awk' program may have open to just one. In 'gawk', there + is no such limit. You can open as many pipelines (and coprocesses) + as the underlying operating system permits. + + * An interesting side effect occurs if you use 'getline' without a + redirection inside a 'BEGIN' rule. Because an unredirected + 'getline' reads from the command-line data files, the first + 'getline' command causes 'awk' to set the value of 'FILENAME'. + Normally, 'FILENAME' does not have a value inside 'BEGIN' rules, + because you have not yet started to process the command-line data + files. (d.c.) (See *note BEGIN/END::; also *note Auto-set::.) + + * Using 'FILENAME' with 'getline' ('getline < FILENAME') is likely to + be a source of confusion. 'awk' opens a separate input stream from + the current input file. However, by not using a variable, '$0' and + 'NF' are still updated. If you're doing this, it's probably by + accident, and you should reconsider what it is you're trying to + accomplish. + + * *note Getline Summary::, presents a table summarizing the 'getline' + variants and which variables they can affect. It is worth noting + that those variants that do not use redirection can cause + 'FILENAME' to be updated if they cause 'awk' to start reading a new + input file. + + * If the variable being assigned is an expression with side effects, + different versions of 'awk' behave differently upon encountering + end-of-file. Some versions don't evaluate the expression; many + versions (including 'gawk') do. Here is an example, courtesy of + Duncan Moore: + + BEGIN { + system("echo 1 > f") + while ((getline a[++c] < "f") > 0) { } + print c + } + + Here, the side effect is the '++c'. Is 'c' incremented if + end-of-file is encountered before the element in 'a' is assigned? + + 'gawk' treats 'getline' like a function call, and evaluates the + expression 'a[++c]' before attempting to read from 'f'. However, + some versions of 'awk' only evaluate the expression once they know + that there is a string value to be assigned. + + +File: gawk.info, Node: Getline Summary, Prev: Getline Notes, Up: Getline + +4.9.10 Summary of 'getline' Variants +------------------------------------ + +*note Table 4.1: table-getline-variants. summarizes the eight variants +of 'getline', listing which predefined variables are set by each one, +and whether the variant is standard or a 'gawk' extension. Note: for +each variant, 'gawk' sets the 'RT' predefined variable. + +Variant Effect 'awk' / 'gawk' +------------------------------------------------------------------------- +'getline' Sets '$0', 'NF', 'FNR', 'awk' + 'NR', and 'RT' +'getline' VAR Sets VAR, 'FNR', 'NR', 'awk' + and 'RT' +'getline <' FILE Sets '$0', 'NF', and 'RT' 'awk' +'getline VAR < FILE' Sets VAR and 'RT' 'awk' +COMMAND '| getline' Sets '$0', 'NF', and 'RT' 'awk' +COMMAND '| getline' Sets VAR and 'RT' 'awk' +VAR +COMMAND '|& getline' Sets '$0', 'NF', and 'RT' 'gawk' +COMMAND '|& getline' Sets VAR and 'RT' 'gawk' +VAR + +Table 4.1: 'getline' variants and what they set + + +File: gawk.info, Node: Read Timeout, Next: Retrying Input, Prev: Getline, Up: Reading Files + +4.10 Reading Input with a Timeout +================================= + +This minor node describes a feature that is specific to 'gawk'. + + You may specify a timeout in milliseconds for reading input from the +keyboard, a pipe, or two-way communication, including TCP/IP sockets. +This can be done on a per-input, per-command, or per-connection basis, +by setting a special element in the 'PROCINFO' array (*note Auto-set::): + + PROCINFO["input_name", "READ_TIMEOUT"] = TIMEOUT IN MILLISECONDS + + When set, this causes 'gawk' to time out and return failure if no +data is available to read within the specified timeout period. For +example, a TCP client can decide to give up on receiving any response +from the server after a certain amount of time: + + Service = "/inet/tcp/0/localhost/daytime" + PROCINFO[Service, "READ_TIMEOUT"] = 100 + if ((Service |& getline) > 0) + print $0 + else if (ERRNO != "") + print ERRNO + + Here is how to read interactively from the user(1) without waiting +for more than five seconds: + + PROCINFO["/dev/stdin", "READ_TIMEOUT"] = 5000 + while ((getline < "/dev/stdin") > 0) + print $0 + + 'gawk' terminates the read operation if input does not arrive after +waiting for the timeout period, returns failure, and sets 'ERRNO' to an +appropriate string value. A negative or zero value for the timeout is +the same as specifying no timeout at all. + + A timeout can also be set for reading from the keyboard in the +implicit loop that reads input records and matches them against +patterns, like so: + + $ gawk 'BEGIN { PROCINFO["-", "READ_TIMEOUT"] = 5000 } + > { print "You entered: " $0 }' + gawk + -| You entered: gawk + + In this case, failure to respond within five seconds results in the +following error message: + + error-> gawk: cmd. line:2: (FILENAME=- FNR=1) fatal: error reading input file `-': Connection timed out + + The timeout can be set or changed at any time, and will take effect +on the next attempt to read from the input device. In the following +example, we start with a timeout value of one second, and progressively +reduce it by one-tenth of a second until we wait indefinitely for the +input to arrive: + + PROCINFO[Service, "READ_TIMEOUT"] = 1000 + while ((Service |& getline) > 0) { + print $0 + PROCINFO[Service, "READ_TIMEOUT"] -= 100 + } + + NOTE: You should not assume that the read operation will block + exactly after the tenth record has been printed. It is possible + that 'gawk' will read and buffer more than one record's worth of + data the first time. Because of this, changing the value of + timeout like in the preceding example is not very useful. + + If the 'PROCINFO' element is not present and the 'GAWK_READ_TIMEOUT' +environment variable exists, 'gawk' uses its value to initialize the +timeout value. The exclusive use of the environment variable to specify +timeout has the disadvantage of not being able to control it on a +per-command or per-connection basis. + + 'gawk' considers a timeout event to be an error even though the +attempt to read from the underlying device may succeed in a later +attempt. This is a limitation, and it also means that you cannot use +this to multiplex input from two or more sources. *Note Retrying +Input:: for a way to enable later I/O attempts to succeed. + + Assigning a timeout value prevents read operations from blocking +indefinitely. But bear in mind that there are other ways 'gawk' can +stall waiting for an input device to be ready. A network client can +sometimes take a long time to establish a connection before it can start +reading any data, or the attempt to open a FIFO special file for reading +can block indefinitely until some other process opens it for writing. + + ---------- Footnotes ---------- + + (1) This assumes that standard input is the keyboard. + + +File: gawk.info, Node: Retrying Input, Next: Command-line directories, Prev: Read Timeout, Up: Reading Files + +4.11 Retrying Reads After Certain Input Errors +============================================== + +This minor node describes a feature that is specific to 'gawk'. + + When 'gawk' encounters an error while reading input, by default +'getline' returns -1, and subsequent attempts to read from that file +result in an end-of-file indication. However, you may optionally +instruct 'gawk' to allow I/O to be retried when certain errors are +encountered by setting a special element in the 'PROCINFO' array (*note +Auto-set::): + + PROCINFO["INPUT_NAME", "RETRY"] = 1 + + When this element exists, 'gawk' checks the value of the system (C +language) 'errno' variable when an I/O error occurs. If 'errno' +indicates a subsequent I/O attempt may succeed, 'getline' instead +returns -2 and further calls to 'getline' may succeed. This applies to +the 'errno' values 'EAGAIN', 'EWOULDBLOCK', 'EINTR', or 'ETIMEDOUT'. + + This feature is useful in conjunction with 'PROCINFO["INPUT_NAME", +"READ_TIMEOUT"]' or situations where a file descriptor has been +configured to behave in a non-blocking fashion. + + +File: gawk.info, Node: Command-line directories, Next: Input Summary, Prev: Retrying Input, Up: Reading Files + +4.12 Directories on the Command Line +==================================== + +According to the POSIX standard, files named on the 'awk' command line +must be text files; it is a fatal error if they are not. Most versions +of 'awk' treat a directory on the command line as a fatal error. + + By default, 'gawk' produces a warning for a directory on the command +line, but otherwise ignores it. This makes it easier to use shell +wildcards with your 'awk' program: + + $ gawk -f whizprog.awk * Directories could kill this program + + If either of the '--posix' or '--traditional' options is given, then +'gawk' reverts to treating a directory on the command line as a fatal +error. + + *Note Extension Sample Readdir:: for a way to treat directories as +usable data from an 'awk' program. + + +File: gawk.info, Node: Input Summary, Next: Input Exercises, Prev: Command-line directories, Up: Reading Files + +4.13 Summary +============ + + * Input is split into records based on the value of 'RS'. The + possibilities are as follows: + + Value of 'RS' Records are split on 'awk' / 'gawk' + ... + --------------------------------------------------------------------------- + Any single That character 'awk' + character + The empty string Runs of two or more 'awk' + ('""') newlines + A regexp Text that matches the 'gawk' + regexp + + * 'FNR' indicates how many records have been read from the current + input file; 'NR' indicates how many records have been read in + total. + + * 'gawk' sets 'RT' to the text matched by 'RS'. + + * After splitting the input into records, 'awk' further splits the + records into individual fields, named '$1', '$2', and so on. '$0' + is the whole record, and 'NF' indicates how many fields there are. + The default way to split fields is between whitespace characters. + + * Fields may be referenced using a variable, as in '$NF'. Fields may + also be assigned values, which causes the value of '$0' to be + recomputed when it is later referenced. Assigning to a field with + a number greater than 'NF' creates the field and rebuilds the + record, using 'OFS' to separate the fields. Incrementing 'NF' does + the same thing. Decrementing 'NF' throws away fields and rebuilds + the record. + + * Field splitting is more complicated than record splitting: + + Field separator value Fields are split ... 'awk' / + 'gawk' + --------------------------------------------------------------------------- + 'FS == " "' On runs of whitespace 'awk' + 'FS == ANY SINGLE On that character 'awk' + CHARACTER' + 'FS == REGEXP' On text matching the regexp 'awk' + 'FS == ""' Such that each individual 'gawk' + character is a separate + field + 'FIELDWIDTHS == LIST OF Based on character position 'gawk' + COLUMNS' + 'FPAT == REGEXP' On the text surrounding 'gawk' + text matching the regexp + + * Using 'FS = "\n"' causes the entire record to be a single field + (assuming that newlines separate records). + + * 'FS' may be set from the command line using the '-F' option. This + can also be done using command-line variable assignment. + + * Use 'PROCINFO["FS"]' to see how fields are being split. + + * Use 'getline' in its various forms to read additional records from + the default input stream, from a file, or from a pipe or coprocess. + + * Use 'PROCINFO[FILE, "READ_TIMEOUT"]' to cause reads to time out for + FILE. + + * Directories on the command line are fatal for standard 'awk'; + 'gawk' ignores them if not in POSIX mode. + + +File: gawk.info, Node: Input Exercises, Prev: Input Summary, Up: Reading Files + +4.14 Exercises +============== + + 1. Using the 'FIELDWIDTHS' variable (*note Constant Size::), write a + program to read election data, where each record represents one + voter's votes. Come up with a way to define which columns are + associated with each ballot item, and print the total votes, + including abstentions, for each item. + + 2. *note Plain Getline::, presented a program to remove C-style + comments ('/* ... */') from the input. That program does not work + if one comment ends on one line and another one starts later on the + same line. That can be fixed by making one simple change. What is + it? + + +File: gawk.info, Node: Printing, Next: Expressions, Prev: Reading Files, Up: Top + +5 Printing Output +***************** + +One of the most common programming actions is to "print", or output, +some or all of the input. Use the 'print' statement for simple output, +and the 'printf' statement for fancier formatting. The 'print' +statement is not limited when computing _which_ values to print. +However, with two exceptions, you cannot specify _how_ to print +them--how many columns, whether to use exponential notation or not, and +so on. (For the exceptions, *note Output Separators:: and *note +OFMT::.) For printing with specifications, you need the 'printf' +statement (*note Printf::). + + Besides basic and formatted printing, this major node also covers I/O +redirections to files and pipes, introduces the special file names that +'gawk' processes internally, and discusses the 'close()' built-in +function. + +* Menu: + +* Print:: The 'print' statement. +* Print Examples:: Simple examples of 'print' statements. +* Output Separators:: The output separators and how to change them. +* OFMT:: Controlling Numeric Output With 'print'. +* Printf:: The 'printf' statement. +* Redirection:: How to redirect output to multiple files and + pipes. +* Special FD:: Special files for I/O. +* Special Files:: File name interpretation in 'gawk'. + 'gawk' allows access to inherited file + descriptors. +* Close Files And Pipes:: Closing Input and Output Files and Pipes. +* Nonfatal:: Enabling Nonfatal Output. +* Output Summary:: Output summary. +* Output Exercises:: Exercises. + + +File: gawk.info, Node: Print, Next: Print Examples, Up: Printing + +5.1 The 'print' Statement +========================= + +Use the 'print' statement to produce output with simple, standardized +formatting. You specify only the strings or numbers to print, in a list +separated by commas. They are output, separated by single spaces, +followed by a newline. The statement looks like this: + + print ITEM1, ITEM2, ... + +The entire list of items may be optionally enclosed in parentheses. The +parentheses are necessary if any of the item expressions uses the '>' +relational operator; otherwise it could be confused with an output +redirection (*note Redirection::). + + The items to print can be constant strings or numbers, fields of the +current record (such as '$1'), variables, or any 'awk' expression. +Numeric values are converted to strings and then printed. + + The simple statement 'print' with no items is equivalent to 'print +$0': it prints the entire current record. To print a blank line, use +'print ""'. To print a fixed piece of text, use a string constant, such +as '"Don't Panic"', as one item. If you forget to use the double-quote +characters, your text is taken as an 'awk' expression, and you will +probably get an error. Keep in mind that a space is printed between any +two items. + + Note that the 'print' statement is a statement and not an +expression--you can't use it in the pattern part of a pattern-action +statement, for example. + + +File: gawk.info, Node: Print Examples, Next: Output Separators, Prev: Print, Up: Printing + +5.2 'print' Statement Examples +============================== + +Each 'print' statement makes at least one line of output. However, it +isn't limited to only one line. If an item value is a string containing +a newline, the newline is output along with the rest of the string. A +single 'print' statement can make any number of lines this way. + + The following is an example of printing a string that contains +embedded newlines (the '\n' is an escape sequence, used to represent the +newline character; *note Escape Sequences::): + + $ awk 'BEGIN { print "line one\nline two\nline three" }' + -| line one + -| line two + -| line three + + The next example, which is run on the 'inventory-shipped' file, +prints the first two fields of each input record, with a space between +them: + + $ awk '{ print $1, $2 }' inventory-shipped + -| Jan 13 + -| Feb 15 + -| Mar 15 + ... + + A common mistake in using the 'print' statement is to omit the comma +between two items. This often has the effect of making the items run +together in the output, with no space. The reason for this is that +juxtaposing two string expressions in 'awk' means to concatenate them. +Here is the same program, without the comma: + + $ awk '{ print $1 $2 }' inventory-shipped + -| Jan13 + -| Feb15 + -| Mar15 + ... + + To someone unfamiliar with the 'inventory-shipped' file, neither +example's output makes much sense. A heading line at the beginning +would make it clearer. Let's add some headings to our table of months +('$1') and green crates shipped ('$2'). We do this using a 'BEGIN' rule +(*note BEGIN/END::) so that the headings are only printed once: + + awk 'BEGIN { print "Month Crates" + print "----- ------" } + { print $1, $2 }' inventory-shipped + +When run, the program prints the following: + + Month Crates + ----- ------ + Jan 13 + Feb 15 + Mar 15 + ... + +The only problem, however, is that the headings and the table data don't +line up! We can fix this by printing some spaces between the two +fields: + + awk 'BEGIN { print "Month Crates" + print "----- ------" } + { print $1, " ", $2 }' inventory-shipped + + Lining up columns this way can get pretty complicated when there are +many columns to fix. Counting spaces for two or three columns is +simple, but any more than this can take up a lot of time. This is why +the 'printf' statement was created (*note Printf::); one of its +specialties is lining up columns of data. + + NOTE: You can continue either a 'print' or 'printf' statement + simply by putting a newline after any comma (*note + Statements/Lines::). + + +File: gawk.info, Node: Output Separators, Next: OFMT, Prev: Print Examples, Up: Printing + +5.3 Output Separators +===================== + +As mentioned previously, a 'print' statement contains a list of items +separated by commas. In the output, the items are normally separated by +single spaces. However, this doesn't need to be the case; a single +space is simply the default. Any string of characters may be used as +the "output field separator" by setting the predefined variable 'OFS'. +The initial value of this variable is the string '" "' (i.e., a single +space). + + The output from an entire 'print' statement is called an "output +record". Each 'print' statement outputs one output record, and then +outputs a string called the "output record separator" (or 'ORS'). The +initial value of 'ORS' is the string '"\n"' (i.e., a newline character). +Thus, each 'print' statement normally makes a separate line. + + In order to change how output fields and records are separated, +assign new values to the variables 'OFS' and 'ORS'. The usual place to +do this is in the 'BEGIN' rule (*note BEGIN/END::), so that it happens +before any input is processed. It can also be done with assignments on +the command line, before the names of the input files, or using the '-v' +command-line option (*note Options::). The following example prints the +first and second fields of each input record, separated by a semicolon, +with a blank line added after each newline: + + $ awk 'BEGIN { OFS = ";"; ORS = "\n\n" } + > { print $1, $2 }' mail-list + -| Amelia;555-5553 + -| + -| Anthony;555-3412 + -| + -| Becky;555-7685 + -| + -| Bill;555-1675 + -| + -| Broderick;555-0542 + -| + -| Camilla;555-2912 + -| + -| Fabius;555-1234 + -| + -| Julie;555-6699 + -| + -| Martin;555-6480 + -| + -| Samuel;555-3430 + -| + -| Jean-Paul;555-2127 + -| + + If the value of 'ORS' does not contain a newline, the program's +output runs together on a single line. + + +File: gawk.info, Node: OFMT, Next: Printf, Prev: Output Separators, Up: Printing + +5.4 Controlling Numeric Output with 'print' +=========================================== + +When printing numeric values with the 'print' statement, 'awk' +internally converts each number to a string of characters and prints +that string. 'awk' uses the 'sprintf()' function to do this conversion +(*note String Functions::). For now, it suffices to say that the +'sprintf()' function accepts a "format specification" that tells it how +to format numbers (or strings), and that there are a number of different +ways in which numbers can be formatted. The different format +specifications are discussed more fully in *note Control Letters::. + + The predefined variable 'OFMT' contains the format specification that +'print' uses with 'sprintf()' when it wants to convert a number to a +string for printing. The default value of 'OFMT' is '"%.6g"'. The way +'print' prints numbers can be changed by supplying a different format +specification for the value of 'OFMT', as shown in the following +example: + + $ awk 'BEGIN { + > OFMT = "%.0f" # print numbers as integers (rounds) + > print 17.23, 17.54 }' + -| 17 18 + +According to the POSIX standard, 'awk''s behavior is undefined if 'OFMT' +contains anything but a floating-point conversion specification. (d.c.) + + +File: gawk.info, Node: Printf, Next: Redirection, Prev: OFMT, Up: Printing + +5.5 Using 'printf' Statements for Fancier Printing +================================================== + +For more precise control over the output format than what is provided by +'print', use 'printf'. With 'printf' you can specify the width to use +for each item, as well as various formatting choices for numbers (such +as what output base to use, whether to print an exponent, whether to +print a sign, and how many digits to print after the decimal point). + +* Menu: + +* Basic Printf:: Syntax of the 'printf' statement. +* Control Letters:: Format-control letters. +* Format Modifiers:: Format-specification modifiers. +* Printf Examples:: Several examples. + + +File: gawk.info, Node: Basic Printf, Next: Control Letters, Up: Printf + +5.5.1 Introduction to the 'printf' Statement +-------------------------------------------- + +A simple 'printf' statement looks like this: + + printf FORMAT, ITEM1, ITEM2, ... + +As for 'print', the entire list of arguments may optionally be enclosed +in parentheses. Here too, the parentheses are necessary if any of the +item expressions uses the '>' relational operator; otherwise, it can be +confused with an output redirection (*note Redirection::). + + The difference between 'printf' and 'print' is the FORMAT argument. +This is an expression whose value is taken as a string; it specifies how +to output each of the other arguments. It is called the "format +string". + + The format string is very similar to that in the ISO C library +function 'printf()'. Most of FORMAT is text to output verbatim. +Scattered among this text are "format specifiers"--one per item. Each +format specifier says to output the next item in the argument list at +that place in the format. + + The 'printf' statement does not automatically append a newline to its +output. It outputs only what the format string specifies. So if a +newline is needed, you must include one in the format string. The +output separator variables 'OFS' and 'ORS' have no effect on 'printf' +statements. For example: + + $ awk 'BEGIN { + > ORS = "\nOUCH!\n"; OFS = "+" + > msg = "Don\47t Panic!" + > printf "%s\n", msg + > }' + -| Don't Panic! + +Here, neither the '+' nor the 'OUCH!' appears in the output message. + + +File: gawk.info, Node: Control Letters, Next: Format Modifiers, Prev: Basic Printf, Up: Printf + +5.5.2 Format-Control Letters +---------------------------- + +A format specifier starts with the character '%' and ends with a +"format-control letter"--it tells the 'printf' statement how to output +one item. The format-control letter specifies what _kind_ of value to +print. The rest of the format specifier is made up of optional +"modifiers" that control _how_ to print the value, such as the field +width. Here is a list of the format-control letters: + +'%c' + Print a number as a character; thus, 'printf "%c", 65' outputs the + letter 'A'. The output for a string value is the first character + of the string. + + NOTE: The POSIX standard says the first character of a string + is printed. In locales with multibyte characters, 'gawk' + attempts to convert the leading bytes of the string into a + valid wide character and then to print the multibyte encoding + of that character. Similarly, when printing a numeric value, + 'gawk' allows the value to be within the numeric range of + values that can be held in a wide character. If the + conversion to multibyte encoding fails, 'gawk' uses the low + eight bits of the value as the character to print. + + Other 'awk' versions generally restrict themselves to printing + the first byte of a string or to numeric values within the + range of a single byte (0-255). + +'%d', '%i' + Print a decimal integer. The two control letters are equivalent. + (The '%i' specification is for compatibility with ISO C.) + +'%e', '%E' + Print a number in scientific (exponential) notation. For example: + + printf "%4.3e\n", 1950 + + prints '1.950e+03', with a total of four significant figures, three + of which follow the decimal point. (The '4.3' represents two + modifiers, discussed in the next node.) '%E' uses 'E' instead of + 'e' in the output. + +'%f' + Print a number in floating-point notation. For example: + + printf "%4.3f", 1950 + + prints '1950.000', with a total of four significant figures, three + of which follow the decimal point. (The '4.3' represents two + modifiers, discussed in the next node.) + + On systems supporting IEEE 754 floating-point format, values + representing negative infinity are formatted as '-inf' or + '-infinity', and positive infinity as 'inf' or 'infinity'. The + special "not a number" value formats as '-nan' or 'nan' (*note Math + Definitions::). + +'%F' + Like '%f', but the infinity and "not a number" values are spelled + using uppercase letters. + + The '%F' format is a POSIX extension to ISO C; not all systems + support it. On those that don't, 'gawk' uses '%f' instead. + +'%g', '%G' + Print a number in either scientific notation or in floating-point + notation, whichever uses fewer characters; if the result is printed + in scientific notation, '%G' uses 'E' instead of 'e'. + +'%o' + Print an unsigned octal integer (*note Nondecimal-numbers::). + +'%s' + Print a string. + +'%u' + Print an unsigned decimal integer. (This format is of marginal + use, because all numbers in 'awk' are floating point; it is + provided primarily for compatibility with C.) + +'%x', '%X' + Print an unsigned hexadecimal integer; '%X' uses the letters 'A' + through 'F' instead of 'a' through 'f' (*note + Nondecimal-numbers::). + +'%%' + Print a single '%'. This does not consume an argument and it + ignores any modifiers. + + NOTE: When using the integer format-control letters for values that + are outside the range of the widest C integer type, 'gawk' switches + to the '%g' format specifier. If '--lint' is provided on the + command line (*note Options::), 'gawk' warns about this. Other + versions of 'awk' may print invalid values or do something else + entirely. (d.c.) + + +File: gawk.info, Node: Format Modifiers, Next: Printf Examples, Prev: Control Letters, Up: Printf + +5.5.3 Modifiers for 'printf' Formats +------------------------------------ + +A format specification can also include "modifiers" that can control how +much of the item's value is printed, as well as how much space it gets. +The modifiers come between the '%' and the format-control letter. We +use the bullet symbol "*" in the following examples to represent spaces +in the output. Here are the possible modifiers, in the order in which +they may appear: + +'N$' + An integer constant followed by a '$' is a "positional specifier". + Normally, format specifications are applied to arguments in the + order given in the format string. With a positional specifier, the + format specification is applied to a specific argument, instead of + what would be the next argument in the list. Positional specifiers + begin counting with one. Thus: + + printf "%s %s\n", "don't", "panic" + printf "%2$s %1$s\n", "panic", "don't" + + prints the famous friendly message twice. + + At first glance, this feature doesn't seem to be of much use. It + is in fact a 'gawk' extension, intended for use in translating + messages at runtime. *Note Printf Ordering::, which describes how + and why to use positional specifiers. For now, we ignore them. + +'-' (Minus) + The minus sign, used before the width modifier (see later on in + this list), says to left-justify the argument within its specified + width. Normally, the argument is printed right-justified in the + specified width. Thus: + + printf "%-4s", "foo" + + prints 'foo*'. + +SPACE + For numeric conversions, prefix positive values with a space and + negative values with a minus sign. + +'+' + The plus sign, used before the width modifier (see later on in this + list), says to always supply a sign for numeric conversions, even + if the data to format is positive. The '+' overrides the space + modifier. + +'#' + Use an "alternative form" for certain control letters. For '%o', + supply a leading zero. For '%x' and '%X', supply a leading '0x' or + '0X' for a nonzero result. For '%e', '%E', '%f', and '%F', the + result always contains a decimal point. For '%g' and '%G', + trailing zeros are not removed from the result. + +'0' + A leading '0' (zero) acts as a flag indicating that output should + be padded with zeros instead of spaces. This applies only to the + numeric output formats. This flag only has an effect when the + field width is wider than the value to print. + +''' + A single quote or apostrophe character is a POSIX extension to ISO + C. It indicates that the integer part of a floating-point value, or + the entire part of an integer decimal value, should have a + thousands-separator character in it. This only works in locales + that support such characters. For example: + + $ cat thousands.awk Show source program + -| BEGIN { printf "%'d\n", 1234567 } + $ LC_ALL=C gawk -f thousands.awk + -| 1234567 Results in "C" locale + $ LC_ALL=en_US.UTF-8 gawk -f thousands.awk + -| 1,234,567 Results in US English UTF locale + + For more information about locales and internationalization issues, + see *note Locales::. + + NOTE: The ''' flag is a nice feature, but its use complicates + things: it becomes difficult to use it in command-line + programs. For information on appropriate quoting tricks, see + *note Quoting::. + +WIDTH + This is a number specifying the desired minimum width of a field. + Inserting any number between the '%' sign and the format-control + character forces the field to expand to this width. The default + way to do this is to pad with spaces on the left. For example: + + printf "%4s", "foo" + + prints '*foo'. + + The value of WIDTH is a minimum width, not a maximum. If the item + value requires more than WIDTH characters, it can be as wide as + necessary. Thus, the following: + + printf "%4s", "foobar" + + prints 'foobar'. + + Preceding the WIDTH with a minus sign causes the output to be + padded with spaces on the right, instead of on the left. + +'.PREC' + A period followed by an integer constant specifies the precision to + use when printing. The meaning of the precision varies by control + letter: + + '%d', '%i', '%o', '%u', '%x', '%X' + Minimum number of digits to print. + + '%e', '%E', '%f', '%F' + Number of digits to the right of the decimal point. + + '%g', '%G' + Maximum number of significant digits. + + '%s' + Maximum number of characters from the string that should + print. + + Thus, the following: + + printf "%.4s", "foobar" + + prints 'foob'. + + The C library 'printf''s dynamic WIDTH and PREC capability (e.g., +'"%*.*s"') is supported. Instead of supplying explicit WIDTH and/or +PREC values in the format string, they are passed in the argument list. +For example: + + w = 5 + p = 3 + s = "abcdefg" + printf "%*.*s\n", w, p, s + +is exactly equivalent to: + + s = "abcdefg" + printf "%5.3s\n", s + +Both programs output '**abc'. Earlier versions of 'awk' did not support +this capability. If you must use such a version, you may simulate this +feature by using concatenation to build up the format string, like so: + + w = 5 + p = 3 + s = "abcdefg" + printf "%" w "." p "s\n", s + +This is not particularly easy to read, but it does work. + + C programmers may be used to supplying additional modifiers ('h', +'j', 'l', 'L', 't', and 'z') in 'printf' format strings. These are not +valid in 'awk'. Most 'awk' implementations silently ignore them. If +'--lint' is provided on the command line (*note Options::), 'gawk' warns +about their use. If '--posix' is supplied, their use is a fatal error. + + +File: gawk.info, Node: Printf Examples, Prev: Format Modifiers, Up: Printf + +5.5.4 Examples Using 'printf' +----------------------------- + +The following simple example shows how to use 'printf' to make an +aligned table: + + awk '{ printf "%-10s %s\n", $1, $2 }' mail-list + +This command prints the names of the people ('$1') in the file +'mail-list' as a string of 10 characters that are left-justified. It +also prints the phone numbers ('$2') next on the line. This produces an +aligned two-column table of names and phone numbers, as shown here: + + $ awk '{ printf "%-10s %s\n", $1, $2 }' mail-list + -| Amelia 555-5553 + -| Anthony 555-3412 + -| Becky 555-7685 + -| Bill 555-1675 + -| Broderick 555-0542 + -| Camilla 555-2912 + -| Fabius 555-1234 + -| Julie 555-6699 + -| Martin 555-6480 + -| Samuel 555-3430 + -| Jean-Paul 555-2127 + + In this case, the phone numbers had to be printed as strings because +the numbers are separated by dashes. Printing the phone numbers as +numbers would have produced just the first three digits: '555'. This +would have been pretty confusing. + + It wasn't necessary to specify a width for the phone numbers because +they are last on their lines. They don't need to have spaces after +them. + + The table could be made to look even nicer by adding headings to the +tops of the columns. This is done using a 'BEGIN' rule (*note +BEGIN/END::) so that the headers are only printed once, at the beginning +of the 'awk' program: + + awk 'BEGIN { print "Name Number" + print "---- ------" } + { printf "%-10s %s\n", $1, $2 }' mail-list + + The preceding example mixes 'print' and 'printf' statements in the +same program. Using just 'printf' statements can produce the same +results: + + awk 'BEGIN { printf "%-10s %s\n", "Name", "Number" + printf "%-10s %s\n", "----", "------" } + { printf "%-10s %s\n", $1, $2 }' mail-list + +Printing each column heading with the same format specification used for +the column elements ensures that the headings are aligned just like the +columns. + + The fact that the same format specification is used three times can +be emphasized by storing it in a variable, like this: + + awk 'BEGIN { format = "%-10s %s\n" + printf format, "Name", "Number" + printf format, "----", "------" } + { printf format, $1, $2 }' mail-list + + +File: gawk.info, Node: Redirection, Next: Special FD, Prev: Printf, Up: Printing + +5.6 Redirecting Output of 'print' and 'printf' +============================================== + +So far, the output from 'print' and 'printf' has gone to the standard +output, usually the screen. Both 'print' and 'printf' can also send +their output to other places. This is called "redirection". + + NOTE: When '--sandbox' is specified (*note Options::), redirecting + output to files, pipes, and coprocesses is disabled. + + A redirection appears after the 'print' or 'printf' statement. +Redirections in 'awk' are written just like redirections in shell +commands, except that they are written inside the 'awk' program. + + There are four forms of output redirection: output to a file, output +appended to a file, output through a pipe to another command, and output +to a coprocess. We show them all for the 'print' statement, but they +work identically for 'printf': + +'print ITEMS > OUTPUT-FILE' + This redirection prints the items into the output file named + OUTPUT-FILE. The file name OUTPUT-FILE can be any expression. Its + value is changed to a string and then used as a file name (*note + Expressions::). + + When this type of redirection is used, the OUTPUT-FILE is erased + before the first output is written to it. Subsequent writes to the + same OUTPUT-FILE do not erase OUTPUT-FILE, but append to it. (This + is different from how you use redirections in shell scripts.) If + OUTPUT-FILE does not exist, it is created. For example, here is + how an 'awk' program can write a list of peoples' names to one file + named 'name-list', and a list of phone numbers to another file + named 'phone-list': + + $ awk '{ print $2 > "phone-list" + > print $1 > "name-list" }' mail-list + $ cat phone-list + -| 555-5553 + -| 555-3412 + ... + $ cat name-list + -| Amelia + -| Anthony + ... + + Each output file contains one name or number per line. + +'print ITEMS >> OUTPUT-FILE' + This redirection prints the items into the preexisting output file + named OUTPUT-FILE. The difference between this and the single-'>' + redirection is that the old contents (if any) of OUTPUT-FILE are + not erased. Instead, the 'awk' output is appended to the file. If + OUTPUT-FILE does not exist, then it is created. + +'print ITEMS | COMMAND' + It is possible to send output to another program through a pipe + instead of into a file. This redirection opens a pipe to COMMAND, + and writes the values of ITEMS through this pipe to another process + created to execute COMMAND. + + The redirection argument COMMAND is actually an 'awk' expression. + Its value is converted to a string whose contents give the shell + command to be run. For example, the following produces two files, + one unsorted list of peoples' names, and one list sorted in reverse + alphabetical order: + + awk '{ print $1 > "names.unsorted" + command = "sort -r > names.sorted" + print $1 | command }' mail-list + + The unsorted list is written with an ordinary redirection, while + the sorted list is written by piping through the 'sort' utility. + + The next example uses redirection to mail a message to the mailing + list 'bug-system'. This might be useful when trouble is + encountered in an 'awk' script run periodically for system + maintenance: + + report = "mail bug-system" + print("Awk script failed:", $0) | report + print("at record number", FNR, "of", FILENAME) | report + close(report) + + The 'close()' function is called here because it's a good idea to + close the pipe as soon as all the intended output has been sent to + it. *Note Close Files And Pipes:: for more information. + + This example also illustrates the use of a variable to represent a + FILE or COMMAND--it is not necessary to always use a string + constant. Using a variable is generally a good idea, because (if + you mean to refer to that same file or command) 'awk' requires that + the string value be written identically every time. + +'print ITEMS |& COMMAND' + This redirection prints the items to the input of COMMAND. The + difference between this and the single-'|' redirection is that the + output from COMMAND can be read with 'getline'. Thus, COMMAND is a + "coprocess", which works together with but is subsidiary to the + 'awk' program. + + This feature is a 'gawk' extension, and is not available in POSIX + 'awk'. *Note Getline/Coprocess::, for a brief discussion. *Note + Two-way I/O::, for a more complete discussion. + + Redirecting output using '>', '>>', '|', or '|&' asks the system to +open a file, pipe, or coprocess only if the particular FILE or COMMAND +you specify has not already been written to by your program or if it has +been closed since it was last written to. + + It is a common error to use '>' redirection for the first 'print' to +a file, and then to use '>>' for subsequent output: + + # clear the file + print "Don't panic" > "guide.txt" + ... + # append + print "Avoid improbability generators" >> "guide.txt" + +This is indeed how redirections must be used from the shell. But in +'awk', it isn't necessary. In this kind of case, a program should use +'>' for all the 'print' statements, because the output file is only +opened once. (It happens that if you mix '>' and '>>' output is +produced in the expected order. However, mixing the operators for the +same file is definitely poor style, and is confusing to readers of your +program.) + + Many older 'awk' implementations limit the number of pipelines that +an 'awk' program may have open to just one! In 'gawk', there is no such +limit. 'gawk' allows a program to open as many pipelines as the +underlying operating system permits. + + Piping into 'sh' + + A particularly powerful way to use redirection is to build command +lines and pipe them into the shell, 'sh'. For example, suppose you have +a list of files brought over from a system where all the file names are +stored in uppercase, and you wish to rename them to have names in all +lowercase. The following program is both simple and efficient: + + { printf("mv %s %s\n", $0, tolower($0)) | "sh" } + + END { close("sh") } + + The 'tolower()' function returns its argument string with all +uppercase characters converted to lowercase (*note String Functions::). +The program builds up a list of command lines, using the 'mv' utility to +rename the files. It then sends the list to the shell for execution. + + *Note Shell Quoting:: for a function that can help in generating +command lines to be fed to the shell. + + +File: gawk.info, Node: Special FD, Next: Special Files, Prev: Redirection, Up: Printing + +5.7 Special Files for Standard Preopened Data Streams +===================================================== + +Running programs conventionally have three input and output streams +already available to them for reading and writing. These are known as +the "standard input", "standard output", and "standard error output". +These open streams (and any other open files or pipes) are often +referred to by the technical term "file descriptors". + + These streams are, by default, connected to your keyboard and screen, +but they are often redirected with the shell, via the '<', '<<', '>', +'>>', '>&', and '|' operators. Standard error is typically used for +writing error messages; the reason there are two separate streams, +standard output and standard error, is so that they can be redirected +separately. + + In traditional implementations of 'awk', the only way to write an +error message to standard error in an 'awk' program is as follows: + + print "Serious error detected!" | "cat 1>&2" + +This works by opening a pipeline to a shell command that can access the +standard error stream that it inherits from the 'awk' process. This is +far from elegant, and it also requires a separate process. So people +writing 'awk' programs often don't do this. Instead, they send the +error messages to the screen, like this: + + print "Serious error detected!" > "/dev/tty" + +('/dev/tty' is a special file supplied by the operating system that is +connected to your keyboard and screen. It represents the "terminal,"(1) +which on modern systems is a keyboard and screen, not a serial console.) +This generally has the same effect, but not always: although the +standard error stream is usually the screen, it can be redirected; when +that happens, writing to the screen is not correct. In fact, if 'awk' +is run from a background job, it may not have a terminal at all. Then +opening '/dev/tty' fails. + + 'gawk', BWK 'awk', and 'mawk' provide special file names for +accessing the three standard streams. If the file name matches one of +these special names when 'gawk' (or one of the others) redirects input +or output, then it directly uses the descriptor that the file name +stands for. These special file names work for all operating systems +that 'gawk' has been ported to, not just those that are POSIX-compliant: + +'/dev/stdin' + The standard input (file descriptor 0). + +'/dev/stdout' + The standard output (file descriptor 1). + +'/dev/stderr' + The standard error output (file descriptor 2). + + With these facilities, the proper way to write an error message then +becomes: + + print "Serious error detected!" > "/dev/stderr" + + Note the use of quotes around the file name. Like with any other +redirection, the value must be a string. It is a common error to omit +the quotes, which leads to confusing results. + + 'gawk' does not treat these file names as special when in +POSIX-compatibility mode. However, because BWK 'awk' supports them, +'gawk' does support them even when invoked with the '--traditional' +option (*note Options::). + + ---------- Footnotes ---------- + + (1) The "tty" in '/dev/tty' stands for "Teletype," a serial terminal. + + +File: gawk.info, Node: Special Files, Next: Close Files And Pipes, Prev: Special FD, Up: Printing + +5.8 Special File names in 'gawk' +================================ + +Besides access to standard input, standard output, and standard error, +'gawk' provides access to any open file descriptor. Additionally, there +are special file names reserved for TCP/IP networking. + +* Menu: + +* Other Inherited Files:: Accessing other open files with + 'gawk'. +* Special Network:: Special files for network communications. +* Special Caveats:: Things to watch out for. + + +File: gawk.info, Node: Other Inherited Files, Next: Special Network, Up: Special Files + +5.8.1 Accessing Other Open Files with 'gawk' +-------------------------------------------- + +Besides the '/dev/stdin', '/dev/stdout', and '/dev/stderr' special file +names mentioned earlier, 'gawk' provides syntax for accessing any other +inherited open file: + +'/dev/fd/N' + The file associated with file descriptor N. Such a file must be + opened by the program initiating the 'awk' execution (typically the + shell). Unless special pains are taken in the shell from which + 'gawk' is invoked, only descriptors 0, 1, and 2 are available. + + The file names '/dev/stdin', '/dev/stdout', and '/dev/stderr' are +essentially aliases for '/dev/fd/0', '/dev/fd/1', and '/dev/fd/2', +respectively. However, those names are more self-explanatory. + + Note that using 'close()' on a file name of the form '"/dev/fd/N"', +for file descriptor numbers above two, does actually close the given +file descriptor. + + +File: gawk.info, Node: Special Network, Next: Special Caveats, Prev: Other Inherited Files, Up: Special Files + +5.8.2 Special Files for Network Communications +---------------------------------------------- + +'gawk' programs can open a two-way TCP/IP connection, acting as either a +client or a server. This is done using a special file name of the form: + + /NET-TYPE/PROTOCOL/LOCAL-PORT/REMOTE-HOST/REMOTE-PORT + + The NET-TYPE is one of 'inet', 'inet4', or 'inet6'. The PROTOCOL is +one of 'tcp' or 'udp', and the other fields represent the other +essential pieces of information for making a networking connection. +These file names are used with the '|&' operator for communicating with +a coprocess (*note Two-way I/O::). This is an advanced feature, +mentioned here only for completeness. Full discussion is delayed until +*note TCP/IP Networking::. + + +File: gawk.info, Node: Special Caveats, Prev: Special Network, Up: Special Files + +5.8.3 Special File name Caveats +------------------------------- + +Here are some things to bear in mind when using the special file names +that 'gawk' provides: + + * Recognition of the file names for the three standard preopened + files is disabled only in POSIX mode. + + * Recognition of the other special file names is disabled if 'gawk' + is in compatibility mode (either '--traditional' or '--posix'; + *note Options::). + + * 'gawk' _always_ interprets these special file names. For example, + using '/dev/fd/4' for output actually writes on file descriptor 4, + and not on a new file descriptor that is 'dup()'ed from file + descriptor 4. Most of the time this does not matter; however, it + is important to _not_ close any of the files related to file + descriptors 0, 1, and 2. Doing so results in unpredictable + behavior. + + +File: gawk.info, Node: Close Files And Pipes, Next: Nonfatal, Prev: Special Files, Up: Printing + +5.9 Closing Input and Output Redirections +========================================= + +If the same file name or the same shell command is used with 'getline' +more than once during the execution of an 'awk' program (*note +Getline::), the file is opened (or the command is executed) the first +time only. At that time, the first record of input is read from that +file or command. The next time the same file or command is used with +'getline', another record is read from it, and so on. + + Similarly, when a file or pipe is opened for output, 'awk' remembers +the file name or command associated with it, and subsequent writes to +the same file or command are appended to the previous writes. The file +or pipe stays open until 'awk' exits. + + This implies that special steps are necessary in order to read the +same file again from the beginning, or to rerun a shell command (rather +than reading more output from the same command). The 'close()' function +makes these things possible: + + close(FILENAME) + +or: + + close(COMMAND) + + The argument FILENAME or COMMAND can be any expression. Its value +must _exactly_ match the string that was used to open the file or start +the command (spaces and other "irrelevant" characters included). For +example, if you open a pipe with this: + + "sort -r names" | getline foo + +then you must close it with this: + + close("sort -r names") + + Once this function call is executed, the next 'getline' from that +file or command, or the next 'print' or 'printf' to that file or +command, reopens the file or reruns the command. Because the expression +that you use to close a file or pipeline must exactly match the +expression used to open the file or run the command, it is good practice +to use a variable to store the file name or command. The previous +example becomes the following: + + sortcom = "sort -r names" + sortcom | getline foo + ... + close(sortcom) + +This helps avoid hard-to-find typographical errors in your 'awk' +programs. Here are some of the reasons for closing an output file: + + * To write a file and read it back later on in the same 'awk' + program. Close the file after writing it, then begin reading it + with 'getline'. + + * To write numerous files, successively, in the same 'awk' program. + If the files aren't closed, eventually 'awk' may exceed a system + limit on the number of open files in one process. It is best to + close each one when the program has finished writing it. + + * To make a command finish. When output is redirected through a + pipe, the command reading the pipe normally continues to try to + read input as long as the pipe is open. Often this means the + command cannot really do its work until the pipe is closed. For + example, if output is redirected to the 'mail' program, the message + is not actually sent until the pipe is closed. + + * To run the same program a second time, with the same arguments. + This is not the same thing as giving more input to the first run! + + For example, suppose a program pipes output to the 'mail' program. + If it outputs several lines redirected to this pipe without closing + it, they make a single message of several lines. By contrast, if + the program closes the pipe after each line of output, then each + line makes a separate message. + + If you use more files than the system allows you to have open, 'gawk' +attempts to multiplex the available open files among your data files. +'gawk''s ability to do this depends upon the facilities of your +operating system, so it may not always work. It is therefore both good +practice and good portability advice to always use 'close()' on your +files when you are done with them. In fact, if you are using a lot of +pipes, it is essential that you close commands when done. For example, +consider something like this: + + { + ... + command = ("grep " $1 " /some/file | my_prog -q " $3) + while ((command | getline) > 0) { + PROCESS OUTPUT OF command + } + # need close(command) here + } + + This example creates a new pipeline based on data in _each_ record. +Without the call to 'close()' indicated in the comment, 'awk' creates +child processes to run the commands, until it eventually runs out of +file descriptors for more pipelines. + + Even though each command has finished (as indicated by the +end-of-file return status from 'getline'), the child process is not +terminated;(1) more importantly, the file descriptor for the pipe is not +closed and released until 'close()' is called or 'awk' exits. + + 'close()' silently does nothing if given an argument that does not +represent a file, pipe, or coprocess that was opened with a redirection. +In such a case, it returns a negative value, indicating an error. In +addition, 'gawk' sets 'ERRNO' to a string indicating the error. + + Note also that 'close(FILENAME)' has no "magic" effects on the +implicit loop that reads through the files named on the command line. +It is, more likely, a close of a file that was never opened with a +redirection, so 'awk' silently does nothing, except return a negative +value. + + When using the '|&' operator to communicate with a coprocess, it is +occasionally useful to be able to close one end of the two-way pipe +without closing the other. This is done by supplying a second argument +to 'close()'. As in any other call to 'close()', the first argument is +the name of the command or special file used to start the coprocess. +The second argument should be a string, with either of the values '"to"' +or '"from"'. Case does not matter. As this is an advanced feature, +discussion is delayed until *note Two-way I/O::, which describes it in +more detail and gives an example. + + Using 'close()''s Return Value + + In many older versions of Unix 'awk', the 'close()' function is +actually a statement. (d.c.) It is a syntax error to try and use the +return value from 'close()': + + command = "..." + command | getline info + retval = close(command) # syntax error in many Unix awks + + 'gawk' treats 'close()' as a function. The return value is -1 if the +argument names something that was never opened with a redirection, or if +there is a system problem closing the file or process. In these cases, +'gawk' sets the predefined variable 'ERRNO' to a string describing the +problem. + + In 'gawk', starting with version 4.2, when closing a pipe or +coprocess (input or output), the return value is the exit status of the +command, as described in *note Table 5.1: +table-close-pipe-return-values.(2) Otherwise, it is the return value +from the system's 'close()' or 'fclose()' C functions when closing input +or output files, respectively. This value is zero if the close +succeeds, or -1 if it fails. + +Situation Return value from 'close()' +-------------------------------------------------------------------------- +Normal exit of command Command's exit status +Death by signal of command 256 + number of murderous signal +Death by signal of command 512 + number of murderous signal +with core dump +Some kind of error -1 + +Table 5.1: Return values from 'close()' of a pipe + + The POSIX standard is very vague; it says that 'close()' returns zero +on success and a nonzero value otherwise. In general, different +implementations vary in what they report when closing pipes; thus, the +return value cannot be used portably. (d.c.) In POSIX mode (*note +Options::), 'gawk' just returns zero when closing a pipe. + + ---------- Footnotes ---------- + + (1) The technical terminology is rather morbid. The finished child +is called a "zombie," and cleaning up after it is referred to as +"reaping." + + (2) Prior to version 4.2, the return value from closing a pipe or +co-process was the full 16-bit exit value as defined by the 'wait()' +system call. + + +File: gawk.info, Node: Nonfatal, Next: Output Summary, Prev: Close Files And Pipes, Up: Printing + +5.10 Enabling Nonfatal Output +============================= + +This minor node describes a 'gawk'-specific feature. + + In standard 'awk', output with 'print' or 'printf' to a nonexistent +file, or some other I/O error (such as filling up the disk) is a fatal +error. + + $ gawk 'BEGIN { print "hi" > "/no/such/file" }' + error-> gawk: cmd. line:1: fatal: can't redirect to `/no/such/file' (No such file or directory) + + 'gawk' makes it possible to detect that an error has occurred, +allowing you to possibly recover from the error, or at least print an +error message of your choosing before exiting. You can do this in one +of two ways: + + * For all output files, by assigning any value to + 'PROCINFO["NONFATAL"]'. + + * On a per-file basis, by assigning any value to 'PROCINFO[FILENAME, + "NONFATAL"]'. Here, FILENAME is the name of the file to which you + wish output to be nonfatal. + + Once you have enabled nonfatal output, you must check 'ERRNO' after +every relevant 'print' or 'printf' statement to see if something went +wrong. It is also a good idea to initialize 'ERRNO' to zero before +attempting the output. For example: + + $ gawk ' + > BEGIN { + > PROCINFO["NONFATAL"] = 1 + > ERRNO = 0 + > print "hi" > "/no/such/file" + > if (ERRNO) { + > print("Output failed:", ERRNO) > "/dev/stderr" + > exit 1 + > } + > }' + error-> Output failed: No such file or directory + + Here, 'gawk' did not produce a fatal error; instead it let the 'awk' +program code detect the problem and handle it. + + This mechanism works also for standard output and standard error. +For standard output, you may use 'PROCINFO["-", "NONFATAL"]' or +'PROCINFO["/dev/stdout", "NONFATAL"]'. For standard error, use +'PROCINFO["/dev/stderr", "NONFATAL"]'. + + When attempting to open a TCP/IP socket (*note TCP/IP Networking::), +'gawk' tries multiple times. The 'GAWK_SOCK_RETRIES' environment +variable (*note Other Environment Variables::) allows you to override +'gawk''s builtin default number of attempts. However, once nonfatal I/O +is enabled for a given socket, 'gawk' only retries once, relying on +'awk'-level code to notice that there was a problem. + + +File: gawk.info, Node: Output Summary, Next: Output Exercises, Prev: Nonfatal, Up: Printing + +5.11 Summary +============ + + * The 'print' statement prints comma-separated expressions. Each + expression is separated by the value of 'OFS' and terminated by the + value of 'ORS'. 'OFMT' provides the conversion format for numeric + values for the 'print' statement. + + * The 'printf' statement provides finer-grained control over output, + with format-control letters for different data types and various + flags that modify the behavior of the format-control letters. + + * Output from both 'print' and 'printf' may be redirected to files, + pipes, and coprocesses. + + * 'gawk' provides special file names for access to standard input, + output, and error, and for network communications. + + * Use 'close()' to close open file, pipe, and coprocess redirections. + For coprocesses, it is possible to close only one direction of the + communications. + + * Normally errors with 'print' or 'printf' are fatal. 'gawk' lets + you make output errors be nonfatal either for all files or on a + per-file basis. You must then check for errors after every + relevant output statement. + + +File: gawk.info, Node: Output Exercises, Prev: Output Summary, Up: Printing + +5.12 Exercises +============== + + 1. Rewrite the program: + + awk 'BEGIN { print "Month Crates" + print "----- ------" } + { print $1, " ", $2 }' inventory-shipped + + from *note Output Separators::, by using a new value of 'OFS'. + + 2. Use the 'printf' statement to line up the headings and table data + for the 'inventory-shipped' example that was covered in *note + Print::. + + 3. What happens if you forget the double quotes when redirecting + output, as follows: + + BEGIN { print "Serious error detected!" > /dev/stderr } + + +File: gawk.info, Node: Expressions, Next: Patterns and Actions, Prev: Printing, Up: Top + +6 Expressions +************* + +Expressions are the basic building blocks of 'awk' patterns and actions. +An expression evaluates to a value that you can print, test, or pass to +a function. Additionally, an expression can assign a new value to a +variable or a field by using an assignment operator. + + An expression can serve as a pattern or action statement on its own. +Most other kinds of statements contain one or more expressions that +specify the data on which to operate. As in other languages, +expressions in 'awk' can include variables, array references, constants, +and function calls, as well as combinations of these with various +operators. + +* Menu: + +* Values:: Constants, Variables, and Regular Expressions. +* All Operators:: 'gawk''s operators. +* Truth Values and Conditions:: Testing for true and false. +* Function Calls:: A function call is an expression. +* Precedence:: How various operators nest. +* Locales:: How the locale affects things. +* Expressions Summary:: Expressions summary. + + +File: gawk.info, Node: Values, Next: All Operators, Up: Expressions + +6.1 Constants, Variables, and Conversions +========================================= + +Expressions are built up from values and the operations performed upon +them. This minor node describes the elementary objects that provide the +values used in expressions. + +* Menu: + +* Constants:: String, numeric and regexp constants. +* Using Constant Regexps:: When and how to use a regexp constant. +* Variables:: Variables give names to values for later use. +* Conversion:: The conversion of strings to numbers and vice + versa. + + +File: gawk.info, Node: Constants, Next: Using Constant Regexps, Up: Values + +6.1.1 Constant Expressions +-------------------------- + +The simplest type of expression is the "constant", which always has the +same value. There are three types of constants: numeric, string, and +regular expression. + + Each is used in the appropriate context when you need a data value +that isn't going to change. Numeric constants can have different forms, +but are internally stored in an identical manner. + +* Menu: + +* Scalar Constants:: Numeric and string constants. +* Nondecimal-numbers:: What are octal and hex numbers. +* Regexp Constants:: Regular Expression constants. + + +File: gawk.info, Node: Scalar Constants, Next: Nondecimal-numbers, Up: Constants + +6.1.1.1 Numeric and String Constants +.................................... + +A "numeric constant" stands for a number. This number can be an +integer, a decimal fraction, or a number in scientific (exponential) +notation.(1) Here are some examples of numeric constants that all have +the same value: + + 105 + 1.05e+2 + 1050e-1 + + A "string constant" consists of a sequence of characters enclosed in +double quotation marks. For example: + + "parrot" + +represents the string whose contents are 'parrot'. Strings in 'gawk' +can be of any length, and they can contain any of the possible eight-bit +ASCII characters, including ASCII NUL (character code zero). Other +'awk' implementations may have difficulty with some character codes. + + ---------- Footnotes ---------- + + (1) The internal representation of all numbers, including integers, +uses double-precision floating-point numbers. On most modern systems, +these are in IEEE 754 standard format. *Note Arbitrary Precision +Arithmetic::, for much more information. + + +File: gawk.info, Node: Nondecimal-numbers, Next: Regexp Constants, Prev: Scalar Constants, Up: Constants + +6.1.1.2 Octal and Hexadecimal Numbers +..................................... + +In 'awk', all numbers are in decimal (i.e., base 10). Many other +programming languages allow you to specify numbers in other bases, often +octal (base 8) and hexadecimal (base 16). In octal, the numbers go 0, +1, 2, 3, 4, 5, 6, 7, 10, 11, 12, and so on. Just as '11' in decimal is +1 times 10 plus 1, so '11' in octal is 1 times 8 plus 1. This equals 9 +in decimal. In hexadecimal, there are 16 digits. Because the everyday +decimal number system only has ten digits ('0'-'9'), the letters 'a' +through 'f' are used to represent the rest. (Case in the letters is +usually irrelevant; hexadecimal 'a' and 'A' have the same value.) Thus, +'11' in hexadecimal is 1 times 16 plus 1, which equals 17 in decimal. + + Just by looking at plain '11', you can't tell what base it's in. So, +in C, C++, and other languages derived from C, there is a special +notation to signify the base. Octal numbers start with a leading '0', +and hexadecimal numbers start with a leading '0x' or '0X': + +'11' + Decimal value 11 + +'011' + Octal 11, decimal value 9 + +'0x11' + Hexadecimal 11, decimal value 17 + + This example shows the difference: + + $ gawk 'BEGIN { printf "%d, %d, %d\n", 011, 11, 0x11 }' + -| 9, 11, 17 + + Being able to use octal and hexadecimal constants in your programs is +most useful when working with data that cannot be represented +conveniently as characters or as regular numbers, such as binary data of +various sorts. + + 'gawk' allows the use of octal and hexadecimal constants in your +program text. However, such numbers in the input data are not treated +differently; doing so by default would break old programs. (If you +really need to do this, use the '--non-decimal-data' command-line +option; *note Nondecimal Data::.) If you have octal or hexadecimal +data, you can use the 'strtonum()' function (*note String Functions::) +to convert the data into a number. Most of the time, you will want to +use octal or hexadecimal constants when working with the built-in +bit-manipulation functions; see *note Bitwise Functions:: for more +information. + + Unlike in some early C implementations, '8' and '9' are not valid in +octal constants. For example, 'gawk' treats '018' as decimal 18: + + $ gawk 'BEGIN { print "021 is", 021 ; print 018 }' + -| 021 is 17 + -| 18 + + Octal and hexadecimal source code constants are a 'gawk' extension. +If 'gawk' is in compatibility mode (*note Options::), they are not +available. + + A Constant's Base Does Not Affect Its Value + + Once a numeric constant has been converted internally into a number, +'gawk' no longer remembers what the original form of the constant was; +the internal value is always used. This has particular consequences for +conversion of numbers to strings: + + $ gawk 'BEGIN { printf "0x11 is <%s>\n", 0x11 }' + -| 0x11 is <17> + + +File: gawk.info, Node: Regexp Constants, Prev: Nondecimal-numbers, Up: Constants + +6.1.1.3 Regular Expression Constants +.................................... + +A "regexp constant" is a regular expression description enclosed in +slashes, such as '/^beginning and end$/'. Most regexps used in 'awk' +programs are constant, but the '~' and '!~' matching operators can also +match computed or dynamic regexps (which are typically just ordinary +strings or variables that contain a regexp, but could be more complex +expressions). + + +File: gawk.info, Node: Using Constant Regexps, Next: Variables, Prev: Constants, Up: Values + +6.1.2 Using Regular Expression Constants +---------------------------------------- + +When used on the righthand side of the '~' or '!~' operators, a regexp +constant merely stands for the regexp that is to be matched. However, +regexp constants (such as '/foo/') may be used like simple expressions. +When a regexp constant appears by itself, it has the same meaning as if +it appeared in a pattern (i.e., '($0 ~ /foo/)'). (d.c.) *Note +Expression Patterns::. This means that the following two code segments: + + if ($0 ~ /barfly/ || $0 ~ /camelot/) + print "found" + +and: + + if (/barfly/ || /camelot/) + print "found" + +are exactly equivalent. One rather bizarre consequence of this rule is +that the following Boolean expression is valid, but does not do what its +author probably intended: + + # Note that /foo/ is on the left of the ~ + if (/foo/ ~ $1) print "found foo" + +This code is "obviously" testing '$1' for a match against the regexp +'/foo/'. But in fact, the expression '/foo/ ~ $1' really means '($0 ~ +/foo/) ~ $1'. In other words, first match the input record against the +regexp '/foo/'. The result is either zero or one, depending upon the +success or failure of the match. That result is then matched against +the first field in the record. Because it is unlikely that you would +ever really want to make this kind of test, 'gawk' issues a warning when +it sees this construct in a program. Another consequence of this rule +is that the assignment statement: + + matches = /foo/ + +assigns either zero or one to the variable 'matches', depending upon the +contents of the current input record. + + Constant regular expressions are also used as the first argument for +the 'gensub()', 'sub()', and 'gsub()' functions, as the second argument +of the 'match()' function, and as the third argument of the 'split()' +and 'patsplit()' functions (*note String Functions::). Modern +implementations of 'awk', including 'gawk', allow the third argument of +'split()' to be a regexp constant, but some older implementations do +not. (d.c.) Because some built-in functions accept regexp constants as +arguments, confusion can arise when attempting to use regexp constants +as arguments to user-defined functions (*note User-defined::). For +example: + + function mysub(pat, repl, str, global) + { + if (global) + gsub(pat, repl, str) + else + sub(pat, repl, str) + return str + } + + { + ... + text = "hi! hi yourself!" + mysub(/hi/, "howdy", text, 1) + ... + } + + In this example, the programmer wants to pass a regexp constant to +the user-defined function 'mysub()', which in turn passes it on to +either 'sub()' or 'gsub()'. However, what really happens is that the +'pat' parameter is assigned a value of either one or zero, depending +upon whether or not '$0' matches '/hi/'. 'gawk' issues a warning when +it sees a regexp constant used as a parameter to a user-defined +function, because passing a truth value in this way is probably not what +was intended. + + +File: gawk.info, Node: Variables, Next: Conversion, Prev: Using Constant Regexps, Up: Values + +6.1.3 Variables +--------------- + +"Variables" are ways of storing values at one point in your program for +use later in another part of your program. They can be manipulated +entirely within the program text, and they can also be assigned values +on the 'awk' command line. + +* Menu: + +* Using Variables:: Using variables in your programs. +* Assignment Options:: Setting variables on the command line and a + summary of command-line syntax. This is an + advanced method of input. + + +File: gawk.info, Node: Using Variables, Next: Assignment Options, Up: Variables + +6.1.3.1 Using Variables in a Program +.................................... + +Variables let you give names to values and refer to them later. +Variables have already been used in many of the examples. The name of a +variable must be a sequence of letters, digits, or underscores, and it +may not begin with a digit. Here, a "letter" is any one of the 52 +upper- and lowercase English letters. Other characters that may be +defined as letters in non-English locales are not valid in variable +names. Case is significant in variable names; 'a' and 'A' are distinct +variables. + + A variable name is a valid expression by itself; it represents the +variable's current value. Variables are given new values with +"assignment operators", "increment operators", and "decrement operators" +(*note Assignment Ops::). In addition, the 'sub()' and 'gsub()' +functions can change a variable's value, and the 'match()', 'split()', +and 'patsplit()' functions can change the contents of their array +parameters (*note String Functions::). + + A few variables have special built-in meanings, such as 'FS' (the +field separator) and 'NF' (the number of fields in the current input +record). *Note Built-in Variables:: for a list of the predefined +variables. These predefined variables can be used and assigned just +like all other variables, but their values are also used or changed +automatically by 'awk'. All predefined variables' names are entirely +uppercase. + + Variables in 'awk' can be assigned either numeric or string values. +The kind of value a variable holds can change over the life of a +program. By default, variables are initialized to the empty string, +which is zero if converted to a number. There is no need to explicitly +initialize a variable in 'awk', which is what you would do in C and in +most other traditional languages. + + +File: gawk.info, Node: Assignment Options, Prev: Using Variables, Up: Variables + +6.1.3.2 Assigning Variables on the Command Line +............................................... + +Any 'awk' variable can be set by including a "variable assignment" among +the arguments on the command line when 'awk' is invoked (*note Other +Arguments::). Such an assignment has the following form: + + VARIABLE=TEXT + +With it, a variable is set either at the beginning of the 'awk' run or +in between input files. When the assignment is preceded with the '-v' +option, as in the following: + + -v VARIABLE=TEXT + +the variable is set at the very beginning, even before the 'BEGIN' rules +execute. The '-v' option and its assignment must precede all the file +name arguments, as well as the program text. (*Note Options:: for more +information about the '-v' option.) Otherwise, the variable assignment +is performed at a time determined by its position among the input file +arguments--after the processing of the preceding input file argument. +For example: + + awk '{ print $n }' n=4 inventory-shipped n=2 mail-list + +prints the value of field number 'n' for all input records. Before the +first file is read, the command line sets the variable 'n' equal to +four. This causes the fourth field to be printed in lines from +'inventory-shipped'. After the first file has finished, but before the +second file is started, 'n' is set to two, so that the second field is +printed in lines from 'mail-list': + + $ awk '{ print $n }' n=4 inventory-shipped n=2 mail-list + -| 15 + -| 24 + ... + -| 555-5553 + -| 555-3412 + ... + + Command-line arguments are made available for explicit examination by +the 'awk' program in the 'ARGV' array (*note ARGC and ARGV::). 'awk' +processes the values of command-line assignments for escape sequences +(*note Escape Sequences::). (d.c.) + + +File: gawk.info, Node: Conversion, Prev: Variables, Up: Values + +6.1.4 Conversion of Strings and Numbers +--------------------------------------- + +Number-to-string and string-to-number conversion are generally +straightforward. There can be subtleties to be aware of; this minor +node discusses this important facet of 'awk'. + +* Menu: + +* Strings And Numbers:: How 'awk' Converts Between Strings And + Numbers. +* Locale influences conversions:: How the locale may affect conversions. + + +File: gawk.info, Node: Strings And Numbers, Next: Locale influences conversions, Up: Conversion + +6.1.4.1 How 'awk' Converts Between Strings and Numbers +...................................................... + +Strings are converted to numbers and numbers are converted to strings, +if the context of the 'awk' program demands it. For example, if the +value of either 'foo' or 'bar' in the expression 'foo + bar' happens to +be a string, it is converted to a number before the addition is +performed. If numeric values appear in string concatenation, they are +converted to strings. Consider the following: + + two = 2; three = 3 + print (two three) + 4 + +This prints the (numeric) value 27. The numeric values of the variables +'two' and 'three' are converted to strings and concatenated together. +The resulting string is converted back to the number 23, to which 4 is +then added. + + If, for some reason, you need to force a number to be converted to a +string, concatenate that number with the empty string, '""'. To force a +string to be converted to a number, add zero to that string. A string +is converted to a number by interpreting any numeric prefix of the +string as numerals: '"2.5"' converts to 2.5, '"1e3"' converts to 1,000, +and '"25fix"' has a numeric value of 25. Strings that can't be +interpreted as valid numbers convert to zero. + + The exact manner in which numbers are converted into strings is +controlled by the 'awk' predefined variable 'CONVFMT' (*note Built-in +Variables::). Numbers are converted using the 'sprintf()' function with +'CONVFMT' as the format specifier (*note String Functions::). + + 'CONVFMT''s default value is '"%.6g"', which creates a value with at +most six significant digits. For some applications, you might want to +change it to specify more precision. On most modern machines, 17 digits +is usually enough to capture a floating-point number's value exactly.(1) + + Strange results can occur if you set 'CONVFMT' to a string that +doesn't tell 'sprintf()' how to format floating-point numbers in a +useful way. For example, if you forget the '%' in the format, 'awk' +converts all numbers to the same constant string. + + As a special case, if a number is an integer, then the result of +converting it to a string is _always_ an integer, no matter what the +value of 'CONVFMT' may be. Given the following code fragment: + + CONVFMT = "%2.2f" + a = 12 + b = a "" + +'b' has the value '"12"', not '"12.00"'. (d.c.) + + Pre-POSIX 'awk' Used 'OFMT' for String Conversion + + Prior to the POSIX standard, 'awk' used the value of 'OFMT' for +converting numbers to strings. 'OFMT' specifies the output format to +use when printing numbers with 'print'. 'CONVFMT' was introduced in +order to separate the semantics of conversion from the semantics of +printing. Both 'CONVFMT' and 'OFMT' have the same default value: +'"%.6g"'. In the vast majority of cases, old 'awk' programs do not +change their behavior. *Note Print:: for more information on the +'print' statement. + + ---------- Footnotes ---------- + + (1) Pathological cases can require up to 752 digits (!), but we doubt +that you need to worry about this. + + +File: gawk.info, Node: Locale influences conversions, Prev: Strings And Numbers, Up: Conversion + +6.1.4.2 Locales Can Influence Conversion +........................................ + +Where you are can matter when it comes to converting between numbers and +strings. The local character set and language--the "locale"--can affect +numeric formats. In particular, for 'awk' programs, it affects the +decimal point character and the thousands-separator character. The +'"C"' locale, and most English-language locales, use the period +character ('.') as the decimal point and don't have a thousands +separator. However, many (if not most) European and non-English locales +use the comma (',') as the decimal point character. European locales +often use either a space or a period as the thousands separator, if they +have one. + + The POSIX standard says that 'awk' always uses the period as the +decimal point when reading the 'awk' program source code, and for +command-line variable assignments (*note Other Arguments::). However, +when interpreting input data, for 'print' and 'printf' output, and for +number-to-string conversion, the local decimal point character is used. +(d.c.) In all cases, numbers in source code and in input data cannot +have a thousands separator. Here are some examples indicating the +difference in behavior, on a GNU/Linux system: + + $ export POSIXLY_CORRECT=1 Force POSIX behavior + $ gawk 'BEGIN { printf "%g\n", 3.1415927 }' + -| 3.14159 + $ LC_ALL=en_DK.utf-8 gawk 'BEGIN { printf "%g\n", 3.1415927 }' + -| 3,14159 + $ echo 4,321 | gawk '{ print $1 + 1 }' + -| 5 + $ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }' + -| 5,321 + +The 'en_DK.utf-8' locale is for English in Denmark, where the comma acts +as the decimal point separator. In the normal '"C"' locale, 'gawk' +treats '4,321' as 4, while in the Danish locale, it's treated as the +full number including the fractional part, 4.321. + + Some earlier versions of 'gawk' fully complied with this aspect of +the standard. However, many users in non-English locales complained +about this behavior, because their data used a period as the decimal +point, so the default behavior was restored to use a period as the +decimal point character. You can use the '--use-lc-numeric' option +(*note Options::) to force 'gawk' to use the locale's decimal point +character. ('gawk' also uses the locale's decimal point character when +in POSIX mode, either via '--posix' or the 'POSIXLY_CORRECT' environment +variable, as shown previously.) + + *note Table 6.1: table-locale-affects. describes the cases in which +the locale's decimal point character is used and when a period is used. +Some of these features have not been described yet. + +Feature Default '--posix' or + '--use-lc-numeric' +------------------------------------------------------------ +'%'g' Use locale Use locale +'%g' Use period Use locale +Input Use period Use locale +'strtonum()'Use period Use locale + +Table 6.1: Locale decimal point versus a period + + Finally, modern-day formal standards and the IEEE standard +floating-point representation can have an unusual but important effect +on the way 'gawk' converts some special string values to numbers. The +details are presented in *note POSIX Floating Point Problems::. + + +File: gawk.info, Node: All Operators, Next: Truth Values and Conditions, Prev: Values, Up: Expressions + +6.2 Operators: Doing Something with Values +========================================== + +This minor node introduces the "operators" that make use of the values +provided by constants and variables. + +* Menu: + +* Arithmetic Ops:: Arithmetic operations ('+', '-', + etc.) +* Concatenation:: Concatenating strings. +* Assignment Ops:: Changing the value of a variable or a field. +* Increment Ops:: Incrementing the numeric value of a variable. + + +File: gawk.info, Node: Arithmetic Ops, Next: Concatenation, Up: All Operators + +6.2.1 Arithmetic Operators +-------------------------- + +The 'awk' language uses the common arithmetic operators when evaluating +expressions. All of these arithmetic operators follow normal precedence +rules and work as you would expect them to. + + The following example uses a file named 'grades', which contains a +list of student names as well as three test scores per student (it's a +small class): + + Pat 100 97 58 + Sandy 84 72 93 + Chris 72 92 89 + +This program takes the file 'grades' and prints the average of the +scores: + + $ awk '{ sum = $2 + $3 + $4 ; avg = sum / 3 + > print $1, avg }' grades + -| Pat 85 + -| Sandy 83 + -| Chris 84.3333 + + The following list provides the arithmetic operators in 'awk', in +order from the highest precedence to the lowest: + +'X ^ Y' +'X ** Y' + Exponentiation; X raised to the Y power. '2 ^ 3' has the value + eight; the character sequence '**' is equivalent to '^'. (c.e.) + +'- X' + Negation. + +'+ X' + Unary plus; the expression is converted to a number. + +'X * Y' + Multiplication. + +'X / Y' + Division; because all numbers in 'awk' are floating-point numbers, + the result is _not_ rounded to an integer--'3 / 4' has the value + 0.75. (It is a common mistake, especially for C programmers, to + forget that _all_ numbers in 'awk' are floating point, and that + division of integer-looking constants produces a real number, not + an integer.) + +'X % Y' + Remainder; further discussion is provided in the text, just after + this list. + +'X + Y' + Addition. + +'X - Y' + Subtraction. + + Unary plus and minus have the same precedence, the multiplication +operators all have the same precedence, and addition and subtraction +have the same precedence. + + When computing the remainder of 'X % Y', the quotient is rounded +toward zero to an integer and multiplied by Y. This result is +subtracted from X; this operation is sometimes known as "trunc-mod." +The following relation always holds: + + b * int(a / b) + (a % b) == a + + One possibly undesirable effect of this definition of remainder is +that 'X % Y' is negative if X is negative. Thus: + + -17 % 8 = -1 + + In other 'awk' implementations, the signedness of the remainder may +be machine-dependent. + + NOTE: The POSIX standard only specifies the use of '^' for + exponentiation. For maximum portability, do not use the '**' + operator. + + +File: gawk.info, Node: Concatenation, Next: Assignment Ops, Prev: Arithmetic Ops, Up: All Operators + +6.2.2 String Concatenation +-------------------------- + + It seemed like a good idea at the time. + -- _Brian Kernighan_ + + There is only one string operation: concatenation. It does not have +a specific operator to represent it. Instead, concatenation is +performed by writing expressions next to one another, with no operator. +For example: + + $ awk '{ print "Field number one: " $1 }' mail-list + -| Field number one: Amelia + -| Field number one: Anthony + ... + + Without the space in the string constant after the ':', the line runs +together. For example: + + $ awk '{ print "Field number one:" $1 }' mail-list + -| Field number one:Amelia + -| Field number one:Anthony + ... + + Because string concatenation does not have an explicit operator, it +is often necessary to ensure that it happens at the right time by using +parentheses to enclose the items to concatenate. For example, you might +expect that the following code fragment concatenates 'file' and 'name': + + file = "file" + name = "name" + print "something meaningful" > file name + +This produces a syntax error with some versions of Unix 'awk'.(1) It is +necessary to use the following: + + print "something meaningful" > (file name) + + Parentheses should be used around concatenation in all but the most +common contexts, such as on the righthand side of '='. Be careful about +the kinds of expressions used in string concatenation. In particular, +the order of evaluation of expressions used for concatenation is +undefined in the 'awk' language. Consider this example: + + BEGIN { + a = "don't" + print (a " " (a = "panic")) + } + +It is not defined whether the second assignment to 'a' happens before or +after the value of 'a' is retrieved for producing the concatenated +value. The result could be either 'don't panic', or 'panic panic'. + + The precedence of concatenation, when mixed with other operators, is +often counter-intuitive. Consider this example: + + $ awk 'BEGIN { print -12 " " -24 }' + -| -12-24 + + This "obviously" is concatenating -12, a space, and -24. But where +did the space disappear to? The answer lies in the combination of +operator precedences and 'awk''s automatic conversion rules. To get the +desired result, write the program this way: + + $ awk 'BEGIN { print -12 " " (-24) }' + -| -12 -24 + + This forces 'awk' to treat the '-' on the '-24' as unary. Otherwise, +it's parsed as follows: + + -12 ('" "' - 24) + => -12 (0 - 24) + => -12 (-24) + => -12-24 + + As mentioned earlier, when mixing concatenation with other operators, +_parenthesize_. Otherwise, you're never quite sure what you'll get. + + ---------- Footnotes ---------- + + (1) It happens that BWK 'awk', 'gawk', and 'mawk' all "get it right," +but you should not rely on this. + + +File: gawk.info, Node: Assignment Ops, Next: Increment Ops, Prev: Concatenation, Up: All Operators + +6.2.3 Assignment Expressions +---------------------------- + +An "assignment" is an expression that stores a (usually different) value +into a variable. For example, let's assign the value one to the +variable 'z': + + z = 1 + + After this expression is executed, the variable 'z' has the value +one. Whatever old value 'z' had before the assignment is forgotten. + + Assignments can also store string values. For example, the following +stores the value '"this food is good"' in the variable 'message': + + thing = "food" + predicate = "good" + message = "this " thing " is " predicate + +This also illustrates string concatenation. The '=' sign is called an +"assignment operator". It is the simplest assignment operator because +the value of the righthand operand is stored unchanged. Most operators +(addition, concatenation, and so on) have no effect except to compute a +value. If the value isn't used, there's no reason to use the operator. +An assignment operator is different; it does produce a value, but even +if you ignore it, the assignment still makes itself felt through the +alteration of the variable. We call this a "side effect". + + The lefthand operand of an assignment need not be a variable (*note +Variables::); it can also be a field (*note Changing Fields::) or an +array element (*note Arrays::). These are all called "lvalues", which +means they can appear on the lefthand side of an assignment operator. +The righthand operand may be any expression; it produces the new value +that the assignment stores in the specified variable, field, or array +element. (Such values are called "rvalues".) + + It is important to note that variables do _not_ have permanent types. +A variable's type is simply the type of whatever value was last assigned +to it. In the following program fragment, the variable 'foo' has a +numeric value at first, and a string value later on: + + foo = 1 + print foo + foo = "bar" + print foo + +When the second assignment gives 'foo' a string value, the fact that it +previously had a numeric value is forgotten. + + String values that do not begin with a digit have a numeric value of +zero. After executing the following code, the value of 'foo' is five: + + foo = "a string" + foo = foo + 5 + + NOTE: Using a variable as a number and then later as a string can + be confusing and is poor programming style. The previous two + examples illustrate how 'awk' works, _not_ how you should write + your programs! + + An assignment is an expression, so it has a value--the same value +that is assigned. Thus, 'z = 1' is an expression with the value one. +One consequence of this is that you can write multiple assignments +together, such as: + + x = y = z = 5 + +This example stores the value five in all three variables ('x', 'y', and +'z'). It does so because the value of 'z = 5', which is five, is stored +into 'y' and then the value of 'y = z = 5', which is five, is stored +into 'x'. + + Assignments may be used anywhere an expression is called for. For +example, it is valid to write 'x != (y = 1)' to set 'y' to one, and then +test whether 'x' equals one. But this style tends to make programs hard +to read; such nesting of assignments should be avoided, except perhaps +in a one-shot program. + + Aside from '=', there are several other assignment operators that do +arithmetic with the old value of the variable. For example, the +operator '+=' computes a new value by adding the righthand value to the +old value of the variable. Thus, the following assignment adds five to +the value of 'foo': + + foo += 5 + +This is equivalent to the following: + + foo = foo + 5 + +Use whichever makes the meaning of your program clearer. + + There are situations where using '+=' (or any assignment operator) is +_not_ the same as simply repeating the lefthand operand in the righthand +expression. For example: + + # Thanks to Pat Rankin for this example + BEGIN { + foo[rand()] += 5 + for (x in foo) + print x, foo[x] + + bar[rand()] = bar[rand()] + 5 + for (x in bar) + print x, bar[x] + } + +The indices of 'bar' are practically guaranteed to be different, because +'rand()' returns different values each time it is called. (Arrays and +the 'rand()' function haven't been covered yet. *Note Arrays::, and +*note Numeric Functions:: for more information.) This example +illustrates an important fact about assignment operators: the lefthand +expression is only evaluated _once_. + + It is up to the implementation as to which expression is evaluated +first, the lefthand or the righthand. Consider this example: + + i = 1 + a[i += 2] = i + 1 + +The value of 'a[3]' could be either two or four. + + *note Table 6.2: table-assign-ops. lists the arithmetic assignment +operators. In each case, the righthand operand is an expression whose +value is converted to a number. + +Operator Effect +-------------------------------------------------------------------------- +LVALUE '+=' Add INCREMENT to the value of LVALUE. +INCREMENT +LVALUE '-=' Subtract DECREMENT from the value of LVALUE. +DECREMENT +LVALUE '*=' Multiply the value of LVALUE by COEFFICIENT. +COEFFICIENT +LVALUE '/=' DIVISOR Divide the value of LVALUE by DIVISOR. +LVALUE '%=' MODULUS Set LVALUE to its remainder by MODULUS. +LVALUE '^=' POWER Raise LVALUE to the power POWER. +LVALUE '**=' POWER Raise LVALUE to the power POWER. (c.e.) + +Table 6.2: Arithmetic assignment operators + + NOTE: Only the '^=' operator is specified by POSIX. For maximum + portability, do not use the '**=' operator. + + Syntactic Ambiguities Between '/=' and Regular Expressions + + There is a syntactic ambiguity between the '/=' assignment operator +and regexp constants whose first character is an '='. (d.c.) This is +most notable in some commercial 'awk' versions. For example: + + $ awk /==/ /dev/null + error-> awk: syntax error at source line 1 + error-> context is + error-> >>> /= <<< + error-> awk: bailing out at source line 1 + +A workaround is: + + awk '/[=]=/' /dev/null + + 'gawk' does not have this problem; BWK 'awk' and 'mawk' also do not. + + +File: gawk.info, Node: Increment Ops, Prev: Assignment Ops, Up: All Operators + +6.2.4 Increment and Decrement Operators +--------------------------------------- + +"Increment" and "decrement operators" increase or decrease the value of +a variable by one. An assignment operator can do the same thing, so the +increment operators add no power to the 'awk' language; however, they +are convenient abbreviations for very common operations. + + The operator used for adding one is written '++'. It can be used to +increment a variable either before or after taking its value. To +"pre-increment" a variable 'v', write '++v'. This adds one to the value +of 'v'--that new value is also the value of the expression. (The +assignment expression 'v += 1' is completely equivalent.) Writing the +'++' after the variable specifies "post-increment". This increments the +variable value just the same; the difference is that the value of the +increment expression itself is the variable's _old_ value. Thus, if +'foo' has the value four, then the expression 'foo++' has the value +four, but it changes the value of 'foo' to five. In other words, the +operator returns the old value of the variable, but with the side effect +of incrementing it. + + The post-increment 'foo++' is nearly the same as writing '(foo += 1) +- 1'. It is not perfectly equivalent because all numbers in 'awk' are +floating point--in floating point, 'foo + 1 - 1' does not necessarily +equal 'foo'. But the difference is minute as long as you stick to +numbers that are fairly small (less than 10e12). + + Fields and array elements are incremented just like variables. (Use +'$(i++)' when you want to do a field reference and a variable increment +at the same time. The parentheses are necessary because of the +precedence of the field reference operator '$'.) + + The decrement operator '--' works just like '++', except that it +subtracts one instead of adding it. As with '++', it can be used before +the lvalue to pre-decrement or after it to post-decrement. Following is +a summary of increment and decrement expressions: + +'++LVALUE' + Increment LVALUE, returning the new value as the value of the + expression. + +'LVALUE++' + Increment LVALUE, returning the _old_ value of LVALUE as the value + of the expression. + +'--LVALUE' + Decrement LVALUE, returning the new value as the value of the + expression. (This expression is like '++LVALUE', but instead of + adding, it subtracts.) + +'LVALUE--' + Decrement LVALUE, returning the _old_ value of LVALUE as the value + of the expression. (This expression is like 'LVALUE++', but + instead of adding, it subtracts.) + + Operator Evaluation Order + + Doctor, it hurts when I do this! + Then don't do that! + -- _Groucho Marx_ + +What happens for something like the following? + + b = 6 + print b += b++ + +Or something even stranger? + + b = 6 + b += ++b + b++ + print b + + In other words, when do the various side effects prescribed by the +postfix operators ('b++') take effect? When side effects happen is +"implementation-defined". In other words, it is up to the particular +version of 'awk'. The result for the first example may be 12 or 13, and +for the second, it may be 22 or 23. + + In short, doing things like this is not recommended and definitely +not anything that you can rely upon for portability. You should avoid +such things in your own programs. + + +File: gawk.info, Node: Truth Values and Conditions, Next: Function Calls, Prev: All Operators, Up: Expressions + +6.3 Truth Values and Conditions +=============================== + +In certain contexts, expression values also serve as "truth values"; +i.e., they determine what should happen next as the program runs. This +minor node describes how 'awk' defines "true" and "false" and how values +are compared. + +* Menu: + +* Truth Values:: What is "true" and what is "false". +* Typing and Comparison:: How variables acquire types and how this + affects comparison of numbers and strings with + '<', etc. +* Boolean Ops:: Combining comparison expressions using boolean + operators '||' ("or"), '&&' + ("and") and '!' ("not"). +* Conditional Exp:: Conditional expressions select between two + subexpressions under control of a third + subexpression. + + +File: gawk.info, Node: Truth Values, Next: Typing and Comparison, Up: Truth Values and Conditions + +6.3.1 True and False in 'awk' +----------------------------- + +Many programming languages have a special representation for the +concepts of "true" and "false." Such languages usually use the special +constants 'true' and 'false', or perhaps their uppercase equivalents. +However, 'awk' is different. It borrows a very simple concept of true +and false from C. In 'awk', any nonzero numeric value _or_ any nonempty +string value is true. Any other value (zero or the null string, '""') +is false. The following program prints 'A strange truth value' three +times: + + BEGIN { + if (3.1415927) + print "A strange truth value" + if ("Four Score And Seven Years Ago") + print "A strange truth value" + if (j = 57) + print "A strange truth value" + } + + There is a surprising consequence of the "nonzero or non-null" rule: +the string constant '"0"' is actually true, because it is non-null. +(d.c.) + + +File: gawk.info, Node: Typing and Comparison, Next: Boolean Ops, Prev: Truth Values, Up: Truth Values and Conditions + +6.3.2 Variable Typing and Comparison Expressions +------------------------------------------------ + + The Guide is definitive. Reality is frequently inaccurate. + -- _Douglas Adams, 'The Hitchhiker's Guide to the Galaxy'_ + + Unlike in other programming languages, in 'awk' variables do not have +a fixed type. Instead, they can be either a number or a string, +depending upon the value that is assigned to them. We look now at how +variables are typed, and how 'awk' compares variables. + +* Menu: + +* Variable Typing:: String type versus numeric type. +* Comparison Operators:: The comparison operators. +* POSIX String Comparison:: String comparison with POSIX rules. + + +File: gawk.info, Node: Variable Typing, Next: Comparison Operators, Up: Typing and Comparison + +6.3.2.1 String Type versus Numeric Type +....................................... + +The POSIX standard introduced the concept of a "numeric string", which +is simply a string that looks like a number--for example, '" +2"'. This +concept is used for determining the type of a variable. The type of the +variable is important because the types of two variables determine how +they are compared. Variable typing follows these rules: + + * A numeric constant or the result of a numeric operation has the + "numeric" attribute. + + * A string constant or the result of a string operation has the + "string" attribute. + + * Fields, 'getline' input, 'FILENAME', 'ARGV' elements, 'ENVIRON' + elements, and the elements of an array created by 'match()', + 'split()', and 'patsplit()' that are numeric strings have the + "strnum" attribute. Otherwise, they have the "string" attribute. + Uninitialized variables also have the "strnum" attribute. + + * Attributes propagate across assignments but are not changed by any + use. + + The last rule is particularly important. In the following program, +'a' has numeric type, even though it is later used in a string +operation: + + BEGIN { + a = 12.345 + b = a " is a cute number" + print b + } + + When two operands are compared, either string comparison or numeric +comparison may be used. This depends upon the attributes of the +operands, according to the following symmetric matrix: + + +------------------------------- + | STRING NUMERIC STRNUM + -----+------------------------------- + | + STRING | string string string + | + NUMERIC | string numeric numeric + | + STRNUM | string numeric numeric + -----+------------------------------- + + The basic idea is that user input that looks numeric--and _only_ user +input--should be treated as numeric, even though it is actually made of +characters and is therefore also a string. Thus, for example, the +string constant '" +3.14"', when it appears in program source code, is a +string--even though it looks numeric--and is _never_ treated as a number +for comparison purposes. + + In short, when one operand is a "pure" string, such as a string +constant, then a string comparison is performed. Otherwise, a numeric +comparison is performed. + + This point bears additional emphasis: All user input is made of +characters, and so is first and foremost of string type; input strings +that look numeric are additionally given the strnum attribute. Thus, +the six-character input string ' +3.14' receives the strnum attribute. +In contrast, the eight characters '" +3.14"' appearing in program text +comprise a string constant. The following examples print '1' when the +comparison between the two different constants is true, and '0' +otherwise: + + $ echo ' +3.14' | awk '{ print($0 == " +3.14") }' True + -| 1 + $ echo ' +3.14' | awk '{ print($0 == "+3.14") }' False + -| 0 + $ echo ' +3.14' | awk '{ print($0 == "3.14") }' False + -| 0 + $ echo ' +3.14' | awk '{ print($0 == 3.14) }' True + -| 1 + $ echo ' +3.14' | awk '{ print($1 == " +3.14") }' False + -| 0 + $ echo ' +3.14' | awk '{ print($1 == "+3.14") }' True + -| 1 + $ echo ' +3.14' | awk '{ print($1 == "3.14") }' False + -| 0 + $ echo ' +3.14' | awk '{ print($1 == 3.14) }' True + -| 1 + + +File: gawk.info, Node: Comparison Operators, Next: POSIX String Comparison, Prev: Variable Typing, Up: Typing and Comparison + +6.3.2.2 Comparison Operators +............................ + +"Comparison expressions" compare strings or numbers for relationships +such as equality. They are written using "relational operators", which +are a superset of those in C. *note Table 6.3: table-relational-ops. +describes them. + +Expression Result +-------------------------------------------------------------------------- +X '<' Y True if X is less than Y +X '<=' Y True if X is less than or equal to Y +X '>' Y True if X is greater than Y +X '>=' Y True if X is greater than or equal to Y +X '==' Y True if X is equal to Y +X '!=' Y True if X is not equal to Y +X '~' Y True if the string X matches the regexp denoted by Y +X '!~' Y True if the string X does not match the regexp + denoted by Y +SUBSCRIPT 'in' True if the array ARRAY has an element with the +ARRAY subscript SUBSCRIPT + +Table 6.3: Relational operators + + Comparison expressions have the value one if true and zero if false. +When comparing operands of mixed types, numeric operands are converted +to strings using the value of 'CONVFMT' (*note Conversion::). + + Strings are compared by comparing the first character of each, then +the second character of each, and so on. Thus, '"10"' is less than +'"9"'. If there are two strings where one is a prefix of the other, the +shorter string is less than the longer one. Thus, '"abc"' is less than +'"abcd"'. + + It is very easy to accidentally mistype the '==' operator and leave +off one of the '=' characters. The result is still valid 'awk' code, +but the program does not do what is intended: + + if (a = b) # oops! should be a == b + ... + else + ... + +Unless 'b' happens to be zero or the null string, the 'if' part of the +test always succeeds. Because the operators are so similar, this kind +of error is very difficult to spot when scanning the source code. + + The following list of expressions illustrates the kinds of +comparisons 'awk' performs, as well as what the result of each +comparison is: + +'1.5 <= 2.0' + Numeric comparison (true) + +'"abc" >= "xyz"' + String comparison (false) + +'1.5 != " +2"' + String comparison (true) + +'"1e2" < "3"' + String comparison (true) + +'a = 2; b = "2"' +'a == b' + String comparison (true) + +'a = 2; b = " +2"' +'a == b' + String comparison (false) + + In this example: + + $ echo 1e2 3 | awk '{ print ($1 < $2) ? "true" : "false" }' + -| false + +the result is 'false' because both '$1' and '$2' are user input. They +are numeric strings--therefore both have the strnum attribute, dictating +a numeric comparison. The purpose of the comparison rules and the use +of numeric strings is to attempt to produce the behavior that is "least +surprising," while still "doing the right thing." + + String comparisons and regular expression comparisons are very +different. For example: + + x == "foo" + +has the value one, or is true if the variable 'x' is precisely 'foo'. +By contrast: + + x ~ /foo/ + +has the value one if 'x' contains 'foo', such as '"Oh, what a fool am +I!"'. + + The righthand operand of the '~' and '!~' operators may be either a +regexp constant ('/'...'/') or an ordinary expression. In the latter +case, the value of the expression as a string is used as a dynamic +regexp (*note Regexp Usage::; also *note Computed Regexps::). + + A constant regular expression in slashes by itself is also an +expression. '/REGEXP/' is an abbreviation for the following comparison +expression: + + $0 ~ /REGEXP/ + + One special place where '/foo/' is _not_ an abbreviation for '$0 ~ +/foo/' is when it is the righthand operand of '~' or '!~'. *Note Using +Constant Regexps::, where this is discussed in more detail. + + +File: gawk.info, Node: POSIX String Comparison, Prev: Comparison Operators, Up: Typing and Comparison + +6.3.2.3 String Comparison Based on Locale Collating Order +......................................................... + +The POSIX standard used to say that all string comparisons are performed +based on the locale's "collating order". This is the order in which +characters sort, as defined by the locale (for more discussion, *note +Locales::). This order is usually very different from the results +obtained when doing straight byte-by-byte comparison.(1) + + Because this behavior differs considerably from existing practice, +'gawk' only implemented it when in POSIX mode (*note Options::). Here +is an example to illustrate the difference, in an 'en_US.UTF-8' locale: + + $ gawk 'BEGIN { printf("ABC < abc = %s\n", + > ("ABC" < "abc" ? "TRUE" : "FALSE")) }' + -| ABC < abc = TRUE + $ gawk --posix 'BEGIN { printf("ABC < abc = %s\n", + > ("ABC" < "abc" ? "TRUE" : "FALSE")) }' + -| ABC < abc = FALSE + + Fortunately, as of August 2016, comparison based on locale collating +order is no longer required for the '==' and '!=' operators.(2) +However, comparison based on locales is still required for '<', '<=', +'>', and '>='. POSIX thus recommends as follows: + + Since the '==' operator checks whether strings are identical, not + whether they collate equally, applications needing to check whether + strings collate equally can use: + + a <= b && a >= b + + As of version 4.2, 'gawk' continues to use locale collating order for +'<', '<=', '>', and '>=' only in POSIX mode. + + ---------- Footnotes ---------- + + (1) Technically, string comparison is supposed to behave the same way +as if the strings were compared with the C 'strcoll()' function. + + (2) See the Austin Group website +(http://austingroupbugs.net/view.php?id=1070). + + +File: gawk.info, Node: Boolean Ops, Next: Conditional Exp, Prev: Typing and Comparison, Up: Truth Values and Conditions + +6.3.3 Boolean Expressions +------------------------- + +A "Boolean expression" is a combination of comparison expressions or +matching expressions, using the Boolean operators "or" ('||'), "and" +('&&'), and "not" ('!'), along with parentheses to control nesting. The +truth value of the Boolean expression is computed by combining the truth +values of the component expressions. Boolean expressions are also +referred to as "logical expressions". The terms are equivalent. + + Boolean expressions can be used wherever comparison and matching +expressions can be used. They can be used in 'if', 'while', 'do', and +'for' statements (*note Statements::). They have numeric values (one if +true, zero if false) that come into play if the result of the Boolean +expression is stored in a variable or used in arithmetic. + + In addition, every Boolean expression is also a valid pattern, so you +can use one as a pattern to control the execution of rules. The Boolean +operators are: + +'BOOLEAN1 && BOOLEAN2' + True if both BOOLEAN1 and BOOLEAN2 are true. For example, the + following statement prints the current input record if it contains + both 'edu' and 'li': + + if ($0 ~ /edu/ && $0 ~ /li/) print + + The subexpression BOOLEAN2 is evaluated only if BOOLEAN1 is true. + This can make a difference when BOOLEAN2 contains expressions that + have side effects. In the case of '$0 ~ /foo/ && ($2 == bar++)', + the variable 'bar' is not incremented if there is no substring + 'foo' in the record. + +'BOOLEAN1 || BOOLEAN2' + True if at least one of BOOLEAN1 or BOOLEAN2 is true. For example, + the following statement prints all records in the input that + contain _either_ 'edu' or 'li': + + if ($0 ~ /edu/ || $0 ~ /li/) print + + The subexpression BOOLEAN2 is evaluated only if BOOLEAN1 is false. + This can make a difference when BOOLEAN2 contains expressions that + have side effects. (Thus, this test never really distinguishes + records that contain both 'edu' and 'li'--as soon as 'edu' is + matched, the full test succeeds.) + +'! BOOLEAN' + True if BOOLEAN is false. For example, the following program + prints 'no home!' in the unusual event that the 'HOME' environment + variable is not defined: + + BEGIN { if (! ("HOME" in ENVIRON)) + print "no home!" } + + (The 'in' operator is described in *note Reference to Elements::.) + + The '&&' and '||' operators are called "short-circuit" operators +because of the way they work. Evaluation of the full expression is +"short-circuited" if the result can be determined partway through its +evaluation. + + Statements that end with '&&' or '||' can be continued simply by +putting a newline after them. But you cannot put a newline in front of +either of these operators without using backslash continuation (*note +Statements/Lines::). + + The actual value of an expression using the '!' operator is either +one or zero, depending upon the truth value of the expression it is +applied to. The '!' operator is often useful for changing the sense of +a flag variable from false to true and back again. For example, the +following program is one way to print lines in between special +bracketing lines: + + $1 == "START" { interested = ! interested; next } + interested { print } + $1 == "END" { interested = ! interested; next } + +The variable 'interested', as with all 'awk' variables, starts out +initialized to zero, which is also false. When a line is seen whose +first field is 'START', the value of 'interested' is toggled to true, +using '!'. The next rule prints lines as long as 'interested' is true. +When a line is seen whose first field is 'END', 'interested' is toggled +back to false.(1) + + Most commonly, the '!' operator is used in the conditions of 'if' and +'while' statements, where it often makes more sense to phrase the logic +in the negative: + + if (! SOME CONDITION || SOME OTHER CONDITION) { + ... DO WHATEVER PROCESSING ... + } + + NOTE: The 'next' statement is discussed in *note Next Statement::. + 'next' tells 'awk' to skip the rest of the rules, get the next + record, and start processing the rules over again at the top. The + reason it's there is to avoid printing the bracketing 'START' and + 'END' lines. + + ---------- Footnotes ---------- + + (1) This program has a bug; it prints lines starting with 'END'. How +would you fix it? + + +File: gawk.info, Node: Conditional Exp, Prev: Boolean Ops, Up: Truth Values and Conditions + +6.3.4 Conditional Expressions +----------------------------- + +A "conditional expression" is a special kind of expression that has +three operands. It allows you to use one expression's value to select +one of two other expressions. The conditional expression in 'awk' is +the same as in the C language, as shown here: + + SELECTOR ? IF-TRUE-EXP : IF-FALSE-EXP + +There are three subexpressions. The first, SELECTOR, is always computed +first. If it is "true" (not zero or not null), then IF-TRUE-EXP is +computed next, and its value becomes the value of the whole expression. +Otherwise, IF-FALSE-EXP is computed next, and its value becomes the +value of the whole expression. For example, the following expression +produces the absolute value of 'x': + + x >= 0 ? x : -x + + Each time the conditional expression is computed, only one of +IF-TRUE-EXP and IF-FALSE-EXP is used; the other is ignored. This is +important when the expressions have side effects. For example, this +conditional expression examines element 'i' of either array 'a' or array +'b', and increments 'i': + + x == y ? a[i++] : b[i++] + +This is guaranteed to increment 'i' exactly once, because each time only +one of the two increment expressions is executed and the other is not. +*Note Arrays::, for more information about arrays. + + As a minor 'gawk' extension, a statement that uses '?:' can be +continued simply by putting a newline after either character. However, +putting a newline in front of either character does not work without +using backslash continuation (*note Statements/Lines::). If '--posix' +is specified (*note Options::), this extension is disabled. + + +File: gawk.info, Node: Function Calls, Next: Precedence, Prev: Truth Values and Conditions, Up: Expressions + +6.4 Function Calls +================== + +A "function" is a name for a particular calculation. This enables you +to ask for it by name at any point in the program. For example, the +function 'sqrt()' computes the square root of a number. + + A fixed set of functions are "built in", which means they are +available in every 'awk' program. The 'sqrt()' function is one of +these. *Note Built-in:: for a list of built-in functions and their +descriptions. In addition, you can define functions for use in your +program. *Note User-defined:: for instructions on how to do this. +Finally, 'gawk' lets you write functions in C or C++ that may be called +from your program (*note Dynamic Extensions::). + + The way to use a function is with a "function call" expression, which +consists of the function name followed immediately by a list of +"arguments" in parentheses. The arguments are expressions that provide +the raw materials for the function's calculations. When there is more +than one argument, they are separated by commas. If there are no +arguments, just write '()' after the function name. The following +examples show function calls with and without arguments: + + sqrt(x^2 + y^2) one argument + atan2(y, x) two arguments + rand() no arguments + + CAUTION: Do not put any space between the function name and the + opening parenthesis! A user-defined function name looks just like + the name of a variable--a space would make the expression look like + concatenation of a variable with an expression inside parentheses. + With built-in functions, space before the parenthesis is harmless, + but it is best not to get into the habit of using space to avoid + mistakes with user-defined functions. + + Each function expects a particular number of arguments. For example, +the 'sqrt()' function must be called with a single argument, the number +of which to take the square root: + + sqrt(ARGUMENT) + + Some of the built-in functions have one or more optional arguments. +If those arguments are not supplied, the functions use a reasonable +default value. *Note Built-in:: for full details. If arguments are +omitted in calls to user-defined functions, then those arguments are +treated as local variables. Such local variables act like the empty +string if referenced where a string value is required, and like zero if +referenced where a numeric value is required (*note User-defined::). + + As an advanced feature, 'gawk' provides indirect function calls, +which is a way to choose the function to call at runtime, instead of +when you write the source code to your program. We defer discussion of +this feature until later; see *note Indirect Calls::. + + Like every other expression, the function call has a value, often +called the "return value", which is computed by the function based on +the arguments you give it. In this example, the return value of +'sqrt(ARGUMENT)' is the square root of ARGUMENT. The following program +reads numbers, one number per line, and prints the square root of each +one: + + $ awk '{ print "The square root of", $1, "is", sqrt($1) }' + 1 + -| The square root of 1 is 1 + 3 + -| The square root of 3 is 1.73205 + 5 + -| The square root of 5 is 2.23607 + Ctrl-d + + A function can also have side effects, such as assigning values to +certain variables or doing I/O. This program shows how the 'match()' +function (*note String Functions::) changes the variables 'RSTART' and +'RLENGTH': + + { + if (match($1, $2)) + print RSTART, RLENGTH + else + print "no match" + } + +Here is a sample run: + + $ awk -f matchit.awk + aaccdd c+ + -| 3 2 + foo bar + -| no match + abcdefg e + -| 5 1 + + +File: gawk.info, Node: Precedence, Next: Locales, Prev: Function Calls, Up: Expressions + +6.5 Operator Precedence (How Operators Nest) +============================================ + +"Operator precedence" determines how operators are grouped when +different operators appear close by in one expression. For example, '*' +has higher precedence than '+'; thus, 'a + b * c' means to multiply 'b' +and 'c', and then add 'a' to the product (i.e., 'a + (b * c)'). + + The normal precedence of the operators can be overruled by using +parentheses. Think of the precedence rules as saying where the +parentheses are assumed to be. In fact, it is wise to always use +parentheses whenever there is an unusual combination of operators, +because other people who read the program may not remember what the +precedence is in this case. Even experienced programmers occasionally +forget the exact rules, which leads to mistakes. Explicit parentheses +help prevent any such mistakes. + + When operators of equal precedence are used together, the leftmost +operator groups first, except for the assignment, conditional, and +exponentiation operators, which group in the opposite order. Thus, 'a - +b + c' groups as '(a - b) + c' and 'a = b = c' groups as 'a = (b = c)'. + + Normally the precedence of prefix unary operators does not matter, +because there is only one way to interpret them: innermost first. Thus, +'$++i' means '$(++i)' and '++$x' means '++($x)'. However, when another +operator follows the operand, then the precedence of the unary operators +can matter. '$x^2' means '($x)^2', but '-x^2' means '-(x^2)', because +'-' has lower precedence than '^', whereas '$' has higher precedence. +Also, operators cannot be combined in a way that violates the precedence +rules; for example, '$$0++--' is not a valid expression because the +first '$' has higher precedence than the '++'; to avoid the problem the +expression can be rewritten as '$($0++)--'. + + This list presents 'awk''s operators, in order of highest to lowest +precedence: + +'('...')' + Grouping. + +'$' + Field reference. + +'++ --' + Increment, decrement. + +'^ **' + Exponentiation. These operators group right to left. + +'+ - !' + Unary plus, minus, logical "not." + +'* / %' + Multiplication, division, remainder. + +'+ -' + Addition, subtraction. + +String concatenation + There is no special symbol for concatenation. The operands are + simply written side by side (*note Concatenation::). + +'< <= == != > >= >> | |&' + Relational and redirection. The relational operators and the + redirections have the same precedence level. Characters such as + '>' serve both as relationals and as redirections; the context + distinguishes between the two meanings. + + Note that the I/O redirection operators in 'print' and 'printf' + statements belong to the statement level, not to expressions. The + redirection does not produce an expression that could be the + operand of another operator. As a result, it does not make sense + to use a redirection operator near another operator of lower + precedence without parentheses. Such combinations (e.g., 'print + foo > a ? b : c') result in syntax errors. The correct way to + write this statement is 'print foo > (a ? b : c)'. + +'~ !~' + Matching, nonmatching. + +'in' + Array membership. + +'&&' + Logical "and." + +'||' + Logical "or." + +'?:' + Conditional. This operator groups right to left. + +'= += -= *= /= %= ^= **=' + Assignment. These operators group right to left. + + NOTE: The '|&', '**', and '**=' operators are not specified by + POSIX. For maximum portability, do not use them. + + +File: gawk.info, Node: Locales, Next: Expressions Summary, Prev: Precedence, Up: Expressions + +6.6 Where You Are Makes a Difference +==================================== + +Modern systems support the notion of "locales": a way to tell the system +about the local character set and language. The ISO C standard defines +a default '"C"' locale, which is an environment that is typical of what +many C programmers are used to. + + Once upon a time, the locale setting used to affect regexp matching, +but this is no longer true (*note Ranges and Locales::). + + Locales can affect record splitting. For the normal case of 'RS = +"\n"', the locale is largely irrelevant. For other single-character +record separators, setting 'LC_ALL=C' in the environment will give you +much better performance when reading records. Otherwise, 'gawk' has to +make several function calls, _per input character_, to find the record +terminator. + + Locales can affect how dates and times are formatted (*note Time +Functions::). For example, a common way to abbreviate the date +September 4, 2015, in the United States is "9/4/15." In many countries +in Europe, however, it is abbreviated "4.9.15." Thus, the '%x' +specification in a '"US"' locale might produce '9/4/15', while in a +'"EUROPE"' locale, it might produce '4.9.15'. + + According to POSIX, string comparison is also affected by locales +(similar to regular expressions). The details are presented in *note +POSIX String Comparison::. + + Finally, the locale affects the value of the decimal point character +used when 'gawk' parses input data. This is discussed in detail in +*note Conversion::. + + +File: gawk.info, Node: Expressions Summary, Prev: Locales, Up: Expressions + +6.7 Summary +=========== + + * Expressions are the basic elements of computation in programs. + They are built from constants, variables, function calls, and + combinations of the various kinds of values with operators. + + * 'awk' supplies three kinds of constants: numeric, string, and + regexp. 'gawk' lets you specify numeric constants in octal and + hexadecimal (bases 8 and 16) as well as decimal (base 10). In + certain contexts, a standalone regexp constant such as '/foo/' has + the same meaning as '$0 ~ /foo/'. + + * Variables hold values between uses in computations. A number of + built-in variables provide information to your 'awk' program, and a + number of others let you control how 'awk' behaves. + + * Numbers are automatically converted to strings, and strings to + numbers, as needed by 'awk'. Numeric values are converted as if + they were formatted with 'sprintf()' using the format in 'CONVFMT'. + Locales can influence the conversions. + + * 'awk' provides the usual arithmetic operators (addition, + subtraction, multiplication, division, modulus), and unary plus and + minus. It also provides comparison operators, Boolean operators, + an array membership testing operator, and regexp matching + operators. String concatenation is accomplished by placing two + expressions next to each other; there is no explicit operator. The + three-operand '?:' operator provides an "if-else" test within + expressions. + + * Assignment operators provide convenient shorthands for common + arithmetic operations. + + * In 'awk', a value is considered to be true if it is nonzero _or_ + non-null. Otherwise, the value is false. + + * A variable's type is set upon each assignment and may change over + its lifetime. The type determines how it behaves in comparisons + (string or numeric). + + * Function calls return a value that may be used as part of a larger + expression. Expressions used to pass parameter values are fully + evaluated before the function is called. 'awk' provides built-in + and user-defined functions; this is described in *note Functions::. + + * Operator precedence specifies the order in which operations are + performed, unless explicitly overridden by parentheses. 'awk''s + operator precedence is compatible with that of C. + + * Locales can affect the format of data as output by an 'awk' + program, and occasionally the format for data read as input. + + +File: gawk.info, Node: Patterns and Actions, Next: Arrays, Prev: Expressions, Up: Top + +7 Patterns, Actions, and Variables +********************************** + +As you have already seen, each 'awk' statement consists of a pattern +with an associated action. This major node describes how you build +patterns and actions, what kinds of things you can do within actions, +and 'awk''s predefined variables. + + The pattern-action rules and the statements available for use within +actions form the core of 'awk' programming. In a sense, everything +covered up to here has been the foundation that programs are built on +top of. Now it's time to start building something useful. + +* Menu: + +* Pattern Overview:: What goes into a pattern. +* Using Shell Variables:: How to use shell variables with 'awk'. +* Action Overview:: What goes into an action. +* Statements:: Describes the various control statements in + detail. +* Built-in Variables:: Summarizes the predefined variables. +* Pattern Action Summary:: Patterns and Actions summary. + + +File: gawk.info, Node: Pattern Overview, Next: Using Shell Variables, Up: Patterns and Actions + +7.1 Pattern Elements +==================== + +* Menu: + +* Regexp Patterns:: Using regexps as patterns. +* Expression Patterns:: Any expression can be used as a pattern. +* Ranges:: Pairs of patterns specify record ranges. +* BEGIN/END:: Specifying initialization and cleanup rules. +* BEGINFILE/ENDFILE:: Two special patterns for advanced control. +* Empty:: The empty pattern, which matches every record. + +Patterns in 'awk' control the execution of rules--a rule is executed +when its pattern matches the current input record. The following is a +summary of the types of 'awk' patterns: + +'/REGULAR EXPRESSION/' + A regular expression. It matches when the text of the input record + fits the regular expression. (*Note Regexp::.) + +'EXPRESSION' + A single expression. It matches when its value is nonzero (if a + number) or non-null (if a string). (*Note Expression Patterns::.) + +'BEGPAT, ENDPAT' + A pair of patterns separated by a comma, specifying a "range" of + records. The range includes both the initial record that matches + BEGPAT and the final record that matches ENDPAT. (*Note Ranges::.) + +'BEGIN' +'END' + Special patterns for you to supply startup or cleanup actions for + your 'awk' program. (*Note BEGIN/END::.) + +'BEGINFILE' +'ENDFILE' + Special patterns for you to supply startup or cleanup actions to be + done on a per-file basis. (*Note BEGINFILE/ENDFILE::.) + +'EMPTY' + The empty pattern matches every input record. (*Note Empty::.) + + +File: gawk.info, Node: Regexp Patterns, Next: Expression Patterns, Up: Pattern Overview + +7.1.1 Regular Expressions as Patterns +------------------------------------- + +Regular expressions are one of the first kinds of patterns presented in +this book. This kind of pattern is simply a regexp constant in the +pattern part of a rule. Its meaning is '$0 ~ /PATTERN/'. The pattern +matches when the input record matches the regexp. For example: + + /foo|bar|baz/ { buzzwords++ } + END { print buzzwords, "buzzwords seen" } + + +File: gawk.info, Node: Expression Patterns, Next: Ranges, Prev: Regexp Patterns, Up: Pattern Overview + +7.1.2 Expressions as Patterns +----------------------------- + +Any 'awk' expression is valid as an 'awk' pattern. The pattern matches +if the expression's value is nonzero (if a number) or non-null (if a +string). The expression is reevaluated each time the rule is tested +against a new input record. If the expression uses fields such as '$1', +the value depends directly on the new input record's text; otherwise, it +depends on only what has happened so far in the execution of the 'awk' +program. + + Comparison expressions, using the comparison operators described in +*note Typing and Comparison::, are a very common kind of pattern. +Regexp matching and nonmatching are also very common expressions. The +left operand of the '~' and '!~' operators is a string. The right +operand is either a constant regular expression enclosed in slashes +('/REGEXP/'), or any expression whose string value is used as a dynamic +regular expression (*note Computed Regexps::). The following example +prints the second field of each input record whose first field is +precisely 'li': + + $ awk '$1 == "li" { print $2 }' mail-list + +(There is no output, because there is no person with the exact name +'li'.) Contrast this with the following regular expression match, which +accepts any record with a first field that contains 'li': + + $ awk '$1 ~ /li/ { print $2 }' mail-list + -| 555-5553 + -| 555-6699 + + A regexp constant as a pattern is also a special case of an +expression pattern. The expression '/li/' has the value one if 'li' +appears in the current input record. Thus, as a pattern, '/li/' matches +any record containing 'li'. + + Boolean expressions are also commonly used as patterns. Whether the +pattern matches an input record depends on whether its subexpressions +match. For example, the following command prints all the records in +'mail-list' that contain both 'edu' and 'li': + + $ awk '/edu/ && /li/' mail-list + -| Samuel 555-3430 samuel.lanceolis@shu.edu A + + The following command prints all records in 'mail-list' that contain +_either_ 'edu' or 'li' (or both, of course): + + $ awk '/edu/ || /li/' mail-list + -| Amelia 555-5553 amelia.zodiacusque@gmail.com F + -| Broderick 555-0542 broderick.aliquotiens@yahoo.com R + -| Fabius 555-1234 fabius.undevicesimus@ucb.edu F + -| Julie 555-6699 julie.perscrutabor@skeeve.com F + -| Samuel 555-3430 samuel.lanceolis@shu.edu A + -| Jean-Paul 555-2127 jeanpaul.campanorum@nyu.edu R + + The following command prints all records in 'mail-list' that do _not_ +contain the string 'li': + + $ awk '! /li/' mail-list + -| Anthony 555-3412 anthony.asserturo@hotmail.com A + -| Becky 555-7685 becky.algebrarum@gmail.com A + -| Bill 555-1675 bill.drowning@hotmail.com A + -| Camilla 555-2912 camilla.infusarum@skynet.be R + -| Fabius 555-1234 fabius.undevicesimus@ucb.edu F + -| Martin 555-6480 martin.codicibus@hotmail.com A + -| Jean-Paul 555-2127 jeanpaul.campanorum@nyu.edu R + + The subexpressions of a Boolean operator in a pattern can be constant +regular expressions, comparisons, or any other 'awk' expressions. Range +patterns are not expressions, so they cannot appear inside Boolean +patterns. Likewise, the special patterns 'BEGIN', 'END', 'BEGINFILE', +and 'ENDFILE', which never match any input record, are not expressions +and cannot appear inside Boolean patterns. + + The precedence of the different operators that can appear in patterns +is described in *note Precedence::. + + +File: gawk.info, Node: Ranges, Next: BEGIN/END, Prev: Expression Patterns, Up: Pattern Overview + +7.1.3 Specifying Record Ranges with Patterns +-------------------------------------------- + +A "range pattern" is made of two patterns separated by a comma, in the +form 'BEGPAT, ENDPAT'. It is used to match ranges of consecutive input +records. The first pattern, BEGPAT, controls where the range begins, +while ENDPAT controls where the pattern ends. For example, the +following: + + awk '$1 == "on", $1 == "off"' myfile + +prints every record in 'myfile' between 'on'/'off' pairs, inclusive. + + A range pattern starts out by matching BEGPAT against every input +record. When a record matches BEGPAT, the range pattern is "turned on", +and the range pattern matches this record as well. As long as the range +pattern stays turned on, it automatically matches every input record +read. The range pattern also matches ENDPAT against every input record; +when this succeeds, the range pattern is "turned off" again for the +following record. Then the range pattern goes back to checking BEGPAT +against each record. + + The record that turns on the range pattern and the one that turns it +off both match the range pattern. If you don't want to operate on these +records, you can write 'if' statements in the rule's action to +distinguish them from the records you are interested in. + + It is possible for a pattern to be turned on and off by the same +record. If the record satisfies both conditions, then the action is +executed for just that record. For example, suppose there is text +between two identical markers (e.g., the '%' symbol), each on its own +line, that should be ignored. A first attempt would be to combine a +range pattern that describes the delimited text with the 'next' +statement (not discussed yet, *note Next Statement::). This causes +'awk' to skip any further processing of the current record and start +over again with the next input record. Such a program looks like this: + + /^%$/,/^%$/ { next } + { print } + +This program fails because the range pattern is both turned on and +turned off by the first line, which just has a '%' on it. To accomplish +this task, write the program in the following manner, using a flag: + + /^%$/ { skip = ! skip; next } + skip == 1 { next } # skip lines with `skip' set + + In a range pattern, the comma (',') has the lowest precedence of all +the operators (i.e., it is evaluated last). Thus, the following program +attempts to combine a range pattern with another, simpler test: + + echo Yes | awk '/1/,/2/ || /Yes/' + + The intent of this program is '(/1/,/2/) || /Yes/'. However, 'awk' +interprets this as '/1/, (/2/ || /Yes/)'. This cannot be changed or +worked around; range patterns do not combine with other patterns: + + $ echo Yes | gawk '(/1/,/2/) || /Yes/' + error-> gawk: cmd. line:1: (/1/,/2/) || /Yes/ + error-> gawk: cmd. line:1: ^ syntax error + + As a minor point of interest, although it is poor style, POSIX allows +you to put a newline after the comma in a range pattern. (d.c.) + + +File: gawk.info, Node: BEGIN/END, Next: BEGINFILE/ENDFILE, Prev: Ranges, Up: Pattern Overview + +7.1.4 The 'BEGIN' and 'END' Special Patterns +-------------------------------------------- + +All the patterns described so far are for matching input records. The +'BEGIN' and 'END' special patterns are different. They supply startup +and cleanup actions for 'awk' programs. 'BEGIN' and 'END' rules must +have actions; there is no default action for these rules because there +is no current record when they run. 'BEGIN' and 'END' rules are often +referred to as "'BEGIN' and 'END' blocks" by longtime 'awk' programmers. + +* Menu: + +* Using BEGIN/END:: How and why to use BEGIN/END rules. +* I/O And BEGIN/END:: I/O issues in BEGIN/END rules. + + +File: gawk.info, Node: Using BEGIN/END, Next: I/O And BEGIN/END, Up: BEGIN/END + +7.1.4.1 Startup and Cleanup Actions +................................... + +A 'BEGIN' rule is executed once only, before the first input record is +read. Likewise, an 'END' rule is executed once only, after all the +input is read. For example: + + $ awk ' + > BEGIN { print "Analysis of \"li\"" } + > /li/ { ++n } + > END { print "\"li\" appears in", n, "records." }' mail-list + -| Analysis of "li" + -| "li" appears in 4 records. + + This program finds the number of records in the input file +'mail-list' that contain the string 'li'. The 'BEGIN' rule prints a +title for the report. There is no need to use the 'BEGIN' rule to +initialize the counter 'n' to zero, as 'awk' does this automatically +(*note Variables::). The second rule increments the variable 'n' every +time a record containing the pattern 'li' is read. The 'END' rule +prints the value of 'n' at the end of the run. + + The special patterns 'BEGIN' and 'END' cannot be used in ranges or +with Boolean operators (indeed, they cannot be used with any operators). +An 'awk' program may have multiple 'BEGIN' and/or 'END' rules. They are +executed in the order in which they appear: all the 'BEGIN' rules at +startup and all the 'END' rules at termination. 'BEGIN' and 'END' rules +may be intermixed with other rules. This feature was added in the 1987 +version of 'awk' and is included in the POSIX standard. The original +(1978) version of 'awk' required the 'BEGIN' rule to be placed at the +beginning of the program, the 'END' rule to be placed at the end, and +only allowed one of each. This is no longer required, but it is a good +idea to follow this template in terms of program organization and +readability. + + Multiple 'BEGIN' and 'END' rules are useful for writing library +functions, because each library file can have its own 'BEGIN' and/or +'END' rule to do its own initialization and/or cleanup. The order in +which library functions are named on the command line controls the order +in which their 'BEGIN' and 'END' rules are executed. Therefore, you +have to be careful when writing such rules in library files so that the +order in which they are executed doesn't matter. *Note Options:: for +more information on using library functions. *Note Library Functions::, +for a number of useful library functions. + + If an 'awk' program has only 'BEGIN' rules and no other rules, then +the program exits after the 'BEGIN' rules are run.(1) However, if an +'END' rule exists, then the input is read, even if there are no other +rules in the program. This is necessary in case the 'END' rule checks +the 'FNR' and 'NR' variables. + + ---------- Footnotes ---------- + + (1) The original version of 'awk' kept reading and ignoring input +until the end of the file was seen. + + +File: gawk.info, Node: I/O And BEGIN/END, Prev: Using BEGIN/END, Up: BEGIN/END + +7.1.4.2 Input/Output from 'BEGIN' and 'END' Rules +................................................. + +There are several (sometimes subtle) points to be aware of when doing +I/O from a 'BEGIN' or 'END' rule. The first has to do with the value of +'$0' in a 'BEGIN' rule. Because 'BEGIN' rules are executed before any +input is read, there simply is no input record, and therefore no fields, +when executing 'BEGIN' rules. References to '$0' and the fields yield a +null string or zero, depending upon the context. One way to give '$0' a +real value is to execute a 'getline' command without a variable (*note +Getline::). Another way is simply to assign a value to '$0'. + + The second point is similar to the first, but from the other +direction. Traditionally, due largely to implementation issues, '$0' +and 'NF' were _undefined_ inside an 'END' rule. The POSIX standard +specifies that 'NF' is available in an 'END' rule. It contains the +number of fields from the last input record. Most probably due to an +oversight, the standard does not say that '$0' is also preserved, +although logically one would think that it should be. In fact, all of +BWK 'awk', 'mawk', and 'gawk' preserve the value of '$0' for use in +'END' rules. Be aware, however, that some other implementations and +many older versions of Unix 'awk' do not. + + The third point follows from the first two. The meaning of 'print' +inside a 'BEGIN' or 'END' rule is the same as always: 'print $0'. If +'$0' is the null string, then this prints an empty record. Many +longtime 'awk' programmers use an unadorned 'print' in 'BEGIN' and 'END' +rules, to mean 'print ""', relying on '$0' being null. Although one +might generally get away with this in 'BEGIN' rules, it is a very bad +idea in 'END' rules, at least in 'gawk'. It is also poor style, because +if an empty line is needed in the output, the program should print one +explicitly. + + Finally, the 'next' and 'nextfile' statements are not allowed in a +'BEGIN' rule, because the implicit +read-a-record-and-match-against-the-rules loop has not started yet. +Similarly, those statements are not valid in an 'END' rule, because all +the input has been read. (*Note Next Statement:: and *note Nextfile +Statement::.) + + +File: gawk.info, Node: BEGINFILE/ENDFILE, Next: Empty, Prev: BEGIN/END, Up: Pattern Overview + +7.1.5 The 'BEGINFILE' and 'ENDFILE' Special Patterns +---------------------------------------------------- + +This minor node describes a 'gawk'-specific feature. + + Two special kinds of rule, 'BEGINFILE' and 'ENDFILE', give you +"hooks" into 'gawk''s command-line file processing loop. As with the +'BEGIN' and 'END' rules (*note BEGIN/END::), all 'BEGINFILE' rules in a +program are merged, in the order they are read by 'gawk', and all +'ENDFILE' rules are merged as well. + + The body of the 'BEGINFILE' rules is executed just before 'gawk' +reads the first record from a file. 'FILENAME' is set to the name of +the current file, and 'FNR' is set to zero. + + The 'BEGINFILE' rule provides you the opportunity to accomplish two +tasks that would otherwise be difficult or impossible to perform: + + * You can test if the file is readable. Normally, it is a fatal + error if a file named on the command line cannot be opened for + reading. However, you can bypass the fatal error and move on to + the next file on the command line. + + You do this by checking if the 'ERRNO' variable is not the empty + string; if so, then 'gawk' was not able to open the file. In this + case, your program can execute the 'nextfile' statement (*note + Nextfile Statement::). This causes 'gawk' to skip the file + entirely. Otherwise, 'gawk' exits with the usual fatal error. + + * If you have written extensions that modify the record handling (by + inserting an "input parser"; *note Input Parsers::), you can invoke + them at this point, before 'gawk' has started processing the file. + (This is a _very_ advanced feature, currently used only by the + 'gawkextlib' project (http://sourceforge.net/projects/gawkextlib).) + + The 'ENDFILE' rule is called when 'gawk' has finished processing the +last record in an input file. For the last input file, it will be +called before any 'END' rules. The 'ENDFILE' rule is executed even for +empty input files. + + Normally, when an error occurs when reading input in the normal +input-processing loop, the error is fatal. However, if an 'ENDFILE' +rule is present, the error becomes non-fatal, and instead 'ERRNO' is +set. This makes it possible to catch and process I/O errors at the +level of the 'awk' program. + + The 'next' statement (*note Next Statement::) is not allowed inside +either a 'BEGINFILE' or an 'ENDFILE' rule. The 'nextfile' statement is +allowed only inside a 'BEGINFILE' rule, not inside an 'ENDFILE' rule. + + The 'getline' statement (*note Getline::) is restricted inside both +'BEGINFILE' and 'ENDFILE': only redirected forms of 'getline' are +allowed. + + 'BEGINFILE' and 'ENDFILE' are 'gawk' extensions. In most other 'awk' +implementations, or if 'gawk' is in compatibility mode (*note +Options::), they are not special. + + +File: gawk.info, Node: Empty, Prev: BEGINFILE/ENDFILE, Up: Pattern Overview + +7.1.6 The Empty Pattern +----------------------- + +An empty (i.e., nonexistent) pattern is considered to match _every_ +input record. For example, the program: + + awk '{ print $1 }' mail-list + +prints the first field of every record. + + +File: gawk.info, Node: Using Shell Variables, Next: Action Overview, Prev: Pattern Overview, Up: Patterns and Actions + +7.2 Using Shell Variables in Programs +===================================== + +'awk' programs are often used as components in larger programs written +in shell. For example, it is very common to use a shell variable to +hold a pattern that the 'awk' program searches for. There are two ways +to get the value of the shell variable into the body of the 'awk' +program. + + A common method is to use shell quoting to substitute the variable's +value into the program inside the script. For example, consider the +following program: + + printf "Enter search pattern: " + read pattern + awk "/$pattern/ "'{ nmatches++ } + END { print nmatches, "found" }' /path/to/data + +The 'awk' program consists of two pieces of quoted text that are +concatenated together to form the program. The first part is +double-quoted, which allows substitution of the 'pattern' shell variable +inside the quotes. The second part is single-quoted. + + Variable substitution via quoting works, but can potentially be +messy. It requires a good understanding of the shell's quoting rules +(*note Quoting::), and it's often difficult to correctly match up the +quotes when reading the program. + + A better method is to use 'awk''s variable assignment feature (*note +Assignment Options::) to assign the shell variable's value to an 'awk' +variable. Then use dynamic regexps to match the pattern (*note Computed +Regexps::). The following shows how to redo the previous example using +this technique: + + printf "Enter search pattern: " + read pattern + awk -v pat="$pattern" '$0 ~ pat { nmatches++ } + END { print nmatches, "found" }' /path/to/data + +Now, the 'awk' program is just one single-quoted string. The assignment +'-v pat="$pattern"' still requires double quotes, in case there is +whitespace in the value of '$pattern'. The 'awk' variable 'pat' could +be named 'pattern' too, but that would be more confusing. Using a +variable also provides more flexibility, as the variable can be used +anywhere inside the program--for printing, as an array subscript, or for +any other use--without requiring the quoting tricks at every point in +the program. + + +File: gawk.info, Node: Action Overview, Next: Statements, Prev: Using Shell Variables, Up: Patterns and Actions + +7.3 Actions +=========== + +An 'awk' program or script consists of a series of rules and function +definitions interspersed. (Functions are described later. *Note +User-defined::.) A rule contains a pattern and an action, either of +which (but not both) may be omitted. The purpose of the "action" is to +tell 'awk' what to do once a match for the pattern is found. Thus, in +outline, an 'awk' program generally looks like this: + + [PATTERN] '{ ACTION }' + PATTERN ['{ ACTION }'] + ... + 'function NAME(ARGS) { ... }' + ... + + An action consists of one or more 'awk' "statements", enclosed in +braces ('{...}'). Each statement specifies one thing to do. The +statements are separated by newlines or semicolons. The braces around +an action must be used even if the action contains only one statement, +or if it contains no statements at all. However, if you omit the action +entirely, omit the braces as well. An omitted action is equivalent to +'{ print $0 }': + + /foo/ { } match 'foo', do nothing -- empty action + /foo/ match 'foo', print the record -- omitted action + + The following types of statements are supported in 'awk': + +Expressions + Call functions or assign values to variables (*note Expressions::). + Executing this kind of statement simply computes the value of the + expression. This is useful when the expression has side effects + (*note Assignment Ops::). + +Control statements + Specify the control flow of 'awk' programs. The 'awk' language + gives you C-like constructs ('if', 'for', 'while', and 'do') as + well as a few special ones (*note Statements::). + +Compound statements + Enclose one or more statements in braces. A compound statement is + used in order to put several statements together in the body of an + 'if', 'while', 'do', or 'for' statement. + +Input statements + Use the 'getline' command (*note Getline::). Also supplied in + 'awk' are the 'next' statement (*note Next Statement::) and the + 'nextfile' statement (*note Nextfile Statement::). + +Output statements + Such as 'print' and 'printf'. *Note Printing::. + +Deletion statements + For deleting array elements. *Note Delete::. + + +File: gawk.info, Node: Statements, Next: Built-in Variables, Prev: Action Overview, Up: Patterns and Actions + +7.4 Control Statements in Actions +================================= + +"Control statements", such as 'if', 'while', and so on, control the flow +of execution in 'awk' programs. Most of 'awk''s control statements are +patterned after similar statements in C. + + All the control statements start with special keywords, such as 'if' +and 'while', to distinguish them from simple expressions. Many control +statements contain other statements. For example, the 'if' statement +contains another statement that may or may not be executed. The +contained statement is called the "body". To include more than one +statement in the body, group them into a single "compound statement" +with braces, separating them with newlines or semicolons. + +* Menu: + +* If Statement:: Conditionally execute some 'awk' + statements. +* While Statement:: Loop until some condition is satisfied. +* Do Statement:: Do specified action while looping until some + condition is satisfied. +* For Statement:: Another looping statement, that provides + initialization and increment clauses. +* Switch Statement:: Switch/case evaluation for conditional + execution of statements based on a value. +* Break Statement:: Immediately exit the innermost enclosing loop. +* Continue Statement:: Skip to the end of the innermost enclosing + loop. +* Next Statement:: Stop processing the current input record. +* Nextfile Statement:: Stop processing the current file. +* Exit Statement:: Stop execution of 'awk'. + + +File: gawk.info, Node: If Statement, Next: While Statement, Up: Statements + +7.4.1 The 'if'-'else' Statement +------------------------------- + +The 'if'-'else' statement is 'awk''s decision-making statement. It +looks like this: + + 'if (CONDITION) THEN-BODY' ['else ELSE-BODY'] + +The CONDITION is an expression that controls what the rest of the +statement does. If the CONDITION is true, THEN-BODY is executed; +otherwise, ELSE-BODY is executed. The 'else' part of the statement is +optional. The condition is considered false if its value is zero or the +null string; otherwise, the condition is true. Refer to the following: + + if (x % 2 == 0) + print "x is even" + else + print "x is odd" + + In this example, if the expression 'x % 2 == 0' is true (i.e., if the +value of 'x' is evenly divisible by two), then the first 'print' +statement is executed; otherwise, the second 'print' statement is +executed. If the 'else' keyword appears on the same line as THEN-BODY +and THEN-BODY is not a compound statement (i.e., not surrounded by +braces), then a semicolon must separate THEN-BODY from the 'else'. To +illustrate this, the previous example can be rewritten as: + + if (x % 2 == 0) print "x is even"; else + print "x is odd" + +If the ';' is left out, 'awk' can't interpret the statement and it +produces a syntax error. Don't actually write programs this way, +because a human reader might fail to see the 'else' if it is not the +first thing on its line. + + +File: gawk.info, Node: While Statement, Next: Do Statement, Prev: If Statement, Up: Statements + +7.4.2 The 'while' Statement +--------------------------- + +In programming, a "loop" is a part of a program that can be executed two +or more times in succession. The 'while' statement is the simplest +looping statement in 'awk'. It repeatedly executes a statement as long +as a condition is true. For example: + + while (CONDITION) + BODY + +BODY is a statement called the "body" of the loop, and CONDITION is an +expression that controls how long the loop keeps running. The first +thing the 'while' statement does is test the CONDITION. If the +CONDITION is true, it executes the statement BODY. (The CONDITION is +true when the value is not zero and not a null string.) After BODY has +been executed, CONDITION is tested again, and if it is still true, BODY +executes again. This process repeats until the CONDITION is no longer +true. If the CONDITION is initially false, the body of the loop never +executes and 'awk' continues with the statement following the loop. +This example prints the first three fields of each record, one per line: + + awk ' + { + i = 1 + while (i <= 3) { + print $i + i++ + } + }' inventory-shipped + +The body of this loop is a compound statement enclosed in braces, +containing two statements. The loop works in the following manner: +first, the value of 'i' is set to one. Then, the 'while' statement +tests whether 'i' is less than or equal to three. This is true when 'i' +equals one, so the 'i'th field is printed. Then the 'i++' increments +the value of 'i' and the loop repeats. The loop terminates when 'i' +reaches four. + + A newline is not required between the condition and the body; +however, using one makes the program clearer unless the body is a +compound statement or else is very simple. The newline after the open +brace that begins the compound statement is not required either, but the +program is harder to read without it. + + +File: gawk.info, Node: Do Statement, Next: For Statement, Prev: While Statement, Up: Statements + +7.4.3 The 'do'-'while' Statement +-------------------------------- + +The 'do' loop is a variation of the 'while' looping statement. The 'do' +loop executes the BODY once and then repeats the BODY as long as the +CONDITION is true. It looks like this: + + do + BODY + while (CONDITION) + + Even if the CONDITION is false at the start, the BODY executes at +least once (and only once, unless executing BODY makes CONDITION true). +Contrast this with the corresponding 'while' statement: + + while (CONDITION) + BODY + +This statement does not execute the BODY even once if the CONDITION is +false to begin with. The following is an example of a 'do' statement: + + { + i = 1 + do { + print $0 + i++ + } while (i <= 10) + } + +This program prints each input record 10 times. However, it isn't a +very realistic example, because in this case an ordinary 'while' would +do just as well. This situation reflects actual experience; only +occasionally is there a real use for a 'do' statement. + + +File: gawk.info, Node: For Statement, Next: Switch Statement, Prev: Do Statement, Up: Statements + +7.4.4 The 'for' Statement +------------------------- + +The 'for' statement makes it more convenient to count iterations of a +loop. The general form of the 'for' statement looks like this: + + for (INITIALIZATION; CONDITION; INCREMENT) + BODY + +The INITIALIZATION, CONDITION, and INCREMENT parts are arbitrary 'awk' +expressions, and BODY stands for any 'awk' statement. + + The 'for' statement starts by executing INITIALIZATION. Then, as +long as the CONDITION is true, it repeatedly executes BODY and then +INCREMENT. Typically, INITIALIZATION sets a variable to either zero or +one, INCREMENT adds one to it, and CONDITION compares it against the +desired number of iterations. For example: + + awk ' + { + for (i = 1; i <= 3; i++) + print $i + }' inventory-shipped + +This prints the first three fields of each input record, with one field +per line. + + It isn't possible to set more than one variable in the INITIALIZATION +part without using a multiple assignment statement such as 'x = y = 0'. +This makes sense only if all the initial values are equal. (But it is +possible to initialize additional variables by writing their assignments +as separate statements preceding the 'for' loop.) + + The same is true of the INCREMENT part. Incrementing additional +variables requires separate statements at the end of the loop. The C +compound expression, using C's comma operator, is useful in this +context, but it is not supported in 'awk'. + + Most often, INCREMENT is an increment expression, as in the previous +example. But this is not required; it can be any expression whatsoever. +For example, the following statement prints all the powers of two +between 1 and 100: + + for (i = 1; i <= 100; i *= 2) + print i + + If there is nothing to be done, any of the three expressions in the +parentheses following the 'for' keyword may be omitted. Thus, +'for (; x > 0;)' is equivalent to 'while (x > 0)'. If the CONDITION is +omitted, it is treated as true, effectively yielding an "infinite loop" +(i.e., a loop that never terminates). + + In most cases, a 'for' loop is an abbreviation for a 'while' loop, as +shown here: + + INITIALIZATION + while (CONDITION) { + BODY + INCREMENT + } + +The only exception is when the 'continue' statement (*note Continue +Statement::) is used inside the loop. Changing a 'for' statement to a +'while' statement in this way can change the effect of the 'continue' +statement inside the loop. + + The 'awk' language has a 'for' statement in addition to a 'while' +statement because a 'for' loop is often both less work to type and more +natural to think of. Counting the number of iterations is very common +in loops. It can be easier to think of this counting as part of looping +rather than as something to do inside the loop. + + There is an alternative version of the 'for' loop, for iterating over +all the indices of an array: + + for (i in array) + DO SOMETHING WITH array[i] + +*Note Scanning an Array:: for more information on this version of the +'for' loop. + + +File: gawk.info, Node: Switch Statement, Next: Break Statement, Prev: For Statement, Up: Statements + +7.4.5 The 'switch' Statement +---------------------------- + +This minor node describes a 'gawk'-specific feature. If 'gawk' is in +compatibility mode (*note Options::), it is not available. + + The 'switch' statement allows the evaluation of an expression and the +execution of statements based on a 'case' match. Case statements are +checked for a match in the order they are defined. If no suitable +'case' is found, the 'default' section is executed, if supplied. + + Each 'case' contains a single constant, be it numeric, string, or +regexp. The 'switch' expression is evaluated, and then each 'case''s +constant is compared against the result in turn. The type of constant +determines the comparison: numeric or string do the usual comparisons. +A regexp constant does a regular expression match against the string +value of the original expression. The general form of the 'switch' +statement looks like this: + + switch (EXPRESSION) { + case VALUE OR REGULAR EXPRESSION: + CASE-BODY + default: + DEFAULT-BODY + } + + Control flow in the 'switch' statement works as it does in C. Once a +match to a given case is made, the case statement bodies execute until a +'break', 'continue', 'next', 'nextfile', or 'exit' is encountered, or +the end of the 'switch' statement itself. For example: + + while ((c = getopt(ARGC, ARGV, "aksx")) != -1) { + switch (c) { + case "a": + # report size of all files + all_files = TRUE; + break + case "k": + BLOCK_SIZE = 1024 # 1K block size + break + case "s": + # do sums only + sum_only = TRUE + break + case "x": + # don't cross filesystems + fts_flags = or(fts_flags, FTS_XDEV) + break + case "?": + default: + usage() + break + } + } + + Note that if none of the statements specified here halt execution of +a matched 'case' statement, execution falls through to the next 'case' +until execution halts. In this example, the 'case' for '"?"' falls +through to the 'default' case, which is to call a function named +'usage()'. (The 'getopt()' function being called here is described in +*note Getopt Function::.) + + +File: gawk.info, Node: Break Statement, Next: Continue Statement, Prev: Switch Statement, Up: Statements + +7.4.6 The 'break' Statement +--------------------------- + +The 'break' statement jumps out of the innermost 'for', 'while', or 'do' +loop that encloses it. The following example finds the smallest divisor +of any integer, and also identifies prime numbers: + + # find smallest divisor of num + { + num = $1 + for (divisor = 2; divisor * divisor <= num; divisor++) { + if (num % divisor == 0) + break + } + if (num % divisor == 0) + printf "Smallest divisor of %d is %d\n", num, divisor + else + printf "%d is prime\n", num + } + + When the remainder is zero in the first 'if' statement, 'awk' +immediately "breaks out" of the containing 'for' loop. This means that +'awk' proceeds immediately to the statement following the loop and +continues processing. (This is very different from the 'exit' +statement, which stops the entire 'awk' program. *Note Exit +Statement::.) + + The following program illustrates how the CONDITION of a 'for' or +'while' statement could be replaced with a 'break' inside an 'if': + + # find smallest divisor of num + { + num = $1 + for (divisor = 2; ; divisor++) { + if (num % divisor == 0) { + printf "Smallest divisor of %d is %d\n", num, divisor + break + } + if (divisor * divisor > num) { + printf "%d is prime\n", num + break + } + } + } + + The 'break' statement is also used to break out of the 'switch' +statement. This is discussed in *note Switch Statement::. + + The 'break' statement has no meaning when used outside the body of a +loop or 'switch'. However, although it was never documented, historical +implementations of 'awk' treated the 'break' statement outside of a loop +as if it were a 'next' statement (*note Next Statement::). (d.c.) +Recent versions of BWK 'awk' no longer allow this usage, nor does +'gawk'. + + +File: gawk.info, Node: Continue Statement, Next: Next Statement, Prev: Break Statement, Up: Statements + +7.4.7 The 'continue' Statement +------------------------------ + +Similar to 'break', the 'continue' statement is used only inside 'for', +'while', and 'do' loops. It skips over the rest of the loop body, +causing the next cycle around the loop to begin immediately. Contrast +this with 'break', which jumps out of the loop altogether. + + The 'continue' statement in a 'for' loop directs 'awk' to skip the +rest of the body of the loop and resume execution with the +increment-expression of the 'for' statement. The following program +illustrates this fact: + + BEGIN { + for (x = 0; x <= 20; x++) { + if (x == 5) + continue + printf "%d ", x + } + print "" + } + +This program prints all the numbers from 0 to 20--except for 5, for +which the 'printf' is skipped. Because the increment 'x++' is not +skipped, 'x' does not remain stuck at 5. Contrast the 'for' loop from +the previous example with the following 'while' loop: + + BEGIN { + x = 0 + while (x <= 20) { + if (x == 5) + continue + printf "%d ", x + x++ + } + print "" + } + +This program loops forever once 'x' reaches 5, because the increment +('x++') is never reached. + + The 'continue' statement has no special meaning with respect to the +'switch' statement, nor does it have any meaning when used outside the +body of a loop. Historical versions of 'awk' treated a 'continue' +statement outside a loop the same way they treated a 'break' statement +outside a loop: as if it were a 'next' statement (*note Next +Statement::). (d.c.) Recent versions of BWK 'awk' no longer work this +way, nor does 'gawk'. + + +File: gawk.info, Node: Next Statement, Next: Nextfile Statement, Prev: Continue Statement, Up: Statements + +7.4.8 The 'next' Statement +-------------------------- + +The 'next' statement forces 'awk' to immediately stop processing the +current record and go on to the next record. This means that no further +rules are executed for the current record, and the rest of the current +rule's action isn't executed. + + Contrast this with the effect of the 'getline' function (*note +Getline::). That also causes 'awk' to read the next record immediately, +but it does not alter the flow of control in any way (i.e., the rest of +the current action executes with a new input record). + + At the highest level, 'awk' program execution is a loop that reads an +input record and then tests each rule's pattern against it. If you +think of this loop as a 'for' statement whose body contains the rules, +then the 'next' statement is analogous to a 'continue' statement. It +skips to the end of the body of this implicit loop and executes the +increment (which reads another record). + + For example, suppose an 'awk' program works only on records with four +fields, and it shouldn't fail when given bad input. To avoid +complicating the rest of the program, write a "weed out" rule near the +beginning, in the following manner: + + NF != 4 { + printf("%s:%d: skipped: NF != 4\n", FILENAME, FNR) > "/dev/stderr" + next + } + +Because of the 'next' statement, the program's subsequent rules won't +see the bad record. The error message is redirected to the standard +error output stream, as error messages should be. For more detail, see +*note Special Files::. + + If the 'next' statement causes the end of the input to be reached, +then the code in any 'END' rules is executed. *Note BEGIN/END::. + + The 'next' statement is not allowed inside 'BEGINFILE' and 'ENDFILE' +rules. *Note BEGINFILE/ENDFILE::. + + According to the POSIX standard, the behavior is undefined if the +'next' statement is used in a 'BEGIN' or 'END' rule. 'gawk' treats it +as a syntax error. Although POSIX does not disallow it, most other +'awk' implementations don't allow the 'next' statement inside function +bodies (*note User-defined::). Just as with any other 'next' statement, +a 'next' statement inside a function body reads the next record and +starts processing it with the first rule in the program. + + +File: gawk.info, Node: Nextfile Statement, Next: Exit Statement, Prev: Next Statement, Up: Statements + +7.4.9 The 'nextfile' Statement +------------------------------ + +The 'nextfile' statement is similar to the 'next' statement. However, +instead of abandoning processing of the current record, the 'nextfile' +statement instructs 'awk' to stop processing the current data file. + + Upon execution of the 'nextfile' statement, 'FILENAME' is updated to +the name of the next data file listed on the command line, 'FNR' is +reset to one, and processing starts over with the first rule in the +program. If the 'nextfile' statement causes the end of the input to be +reached, then the code in any 'END' rules is executed. An exception to +this is when 'nextfile' is invoked during execution of any statement in +an 'END' rule; in this case, it causes the program to stop immediately. +*Note BEGIN/END::. + + The 'nextfile' statement is useful when there are many data files to +process but it isn't necessary to process every record in every file. +Without 'nextfile', in order to move on to the next data file, a program +would have to continue scanning the unwanted records. The 'nextfile' +statement accomplishes this much more efficiently. + + In 'gawk', execution of 'nextfile' causes additional things to +happen: any 'ENDFILE' rules are executed if 'gawk' is not currently in +an 'END' or 'BEGINFILE' rule, 'ARGIND' is incremented, and any +'BEGINFILE' rules are executed. ('ARGIND' hasn't been introduced yet. +*Note Built-in Variables::.) + + With 'gawk', 'nextfile' is useful inside a 'BEGINFILE' rule to skip +over a file that would otherwise cause 'gawk' to exit with a fatal +error. In this case, 'ENDFILE' rules are not executed. *Note +BEGINFILE/ENDFILE::. + + Although it might seem that 'close(FILENAME)' would accomplish the +same as 'nextfile', this isn't true. 'close()' is reserved for closing +files, pipes, and coprocesses that are opened with redirections. It is +not related to the main processing that 'awk' does with the files listed +in 'ARGV'. + + NOTE: For many years, 'nextfile' was a common extension. In + September 2012, it was accepted for inclusion into the POSIX + standard. See the Austin Group website + (http://austingroupbugs.net/view.php?id=607). + + The current version of BWK 'awk' and 'mawk' also support 'nextfile'. +However, they don't allow the 'nextfile' statement inside function +bodies (*note User-defined::). 'gawk' does; a 'nextfile' inside a +function body reads the first record from the next file and starts +processing it with the first rule in the program, just as any other +'nextfile' statement. + + +File: gawk.info, Node: Exit Statement, Prev: Nextfile Statement, Up: Statements + +7.4.10 The 'exit' Statement +--------------------------- + +The 'exit' statement causes 'awk' to immediately stop executing the +current rule and to stop processing input; any remaining input is +ignored. The 'exit' statement is written as follows: + + 'exit' [RETURN CODE] + + When an 'exit' statement is executed from a 'BEGIN' rule, the program +stops processing everything immediately. No input records are read. +However, if an 'END' rule is present, as part of executing the 'exit' +statement, the 'END' rule is executed (*note BEGIN/END::). If 'exit' is +used in the body of an 'END' rule, it causes the program to stop +immediately. + + An 'exit' statement that is not part of a 'BEGIN' or 'END' rule stops +the execution of any further automatic rules for the current record, +skips reading any remaining input records, and executes the 'END' rule +if there is one. 'gawk' also skips any 'ENDFILE' rules; they do not +execute. + + In such a case, if you don't want the 'END' rule to do its job, set a +variable to a nonzero value before the 'exit' statement and check that +variable in the 'END' rule. *Note Assert Function:: for an example that +does this. + + If an argument is supplied to 'exit', its value is used as the exit +status code for the 'awk' process. If no argument is supplied, 'exit' +causes 'awk' to return a "success" status. In the case where an +argument is supplied to a first 'exit' statement, and then 'exit' is +called a second time from an 'END' rule with no argument, 'awk' uses the +previously supplied exit value. (d.c.) *Note Exit Status:: for more +information. + + For example, suppose an error condition occurs that is difficult or +impossible to handle. Conventionally, programs report this by exiting +with a nonzero status. An 'awk' program can do this using an 'exit' +statement with a nonzero argument, as shown in the following example: + + BEGIN { + if (("date" | getline date_now) <= 0) { + print "Can't get system date" > "/dev/stderr" + exit 1 + } + print "current date is", date_now + close("date") + } + + NOTE: For full portability, exit values should be between zero and + 126, inclusive. Negative values, and values of 127 or greater, may + not produce consistent results across different operating systems. + + +File: gawk.info, Node: Built-in Variables, Next: Pattern Action Summary, Prev: Statements, Up: Patterns and Actions + +7.5 Predefined Variables +======================== + +Most 'awk' variables are available to use for your own purposes; they +never change unless your program assigns values to them, and they never +affect anything unless your program examines them. However, a few +variables in 'awk' have special built-in meanings. 'awk' examines some +of these automatically, so that they enable you to tell 'awk' how to do +certain things. Others are set automatically by 'awk', so that they +carry information from the internal workings of 'awk' to your program. + + This minor node documents all of 'gawk''s predefined variables, most +of which are also documented in the major nodes describing their areas +of activity. + +* Menu: + +* User-modified:: Built-in variables that you change to control + 'awk'. +* Auto-set:: Built-in variables where 'awk' gives + you information. +* ARGC and ARGV:: Ways to use 'ARGC' and 'ARGV'. + + +File: gawk.info, Node: User-modified, Next: Auto-set, Up: Built-in Variables + +7.5.1 Built-in Variables That Control 'awk' +------------------------------------------- + +The following is an alphabetical list of variables that you can change +to control how 'awk' does certain things. + + The variables that are specific to 'gawk' are marked with a pound +sign ('#'). These variables are 'gawk' extensions. In other 'awk' +implementations or if 'gawk' is in compatibility mode (*note Options::), +they are not special. (Any exceptions are noted in the description of +each variable.) + +'BINMODE #' + On non-POSIX systems, this variable specifies use of binary mode + for all I/O. Numeric values of one, two, or three specify that + input files, output files, or all files, respectively, should use + binary I/O. A numeric value less than zero is treated as zero, and + a numeric value greater than three is treated as three. + Alternatively, string values of '"r"' or '"w"' specify that input + files and output files, respectively, should use binary I/O. A + string value of '"rw"' or '"wr"' indicates that all files should + use binary I/O. Any other string value is treated the same as + '"rw"', but causes 'gawk' to generate a warning message. 'BINMODE' + is described in more detail in *note PC Using::. 'mawk' (*note + Other Versions::) also supports this variable, but only using + numeric values. + +'CONVFMT' + A string that controls the conversion of numbers to strings (*note + Conversion::). It works by being passed, in effect, as the first + argument to the 'sprintf()' function (*note String Functions::). + Its default value is '"%.6g"'. 'CONVFMT' was introduced by the + POSIX standard. + +'FIELDWIDTHS #' + A space-separated list of columns that tells 'gawk' how to split + input with fixed columnar boundaries. Assigning a value to + 'FIELDWIDTHS' overrides the use of 'FS' and 'FPAT' for field + splitting. *Note Constant Size:: for more information. + +'FPAT #' + A regular expression (as a string) that tells 'gawk' to create the + fields based on text that matches the regular expression. + Assigning a value to 'FPAT' overrides the use of 'FS' and + 'FIELDWIDTHS' for field splitting. *Note Splitting By Content:: + for more information. + +'FS' + The input field separator (*note Field Separators::). The value is + a single-character string or a multicharacter regular expression + that matches the separations between fields in an input record. If + the value is the null string ('""'), then each character in the + record becomes a separate field. (This behavior is a 'gawk' + extension. POSIX 'awk' does not specify the behavior when 'FS' is + the null string. Nonetheless, some other versions of 'awk' also + treat '""' specially.) + + The default value is '" "', a string consisting of a single space. + As a special exception, this value means that any sequence of + spaces, TABs, and/or newlines is a single separator. It also + causes spaces, TABs, and newlines at the beginning and end of a + record to be ignored. + + You can set the value of 'FS' on the command line using the '-F' + option: + + awk -F, 'PROGRAM' INPUT-FILES + + If 'gawk' is using 'FIELDWIDTHS' or 'FPAT' for field splitting, + assigning a value to 'FS' causes 'gawk' to return to the normal, + 'FS'-based field splitting. An easy way to do this is to simply + say 'FS = FS', perhaps with an explanatory comment. + +'IGNORECASE #' + If 'IGNORECASE' is nonzero or non-null, then all string comparisons + and all regular expression matching are case-independent. This + applies to regexp matching with '~' and '!~', the 'gensub()', + 'gsub()', 'index()', 'match()', 'patsplit()', 'split()', and + 'sub()' functions, record termination with 'RS', and field + splitting with 'FS' and 'FPAT'. However, the value of 'IGNORECASE' + does _not_ affect array subscripting and it does not affect field + splitting when using a single-character field separator. *Note + Case-sensitivity::. + +'LINT #' + When this variable is true (nonzero or non-null), 'gawk' behaves as + if the '--lint' command-line option is in effect (*note Options::). + With a value of '"fatal"', lint warnings become fatal errors. With + a value of '"invalid"', only warnings about things that are + actually invalid are issued. (This is not fully implemented yet.) + Any other true value prints nonfatal warnings. Assigning a false + value to 'LINT' turns off the lint warnings. + + This variable is a 'gawk' extension. It is not special in other + 'awk' implementations. Unlike with the other special variables, + changing 'LINT' does affect the production of lint warnings, even + if 'gawk' is in compatibility mode. Much as the '--lint' and + '--traditional' options independently control different aspects of + 'gawk''s behavior, the control of lint warnings during program + execution is independent of the flavor of 'awk' being executed. + +'OFMT' + A string that controls conversion of numbers to strings (*note + Conversion::) for printing with the 'print' statement. It works by + being passed as the first argument to the 'sprintf()' function + (*note String Functions::). Its default value is '"%.6g"'. + Earlier versions of 'awk' used 'OFMT' to specify the format for + converting numbers to strings in general expressions; this is now + done by 'CONVFMT'. + +'OFS' + The output field separator (*note Output Separators::). It is + output between the fields printed by a 'print' statement. Its + default value is '" "', a string consisting of a single space. + +'ORS' + The output record separator. It is output at the end of every + 'print' statement. Its default value is '"\n"', the newline + character. (*Note Output Separators::.) + +'PREC #' + The working precision of arbitrary-precision floating-point + numbers, 53 bits by default (*note Setting precision::). + +'ROUNDMODE #' + The rounding mode to use for arbitrary-precision arithmetic on + numbers, by default '"N"' ('roundTiesToEven' in the IEEE 754 + standard; *note Setting the rounding mode::). + +'RS' + The input record separator. Its default value is a string + containing a single newline character, which means that an input + record consists of a single line of text. It can also be the null + string, in which case records are separated by runs of blank lines. + If it is a regexp, records are separated by matches of the regexp + in the input text. (*Note Records::.) + + The ability for 'RS' to be a regular expression is a 'gawk' + extension. In most other 'awk' implementations, or if 'gawk' is in + compatibility mode (*note Options::), just the first character of + 'RS''s value is used. + +'SUBSEP' + The subscript separator. It has the default value of '"\034"' and + is used to separate the parts of the indices of a multidimensional + array. Thus, the expression 'foo["A", "B"]' really accesses + 'foo["A\034B"]' (*note Multidimensional::). + +'TEXTDOMAIN #' + Used for internationalization of programs at the 'awk' level. It + sets the default text domain for specially marked string constants + in the source text, as well as for the 'dcgettext()', + 'dcngettext()', and 'bindtextdomain()' functions (*note + Internationalization::). The default value of 'TEXTDOMAIN' is + '"messages"'. + + +File: gawk.info, Node: Auto-set, Next: ARGC and ARGV, Prev: User-modified, Up: Built-in Variables + +7.5.2 Built-in Variables That Convey Information +------------------------------------------------ + +The following is an alphabetical list of variables that 'awk' sets +automatically on certain occasions in order to provide information to +your program. + + The variables that are specific to 'gawk' are marked with a pound +sign ('#'). These variables are 'gawk' extensions. In other 'awk' +implementations or if 'gawk' is in compatibility mode (*note Options::), +they are not special: + +'ARGC', 'ARGV' + The command-line arguments available to 'awk' programs are stored + in an array called 'ARGV'. 'ARGC' is the number of command-line + arguments present. *Note Other Arguments::. Unlike most 'awk' + arrays, 'ARGV' is indexed from 0 to 'ARGC' - 1. In the following + example: + + $ awk 'BEGIN { + > for (i = 0; i < ARGC; i++) + > print ARGV[i] + > }' inventory-shipped mail-list + -| awk + -| inventory-shipped + -| mail-list + + 'ARGV[0]' contains 'awk', 'ARGV[1]' contains 'inventory-shipped', + and 'ARGV[2]' contains 'mail-list'. The value of 'ARGC' is three, + one more than the index of the last element in 'ARGV', because the + elements are numbered from zero. + + The names 'ARGC' and 'ARGV', as well as the convention of indexing + the array from 0 to 'ARGC' - 1, are derived from the C language's + method of accessing command-line arguments. + + The value of 'ARGV[0]' can vary from system to system. Also, you + should note that the program text is _not_ included in 'ARGV', nor + are any of 'awk''s command-line options. *Note ARGC and ARGV:: for + information about how 'awk' uses these variables. (d.c.) + +'ARGIND #' + The index in 'ARGV' of the current file being processed. Every + time 'gawk' opens a new data file for processing, it sets 'ARGIND' + to the index in 'ARGV' of the file name. When 'gawk' is processing + the input files, 'FILENAME == ARGV[ARGIND]' is always true. + + This variable is useful in file processing; it allows you to tell + how far along you are in the list of data files as well as to + distinguish between successive instances of the same file name on + the command line. + + While you can change the value of 'ARGIND' within your 'awk' + program, 'gawk' automatically sets it to a new value when it opens + the next file. + +'ENVIRON' + An associative array containing the values of the environment. The + array indices are the environment variable names; the elements are + the values of the particular environment variables. For example, + 'ENVIRON["HOME"]' might be '/home/arnold'. + + For POSIX 'awk', changing this array does not affect the + environment passed on to any programs that 'awk' may spawn via + redirection or the 'system()' function. + + However, beginning with version 4.2, if not in POSIX compatibility + mode, 'gawk' does update its own environment when 'ENVIRON' is + changed, thus changing the environment seen by programs that it + creates. You should therefore be especially careful if you modify + 'ENVIRON["PATH"]', which is the search path for finding executable + programs. + + This can also affect the running 'gawk' program, since some of the + built-in functions may pay attention to certain environment + variables. The most notable instance of this is 'mktime()' (*note + Time Functions::), which pays attention the value of the 'TZ' + environment variable on many systems. + + Some operating systems may not have environment variables. On such + systems, the 'ENVIRON' array is empty (except for + 'ENVIRON["AWKPATH"]' and 'ENVIRON["AWKLIBPATH"]'; *note AWKPATH + Variable:: and *note AWKLIBPATH Variable::). + +'ERRNO #' + If a system error occurs during a redirection for 'getline', during + a read for 'getline', or during a 'close()' operation, then 'ERRNO' + contains a string describing the error. + + In addition, 'gawk' clears 'ERRNO' before opening each command-line + input file. This enables checking if the file is readable inside a + 'BEGINFILE' pattern (*note BEGINFILE/ENDFILE::). + + Otherwise, 'ERRNO' works similarly to the C variable 'errno'. + Except for the case just mentioned, 'gawk' _never_ clears it (sets + it to zero or '""'). Thus, you should only expect its value to be + meaningful when an I/O operation returns a failure value, such as + 'getline' returning -1. You are, of course, free to clear it + yourself before doing an I/O operation. + + If the value of 'ERRNO' corresponds to a system error in the C + 'errno' variable, then 'PROCINFO["errno"]' will be set to the value + of 'errno'. For non-system errors, 'PROCINFO["errno"]' will be + zero. + +'FILENAME' + The name of the current input file. When no data files are listed + on the command line, 'awk' reads from the standard input and + 'FILENAME' is set to '"-"'. 'FILENAME' changes each time a new + file is read (*note Reading Files::). Inside a 'BEGIN' rule, the + value of 'FILENAME' is '""', because there are no input files being + processed yet.(1) (d.c.) Note, though, that using 'getline' + (*note Getline::) inside a 'BEGIN' rule can give 'FILENAME' a + value. + +'FNR' + The current record number in the current file. 'awk' increments + 'FNR' each time it reads a new record (*note Records::). 'awk' + resets 'FNR' to zero each time it starts a new input file. + +'NF' + The number of fields in the current input record. 'NF' is set each + time a new record is read, when a new field is created, or when + '$0' changes (*note Fields::). + + Unlike most of the variables described in this node, assigning a + value to 'NF' has the potential to affect 'awk''s internal + workings. In particular, assignments to 'NF' can be used to create + fields in or remove fields from the current record. *Note Changing + Fields::. + +'FUNCTAB #' + An array whose indices and corresponding values are the names of + all the built-in, user-defined, and extension functions in the + program. + + NOTE: Attempting to use the 'delete' statement with the + 'FUNCTAB' array causes a fatal error. Any attempt to assign + to an element of 'FUNCTAB' also causes a fatal error. + +'NR' + The number of input records 'awk' has processed since the beginning + of the program's execution (*note Records::). 'awk' increments + 'NR' each time it reads a new record. + +'PROCINFO #' + The elements of this array provide access to information about the + running 'awk' program. The following elements (listed + alphabetically) are guaranteed to be available: + + 'PROCINFO["egid"]' + The value of the 'getegid()' system call. + + 'PROCINFO["errno"]' + The value of the C 'errno' variable when 'ERRNO' is set to the + associated error message. + + 'PROCINFO["euid"]' + The value of the 'geteuid()' system call. + + 'PROCINFO["FS"]' + This is '"FS"' if field splitting with 'FS' is in effect, + '"FIELDWIDTHS"' if field splitting with 'FIELDWIDTHS' is in + effect, or '"FPAT"' if field matching with 'FPAT' is in + effect. + + 'PROCINFO["gid"]' + The value of the 'getgid()' system call. + + 'PROCINFO["identifiers"]' + A subarray, indexed by the names of all identifiers used in + the text of the 'awk' program. An "identifier" is simply the + name of a variable (be it scalar or array), built-in function, + user-defined function, or extension function. For each + identifier, the value of the element is one of the following: + + '"array"' + The identifier is an array. + + '"builtin"' + The identifier is a built-in function. + + '"extension"' + The identifier is an extension function loaded via + '@load' or '-l'. + + '"scalar"' + The identifier is a scalar. + + '"untyped"' + The identifier is untyped (could be used as a scalar or + an array; 'gawk' doesn't know yet). + + '"user"' + The identifier is a user-defined function. + + The values indicate what 'gawk' knows about the identifiers + after it has finished parsing the program; they are _not_ + updated while the program runs. + + 'PROCINFO["pgrpid"]' + The process group ID of the current process. + + 'PROCINFO["pid"]' + The process ID of the current process. + + 'PROCINFO["ppid"]' + The parent process ID of the current process. + + 'PROCINFO["strftime"]' + The default time format string for 'strftime()'. Assigning a + new value to this element changes the default. *Note Time + Functions::. + + 'PROCINFO["uid"]' + The value of the 'getuid()' system call. + + 'PROCINFO["version"]' + The version of 'gawk'. + + The following additional elements in the array are available to + provide information about the MPFR and GMP libraries if your + version of 'gawk' supports arbitrary-precision arithmetic (*note + Arbitrary Precision Arithmetic::): + + 'PROCINFO["gmp_version"]' + The version of the GNU MP library. + + 'PROCINFO["mpfr_version"]' + The version of the GNU MPFR library. + + 'PROCINFO["prec_max"]' + The maximum precision supported by MPFR. + + 'PROCINFO["prec_min"]' + The minimum precision required by MPFR. + + The following additional elements in the array are available to + provide information about the version of the extension API, if your + version of 'gawk' supports dynamic loading of extension functions + (*note Dynamic Extensions::): + + 'PROCINFO["api_major"]' + The major version of the extension API. + + 'PROCINFO["api_minor"]' + The minor version of the extension API. + + On some systems, there may be elements in the array, '"group1"' + through '"groupN"' for some N. N is the number of supplementary + groups that the process has. Use the 'in' operator to test for + these elements (*note Reference to Elements::). + + The following elements allow you to change 'gawk''s behavior: + + 'PROCINFO["NONFATAL"]' + If this element exists, then I/O errors for all output + redirections become nonfatal. *Note Nonfatal::. + + 'PROCINFO["OUTPUT_NAME", "NONFATAL"]' + Make output errors for OUTPUT_NAME be nonfatal. *Note + Nonfatal::. + + 'PROCINFO["COMMAND", "pty"]' + For two-way communication to COMMAND, use a pseudo-tty instead + of setting up a two-way pipe. *Note Two-way I/O:: for more + information. + + 'PROCINFO["INPUT_NAME", "READ_TIMEOUT"]' + Set a timeout for reading from input redirection INPUT_NAME. + *Note Read Timeout:: for more information. + + 'PROCINFO["sorted_in"]' + If this element exists in 'PROCINFO', its value controls the + order in which array indices will be processed by 'for (INDX + in ARRAY)' loops. This is an advanced feature, so we defer + the full description until later; see *note Scanning an + Array::. + +'RLENGTH' + The length of the substring matched by the 'match()' function + (*note String Functions::). 'RLENGTH' is set by invoking the + 'match()' function. Its value is the length of the matched string, + or -1 if no match is found. + +'RSTART' + The start index in characters of the substring that is matched by + the 'match()' function (*note String Functions::). 'RSTART' is set + by invoking the 'match()' function. Its value is the position of + the string where the matched substring starts, or zero if no match + was found. + +'RT #' + The input text that matched the text denoted by 'RS', the record + separator. It is set every time a record is read. + +'SYMTAB #' + An array whose indices are the names of all defined global + variables and arrays in the program. 'SYMTAB' makes 'gawk''s + symbol table visible to the 'awk' programmer. It is built as + 'gawk' parses the program and is complete before the program starts + to run. + + The array may be used for indirect access to read or write the + value of a variable: + + foo = 5 + SYMTAB["foo"] = 4 + print foo # prints 4 + + The 'isarray()' function (*note Type Functions::) may be used to + test if an element in 'SYMTAB' is an array. Also, you may not use + the 'delete' statement with the 'SYMTAB' array. + + You may use an index for 'SYMTAB' that is not a predefined + identifier: + + SYMTAB["xxx"] = 5 + print SYMTAB["xxx"] + + This works as expected: in this case 'SYMTAB' acts just like a + regular array. The only difference is that you can't then delete + 'SYMTAB["xxx"]'. + + The 'SYMTAB' array is more interesting than it looks. Andrew + Schorr points out that it effectively gives 'awk' data pointers. + Consider his example: + + # Indirect multiply of any variable by amount, return result + + function multiply(variable, amount) + { + return SYMTAB[variable] *= amount + } + + You would use it like this: + + BEGIN { + answer = 10.5 + multiply("answer", 4) + print "The answer is", answer + } + + When run, this produces: + + $ gawk -f answer.awk + -| The answer is 42 + + NOTE: In order to avoid severe time-travel paradoxes,(2) + neither 'FUNCTAB' nor 'SYMTAB' is available as an element + within the 'SYMTAB' array. + + Changing 'NR' and 'FNR' + + 'awk' increments 'NR' and 'FNR' each time it reads a record, instead +of setting them to the absolute value of the number of records read. +This means that a program can change these variables and their new +values are incremented for each record. (d.c.) The following example +shows this: + + $ echo '1 + > 2 + > 3 + > 4' | awk 'NR == 2 { NR = 17 } + > { print NR }' + -| 1 + -| 17 + -| 18 + -| 19 + +Before 'FNR' was added to the 'awk' language (*note V7/SVR3.1::), many +'awk' programs used this feature to track the number of records in a +file by resetting 'NR' to zero when 'FILENAME' changed. + + ---------- Footnotes ---------- + + (1) Some early implementations of Unix 'awk' initialized 'FILENAME' +to '"-"', even if there were data files to be processed. This behavior +was incorrect and should not be relied upon in your programs. + + (2) Not to mention difficult implementation issues. + + +File: gawk.info, Node: ARGC and ARGV, Prev: Auto-set, Up: Built-in Variables + +7.5.3 Using 'ARGC' and 'ARGV' +----------------------------- + +*note Auto-set:: presented the following program describing the +information contained in 'ARGC' and 'ARGV': + + $ awk 'BEGIN { + > for (i = 0; i < ARGC; i++) + > print ARGV[i] + > }' inventory-shipped mail-list + -| awk + -| inventory-shipped + -| mail-list + +In this example, 'ARGV[0]' contains 'awk', 'ARGV[1]' contains +'inventory-shipped', and 'ARGV[2]' contains 'mail-list'. Notice that +the 'awk' program is not entered in 'ARGV'. The other command-line +options, with their arguments, are also not entered. This includes +variable assignments done with the '-v' option (*note Options::). +Normal variable assignments on the command line _are_ treated as +arguments and do show up in the 'ARGV' array. Given the following +program in a file named 'showargs.awk': + + BEGIN { + printf "A=%d, B=%d\n", A, B + for (i = 0; i < ARGC; i++) + printf "\tARGV[%d] = %s\n", i, ARGV[i] + } + END { printf "A=%d, B=%d\n", A, B } + +Running it produces the following: + + $ awk -v A=1 -f showargs.awk B=2 /dev/null + -| A=1, B=0 + -| ARGV[0] = awk + -| ARGV[1] = B=2 + -| ARGV[2] = /dev/null + -| A=1, B=2 + + A program can alter 'ARGC' and the elements of 'ARGV'. Each time +'awk' reaches the end of an input file, it uses the next element of +'ARGV' as the name of the next input file. By storing a different +string there, a program can change which files are read. Use '"-"' to +represent the standard input. Storing additional elements and +incrementing 'ARGC' causes additional files to be read. + + If the value of 'ARGC' is decreased, that eliminates input files from +the end of the list. By recording the old value of 'ARGC' elsewhere, a +program can treat the eliminated arguments as something other than file +names. + + To eliminate a file from the middle of the list, store the null +string ('""') into 'ARGV' in place of the file's name. As a special +feature, 'awk' ignores file names that have been replaced with the null +string. Another option is to use the 'delete' statement to remove +elements from 'ARGV' (*note Delete::). + + All of these actions are typically done in the 'BEGIN' rule, before +actual processing of the input begins. *Note Split Program:: and *note +Tee Program:: for examples of each way of removing elements from 'ARGV'. + + To actually get options into an 'awk' program, end the 'awk' options +with '--' and then supply the 'awk' program's options, in the following +manner: + + awk -f myprog.awk -- -v -q file1 file2 ... + + The following fragment processes 'ARGV' in order to examine, and then +remove, the previously mentioned command-line options: + + BEGIN { + for (i = 1; i < ARGC; i++) { + if (ARGV[i] == "-v") + verbose = 1 + else if (ARGV[i] == "-q") + debug = 1 + else if (ARGV[i] ~ /^-./) { + e = sprintf("%s: unrecognized option -- %c", + ARGV[0], substr(ARGV[i], 2, 1)) + print e > "/dev/stderr" + } else + break + delete ARGV[i] + } + } + + Ending the 'awk' options with '--' isn't necessary in 'gawk'. Unless +'--posix' has been specified, 'gawk' silently puts any unrecognized +options into 'ARGV' for the 'awk' program to deal with. As soon as it +sees an unknown option, 'gawk' stops looking for other options that it +might otherwise recognize. The previous command line with 'gawk' would +be: + + gawk -f myprog.awk -q -v file1 file2 ... + +Because '-q' is not a valid 'gawk' option, it and the following '-v' are +passed on to the 'awk' program. (*Note Getopt Function:: for an 'awk' +library function that parses command-line options.) + + When designing your program, you should choose options that don't +conflict with 'gawk''s, because it will process any options that it +accepts before passing the rest of the command line on to your program. +Using '#!' with the '-E' option may help (*note Executable Scripts:: and +*note Options::). + + +File: gawk.info, Node: Pattern Action Summary, Prev: Built-in Variables, Up: Patterns and Actions + +7.6 Summary +=========== + + * Pattern-action pairs make up the basic elements of an 'awk' + program. Patterns are either normal expressions, range + expressions, or regexp constants; one of the special keywords + 'BEGIN', 'END', 'BEGINFILE', or 'ENDFILE'; or empty. The action + executes if the current record matches the pattern. Empty + (missing) patterns match all records. + + * I/O from 'BEGIN' and 'END' rules has certain constraints. This is + also true, only more so, for 'BEGINFILE' and 'ENDFILE' rules. The + latter two give you "hooks" into 'gawk''s file processing, allowing + you to recover from a file that otherwise would cause a fatal error + (such as a file that cannot be opened). + + * Shell variables can be used in 'awk' programs by careful use of + shell quoting. It is easier to pass a shell variable into 'awk' by + using the '-v' option and an 'awk' variable. + + * Actions consist of statements enclosed in curly braces. Statements + are built up from expressions, control statements, compound + statements, input and output statements, and deletion statements. + + * The control statements in 'awk' are 'if'-'else', 'while', 'for', + and 'do'-'while'. 'gawk' adds the 'switch' statement. There are + two flavors of 'for' statement: one for performing general looping, + and the other for iterating through an array. + + * 'break' and 'continue' let you exit early or start the next + iteration of a loop (or get out of a 'switch'). + + * 'next' and 'nextfile' let you read the next record and start over + at the top of your program or skip to the next input file and start + over, respectively. + + * The 'exit' statement terminates your program. When executed from + an action (or function body), it transfers control to the 'END' + statements. From an 'END' statement body, it exits immediately. + You may pass an optional numeric value to be used as 'awk''s exit + status. + + * Some predefined variables provide control over 'awk', mainly for + I/O. Other variables convey information from 'awk' to your program. + + * 'ARGC' and 'ARGV' make the command-line arguments available to your + program. Manipulating them from a 'BEGIN' rule lets you control + how 'awk' will process the provided data files. + + +File: gawk.info, Node: Arrays, Next: Functions, Prev: Patterns and Actions, Up: Top + +8 Arrays in 'awk' +***************** + +An "array" is a table of values called "elements". The elements of an +array are distinguished by their "indices". Indices may be either +numbers or strings. + + This major node describes how arrays work in 'awk', how to use array +elements, how to scan through every element in an array, and how to +remove array elements. It also describes how 'awk' simulates +multidimensional arrays, as well as some of the less obvious points +about array usage. The major node moves on to discuss 'gawk''s facility +for sorting arrays, and ends with a brief description of 'gawk''s +ability to support true arrays of arrays. + +* Menu: + +* Array Basics:: The basics of arrays. +* Numeric Array Subscripts:: How to use numbers as subscripts in + 'awk'. +* Uninitialized Subscripts:: Using Uninitialized variables as subscripts. +* Delete:: The 'delete' statement removes an element + from an array. +* Multidimensional:: Emulating multidimensional arrays in + 'awk'. +* Arrays of Arrays:: True multidimensional arrays. +* Arrays Summary:: Summary of arrays. + + +File: gawk.info, Node: Array Basics, Next: Numeric Array Subscripts, Up: Arrays + +8.1 The Basics of Arrays +======================== + +This minor node presents the basics: working with elements in arrays one +at a time, and traversing all of the elements in an array. + +* Menu: + +* Array Intro:: Introduction to Arrays +* Reference to Elements:: How to examine one element of an array. +* Assigning Elements:: How to change an element of an array. +* Array Example:: Basic Example of an Array +* Scanning an Array:: A variation of the 'for' statement. It + loops through the indices of an array's + existing elements. +* Controlling Scanning:: Controlling the order in which arrays are + scanned. + + +File: gawk.info, Node: Array Intro, Next: Reference to Elements, Up: Array Basics + +8.1.1 Introduction to Arrays +---------------------------- + + Doing linear scans over an associative array is like trying to club + someone to death with a loaded Uzi. + -- _Larry Wall_ + + The 'awk' language provides one-dimensional arrays for storing groups +of related strings or numbers. Every 'awk' array must have a name. +Array names have the same syntax as variable names; any valid variable +name would also be a valid array name. But one name cannot be used in +both ways (as an array and as a variable) in the same 'awk' program. + + Arrays in 'awk' superficially resemble arrays in other programming +languages, but there are fundamental differences. In 'awk', it isn't +necessary to specify the size of an array before starting to use it. +Additionally, any number or string, not just consecutive integers, may +be used as an array index. + + In most other languages, arrays must be "declared" before use, +including a specification of how many elements or components they +contain. In such languages, the declaration causes a contiguous block +of memory to be allocated for that many elements. Usually, an index in +the array must be a nonnegative integer. For example, the index zero +specifies the first element in the array, which is actually stored at +the beginning of the block of memory. Index one specifies the second +element, which is stored in memory right after the first element, and so +on. It is impossible to add more elements to the array, because it has +room only for as many elements as given in the declaration. (Some +languages allow arbitrary starting and ending indices--e.g., '15 .. +27'--but the size of the array is still fixed when the array is +declared.) + + A contiguous array of four elements might look like *note Figure 8.1: +figure-array-elements, conceptually, if the element values are eight, +'"foo"', '""', and 30. + + +| 8 | \"foo\" | \"\" | 30 | Value ++---------+---------+--------+---------+ + 0 1 2 3 Index" + +Figure 8.1: A contiguous array + +Only the values are stored; the indices are implicit from the order of +the values. Here, eight is the value at index zero, because eight +appears in the position with zero elements before it. + + Arrays in 'awk' are different--they are "associative". This means +that each array is a collection of pairs--an index and its corresponding +array element value: + + Index Value +------------------------ + '3' '30' + '1' '"foo"' + '0' '8' + '2' '""' + +The pairs are shown in jumbled order because their order is +irrelevant.(1) + + One advantage of associative arrays is that new pairs can be added at +any time. For example, suppose a tenth element is added to the array +whose value is '"number ten"'. The result is: + + Index Value +------------------------------- + '10' '"number + ten"' + '3' '30' + '1' '"foo"' + '0' '8' + '2' '""' + +Now the array is "sparse", which just means some indices are missing. +It has elements 0-3 and 10, but doesn't have elements 4, 5, 6, 7, 8, or +9. + + Another consequence of associative arrays is that the indices don't +have to be nonnegative integers. Any number, or even a string, can be +an index. For example, the following is an array that translates words +from English to French: + + Index Value +------------------------ + '"dog"' '"chien"' + '"cat"' '"chat"' + '"one"' '"un"' + '1' '"un"' + +Here we decided to translate the number one in both spelled-out and +numeric form--thus illustrating that a single array can have both +numbers and strings as indices. (In fact, array subscripts are always +strings. There are some subtleties to how numbers work when used as +array subscripts; this is discussed in more detail in *note Numeric +Array Subscripts::.) Here, the number '1' isn't double-quoted, because +'awk' automatically converts it to a string. + + The value of 'IGNORECASE' has no effect upon array subscripting. The +identical string value used to store an array element must be used to +retrieve it. When 'awk' creates an array (e.g., with the 'split()' +built-in function), that array's indices are consecutive integers +starting at one. (*Note String Functions::.) + + 'awk''s arrays are efficient--the time to access an element is +independent of the number of elements in the array. + + ---------- Footnotes ---------- + + (1) The ordering will vary among 'awk' implementations, which +typically use hash tables to store array elements and values. + + +File: gawk.info, Node: Reference to Elements, Next: Assigning Elements, Prev: Array Intro, Up: Array Basics + +8.1.2 Referring to an Array Element +----------------------------------- + +The principal way to use an array is to refer to one of its elements. +An "array reference" is an expression as follows: + + ARRAY[INDEX-EXPRESSION] + +Here, ARRAY is the name of an array. The expression INDEX-EXPRESSION is +the index of the desired element of the array. + + The value of the array reference is the current value of that array +element. For example, 'foo[4.3]' is an expression referencing the +element of array 'foo' at index '4.3'. + + A reference to an array element that has no recorded value yields a +value of '""', the null string. This includes elements that have not +been assigned any value as well as elements that have been deleted +(*note Delete::). + + NOTE: A reference to an element that does not exist _automatically_ + creates that array element, with the null string as its value. (In + some cases, this is unfortunate, because it might waste memory + inside 'awk'.) + + Novice 'awk' programmers often make the mistake of checking if an + element exists by checking if the value is empty: + + # Check if "foo" exists in a: Incorrect! + if (a["foo"] != "") ... + + This is incorrect for two reasons. First, it _creates_ 'a["foo"]' + if it didn't exist before! Second, it is valid (if a bit unusual) + to set an array element equal to the empty string. + + To determine whether an element exists in an array at a certain +index, use the following expression: + + INDX in ARRAY + +This expression tests whether the particular index INDX exists, without +the side effect of creating that element if it is not present. The +expression has the value one (true) if 'ARRAY[INDX]' exists and zero +(false) if it does not exist. (We use INDX here, because 'index' is the +name of a built-in function.) For example, this statement tests whether +the array 'frequencies' contains the index '2': + + if (2 in frequencies) + print "Subscript 2 is present." + + Note that this is _not_ a test of whether the array 'frequencies' +contains an element whose _value_ is two. There is no way to do that +except to scan all the elements. Also, this _does not_ create +'frequencies[2]', while the following (incorrect) alternative does: + + if (frequencies[2] != "") + print "Subscript 2 is present." + + +File: gawk.info, Node: Assigning Elements, Next: Array Example, Prev: Reference to Elements, Up: Array Basics + +8.1.3 Assigning Array Elements +------------------------------ + +Array elements can be assigned values just like 'awk' variables: + + ARRAY[INDEX-EXPRESSION] = VALUE + +ARRAY is the name of an array. The expression INDEX-EXPRESSION is the +index of the element of the array that is assigned a value. The +expression VALUE is the value to assign to that element of the array. + + +File: gawk.info, Node: Array Example, Next: Scanning an Array, Prev: Assigning Elements, Up: Array Basics + +8.1.4 Basic Array Example +------------------------- + +The following program takes a list of lines, each beginning with a line +number, and prints them out in order of line number. The line numbers +are not in order when they are first read--instead, they are scrambled. +This program sorts the lines by making an array using the line numbers +as subscripts. The program then prints out the lines in sorted order of +their numbers. It is a very simple program and gets confused upon +encountering repeated numbers, gaps, or lines that don't begin with a +number: + + { + if ($1 > max) + max = $1 + arr[$1] = $0 + } + + END { + for (x = 1; x <= max; x++) + print arr[x] + } + + The first rule keeps track of the largest line number seen so far; it +also stores each line into the array 'arr', at an index that is the +line's number. The second rule runs after all the input has been read, +to print out all the lines. When this program is run with the following +input: + + 5 I am the Five man + 2 Who are you? The new number two! + 4 . . . And four on the floor + 1 Who is number one? + 3 I three you. + +Its output is: + + 1 Who is number one? + 2 Who are you? The new number two! + 3 I three you. + 4 . . . And four on the floor + 5 I am the Five man + + If a line number is repeated, the last line with a given number +overrides the others. Gaps in the line numbers can be handled with an +easy improvement to the program's 'END' rule, as follows: + + END { + for (x = 1; x <= max; x++) + if (x in arr) + print arr[x] + } + + +File: gawk.info, Node: Scanning an Array, Next: Controlling Scanning, Prev: Array Example, Up: Array Basics + +8.1.5 Scanning All Elements of an Array +--------------------------------------- + +In programs that use arrays, it is often necessary to use a loop that +executes once for each element of an array. In other languages, where +arrays are contiguous and indices are limited to nonnegative integers, +this is easy: all the valid indices can be found by counting from the +lowest index up to the highest. This technique won't do the job in +'awk', because any number or string can be an array index. So 'awk' has +a special kind of 'for' statement for scanning an array: + + for (VAR in ARRAY) + BODY + +This loop executes BODY once for each index in ARRAY that the program +has previously used, with the variable VAR set to that index. + + The following program uses this form of the 'for' statement. The +first rule scans the input records and notes which words appear (at +least once) in the input, by storing a one into the array 'used' with +the word as the index. The second rule scans the elements of 'used' to +find all the distinct words that appear in the input. It prints each +word that is more than 10 characters long and also prints the number of +such words. *Note String Functions:: for more information on the +built-in function 'length()'. + + # Record a 1 for each word that is used at least once + { + for (i = 1; i <= NF; i++) + used[$i] = 1 + } + + # Find number of distinct words more than 10 characters long + END { + for (x in used) { + if (length(x) > 10) { + ++num_long_words + print x + } + } + print num_long_words, "words longer than 10 characters" + } + +*Note Word Sorting:: for a more detailed example of this type. + + The order in which elements of the array are accessed by this +statement is determined by the internal arrangement of the array +elements within 'awk' and in standard 'awk' cannot be controlled or +changed. This can lead to problems if new elements are added to ARRAY +by statements in the loop body; it is not predictable whether the 'for' +loop will reach them. Similarly, changing VAR inside the loop may +produce strange results. It is best to avoid such things. + + As a point of information, 'gawk' sets up the list of elements to be +iterated over before the loop starts, and does not change it. But not +all 'awk' versions do so. Consider this program, named 'loopcheck.awk': + + BEGIN { + a["here"] = "here" + a["is"] = "is" + a["a"] = "a" + a["loop"] = "loop" + for (i in a) { + j++ + a[j] = j + print i + } + } + + Here is what happens when run with 'gawk' (and 'mawk'): + + $ gawk -f loopcheck.awk + -| here + -| loop + -| a + -| is + + Contrast this to BWK 'awk': + + $ nawk -f loopcheck.awk + -| loop + -| here + -| is + -| a + -| 1 + + +File: gawk.info, Node: Controlling Scanning, Prev: Scanning an Array, Up: Array Basics + +8.1.6 Using Predefined Array Scanning Orders with 'gawk' +-------------------------------------------------------- + +This node describes a feature that is specific to 'gawk'. + + By default, when a 'for' loop traverses an array, the order is +undefined, meaning that the 'awk' implementation determines the order in +which the array is traversed. This order is usually based on the +internal implementation of arrays and will vary from one version of +'awk' to the next. + + Often, though, you may wish to do something simple, such as "traverse +the array by comparing the indices in ascending order," or "traverse the +array by comparing the values in descending order." 'gawk' provides two +mechanisms that give you this control: + + * Set 'PROCINFO["sorted_in"]' to one of a set of predefined values. + We describe this now. + + * Set 'PROCINFO["sorted_in"]' to the name of a user-defined function + to use for comparison of array elements. This advanced feature is + described later in *note Array Sorting::. + + The following special values for 'PROCINFO["sorted_in"]' are +available: + +'"@unsorted"' + Array elements are processed in arbitrary order, which is the + default 'awk' behavior. + +'"@ind_str_asc"' + Order by indices in ascending order compared as strings; this is + the most basic sort. (Internally, array indices are always + strings, so with 'a[2*5] = 1' the index is '"10"' rather than + numeric 10.) + +'"@ind_num_asc"' + Order by indices in ascending order but force them to be treated as + numbers in the process. Any index with a non-numeric value will + end up positioned as if it were zero. + +'"@val_type_asc"' + Order by element values in ascending order (rather than by + indices). Ordering is by the type assigned to the element (*note + Typing and Comparison::). All numeric values come before all + string values, which in turn come before all subarrays. (Subarrays + have not been described yet; *note Arrays of Arrays::.) + +'"@val_str_asc"' + Order by element values in ascending order (rather than by + indices). Scalar values are compared as strings. Subarrays, if + present, come out last. + +'"@val_num_asc"' + Order by element values in ascending order (rather than by + indices). Scalar values are compared as numbers. Subarrays, if + present, come out last. When numeric values are equal, the string + values are used to provide an ordering: this guarantees consistent + results across different versions of the C 'qsort()' function,(1) + which 'gawk' uses internally to perform the sorting. + +'"@ind_str_desc"' + Like '"@ind_str_asc"', but the string indices are ordered from high + to low. + +'"@ind_num_desc"' + Like '"@ind_num_asc"', but the numeric indices are ordered from + high to low. + +'"@val_type_desc"' + Like '"@val_type_asc"', but the element values, based on type, are + ordered from high to low. Subarrays, if present, come out first. + +'"@val_str_desc"' + Like '"@val_str_asc"', but the element values, treated as strings, + are ordered from high to low. Subarrays, if present, come out + first. + +'"@val_num_desc"' + Like '"@val_num_asc"', but the element values, treated as numbers, + are ordered from high to low. Subarrays, if present, come out + first. + + The array traversal order is determined before the 'for' loop starts +to run. Changing 'PROCINFO["sorted_in"]' in the loop body does not +affect the loop. For example: + + $ gawk ' + > BEGIN { + > a[4] = 4 + > a[3] = 3 + > for (i in a) + > print i, a[i] + > }' + -| 4 4 + -| 3 3 + $ gawk ' + > BEGIN { + > PROCINFO["sorted_in"] = "@ind_str_asc" + > a[4] = 4 + > a[3] = 3 + > for (i in a) + > print i, a[i] + > }' + -| 3 3 + -| 4 4 + + When sorting an array by element values, if a value happens to be a +subarray then it is considered to be greater than any string or numeric +value, regardless of what the subarray itself contains, and all +subarrays are treated as being equal to each other. Their order +relative to each other is determined by their index strings. + + Here are some additional things to bear in mind about sorted array +traversal: + + * The value of 'PROCINFO["sorted_in"]' is global. That is, it + affects all array traversal 'for' loops. If you need to change it + within your own code, you should see if it's defined and save and + restore the value: + + ... + if ("sorted_in" in PROCINFO) { + save_sorted = PROCINFO["sorted_in"] + PROCINFO["sorted_in"] = "@val_str_desc" # or whatever + } + ... + if (save_sorted) + PROCINFO["sorted_in"] = save_sorted + + * As already mentioned, the default array traversal order is + represented by '"@unsorted"'. You can also get the default + behavior by assigning the null string to 'PROCINFO["sorted_in"]' or + by just deleting the '"sorted_in"' element from the 'PROCINFO' + array with the 'delete' statement. (The 'delete' statement hasn't + been described yet; *note Delete::.) + + In addition, 'gawk' provides built-in functions for sorting arrays; +see *note Array Sorting Functions::. + + ---------- Footnotes ---------- + + (1) When two elements compare as equal, the C 'qsort()' function does +not guarantee that they will maintain their original relative order +after sorting. Using the string value to provide a unique ordering when +the numeric values are equal ensures that 'gawk' behaves consistently +across different environments. + + +File: gawk.info, Node: Numeric Array Subscripts, Next: Uninitialized Subscripts, Prev: Array Basics, Up: Arrays + +8.2 Using Numbers to Subscript Arrays +===================================== + +An important aspect to remember about arrays is that _array subscripts +are always strings_. When a numeric value is used as a subscript, it is +converted to a string value before being used for subscripting (*note +Conversion::). This means that the value of the predefined variable +'CONVFMT' can affect how your program accesses elements of an array. +For example: + + xyz = 12.153 + data[xyz] = 1 + CONVFMT = "%2.2f" + if (xyz in data) + printf "%s is in data\n", xyz + else + printf "%s is not in data\n", xyz + +This prints '12.15 is not in data'. The first statement gives 'xyz' a +numeric value. Assigning to 'data[xyz]' subscripts 'data' with the +string value '"12.153"' (using the default conversion value of +'CONVFMT', '"%.6g"'). Thus, the array element 'data["12.153"]' is +assigned the value one. The program then changes the value of +'CONVFMT'. The test '(xyz in data)' generates a new string value from +'xyz'--this time '"12.15"'--because the value of 'CONVFMT' only allows +two significant digits. This test fails, because '"12.15"' is different +from '"12.153"'. + + According to the rules for conversions (*note Conversion::), integer +values always convert to strings as integers, no matter what the value +of 'CONVFMT' may happen to be. So the usual case of the following +works: + + for (i = 1; i <= maxsub; i++) + do something with array[i] + + The "integer values always convert to strings as integers" rule has +an additional consequence for array indexing. Octal and hexadecimal +constants (*note Nondecimal-numbers::) are converted internally into +numbers, and their original form is forgotten. This means, for example, +that 'array[17]', 'array[021]', and 'array[0x11]' all refer to the same +element! + + As with many things in 'awk', the majority of the time things work as +you would expect them to. But it is useful to have a precise knowledge +of the actual rules, as they can sometimes have a subtle effect on your +programs. + + +File: gawk.info, Node: Uninitialized Subscripts, Next: Delete, Prev: Numeric Array Subscripts, Up: Arrays + +8.3 Using Uninitialized Variables as Subscripts +=============================================== + +Suppose it's necessary to write a program to print the input data in +reverse order. A reasonable attempt to do so (with some test data) +might look like this: + + $ echo 'line 1 + > line 2 + > line 3' | awk '{ l[lines] = $0; ++lines } + > END { + > for (i = lines - 1; i >= 0; i--) + > print l[i] + > }' + -| line 3 + -| line 2 + + Unfortunately, the very first line of input data did not appear in +the output! + + Upon first glance, we would think that this program should have +worked. The variable 'lines' is uninitialized, and uninitialized +variables have the numeric value zero. So, 'awk' should have printed +the value of 'l[0]'. + + The issue here is that subscripts for 'awk' arrays are _always_ +strings. Uninitialized variables, when used as strings, have the value +'""', not zero. Thus, 'line 1' ends up stored in 'l[""]'. The +following version of the program works correctly: + + { l[lines++] = $0 } + END { + for (i = lines - 1; i >= 0; i--) + print l[i] + } + + Here, the '++' forces 'lines' to be numeric, thus making the "old +value" numeric zero. This is then converted to '"0"' as the array +subscript. + + Even though it is somewhat unusual, the null string ('""') is a valid +array subscript. (d.c.) 'gawk' warns about the use of the null string +as a subscript if '--lint' is provided on the command line (*note +Options::). + + +File: gawk.info, Node: Delete, Next: Multidimensional, Prev: Uninitialized Subscripts, Up: Arrays + +8.4 The 'delete' Statement +========================== + +To remove an individual element of an array, use the 'delete' statement: + + delete ARRAY[INDEX-EXPRESSION] + + Once an array element has been deleted, any value the element once +had is no longer available. It is as if the element had never been +referred to or been given a value. The following is an example of +deleting elements in an array: + + for (i in frequencies) + delete frequencies[i] + +This example removes all the elements from the array 'frequencies'. +Once an element is deleted, a subsequent 'for' statement to scan the +array does not report that element and using the 'in' operator to check +for the presence of that element returns zero (i.e., false): + + delete foo[4] + if (4 in foo) + print "This will never be printed" + + It is important to note that deleting an element is _not_ the same as +assigning it a null value (the empty string, '""'). For example: + + foo[4] = "" + if (4 in foo) + print "This is printed, even though foo[4] is empty" + + It is not an error to delete an element that does not exist. +However, if '--lint' is provided on the command line (*note Options::), +'gawk' issues a warning message when an element that is not in the array +is deleted. + + All the elements of an array may be deleted with a single statement +by leaving off the subscript in the 'delete' statement, as follows: + + delete ARRAY + + Using this version of the 'delete' statement is about three times +more efficient than the equivalent loop that deletes each element one at +a time. + + This form of the 'delete' statement is also supported by BWK 'awk' +and 'mawk', as well as by a number of other implementations. + + NOTE: For many years, using 'delete' without a subscript was a + common extension. In September 2012, it was accepted for inclusion + into the POSIX standard. See the Austin Group website + (http://austingroupbugs.net/view.php?id=544). + + The following statement provides a portable but nonobvious way to +clear out an array:(1) + + split("", array) + + The 'split()' function (*note String Functions::) clears out the +target array first. This call asks it to split apart the null string. +Because there is no data to split out, the function simply clears the +array and then returns. + + CAUTION: Deleting all the elements from an array does not change + its type; you cannot clear an array and then use the array's name + as a scalar (i.e., a regular variable). For example, the following + does not work: + + a[1] = 3 + delete a + a = 3 + + ---------- Footnotes ---------- + + (1) Thanks to Michael Brennan for pointing this out. + + +File: gawk.info, Node: Multidimensional, Next: Arrays of Arrays, Prev: Delete, Up: Arrays + +8.5 Multidimensional Arrays +=========================== + +* Menu: + +* Multiscanning:: Scanning multidimensional arrays. + +A "multidimensional array" is an array in which an element is identified +by a sequence of indices instead of a single index. For example, a +two-dimensional array requires two indices. The usual way (in many +languages, including 'awk') to refer to an element of a two-dimensional +array named 'grid' is with 'grid[X,Y]'. + + Multidimensional arrays are supported in 'awk' through concatenation +of indices into one string. 'awk' converts the indices into strings +(*note Conversion::) and concatenates them together, with a separator +between them. This creates a single string that describes the values of +the separate indices. The combined string is used as a single index +into an ordinary, one-dimensional array. The separator used is the +value of the built-in variable 'SUBSEP'. + + For example, suppose we evaluate the expression 'foo[5,12] = "value"' +when the value of 'SUBSEP' is '"@"'. The numbers 5 and 12 are converted +to strings and concatenated with an '@' between them, yielding '"5@12"'; +thus, the array element 'foo["5@12"]' is set to '"value"'. + + Once the element's value is stored, 'awk' has no record of whether it +was stored with a single index or a sequence of indices. The two +expressions 'foo[5,12]' and 'foo[5 SUBSEP 12]' are always equivalent. + + The default value of 'SUBSEP' is the string '"\034"', which contains +a nonprinting character that is unlikely to appear in an 'awk' program +or in most input data. The usefulness of choosing an unlikely character +comes from the fact that index values that contain a string matching +'SUBSEP' can lead to combined strings that are ambiguous. Suppose that +'SUBSEP' is '"@"'; then 'foo["a@b", "c"]' and 'foo["a", "b@c"]' are +indistinguishable because both are actually stored as 'foo["a@b@c"]'. + + To test whether a particular index sequence exists in a +multidimensional array, use the same operator ('in') that is used for +single-dimensional arrays. Write the whole sequence of indices in +parentheses, separated by commas, as the left operand: + + if ((SUBSCRIPT1, SUBSCRIPT2, ...) in ARRAY) + ... + + Here is an example that treats its input as a two-dimensional array +of fields; it rotates this array 90 degrees clockwise and prints the +result. It assumes that all lines have the same number of elements: + + { + if (max_nf < NF) + max_nf = NF + max_nr = NR + for (x = 1; x <= NF; x++) + vector[x, NR] = $x + } + + END { + for (x = 1; x <= max_nf; x++) { + for (y = max_nr; y >= 1; --y) + printf("%s ", vector[x, y]) + printf("\n") + } + } + +When given the input: + + 1 2 3 4 5 6 + 2 3 4 5 6 1 + 3 4 5 6 1 2 + 4 5 6 1 2 3 + +the program produces the following output: + + 4 3 2 1 + 5 4 3 2 + 6 5 4 3 + 1 6 5 4 + 2 1 6 5 + 3 2 1 6 + + +File: gawk.info, Node: Multiscanning, Up: Multidimensional + +8.5.1 Scanning Multidimensional Arrays +-------------------------------------- + +There is no special 'for' statement for scanning a "multidimensional" +array. There cannot be one, because, in truth, 'awk' does not have +multidimensional arrays or elements--there is only a multidimensional +_way of accessing_ an array. + + However, if your program has an array that is always accessed as +multidimensional, you can get the effect of scanning it by combining the +scanning 'for' statement (*note Scanning an Array::) with the built-in +'split()' function (*note String Functions::). It works in the +following manner: + + for (combined in array) { + split(combined, separate, SUBSEP) + ... + } + +This sets the variable 'combined' to each concatenated combined index in +the array, and splits it into the individual indices by breaking it +apart where the value of 'SUBSEP' appears. The individual indices then +become the elements of the array 'separate'. + + Thus, if a value is previously stored in 'array[1, "foo"]', then an +element with index '"1\034foo"' exists in 'array'. (Recall that the +default value of 'SUBSEP' is the character with code 034.) Sooner or +later, the 'for' statement finds that index and does an iteration with +the variable 'combined' set to '"1\034foo"'. Then the 'split()' +function is called as follows: + + split("1\034foo", separate, "\034") + +The result is to set 'separate[1]' to '"1"' and 'separate[2]' to +'"foo"'. Presto! The original sequence of separate indices is +recovered. + + +File: gawk.info, Node: Arrays of Arrays, Next: Arrays Summary, Prev: Multidimensional, Up: Arrays + +8.6 Arrays of Arrays +==================== + +'gawk' goes beyond standard 'awk''s multidimensional array access and +provides true arrays of arrays. Elements of a subarray are referred to +by their own indices enclosed in square brackets, just like the elements +of the main array. For example, the following creates a two-element +subarray at index '1' of the main array 'a': + + a[1][1] = 1 + a[1][2] = 2 + + This simulates a true two-dimensional array. Each subarray element +can contain another subarray as a value, which in turn can hold other +arrays as well. In this way, you can create arrays of three or more +dimensions. The indices can be any 'awk' expressions, including scalars +separated by commas (i.e., a regular 'awk' simulated multidimensional +subscript). So the following is valid in 'gawk': + + a[1][3][1, "name"] = "barney" + + Each subarray and the main array can be of different length. In +fact, the elements of an array or its subarray do not all have to have +the same type. This means that the main array and any of its subarrays +can be nonrectangular, or jagged in structure. You can assign a scalar +value to the index '4' of the main array 'a', even though 'a[1]' is +itself an array and not a scalar: + + a[4] = "An element in a jagged array" + + The terms "dimension", "row", and "column" are meaningless when +applied to such an array, but we will use "dimension" henceforth to +imply the maximum number of indices needed to refer to an existing +element. The type of any element that has already been assigned cannot +be changed by assigning a value of a different type. You have to first +delete the current element, which effectively makes 'gawk' forget about +the element at that index: + + delete a[4] + a[4][5][6][7] = "An element in a four-dimensional array" + +This removes the scalar value from index '4' and then inserts a +three-level nested subarray containing a scalar. You can also delete an +entire subarray or subarray of subarrays: + + delete a[4][5] + a[4][5] = "An element in subarray a[4]" + + But recall that you can not delete the main array 'a' and then use it +as a scalar. + + The built-in functions that take array arguments can also be used +with subarrays. For example, the following code fragment uses +'length()' (*note String Functions::) to determine the number of +elements in the main array 'a' and its subarrays: + + print length(a), length(a[1]), length(a[1][3]) + +This results in the following output for our main array 'a': + + 2, 3, 1 + +The 'SUBSCRIPT in ARRAY' expression (*note Reference to Elements::) +works similarly for both regular 'awk'-style arrays and arrays of +arrays. For example, the tests '1 in a', '3 in a[1]', and '(1, "name") +in a[1][3]' all evaluate to one (true) for our array 'a'. + + The 'for (item in array)' statement (*note Scanning an Array::) can +be nested to scan all the elements of an array of arrays if it is +rectangular in structure. In order to print the contents (scalar +values) of a two-dimensional array of arrays (i.e., in which each +first-level element is itself an array, not necessarily of the same +length), you could use the following code: + + for (i in array) + for (j in array[i]) + print array[i][j] + + The 'isarray()' function (*note Type Functions::) lets you test if an +array element is itself an array: + + for (i in array) { + if (isarray(array[i]) { + for (j in array[i]) { + print array[i][j] + } + } + else + print array[i] + } + + If the structure of a jagged array of arrays is known in advance, you +can often devise workarounds using control statements. For example, the +following code prints the elements of our main array 'a': + + for (i in a) { + for (j in a[i]) { + if (j == 3) { + for (k in a[i][j]) + print a[i][j][k] + } else + print a[i][j] + } + } + +*Note Walking Arrays:: for a user-defined function that "walks" an +arbitrarily dimensioned array of arrays. + + Recall that a reference to an uninitialized array element yields a +value of '""', the null string. This has one important implication when +you intend to use a subarray as an argument to a function, as +illustrated by the following example: + + $ gawk 'BEGIN { split("a b c d", b[1]); print b[1][1] }' + error-> gawk: cmd. line:1: fatal: split: second argument is not an array + + The way to work around this is to first force 'b[1]' to be an array +by creating an arbitrary index: + + $ gawk 'BEGIN { b[1][1] = ""; split("a b c d", b[1]); print b[1][1] }' + -| a + + +File: gawk.info, Node: Arrays Summary, Prev: Arrays of Arrays, Up: Arrays + +8.7 Summary +=========== + + * Standard 'awk' provides one-dimensional associative arrays (arrays + indexed by string values). All arrays are associative; numeric + indices are converted automatically to strings. + + * Array elements are referenced as 'ARRAY[INDX]'. Referencing an + element creates it if it did not exist previously. + + * The proper way to see if an array has an element with a given index + is to use the 'in' operator: 'INDX in ARRAY'. + + * Use 'for (INDX in ARRAY) ...' to scan through all the individual + elements of an array. In the body of the loop, INDX takes on the + value of each element's index in turn. + + * The order in which a 'for (INDX in ARRAY)' loop traverses an array + is undefined in POSIX 'awk' and varies among implementations. + 'gawk' lets you control the order by assigning special predefined + values to 'PROCINFO["sorted_in"]'. + + * Use 'delete ARRAY[INDX]' to delete an individual element. To + delete all of the elements in an array, use 'delete ARRAY'. This + latter feature has been a common extension for many years and is + now standard, but may not be supported by all commercial versions + of 'awk'. + + * Standard 'awk' simulates multidimensional arrays by separating + subscript values with commas. The values are concatenated into a + single string, separated by the value of 'SUBSEP'. The fact that + such a subscript was created in this way is not retained; thus, + changing 'SUBSEP' may have unexpected consequences. You can use + '(SUB1, SUB2, ...) in ARRAY' to see if such a multidimensional + subscript exists in ARRAY. + + * 'gawk' provides true arrays of arrays. You use a separate set of + square brackets for each dimension in such an array: + 'data[row][col]', for example. Array elements may thus be either + scalar values (number or string) or other arrays. + + * Use the 'isarray()' built-in function to determine if an array + element is itself a subarray. + + +File: gawk.info, Node: Functions, Next: Library Functions, Prev: Arrays, Up: Top + +9 Functions +*********** + +This major node describes 'awk''s built-in functions, which fall into +three categories: numeric, string, and I/O. 'gawk' provides additional +groups of functions to work with values that represent time, do bit +manipulation, sort arrays, provide type information, and +internationalize and localize programs. + + Besides the built-in functions, 'awk' has provisions for writing new +functions that the rest of a program can use. The second half of this +major node describes these "user-defined" functions. Finally, we +explore indirect function calls, a 'gawk'-specific extension that lets +you determine at runtime what function is to be called. + +* Menu: + +* Built-in:: Summarizes the built-in functions. +* User-defined:: Describes User-defined functions in detail. +* Indirect Calls:: Choosing the function to call at runtime. +* Functions Summary:: Summary of functions. + + +File: gawk.info, Node: Built-in, Next: User-defined, Up: Functions + +9.1 Built-in Functions +====================== + +"Built-in" functions are always available for your 'awk' program to +call. This minor node defines all the built-in functions in 'awk'; some +of these are mentioned in other minor nodes but are summarized here for +your convenience. + +* Menu: + +* Calling Built-in:: How to call built-in functions. +* Numeric Functions:: Functions that work with numbers, including + 'int()', 'sin()' and 'rand()'. +* String Functions:: Functions for string manipulation, such as + 'split()', 'match()' and + 'sprintf()'. +* I/O Functions:: Functions for files and shell commands. +* Time Functions:: Functions for dealing with timestamps. +* Bitwise Functions:: Functions for bitwise operations. +* Type Functions:: Functions for type information. +* I18N Functions:: Functions for string translation. + + +File: gawk.info, Node: Calling Built-in, Next: Numeric Functions, Up: Built-in + +9.1.1 Calling Built-in Functions +-------------------------------- + +To call one of 'awk''s built-in functions, write the name of the +function followed by arguments in parentheses. For example, 'atan2(y + +z, 1)' is a call to the function 'atan2()' and has two arguments. + + Whitespace is ignored between the built-in function name and the +opening parenthesis, but nonetheless it is good practice to avoid using +whitespace there. User-defined functions do not permit whitespace in +this way, and it is easier to avoid mistakes by following a simple +convention that always works--no whitespace after a function name. + + Each built-in function accepts a certain number of arguments. In +some cases, arguments can be omitted. The defaults for omitted +arguments vary from function to function and are described under the +individual functions. In some 'awk' implementations, extra arguments +given to built-in functions are ignored. However, in 'gawk', it is a +fatal error to give extra arguments to a built-in function. + + When a function is called, expressions that create the function's +actual parameters are evaluated completely before the call is performed. +For example, in the following code fragment: + + i = 4 + j = sqrt(i++) + +the variable 'i' is incremented to the value five before 'sqrt()' is +called with a value of four for its actual parameter. The order of +evaluation of the expressions used for the function's parameters is +undefined. Thus, avoid writing programs that assume that parameters are +evaluated from left to right or from right to left. For example: + + i = 5 + j = atan2(++i, i *= 2) + + If the order of evaluation is left to right, then 'i' first becomes +six, and then 12, and 'atan2()' is called with the two arguments six and +12. But if the order of evaluation is right to left, 'i' first becomes +10, then 11, and 'atan2()' is called with the two arguments 11 and 10. + + +File: gawk.info, Node: Numeric Functions, Next: String Functions, Prev: Calling Built-in, Up: Built-in + +9.1.2 Numeric Functions +----------------------- + +The following list describes all of the built-in functions that work +with numbers. Optional parameters are enclosed in square +brackets ([ ]): + +'atan2(Y, X)' + Return the arctangent of 'Y / X' in radians. You can use 'pi = + atan2(0, -1)' to retrieve the value of pi. + +'cos(X)' + Return the cosine of X, with X in radians. + +'exp(X)' + Return the exponential of X ('e ^ X') or report an error if X is + out of range. The range of values X can have depends on your + machine's floating-point representation. + +'int(X)' + Return the nearest integer to X, located between X and zero and + truncated toward zero. For example, 'int(3)' is 3, 'int(3.9)' is + 3, 'int(-3.9)' is -3, and 'int(-3)' is -3 as well. + +'intdiv(NUMERATOR, DENOMINATOR, RESULT)' + Perform integer division, similar to the standard C function of the + same name. First, truncate 'numerator' and 'denominator' towards + zero, creating integer values. Clear the 'result' array, and then + set 'result["quotient"]' to the result of 'numerator / + denominator', truncated towards zero to an integer, and set + 'result["remainder"]' to the result of 'numerator % denominator', + truncated towards zero to an integer. This function is primarily + intended for use with arbitrary length integers; it avoids creating + MPFR arbitrary precision floating-point values (*note Arbitrary + Precision Integers::). + + This function is a 'gawk' extension. It is not available in + compatibility mode (*note Options::). + +'log(X)' + Return the natural logarithm of X, if X is positive; otherwise, + return 'NaN' ("not a number") on IEEE 754 systems. Additionally, + 'gawk' prints a warning message when 'x' is negative. + +'rand()' + Return a random number. The values of 'rand()' are uniformly + distributed between zero and one. The value could be zero but is + never one.(1) + + Often random integers are needed instead. Following is a + user-defined function that can be used to obtain a random + nonnegative integer less than N: + + function randint(n) + { + return int(n * rand()) + } + + The multiplication produces a random number greater than or equal + to zero and less than 'n'. Using 'int()', this result is made into + an integer between zero and 'n' - 1, inclusive. + + The following example uses a similar function to produce random + integers between one and N. This program prints a new random + number for each input record: + + # Function to roll a simulated die. + function roll(n) { return 1 + int(rand() * n) } + + # Roll 3 six-sided dice and + # print total number of points. + { + printf("%d points\n", roll(6) + roll(6) + roll(6)) + } + + CAUTION: In most 'awk' implementations, including 'gawk', + 'rand()' starts generating numbers from the same starting + number, or "seed", each time you run 'awk'.(2) Thus, a + program generates the same results each time you run it. The + numbers are random within one 'awk' run but predictable from + run to run. This is convenient for debugging, but if you want + a program to do different things each time it is used, you + must change the seed to a value that is different in each run. + To do this, use 'srand()'. + +'sin(X)' + Return the sine of X, with X in radians. + +'sqrt(X)' + Return the positive square root of X. 'gawk' prints a warning + message if X is negative. Thus, 'sqrt(4)' is 2. + +'srand('[X]')' + Set the starting point, or seed, for generating random numbers to + the value X. + + Each seed value leads to a particular sequence of random + numbers.(3) Thus, if the seed is set to the same value a second + time, the same sequence of random numbers is produced again. + + CAUTION: Different 'awk' implementations use different + random-number generators internally. Don't expect the same + 'awk' program to produce the same series of random numbers + when executed by different versions of 'awk'. + + If the argument X is omitted, as in 'srand()', then the current + date and time of day are used for a seed. This is the way to get + random numbers that are truly unpredictable. + + The return value of 'srand()' is the previous seed. This makes it + easy to keep track of the seeds in case you need to consistently + reproduce sequences of random numbers. + + POSIX does not specify the initial seed; it differs among 'awk' + implementations. + + ---------- Footnotes ---------- + + (1) The C version of 'rand()' on many Unix systems is known to +produce fairly poor sequences of random numbers. However, nothing +requires that an 'awk' implementation use the C 'rand()' to implement +the 'awk' version of 'rand()'. In fact, 'gawk' uses the BSD 'random()' +function, which is considerably better than 'rand()', to produce random +numbers. + + (2) 'mawk' uses a different seed each time. + + (3) Computer-generated random numbers really are not truly random. +They are technically known as "pseudorandom". This means that although +the numbers in a sequence appear to be random, you can in fact generate +the same sequence of random numbers over and over again. + + +File: gawk.info, Node: String Functions, Next: I/O Functions, Prev: Numeric Functions, Up: Built-in + +9.1.3 String-Manipulation Functions +----------------------------------- + +The functions in this minor node look at or change the text of one or +more strings. + + 'gawk' understands locales (*note Locales::) and does all string +processing in terms of _characters_, not _bytes_. This distinction is +particularly important to understand for locales where one character may +be represented by multiple bytes. Thus, for example, 'length()' returns +the number of characters in a string, and not the number of bytes used +to represent those characters. Similarly, 'index()' works with +character indices, and not byte indices. + + CAUTION: A number of functions deal with indices into strings. For + these functions, the first character of a string is at position + (index) one. This is different from C and the languages descended + from it, where the first character is at position zero. You need + to remember this when doing index calculations, particularly if you + are used to C. + + In the following list, optional parameters are enclosed in square +brackets ([ ]). Several functions perform string substitution; the full +discussion is provided in the description of the 'sub()' function, which +comes toward the end, because the list is presented alphabetically. + + Those functions that are specific to 'gawk' are marked with a pound +sign ('#'). They are not available in compatibility mode (*note +Options::): + +* Menu: + +* Gory Details:: More than you want to know about '\' and + '&' with 'sub()', 'gsub()', and + 'gensub()'. + +'asort('SOURCE [',' DEST [',' HOW ] ]') #' +'asorti('SOURCE [',' DEST [',' HOW ] ]') #' + These two functions are similar in behavior, so they are described + together. + + NOTE: The following description ignores the third argument, + HOW, as it requires understanding features that we have not + discussed yet. Thus, the discussion here is a deliberate + simplification. (We do provide all the details later on; see + *note Array Sorting Functions:: for the full story.) + + Both functions return the number of elements in the array SOURCE. + For 'asort()', 'gawk' sorts the values of SOURCE and replaces the + indices of the sorted values of SOURCE with sequential integers + starting with one. If the optional array DEST is specified, then + SOURCE is duplicated into DEST. DEST is then sorted, leaving the + indices of SOURCE unchanged. + + When comparing strings, 'IGNORECASE' affects the sorting (*note + Array Sorting Functions::). If the SOURCE array contains subarrays + as values (*note Arrays of Arrays::), they will come last, after + all scalar values. Subarrays are _not_ recursively sorted. + + For example, if the contents of 'a' are as follows: + + a["last"] = "de" + a["first"] = "sac" + a["middle"] = "cul" + + A call to 'asort()': + + asort(a) + + results in the following contents of 'a': + + a[1] = "cul" + a[2] = "de" + a[3] = "sac" + + The 'asorti()' function works similarly to 'asort()'; however, the + _indices_ are sorted, instead of the values. Thus, in the previous + example, starting with the same initial set of indices and values + in 'a', calling 'asorti(a)' would yield: + + a[1] = "first" + a[2] = "last" + a[3] = "middle" + +'gensub(REGEXP, REPLACEMENT, HOW' [', TARGET']') #' + Search the target string TARGET for matches of the regular + expression REGEXP. If HOW is a string beginning with 'g' or 'G' + (short for "global"), then replace all matches of REGEXP with + REPLACEMENT. Otherwise, HOW is treated as a number indicating + which match of REGEXP to replace. If no TARGET is supplied, use + '$0'. It returns the modified string as the result of the function + and the original target string is _not_ changed. + + 'gensub()' is a general substitution function. Its purpose is to + provide more features than the standard 'sub()' and 'gsub()' + functions. + + 'gensub()' provides an additional feature that is not available in + 'sub()' or 'gsub()': the ability to specify components of a regexp + in the replacement text. This is done by using parentheses in the + regexp to mark the components and then specifying '\N' in the + replacement text, where N is a digit from 1 to 9. For example: + + $ gawk ' + > BEGIN { + > a = "abc def" + > b = gensub(/(.+) (.+)/, "\\2 \\1", "g", a) + > print b + > }' + -| def abc + + As with 'sub()', you must type two backslashes in order to get one + into the string. In the replacement text, the sequence '\0' + represents the entire matched text, as does the character '&'. + + The following example shows how you can use the third argument to + control which match of the regexp should be changed: + + $ echo a b c a b c | + > gawk '{ print gensub(/a/, "AA", 2) }' + -| a b c AA b c + + In this case, '$0' is the default target string. 'gensub()' + returns the new string as its result, which is passed directly to + 'print' for printing. + + If the HOW argument is a string that does not begin with 'g' or + 'G', or if it is a number that is less than or equal to zero, only + one substitution is performed. If HOW is zero, 'gawk' issues a + warning message. + + If REGEXP does not match TARGET, 'gensub()''s return value is the + original unchanged value of TARGET. + +'gsub(REGEXP, REPLACEMENT' [', TARGET']')' + Search TARGET for _all_ of the longest, leftmost, _nonoverlapping_ + matching substrings it can find and replace them with REPLACEMENT. + The 'g' in 'gsub()' stands for "global," which means replace + everywhere. For example: + + { gsub(/Britain/, "United Kingdom"); print } + + replaces all occurrences of the string 'Britain' with 'United + Kingdom' for all input records. + + The 'gsub()' function returns the number of substitutions made. If + the variable to search and alter (TARGET) is omitted, then the + entire input record ('$0') is used. As in 'sub()', the characters + '&' and '\' are special, and the third argument must be assignable. + +'index(IN, FIND)' + Search the string IN for the first occurrence of the string FIND, + and return the position in characters where that occurrence begins + in the string IN. Consider the following example: + + $ awk 'BEGIN { print index("peanut", "an") }' + -| 3 + + If FIND is not found, 'index()' returns zero. + + With BWK 'awk' and 'gawk', it is a fatal error to use a regexp + constant for FIND. Other implementations allow it, simply treating + the regexp constant as an expression meaning '$0 ~ /regexp/'. + (d.c.) + +'length('[STRING]')' + Return the number of characters in STRING. If STRING is a number, + the length of the digit string representing that number is + returned. For example, 'length("abcde")' is five. By contrast, + 'length(15 * 35)' works out to three. In this example, 15 * 35 = + 525, and 525 is then converted to the string '"525"', which has + three characters. + + If no argument is supplied, 'length()' returns the length of '$0'. + + NOTE: In older versions of 'awk', the 'length()' function + could be called without any parentheses. Doing so is + considered poor practice, although the 2008 POSIX standard + explicitly allows it, to support historical practice. For + programs to be maximally portable, always supply the + parentheses. + + If 'length()' is called with a variable that has not been used, + 'gawk' forces the variable to be a scalar. Other implementations + of 'awk' leave the variable without a type. (d.c.) Consider: + + $ gawk 'BEGIN { print length(x) ; x[1] = 1 }' + -| 0 + error-> gawk: fatal: attempt to use scalar `x' as array + + $ nawk 'BEGIN { print length(x) ; x[1] = 1 }' + -| 0 + + If '--lint' has been specified on the command line, 'gawk' issues a + warning about this. + + With 'gawk' and several other 'awk' implementations, when given an + array argument, the 'length()' function returns the number of + elements in the array. (c.e.) This is less useful than it might + seem at first, as the array is not guaranteed to be indexed from + one to the number of elements in it. If '--lint' is provided on + the command line (*note Options::), 'gawk' warns that passing an + array argument is not portable. If '--posix' is supplied, using an + array argument is a fatal error (*note Arrays::). + +'match(STRING, REGEXP' [', ARRAY']')' + Search STRING for the longest, leftmost substring matched by the + regular expression REGEXP and return the character position (index) + at which that substring begins (one, if it starts at the beginning + of STRING). If no match is found, return zero. + + The REGEXP argument may be either a regexp constant ('/'...'/') or + a string constant ('"'...'"'). In the latter case, the string is + treated as a regexp to be matched. *Note Computed Regexps:: for a + discussion of the difference between the two forms, and the + implications for writing your program correctly. + + The order of the first two arguments is the opposite of most other + string functions that work with regular expressions, such as + 'sub()' and 'gsub()'. It might help to remember that for + 'match()', the order is the same as for the '~' operator: 'STRING ~ + REGEXP'. + + The 'match()' function sets the predefined variable 'RSTART' to the + index. It also sets the predefined variable 'RLENGTH' to the + length in characters of the matched substring. If no match is + found, 'RSTART' is set to zero, and 'RLENGTH' to -1. + + For example: + + { + if ($1 == "FIND") + regex = $2 + else { + where = match($0, regex) + if (where != 0) + print "Match of", regex, "found at", where, "in", $0 + } + } + + This program looks for lines that match the regular expression + stored in the variable 'regex'. This regular expression can be + changed. If the first word on a line is 'FIND', 'regex' is changed + to be the second word on that line. Therefore, if given: + + FIND ru+n + My program runs + but not very quickly + FIND Melvin + JF+KM + This line is property of Reality Engineering Co. + Melvin was here. + + 'awk' prints: + + Match of ru+n found at 12 in My program runs + Match of Melvin found at 1 in Melvin was here. + + If ARRAY is present, it is cleared, and then the zeroth element of + ARRAY is set to the entire portion of STRING matched by REGEXP. If + REGEXP contains parentheses, the integer-indexed elements of ARRAY + are set to contain the portion of STRING matching the corresponding + parenthesized subexpression. For example: + + $ echo foooobazbarrrrr | + > gawk '{ match($0, /(fo+).+(bar*)/, arr) + > print arr[1], arr[2] }' + -| foooo barrrrr + + In addition, multidimensional subscripts are available providing + the start index and length of each matched subexpression: + + $ echo foooobazbarrrrr | + > gawk '{ match($0, /(fo+).+(bar*)/, arr) + > print arr[1], arr[2] + > print arr[1, "start"], arr[1, "length"] + > print arr[2, "start"], arr[2, "length"] + > }' + -| foooo barrrrr + -| 1 5 + -| 9 7 + + There may not be subscripts for the start and index for every + parenthesized subexpression, because they may not all have matched + text; thus, they should be tested for with the 'in' operator (*note + Reference to Elements::). + + The ARRAY argument to 'match()' is a 'gawk' extension. In + compatibility mode (*note Options::), using a third argument is a + fatal error. + +'patsplit(STRING, ARRAY' [', FIELDPAT' [', SEPS' ] ]') #' + Divide STRING into pieces defined by FIELDPAT and store the pieces + in ARRAY and the separator strings in the SEPS array. The first + piece is stored in 'ARRAY[1]', the second piece in 'ARRAY[2]', and + so forth. The third argument, FIELDPAT, is a regexp describing the + fields in STRING (just as 'FPAT' is a regexp describing the fields + in input records). It may be either a regexp constant or a string. + If FIELDPAT is omitted, the value of 'FPAT' is used. 'patsplit()' + returns the number of elements created. 'SEPS[I]' is the separator + string between 'ARRAY[I]' and 'ARRAY[I+1]'. Any leading separator + will be in 'SEPS[0]'. + + The 'patsplit()' function splits strings into pieces in a manner + similar to the way input lines are split into fields using 'FPAT' + (*note Splitting By Content::). + + Before splitting the string, 'patsplit()' deletes any previously + existing elements in the arrays ARRAY and SEPS. + +'split(STRING, ARRAY' [', FIELDSEP' [', SEPS' ] ]')' + Divide STRING into pieces separated by FIELDSEP and store the + pieces in ARRAY and the separator strings in the SEPS array. The + first piece is stored in 'ARRAY[1]', the second piece in + 'ARRAY[2]', and so forth. The string value of the third argument, + FIELDSEP, is a regexp describing where to split STRING (much as + 'FS' can be a regexp describing where to split input records). If + FIELDSEP is omitted, the value of 'FS' is used. 'split()' returns + the number of elements created. SEPS is a 'gawk' extension, with + 'SEPS[I]' being the separator string between 'ARRAY[I]' and + 'ARRAY[I+1]'. If FIELDSEP is a single space, then any leading + whitespace goes into 'SEPS[0]' and any trailing whitespace goes + into 'SEPS[N]', where N is the return value of 'split()' (i.e., the + number of elements in ARRAY). + + The 'split()' function splits strings into pieces in a manner + similar to the way input lines are split into fields. For example: + + split("cul-de-sac", a, "-", seps) + + splits the string '"cul-de-sac"' into three fields using '-' as the + separator. It sets the contents of the array 'a' as follows: + + a[1] = "cul" + a[2] = "de" + a[3] = "sac" + + and sets the contents of the array 'seps' as follows: + + seps[1] = "-" + seps[2] = "-" + + The value returned by this call to 'split()' is three. + + As with input field-splitting, when the value of FIELDSEP is '" "', + leading and trailing whitespace is ignored in values assigned to + the elements of ARRAY but not in SEPS, and the elements are + separated by runs of whitespace. Also, as with input field + splitting, if FIELDSEP is the null string, each individual + character in the string is split into its own array element. + (c.e.) + + Note, however, that 'RS' has no effect on the way 'split()' works. + Even though 'RS = ""' causes the newline character to also be an + input field separator, this does not affect how 'split()' splits + strings. + + Modern implementations of 'awk', including 'gawk', allow the third + argument to be a regexp constant ('/'...'/') as well as a string. + (d.c.) The POSIX standard allows this as well. *Note Computed + Regexps:: for a discussion of the difference between using a string + constant or a regexp constant, and the implications for writing + your program correctly. + + Before splitting the string, 'split()' deletes any previously + existing elements in the arrays ARRAY and SEPS. + + If STRING is null, the array has no elements. (So this is a + portable way to delete an entire array with one statement. *Note + Delete::.) + + If STRING does not match FIELDSEP at all (but is not null), ARRAY + has one element only. The value of that element is the original + STRING. + + In POSIX mode (*note Options::), the fourth argument is not + allowed. + +'sprintf(FORMAT, EXPRESSION1, ...)' + Return (without printing) the string that 'printf' would have + printed out with the same arguments (*note Printf::). For example: + + pival = sprintf("pi = %.2f (approx.)", 22/7) + + assigns the string 'pi = 3.14 (approx.)' to the variable 'pival'. + +'strtonum(STR) #' + Examine STR and return its numeric value. If STR begins with a + leading '0', 'strtonum()' assumes that STR is an octal number. If + STR begins with a leading '0x' or '0X', 'strtonum()' assumes that + STR is a hexadecimal number. For example: + + $ echo 0x11 | + > gawk '{ printf "%d\n", strtonum($1) }' + -| 17 + + Using the 'strtonum()' function is _not_ the same as adding zero to + a string value; the automatic coercion of strings to numbers works + only for decimal data, not for octal or hexadecimal.(1) + + Note also that 'strtonum()' uses the current locale's decimal point + for recognizing numbers (*note Locales::). + +'sub(REGEXP, REPLACEMENT' [', TARGET']')' + Search TARGET, which is treated as a string, for the leftmost, + longest substring matched by the regular expression REGEXP. Modify + the entire string by replacing the matched text with REPLACEMENT. + The modified string becomes the new value of TARGET. Return the + number of substitutions made (zero or one). + + The REGEXP argument may be either a regexp constant ('/'...'/') or + a string constant ('"'...'"'). In the latter case, the string is + treated as a regexp to be matched. *Note Computed Regexps:: for a + discussion of the difference between the two forms, and the + implications for writing your program correctly. + + This function is peculiar because TARGET is not simply used to + compute a value, and not just any expression will do--it must be a + variable, field, or array element so that 'sub()' can store a + modified value there. If this argument is omitted, then the + default is to use and alter '$0'.(2) For example: + + str = "water, water, everywhere" + sub(/at/, "ith", str) + + sets 'str' to 'wither, water, everywhere', by replacing the + leftmost longest occurrence of 'at' with 'ith'. + + If the special character '&' appears in REPLACEMENT, it stands for + the precise substring that was matched by REGEXP. (If the regexp + can match more than one string, then this precise substring may + vary.) For example: + + { sub(/candidate/, "& and his wife"); print } + + changes the first occurrence of 'candidate' to 'candidate and his + wife' on each input line. Here is another example: + + $ awk 'BEGIN { + > str = "daabaaa" + > sub(/a+/, "C&C", str) + > print str + > }' + -| dCaaCbaaa + + This shows how '&' can represent a nonconstant string and also + illustrates the "leftmost, longest" rule in regexp matching (*note + Leftmost Longest::). + + The effect of this special character ('&') can be turned off by + putting a backslash before it in the string. As usual, to insert + one backslash in the string, you must write two backslashes. + Therefore, write '\\&' in a string constant to include a literal + '&' in the replacement. For example, the following shows how to + replace the first '|' on each line with an '&': + + { sub(/\|/, "\\&"); print } + + As mentioned, the third argument to 'sub()' must be a variable, + field, or array element. Some versions of 'awk' allow the third + argument to be an expression that is not an lvalue. In such a + case, 'sub()' still searches for the pattern and returns zero or + one, but the result of the substitution (if any) is thrown away + because there is no place to put it. Such versions of 'awk' accept + expressions like the following: + + sub(/USA/, "United States", "the USA and Canada") + + For historical compatibility, 'gawk' accepts such erroneous code. + However, using any other nonchangeable object as the third + parameter causes a fatal error and your program will not run. + + Finally, if the REGEXP is not a regexp constant, it is converted + into a string, and then the value of that string is treated as the + regexp to match. + +'substr(STRING, START' [', LENGTH' ]')' + Return a LENGTH-character-long substring of STRING, starting at + character number START. The first character of a string is + character number one.(3) For example, 'substr("washington", 5, 3)' + returns '"ing"'. + + If LENGTH is not present, 'substr()' returns the whole suffix of + STRING that begins at character number START. For example, + 'substr("washington", 5)' returns '"ington"'. The whole suffix is + also returned if LENGTH is greater than the number of characters + remaining in the string, counting from character START. + + If START is less than one, 'substr()' treats it as if it was one. + (POSIX doesn't specify what to do in this case: BWK 'awk' acts this + way, and therefore 'gawk' does too.) If START is greater than the + number of characters in the string, 'substr()' returns the null + string. Similarly, if LENGTH is present but less than or equal to + zero, the null string is returned. + + The string returned by 'substr()' _cannot_ be assigned. Thus, it + is a mistake to attempt to change a portion of a string, as shown + in the following example: + + string = "abcdef" + # try to get "abCDEf", won't work + substr(string, 3, 3) = "CDE" + + It is also a mistake to use 'substr()' as the third argument of + 'sub()' or 'gsub()': + + gsub(/xyz/, "pdq", substr($0, 5, 20)) # WRONG + + (Some commercial versions of 'awk' treat 'substr()' as assignable, + but doing so is not portable.) + + If you need to replace bits and pieces of a string, combine + 'substr()' with string concatenation, in the following manner: + + string = "abcdef" + ... + string = substr(string, 1, 2) "CDE" substr(string, 6) + +'tolower(STRING)' + Return a copy of STRING, with each uppercase character in the + string replaced with its corresponding lowercase character. + Nonalphabetic characters are left unchanged. For example, + 'tolower("MiXeD cAsE 123")' returns '"mixed case 123"'. + +'toupper(STRING)' + Return a copy of STRING, with each lowercase character in the + string replaced with its corresponding uppercase character. + Nonalphabetic characters are left unchanged. For example, + 'toupper("MiXeD cAsE 123")' returns '"MIXED CASE 123"'. + + Matching the Null String + + In 'awk', the '*' operator can match the null string. This is +particularly important for the 'sub()', 'gsub()', and 'gensub()' +functions. For example: + + $ echo abc | awk '{ gsub(/m*/, "X"); print }' + -| XaXbXcX + +Although this makes a certain amount of sense, it can be surprising. + + ---------- Footnotes ---------- + + (1) Unless you use the '--non-decimal-data' option, which isn't +recommended. *Note Nondecimal Data:: for more information. + + (2) Note that this means that the record will first be regenerated +using the value of 'OFS' if any fields have been changed, and that the +fields will be updated after the substitution, even if the operation is +a "no-op" such as 'sub(/^/, "")'. + + (3) This is different from C and C++, in which the first character is +number zero. + + +File: gawk.info, Node: Gory Details, Up: String Functions + +9.1.3.1 More about '\' and '&' with 'sub()', 'gsub()', and 'gensub()' +..................................................................... + + CAUTION: This subsubsection has been reported to cause headaches. + You might want to skip it upon first reading. + + When using 'sub()', 'gsub()', or 'gensub()', and trying to get +literal backslashes and ampersands into the replacement text, you need +to remember that there are several levels of "escape processing" going +on. + + First, there is the "lexical" level, which is when 'awk' reads your +program and builds an internal copy of it to execute. Then there is the +runtime level, which is when 'awk' actually scans the replacement string +to determine what to generate. + + At both levels, 'awk' looks for a defined set of characters that can +come after a backslash. At the lexical level, it looks for the escape +sequences listed in *note Escape Sequences::. Thus, for every '\' that +'awk' processes at the runtime level, you must type two backslashes at +the lexical level. When a character that is not valid for an escape +sequence follows the '\', BWK 'awk' and 'gawk' both simply remove the +initial '\' and put the next character into the string. Thus, for +example, '"a\qb"' is treated as '"aqb"'. + + At the runtime level, the various functions handle sequences of '\' +and '&' differently. The situation is (sadly) somewhat complex. +Historically, the 'sub()' and 'gsub()' functions treated the +two-character sequence '\&' specially; this sequence was replaced in the +generated text with a single '&'. Any other '\' within the REPLACEMENT +string that did not precede an '&' was passed through unchanged. This +is illustrated in *note Table 9.1: table-sub-escapes. + + You type 'sub()' sees 'sub()' generates + ----- ------- ---------- + '\&' '&' The matched text + '\\&' '\&' A literal '&' + '\\\&' '\&' A literal '&' + '\\\\&' '\\&' A literal '\&' + '\\\\\&' '\\&' A literal '\&' + '\\\\\\&' '\\\&' A literal '\\&' + '\\q' '\q' A literal '\q' + +Table 9.1: Historical escape sequence processing for 'sub()' and +'gsub()' + +This table shows the lexical-level processing, where an odd number of +backslashes becomes an even number at the runtime level, as well as the +runtime processing done by 'sub()'. (For the sake of simplicity, the +rest of the following tables only show the case of even numbers of +backslashes entered at the lexical level.) + + The problem with the historical approach is that there is no way to +get a literal '\' followed by the matched text. + + Several editions of the POSIX standard attempted to fix this problem +but weren't successful. The details are irrelevant at this point in +time. + + At one point, the 'gawk' maintainer submitted proposed text for a +revised standard that reverts to rules that correspond more closely to +the original existing practice. The proposed rules have special cases +that make it possible to produce a '\' preceding the matched text. This +is shown in *note Table 9.2: table-sub-proposed. + + You type 'sub()' sees 'sub()' generates + ----- ------- ---------- + '\\\\\\&' '\\\&' A literal '\&' + '\\\\&' '\\&' A literal '\', followed by the matched text + '\\&' '\&' A literal '&' + '\\q' '\q' A literal '\q' + '\\\\' '\\' '\\' + +Table 9.2: 'gawk' rules for 'sub()' and backslash + + In a nutshell, at the runtime level, there are now three special +sequences of characters ('\\\&', '\\&', and '\&') whereas historically +there was only one. However, as in the historical case, any '\' that is +not part of one of these three sequences is not special and appears in +the output literally. + + 'gawk' 3.0 and 3.1 follow these rules for 'sub()' and 'gsub()'. The +POSIX standard took much longer to be revised than was expected. In +addition, the 'gawk' maintainer's proposal was lost during the +standardization process. The final rules are somewhat simpler. The +results are similar except for one case. + + The POSIX rules state that '\&' in the replacement string produces a +literal '&', '\\' produces a literal '\', and '\' followed by anything +else is not special; the '\' is placed straight into the output. These +rules are presented in *note Table 9.3: table-posix-sub. + + You type 'sub()' sees 'sub()' generates + ----- ------- ---------- + '\\\\\\&' '\\\&' A literal '\&' + '\\\\&' '\\&' A literal '\', followed by the matched text + '\\&' '\&' A literal '&' + '\\q' '\q' A literal '\q' + '\\\\' '\\' '\' + +Table 9.3: POSIX rules for 'sub()' and 'gsub()' + + The only case where the difference is noticeable is the last one: +'\\\\' is seen as '\\' and produces '\' instead of '\\'. + + Starting with version 3.1.4, 'gawk' followed the POSIX rules when +'--posix' was specified (*note Options::). Otherwise, it continued to +follow the proposed rules, as that had been its behavior for many years. + + When version 4.0.0 was released, the 'gawk' maintainer made the POSIX +rules the default, breaking well over a decade's worth of backward +compatibility.(1) Needless to say, this was a bad idea, and as of +version 4.0.1, 'gawk' resumed its historical behavior, and only follows +the POSIX rules when '--posix' is given. + + The rules for 'gensub()' are considerably simpler. At the runtime +level, whenever 'gawk' sees a '\', if the following character is a +digit, then the text that matched the corresponding parenthesized +subexpression is placed in the generated output. Otherwise, no matter +what character follows the '\', it appears in the generated text and the +'\' does not, as shown in *note Table 9.4: table-gensub-escapes. + + You type 'gensub()' sees 'gensub()' generates + ----- --------- ------------ + '&' '&' The matched text + '\\&' '\&' A literal '&' + '\\\\' '\\' A literal '\' + '\\\\&' '\\&' A literal '\', then the matched text + '\\\\\\&' '\\\&' A literal '\&' + '\\q' '\q' A literal 'q' + +Table 9.4: Escape sequence processing for 'gensub()' + + Because of the complexity of the lexical- and runtime-level +processing and the special cases for 'sub()' and 'gsub()', we recommend +the use of 'gawk' and 'gensub()' when you have to do substitutions. + + ---------- Footnotes ---------- + + (1) This was rather naive of him, despite there being a note in this +minor node indicating that the next major version would move to the +POSIX rules. + + +File: gawk.info, Node: I/O Functions, Next: Time Functions, Prev: String Functions, Up: Built-in + +9.1.4 Input/Output Functions +---------------------------- + +The following functions relate to input/output (I/O). Optional +parameters are enclosed in square brackets ([ ]): + +'close('FILENAME [',' HOW]')' + Close the file FILENAME for input or output. Alternatively, the + argument may be a shell command that was used for creating a + coprocess, or for redirecting to or from a pipe; then the coprocess + or pipe is closed. *Note Close Files And Pipes:: for more + information. + + When closing a coprocess, it is occasionally useful to first close + one end of the two-way pipe and then to close the other. This is + done by providing a second argument to 'close()'. This second + argument (HOW) should be one of the two string values '"to"' or + '"from"', indicating which end of the pipe to close. Case in the + string does not matter. *Note Two-way I/O::, which discusses this + feature in more detail and gives an example. + + Note that the second argument to 'close()' is a 'gawk' extension; + it is not available in compatibility mode (*note Options::). + +'fflush('[FILENAME]')' + Flush any buffered output associated with FILENAME, which is either + a file opened for writing or a shell command for redirecting output + to a pipe or coprocess. + + Many utility programs "buffer" their output (i.e., they save + information to write to a disk file or the screen in memory until + there is enough for it to be worthwhile to send the data to the + output device). This is often more efficient than writing every + little bit of information as soon as it is ready. However, + sometimes it is necessary to force a program to "flush" its buffers + (i.e., write the information to its destination, even if a buffer + is not full). This is the purpose of the 'fflush()' + function--'gawk' also buffers its output, and the 'fflush()' + function forces 'gawk' to flush its buffers. + + Brian Kernighan added 'fflush()' to his 'awk' in April 1992. For + two decades, it was a common extension. In December 2012, it was + accepted for inclusion into the POSIX standard. See the Austin + Group website (http://austingroupbugs.net/view.php?id=634). + + POSIX standardizes 'fflush()' as follows: if there is no argument, + or if the argument is the null string ('""'), then 'awk' flushes + the buffers for _all_ open output files and pipes. + + NOTE: Prior to version 4.0.2, 'gawk' would flush only the + standard output if there was no argument, and flush all output + files and pipes if the argument was the null string. This was + changed in order to be compatible with Brian Kernighan's + 'awk', in the hope that standardizing this feature in POSIX + would then be easier (which indeed proved to be the case). + + With 'gawk', you can use 'fflush("/dev/stdout")' if you wish + to flush only the standard output. + + 'fflush()' returns zero if the buffer is successfully flushed; + otherwise, it returns a nonzero value. ('gawk' returns -1.) In + the case where all buffers are flushed, the return value is zero + only if all buffers were flushed successfully. Otherwise, it is + -1, and 'gawk' warns about the problem FILENAME. + + 'gawk' also issues a warning message if you attempt to flush a file + or pipe that was opened for reading (such as with 'getline'), or if + FILENAME is not an open file, pipe, or coprocess. In such a case, + 'fflush()' returns -1, as well. + + Interactive Versus Noninteractive Buffering + + As a side point, buffering issues can be even more confusing if + your program is "interactive" (i.e., communicating with a user + sitting at a keyboard).(1) + + Interactive programs generally "line buffer" their output (i.e., + they write out every line). Noninteractive programs wait until + they have a full buffer, which may be many lines of output. Here + is an example of the difference: + + $ awk '{ print $1 + $2 }' + 1 1 + -| 2 + 2 3 + -| 5 + Ctrl-d + + Each line of output is printed immediately. Compare that behavior + with this example: + + $ awk '{ print $1 + $2 }' | cat + 1 1 + 2 3 + Ctrl-d + -| 2 + -| 5 + + Here, no output is printed until after the 'Ctrl-d' is typed, + because it is all buffered and sent down the pipe to 'cat' in one + shot. + +'system(COMMAND)' + Execute the operating system command COMMAND and then return to the + 'awk' program. Return COMMAND's exit status (see further on). + + For example, if the following fragment of code is put in your 'awk' + program: + + END { + system("date | mail -s 'awk run done' root") + } + + the system administrator is sent mail when the 'awk' program + finishes processing input and begins its end-of-input processing. + + Note that redirecting 'print' or 'printf' into a pipe is often + enough to accomplish your task. If you need to run many commands, + it is more efficient to simply print them down a pipeline to the + shell: + + while (MORE STUFF TO DO) + print COMMAND | "/bin/sh" + close("/bin/sh") + + However, if your 'awk' program is interactive, 'system()' is useful + for running large self-contained programs, such as a shell or an + editor. Some operating systems cannot implement the 'system()' + function. 'system()' causes a fatal error if it is not supported. + + NOTE: When '--sandbox' is specified, the 'system()' function + is disabled (*note Options::). + + On POSIX systems, a command's exit status is a 16-bit number. The + exit value passed to the C 'exit()' function is held in the + high-order eight bits. The low-order bits indicate if the process + was killed by a signal (bit 7) and if so, the guilty signal number + (bits 0-6). + + Traditionally, 'awk''s 'system()' function has simply returned the + exit status value divided by 256. In the normal case this gives + the exit status but in the case of death-by-signal it yields a + fractional floating-point value.(2) POSIX states that 'awk''s + 'system()' should return the full 16-bit value. + + 'gawk' steers a middle ground. The return values are summarized in + *note Table 9.5: table-system-return-values. + + Situation Return value from 'system()' + -------------------------------------------------------------------------- + '--traditional' C 'system()''s value divided by 256 + '--posix' C 'system()''s value + Normal exit of command Command's exit status + Death by signal of command 256 + number of murderous signal + Death by signal of command 512 + number of murderous signal + with core dump + Some kind of error -1 + + Table 9.5: Return values from 'system()' + + Controlling Output Buffering with 'system()' + + The 'fflush()' function provides explicit control over output +buffering for individual files and pipes. However, its use is not +portable to many older 'awk' implementations. An alternative method to +flush output buffers is to call 'system()' with a null string as its +argument: + + system("") # flush output + +'gawk' treats this use of the 'system()' function as a special case and +is smart enough not to run a shell (or other command interpreter) with +the empty command. Therefore, with 'gawk', this idiom is not only +useful, it is also efficient. Although this method should work with +other 'awk' implementations, it does not necessarily avoid starting an +unnecessary shell. (Other implementations may only flush the buffer +associated with the standard output and not necessarily all buffered +output.) + + If you think about what a programmer expects, it makes sense that +'system()' should flush any pending output. The following program: + + BEGIN { + print "first print" + system("echo system echo") + print "second print" + } + +must print: + + first print + system echo + second print + +and not: + + system echo + first print + second print + + If 'awk' did not flush its buffers before calling 'system()', you +would see the latter (undesirable) output. + + ---------- Footnotes ---------- + + (1) A program is interactive if the standard output is connected to a +terminal device. On modern systems, this means your keyboard and +screen. + + (2) In private correspondence, Dr. Kernighan has indicated to me that +the way this was done was probably a mistake. + + +File: gawk.info, Node: Time Functions, Next: Bitwise Functions, Prev: I/O Functions, Up: Built-in + +9.1.5 Time Functions +-------------------- + +'awk' programs are commonly used to process log files containing +timestamp information, indicating when a particular log record was +written. Many programs log their timestamps in the form returned by the +'time()' system call, which is the number of seconds since a particular +epoch. On POSIX-compliant systems, it is the number of seconds since +1970-01-01 00:00:00 UTC, not counting leap seconds.(1) All known +POSIX-compliant systems support timestamps from 0 through 2^31 - 1, +which is sufficient to represent times through 2038-01-19 03:14:07 UTC. +Many systems support a wider range of timestamps, including negative +timestamps that represent times before the epoch. + + In order to make it easier to process such log files and to produce +useful reports, 'gawk' provides the following functions for working with +timestamps. They are 'gawk' extensions; they are not specified in the +POSIX standard.(2) However, recent versions of 'mawk' (*note Other +Versions::) also support these functions. Optional parameters are +enclosed in square brackets ([ ]): + +'mktime(DATESPEC)' + Turn DATESPEC into a timestamp in the same form as is returned by + 'systime()'. It is similar to the function of the same name in ISO + C. The argument, DATESPEC, is a string of the form + '"YYYY MM DD HH MM SS [DST]"'. The string consists of six or seven + numbers representing, respectively, the full year including + century, the month from 1 to 12, the day of the month from 1 to 31, + the hour of the day from 0 to 23, the minute from 0 to 59, the + second from 0 to 60,(3) and an optional daylight-savings flag. + + The values of these numbers need not be within the ranges + specified; for example, an hour of -1 means 1 hour before midnight. + The origin-zero Gregorian calendar is assumed, with year 0 + preceding year 1 and year -1 preceding year 0. The time is assumed + to be in the local time zone. If the daylight-savings flag is + positive, the time is assumed to be daylight savings time; if zero, + the time is assumed to be standard time; and if negative (the + default), 'mktime()' attempts to determine whether daylight savings + time is in effect for the specified time. + + If DATESPEC does not contain enough elements or if the resulting + time is out of range, 'mktime()' returns -1. + +'strftime('[FORMAT [',' TIMESTAMP [',' UTC-FLAG] ] ]')' + Format the time specified by TIMESTAMP based on the contents of the + FORMAT string and return the result. It is similar to the function + of the same name in ISO C. If UTC-FLAG is present and is either + nonzero or non-null, the value is formatted as UTC (Coordinated + Universal Time, formerly GMT or Greenwich Mean Time). Otherwise, + the value is formatted for the local time zone. The TIMESTAMP is + in the same format as the value returned by the 'systime()' + function. If no TIMESTAMP argument is supplied, 'gawk' uses the + current time of day as the timestamp. Without a FORMAT argument, + 'strftime()' uses the value of 'PROCINFO["strftime"]' as the format + string (*note Built-in Variables::). The default string value is + '"%a %b %e %H:%M:%S %Z %Y"'. This format string produces output + that is equivalent to that of the 'date' utility. You can assign a + new value to 'PROCINFO["strftime"]' to change the default format; + see the following list for the various format directives. + +'systime()' + Return the current time as the number of seconds since the system + epoch. On POSIX systems, this is the number of seconds since + 1970-01-01 00:00:00 UTC, not counting leap seconds. It may be a + different number on other systems. + + The 'systime()' function allows you to compare a timestamp from a log +file with the current time of day. In particular, it is easy to +determine how long ago a particular record was logged. It also allows +you to produce log records using the "seconds since the epoch" format. + + The 'mktime()' function allows you to convert a textual +representation of a date and time into a timestamp. This makes it easy +to do before/after comparisons of dates and times, particularly when +dealing with date and time data coming from an external source, such as +a log file. + + The 'strftime()' function allows you to easily turn a timestamp into +human-readable information. It is similar in nature to the 'sprintf()' +function (*note String Functions::), in that it copies nonformat +specification characters verbatim to the returned string, while +substituting date and time values for format specifications in the +FORMAT string. + + 'strftime()' is guaranteed by the 1999 ISO C standard(4) to support +the following date format specifications: + +'%a' + The locale's abbreviated weekday name. + +'%A' + The locale's full weekday name. + +'%b' + The locale's abbreviated month name. + +'%B' + The locale's full month name. + +'%c' + The locale's "appropriate" date and time representation. (This is + '%A %B %d %T %Y' in the '"C"' locale.) + +'%C' + The century part of the current year. This is the year divided by + 100 and truncated to the next lower integer. + +'%d' + The day of the month as a decimal number (01-31). + +'%D' + Equivalent to specifying '%m/%d/%y'. + +'%e' + The day of the month, padded with a space if it is only one digit. + +'%F' + Equivalent to specifying '%Y-%m-%d'. This is the ISO 8601 date + format. + +'%g' + The year modulo 100 of the ISO 8601 week number, as a decimal + number (00-99). For example, January 1, 2012, is in week 53 of + 2011. Thus, the year of its ISO 8601 week number is 2011, even + though its year is 2012. Similarly, December 31, 2012, is in week + 1 of 2013. Thus, the year of its ISO week number is 2013, even + though its year is 2012. + +'%G' + The full year of the ISO week number, as a decimal number. + +'%h' + Equivalent to '%b'. + +'%H' + The hour (24-hour clock) as a decimal number (00-23). + +'%I' + The hour (12-hour clock) as a decimal number (01-12). + +'%j' + The day of the year as a decimal number (001-366). + +'%m' + The month as a decimal number (01-12). + +'%M' + The minute as a decimal number (00-59). + +'%n' + A newline character (ASCII LF). + +'%p' + The locale's equivalent of the AM/PM designations associated with a + 12-hour clock. + +'%r' + The locale's 12-hour clock time. (This is '%I:%M:%S %p' in the + '"C"' locale.) + +'%R' + Equivalent to specifying '%H:%M'. + +'%S' + The second as a decimal number (00-60). + +'%t' + A TAB character. + +'%T' + Equivalent to specifying '%H:%M:%S'. + +'%u' + The weekday as a decimal number (1-7). Monday is day one. + +'%U' + The week number of the year (with the first Sunday as the first day + of week one) as a decimal number (00-53). + +'%V' + The week number of the year (with the first Monday as the first day + of week one) as a decimal number (01-53). The method for + determining the week number is as specified by ISO 8601. (To wit: + if the week containing January 1 has four or more days in the new + year, then it is week one; otherwise it is the last week [52 or 53] + of the previous year and the next week is week one.) + +'%w' + The weekday as a decimal number (0-6). Sunday is day zero. + +'%W' + The week number of the year (with the first Monday as the first day + of week one) as a decimal number (00-53). + +'%x' + The locale's "appropriate" date representation. (This is '%A %B %d + %Y' in the '"C"' locale.) + +'%X' + The locale's "appropriate" time representation. (This is '%T' in + the '"C"' locale.) + +'%y' + The year modulo 100 as a decimal number (00-99). + +'%Y' + The full year as a decimal number (e.g., 2015). + +'%z' + The time zone offset in a '+HHMM' format (e.g., the format + necessary to produce RFC 822/RFC 1036 date headers). + +'%Z' + The time zone name or abbreviation; no characters if no time zone + is determinable. + +'%Ec %EC %Ex %EX %Ey %EY %Od %Oe %OH' +'%OI %Om %OM %OS %Ou %OU %OV %Ow %OW %Oy' + "Alternative representations" for the specifications that use only + the second letter ('%c', '%C', and so on).(5) (These facilitate + compliance with the POSIX 'date' utility.) + +'%%' + A literal '%'. + + If a conversion specifier is not one of those just listed, the +behavior is undefined.(6) + + For systems that are not yet fully standards-compliant, 'gawk' +supplies a copy of 'strftime()' from the GNU C Library. It supports all +of the just-listed format specifications. If that version is used to +compile 'gawk' (*note Installation::), then the following additional +format specifications are available: + +'%k' + The hour (24-hour clock) as a decimal number (0-23). Single-digit + numbers are padded with a space. + +'%l' + The hour (12-hour clock) as a decimal number (1-12). Single-digit + numbers are padded with a space. + +'%s' + The time as a decimal timestamp in seconds since the epoch. + + Additionally, the alternative representations are recognized but +their normal representations are used. + + The following example is an 'awk' implementation of the POSIX 'date' +utility. Normally, the 'date' utility prints the current date and time +of day in a well-known format. However, if you provide an argument to +it that begins with a '+', 'date' copies nonformat specifier characters +to the standard output and interprets the current time according to the +format specifiers in the string. For example: + + $ date '+Today is %A, %B %d, %Y.' + -| Today is Monday, September 22, 2014. + + Here is the 'gawk' version of the 'date' utility. It has a shell +"wrapper" to handle the '-u' option, which requires that 'date' run as +if the time zone is set to UTC: + + #! /bin/sh + # + # date --- approximate the POSIX 'date' command + + case $1 in + -u) TZ=UTC0 # use UTC + export TZ + shift ;; + esac + + gawk 'BEGIN { + format = PROCINFO["strftime"] + exitval = 0 + + if (ARGC > 2) + exitval = 1 + else if (ARGC == 2) { + format = ARGV[1] + if (format ~ /^\+/) + format = substr(format, 2) # remove leading + + } + print strftime(format) + exit exitval + }' "$@" + + ---------- Footnotes ---------- + + (1) *Note Glossary::, especially the entries "Epoch" and "UTC." + + (2) The GNU 'date' utility can also do many of the things described +here. Its use may be preferable for simple time-related operations in +shell scripts. + + (3) Occasionally there are minutes in a year with a leap second, +which is why the seconds can go up to 60. + + (4) Unfortunately, not every system's 'strftime()' necessarily +supports all of the conversions listed here. + + (5) If you don't understand any of this, don't worry about it; these +facilities are meant to make it easier to "internationalize" programs. +Other internationalization features are described in *note +Internationalization::. + + (6) This is because ISO C leaves the behavior of the C version of +'strftime()' undefined and 'gawk' uses the system's version of +'strftime()' if it's there. Typically, the conversion specifier either +does not appear in the returned string or appears literally. + + +File: gawk.info, Node: Bitwise Functions, Next: Type Functions, Prev: Time Functions, Up: Built-in + +9.1.6 Bit-Manipulation Functions +-------------------------------- + + I can explain it for you, but I can't understand it for you. + -- _Anonymous_ + + Many languages provide the ability to perform "bitwise" operations on +two integer numbers. In other words, the operation is performed on each +successive pair of bits in the operands. Three common operations are +bitwise AND, OR, and XOR. The operations are described in *note Table +9.6: table-bitwise-ops. + + Bit operator + | AND | OR | XOR + |--+--+--+--+--+-- + Operands | 0 | 1 | 0 | 1 | 0 | 1 + -------+--+--+--+--+--+-- + 0 | 0 0 | 0 1 | 0 1 + 1 | 0 1 | 1 1 | 1 0 + +Table 9.6: Bitwise operations + + As you can see, the result of an AND operation is 1 only when _both_ +bits are 1. The result of an OR operation is 1 if _either_ bit is 1. +The result of an XOR operation is 1 if either bit is 1, but not both. +The next operation is the "complement"; the complement of 1 is 0 and the +complement of 0 is 1. Thus, this operation "flips" all the bits of a +given value. + + Finally, two other common operations are to shift the bits left or +right. For example, if you have a bit string '10111001' and you shift +it right by three bits, you end up with '00010111'.(1) If you start +over again with '10111001' and shift it left by three bits, you end up +with '11001000'. The following list describes 'gawk''s built-in +functions that implement the bitwise operations. Optional parameters +are enclosed in square brackets ([ ]): + +'and(V1, V2 [, ...])' + Return the bitwise AND of the arguments. There must be at least + two. + +'compl(VAL)' + Return the bitwise complement of VAL. + +'lshift(VAL, COUNT)' + Return the value of VAL, shifted left by COUNT bits. + +'or(V1, V2 [, ...])' + Return the bitwise OR of the arguments. There must be at least + two. + +'rshift(VAL, COUNT)' + Return the value of VAL, shifted right by COUNT bits. + +'xor(V1, V2 [, ...])' + Return the bitwise XOR of the arguments. There must be at least + two. + + CAUTION: Beginning with 'gawk' 4.1 4.2, negative operands are not + allowed for any of these functions. A negative operand produces a + fatal error. See the sidebar "Beware The Smoke and Mirrors!" for + more information as to why. + + Here is a user-defined function (*note User-defined::) that +illustrates the use of these functions: + + # bits2str --- turn a byte into readable ones and zeros + + function bits2str(bits, data, mask) + { + if (bits == 0) + return "0" + + mask = 1 + for (; bits != 0; bits = rshift(bits, 1)) + data = (and(bits, mask) ? "1" : "0") data + + while ((length(data) % 8) != 0) + data = "0" data + + return data + } + + BEGIN { + printf "123 = %s\n", bits2str(123) + printf "0123 = %s\n", bits2str(0123) + printf "0x99 = %s\n", bits2str(0x99) + comp = compl(0x99) + printf "compl(0x99) = %#x = %s\n", comp, bits2str(comp) + shift = lshift(0x99, 2) + printf "lshift(0x99, 2) = %#x = %s\n", shift, bits2str(shift) + shift = rshift(0x99, 2) + printf "rshift(0x99, 2) = %#x = %s\n", shift, bits2str(shift) + } + +This program produces the following output when run: + + $ gawk -f testbits.awk + -| 123 = 01111011 + -| 0123 = 01010011 + -| 0x99 = 10011001 + -| compl(0x99) = 0x3fffffffffff66 = 00111111111111111111111111111111111111111111111101100110 + -| lshift(0x99, 2) = 0x264 = 0000001001100100 + -| rshift(0x99, 2) = 0x26 = 00100110 + + The 'bits2str()' function turns a binary number into a string. +Initializing 'mask' to one creates a binary value where the rightmost +bit is set to one. Using this mask, the function repeatedly checks the +rightmost bit. ANDing the mask with the value indicates whether the +rightmost bit is one or not. If so, a '"1"' is concatenated onto the +front of the string. Otherwise, a '"0"' is added. The value is then +shifted right by one bit and the loop continues until there are no more +one bits. + + If the initial value is zero, it returns a simple '"0"'. Otherwise, +at the end, it pads the value with zeros to represent multiples of 8-bit +quantities. This is typical in modern computers. + + The main code in the 'BEGIN' rule shows the difference between the +decimal and octal values for the same numbers (*note +Nondecimal-numbers::), and then demonstrates the results of the +'compl()', 'lshift()', and 'rshift()' functions. + + Beware The Smoke and Mirrors! + + It other languages, bitwise operations are performed on integer +values, not floating-point values. As a general statement, such +operations work best when performed on unsigned integers. + + 'gawk' attempts to treat the arguments to the bitwise functions as +unsigned integers. For this reason, negative arguments produce a fatal +error. + + In normal operation, for all of these functions, first the +double-precision floating-point value is converted to the widest C +unsigned integer type, then the bitwise operation is performed. If the +result cannot be represented exactly as a C 'double', leading nonzero +bits are removed one by one until it can be represented exactly. The +result is then converted back into a C 'double'.(2) + + However, when using arbitrary precision arithmetic with the '-M' +option (*note Arbitrary Precision Arithmetic::), the results may differ. +This is particularly noticable with the 'compl()' function: + + $ gawk 'BEGIN { print compl(42) }' + -| 9007199254740949 + $ gawk -M 'BEGIN { print compl(42) }' + -| -43 + + What's going on becomes clear when printing the results in +hexadecimal: + + $ gawk 'BEGIN { printf "%#x\n", compl(42) }' + -| 0x1fffffffffffd5 + $ gawk -M 'BEGIN { printf "%#x\n", compl(42) }' + -| 0xffffffffffffffd5 + + When using the '-M' option, under the hood, 'gawk' uses GNU MP +arbitrary precision integers which have at least 64 bits of precision. +When not using '-M', 'gawk' stores integral values in regular +double-precision floating point, which only maintain 53 bits of +precision. Furthermore, the GNU MP library treats (or least seems to +treat) the leading bit as a sign bit; thus the result with '-M' in this +case is a negative number. + + In short, using 'gawk' for any but the simplest kind of bitwise +operations is probably a bad idea; caveat emptor! + + ---------- Footnotes ---------- + + (1) This example shows that zeros come in on the left side. For +'gawk', this is always true, but in some languages, it's possible to +have the left side fill with ones. + + (2) If you don't understand this paragraph, the upshot is that 'gawk' +can only store a particular range of integer values; numbers outside +that range are reduced to fit within the range. + + +File: gawk.info, Node: Type Functions, Next: I18N Functions, Prev: Bitwise Functions, Up: Built-in + +9.1.7 Getting Type Information +------------------------------ + +'gawk' provides two functions that lets you distinguish the type of a +variable. This is necessary for writing code that traverses every +element of an array of arrays (*note Arrays of Arrays::), and in other +contexts. + +'isarray(X)' + Return a true value if X is an array. Otherwise, return false. + +'typeof(X)' + Return one of the following strings, depending upon the type of X: + + '"array"' + X is an array. + + '"number"' + X is a number. + + '"string"' + X is a string. + + '"strnum"' + X is a string that might be a number, such as a field or the + result of calling 'split()'. (I.e., X has the STRNUM + attribute; *note Variable Typing::.) + + '"unassigned"' + X is a scalar variable that has not been assigned a value yet. + For example: + + BEGIN { + a[1] # creates a[1] but it has no assigned value + print typeof(a[1]) # scalar_u + } + + '"untyped"' + X has not yet been used yet at all; it can become a scalar or + an array. For example: + + BEGIN { + print typeof(x) # x never used --> untyped + mk_arr(x) + print typeof(x) # x now an array --> array + } + + function mk_arr(a) { a[1] = 1 } + + 'isarray()' is meant for use in two circumstances. The first is when +traversing a multidimensional array: you can test if an element is +itself an array or not. The second is inside the body of a user-defined +function (not discussed yet; *note User-defined::), to test if a +parameter is an array or not. + + NOTE: Using 'isarray()' at the global level to test variables makes + no sense. Because you are the one writing the program, you are + supposed to know if your variables are arrays or not. And in fact, + due to the way 'gawk' works, if you pass the name of a variable + that has not been previously used to 'isarray()', 'gawk' ends up + turning it into a scalar. + + The 'typeof()' function is general; it allows you to determine if a +variable or function parameter is a scalar, an array. + + 'isarray()' is deprecated; you should use 'typeof()' instead. You +should replace any existing uses of 'isarray(var)' in your code with +'typeof(var) == "array"'. + + +File: gawk.info, Node: I18N Functions, Prev: Type Functions, Up: Built-in + +9.1.8 String-Translation Functions +---------------------------------- + +'gawk' provides facilities for internationalizing 'awk' programs. These +include the functions described in the following list. The descriptions +here are purposely brief. *Note Internationalization::, for the full +story. Optional parameters are enclosed in square brackets ([ ]): + +'bindtextdomain(DIRECTORY' [',' DOMAIN]')' + Set the directory in which 'gawk' will look for message translation + files, in case they will not or cannot be placed in the "standard" + locations (e.g., during testing). It returns the directory in + which DOMAIN is "bound." + + The default DOMAIN is the value of 'TEXTDOMAIN'. If DIRECTORY is + the null string ('""'), then 'bindtextdomain()' returns the current + binding for the given DOMAIN. + +'dcgettext(STRING' [',' DOMAIN [',' CATEGORY] ]')' + Return the translation of STRING in text domain DOMAIN for locale + category CATEGORY. The default value for DOMAIN is the current + value of 'TEXTDOMAIN'. The default value for CATEGORY is + '"LC_MESSAGES"'. + +'dcngettext(STRING1, STRING2, NUMBER' [',' DOMAIN [',' CATEGORY] ]')' + Return the plural form used for NUMBER of the translation of + STRING1 and STRING2 in text domain DOMAIN for locale category + CATEGORY. STRING1 is the English singular variant of a message, + and STRING2 is the English plural variant of the same message. The + default value for DOMAIN is the current value of 'TEXTDOMAIN'. The + default value for CATEGORY is '"LC_MESSAGES"'. + + +File: gawk.info, Node: User-defined, Next: Indirect Calls, Prev: Built-in, Up: Functions + +9.2 User-Defined Functions +========================== + +Complicated 'awk' programs can often be simplified by defining your own +functions. User-defined functions can be called just like built-in ones +(*note Function Calls::), but it is up to you to define them (i.e., to +tell 'awk' what they should do). + +* Menu: + +* Definition Syntax:: How to write definitions and what they mean. +* Function Example:: An example function definition and what it + does. +* Function Caveats:: Things to watch out for. +* Return Statement:: Specifying the value a function returns. +* Dynamic Typing:: How variable types can change at runtime. + + +File: gawk.info, Node: Definition Syntax, Next: Function Example, Up: User-defined + +9.2.1 Function Definition Syntax +-------------------------------- + + It's entirely fair to say that the awk syntax for local variable + definitions is appallingly awful. + -- _Brian Kernighan_ + + Definitions of functions can appear anywhere between the rules of an +'awk' program. Thus, the general form of an 'awk' program is extended +to include sequences of rules _and_ user-defined function definitions. +There is no need to put the definition of a function before all uses of +the function. This is because 'awk' reads the entire program before +starting to execute any of it. + + The definition of a function named NAME looks like this: + + 'function' NAME'('[PARAMETER-LIST]')' + '{' + BODY-OF-FUNCTION + '}' + +Here, NAME is the name of the function to define. A valid function name +is like a valid variable name: a sequence of letters, digits, and +underscores that doesn't start with a digit. Here too, only the 52 +upper- and lowercase English letters may be used in a function name. +Within a single 'awk' program, any particular name can only be used as a +variable, array, or function. + + PARAMETER-LIST is an optional list of the function's arguments and +local variable names, separated by commas. When the function is called, +the argument names are used to hold the argument values given in the +call. + + A function cannot have two parameters with the same name, nor may it +have a parameter with the same name as the function itself. + + CAUTION: According to the POSIX standard, function parameters + cannot have the same name as one of the special predefined + variables (*note Built-in Variables::), nor may a function + parameter have the same name as another function. + + Not all versions of 'awk' enforce these restrictions. 'gawk' + always enforces the first restriction. With '--posix' (*note + Options::), it also enforces the second restriction. + + Local variables act like the empty string if referenced where a +string value is required, and like zero if referenced where a numeric +value is required. This is the same as the behavior of regular +variables that have never been assigned a value. (There is more to +understand about local variables; *note Dynamic Typing::.) + + The BODY-OF-FUNCTION consists of 'awk' statements. It is the most +important part of the definition, because it says what the function +should actually _do_. The argument names exist to give the body a way +to talk about the arguments; local variables exist to give the body +places to keep temporary values. + + Argument names are not distinguished syntactically from local +variable names. Instead, the number of arguments supplied when the +function is called determines how many argument variables there are. +Thus, if three argument values are given, the first three names in +PARAMETER-LIST are arguments and the rest are local variables. + + It follows that if the number of arguments is not the same in all +calls to the function, some of the names in PARAMETER-LIST may be +arguments on some occasions and local variables on others. Another way +to think of this is that omitted arguments default to the null string. + + Usually when you write a function, you know how many names you intend +to use for arguments and how many you intend to use as local variables. +It is conventional to place some extra space between the arguments and +the local variables, in order to document how your function is supposed +to be used. + + During execution of the function body, the arguments and local +variable values hide, or "shadow", any variables of the same names used +in the rest of the program. The shadowed variables are not accessible +in the function definition, because there is no way to name them while +their names have been taken away for the arguments and local variables. +All other variables used in the 'awk' program can be referenced or set +normally in the function's body. + + The arguments and local variables last only as long as the function +body is executing. Once the body finishes, you can once again access +the variables that were shadowed while the function was running. + + The function body can contain expressions that call functions. They +can even call this function, either directly or by way of another +function. When this happens, we say the function is "recursive". The +act of a function calling itself is called "recursion". + + All the built-in functions return a value to their caller. +User-defined functions can do so also, using the 'return' statement, +which is described in detail in *note Return Statement::. Many of the +subsequent examples in this minor node use the 'return' statement. + + In many 'awk' implementations, including 'gawk', the keyword +'function' may be abbreviated 'func'. (c.e.) However, POSIX only +specifies the use of the keyword 'function'. This actually has some +practical implications. If 'gawk' is in POSIX-compatibility mode (*note +Options::), then the following statement does _not_ define a function: + + func foo() { a = sqrt($1) ; print a } + +Instead, it defines a rule that, for each record, concatenates the value +of the variable 'func' with the return value of the function 'foo'. If +the resulting string is non-null, the action is executed. This is +probably not what is desired. ('awk' accepts this input as +syntactically valid, because functions may be used before they are +defined in 'awk' programs.(1)) + + To ensure that your 'awk' programs are portable, always use the +keyword 'function' when defining a function. + + ---------- Footnotes ---------- + + (1) This program won't actually run, because 'foo()' is undefined. + + +File: gawk.info, Node: Function Example, Next: Function Caveats, Prev: Definition Syntax, Up: User-defined + +9.2.2 Function Definition Examples +---------------------------------- + +Here is an example of a user-defined function, called 'myprint()', that +takes a number and prints it in a specific format: + + function myprint(num) + { + printf "%6.3g\n", num + } + +To illustrate, here is an 'awk' rule that uses our 'myprint()' function: + + $3 > 0 { myprint($3) } + +This program prints, in our special format, all the third fields that +contain a positive number in our input. Therefore, when given the +following input: + + 1.2 3.4 5.6 7.8 + 9.10 11.12 -13.14 15.16 + 17.18 19.20 21.22 23.24 + +this program, using our function to format the results, prints: + + 5.6 + 21.2 + + This function deletes all the elements in an array (recall that the +extra whitespace signifies the start of the local variable list): + + function delarray(a, i) + { + for (i in a) + delete a[i] + } + + When working with arrays, it is often necessary to delete all the +elements in an array and start over with a new list of elements (*note +Delete::). Instead of having to repeat this loop everywhere that you +need to clear out an array, your program can just call 'delarray()'. +(This guarantees portability. The use of 'delete ARRAY' to delete the +contents of an entire array is a relatively recent(1) addition to the +POSIX standard.) + + The following is an example of a recursive function. It takes a +string as an input parameter and returns the string in reverse order. +Recursive functions must always have a test that stops the recursion. +In this case, the recursion terminates when the input string is already +empty: + + function rev(str) + { + if (str == "") + return "" + + return (rev(substr(str, 2)) substr(str, 1, 1)) + } + + If this function is in a file named 'rev.awk', it can be tested this +way: + + $ echo "Don't Panic!" | + > gawk -e '{ print rev($0) }' -f rev.awk + -| !cinaP t'noD + + The C 'ctime()' function takes a timestamp and returns it as a +string, formatted in a well-known fashion. The following example uses +the built-in 'strftime()' function (*note Time Functions::) to create an +'awk' version of 'ctime()': + + # ctime.awk + # + # awk version of C ctime(3) function + + function ctime(ts, format) + { + format = "%a %b %e %H:%M:%S %Z %Y" + + if (ts == 0) + ts = systime() # use current time as default + return strftime(format, ts) + } + + You might think that 'ctime()' could use 'PROCINFO["strftime"]' for +its format string. That would be a mistake, because 'ctime()' is +supposed to return the time formatted in a standard fashion, and +user-level code could have changed 'PROCINFO["strftime"]'. + + ---------- Footnotes ---------- + + (1) Late in 2012. + + +File: gawk.info, Node: Function Caveats, Next: Return Statement, Prev: Function Example, Up: User-defined + +9.2.3 Calling User-Defined Functions +------------------------------------ + +"Calling a function" means causing the function to run and do its job. +A function call is an expression and its value is the value returned by +the function. + +* Menu: + +* Calling A Function:: Don't use spaces. +* Variable Scope:: Controlling variable scope. +* Pass By Value/Reference:: Passing parameters. + + +File: gawk.info, Node: Calling A Function, Next: Variable Scope, Up: Function Caveats + +9.2.3.1 Writing a Function Call +............................... + +A function call consists of the function name followed by the arguments +in parentheses. 'awk' expressions are what you write in the call for +the arguments. Each time the call is executed, these expressions are +evaluated, and the values become the actual arguments. For example, +here is a call to 'foo()' with three arguments (the first being a string +concatenation): + + foo(x y, "lose", 4 * z) + + CAUTION: Whitespace characters (spaces and TABs) are not allowed + between the function name and the opening parenthesis of the + argument list. If you write whitespace by mistake, 'awk' might + think that you mean to concatenate a variable with an expression in + parentheses. However, it notices that you used a function name and + not a variable name, and reports an error. + + +File: gawk.info, Node: Variable Scope, Next: Pass By Value/Reference, Prev: Calling A Function, Up: Function Caveats + +9.2.3.2 Controlling Variable Scope +.................................. + +Unlike in many languages, there is no way to make a variable local to a +'{' ... '}' block in 'awk', but you can make a variable local to a +function. It is good practice to do so whenever a variable is needed +only in that function. + + To make a variable local to a function, simply declare the variable +as an argument after the actual function arguments (*note Definition +Syntax::). Look at the following example, where variable 'i' is a +global variable used by both functions 'foo()' and 'bar()': + + function bar() + { + for (i = 0; i < 3; i++) + print "bar's i=" i + } + + function foo(j) + { + i = j + 1 + print "foo's i=" i + bar() + print "foo's i=" i + } + + BEGIN { + i = 10 + print "top's i=" i + foo(0) + print "top's i=" i + } + + Running this script produces the following, because the 'i' in +functions 'foo()' and 'bar()' and at the top level refer to the same +variable instance: + + top's i=10 + foo's i=1 + bar's i=0 + bar's i=1 + bar's i=2 + foo's i=3 + top's i=3 + + If you want 'i' to be local to both 'foo()' and 'bar()', do as +follows (the extra space before 'i' is a coding convention to indicate +that 'i' is a local variable, not an argument): + + function bar( i) + { + for (i = 0; i < 3; i++) + print "bar's i=" i + } + + function foo(j, i) + { + i = j + 1 + print "foo's i=" i + bar() + print "foo's i=" i + } + + BEGIN { + i = 10 + print "top's i=" i + foo(0) + print "top's i=" i + } + + Running the corrected script produces the following: + + top's i=10 + foo's i=1 + bar's i=0 + bar's i=1 + bar's i=2 + foo's i=1 + top's i=10 + + Besides scalar values (strings and numbers), you may also have local +arrays. By using a parameter name as an array, 'awk' treats it as an +array, and it is local to the function. In addition, recursive calls +create new arrays. Consider this example: + + function some_func(p1, a) + { + if (p1++ > 3) + return + + a[p1] = p1 + + some_func(p1) + + printf("At level %d, index %d %s found in a\n", + p1, (p1 - 1), (p1 - 1) in a ? "is" : "is not") + printf("At level %d, index %d %s found in a\n", + p1, p1, p1 in a ? "is" : "is not") + print "" + } + + BEGIN { + some_func(1) + } + + When run, this program produces the following output: + + At level 4, index 3 is not found in a + At level 4, index 4 is found in a + + At level 3, index 2 is not found in a + At level 3, index 3 is found in a + + At level 2, index 1 is not found in a + At level 2, index 2 is found in a + + +File: gawk.info, Node: Pass By Value/Reference, Prev: Variable Scope, Up: Function Caveats + +9.2.3.3 Passing Function Arguments by Value Or by Reference +........................................................... + +In 'awk', when you declare a function, there is no way to declare +explicitly whether the arguments are passed "by value" or "by +reference". + + Instead, the passing convention is determined at runtime when the +function is called, according to the following rule: if the argument is +an array variable, then it is passed by reference. Otherwise, the +argument is passed by value. + + Passing an argument by value means that when a function is called, it +is given a _copy_ of the value of this argument. The caller may use a +variable as the expression for the argument, but the called function +does not know this--it only knows what value the argument had. For +example, if you write the following code: + + foo = "bar" + z = myfunc(foo) + +then you should not think of the argument to 'myfunc()' as being "the +variable 'foo'." Instead, think of the argument as the string value +'"bar"'. If the function 'myfunc()' alters the values of its local +variables, this has no effect on any other variables. Thus, if +'myfunc()' does this: + + function myfunc(str) + { + print str + str = "zzz" + print str + } + +to change its first argument variable 'str', it does _not_ change the +value of 'foo' in the caller. The role of 'foo' in calling 'myfunc()' +ended when its value ('"bar"') was computed. If 'str' also exists +outside of 'myfunc()', the function body cannot alter this outer value, +because it is shadowed during the execution of 'myfunc()' and cannot be +seen or changed from there. + + However, when arrays are the parameters to functions, they are _not_ +copied. Instead, the array itself is made available for direct +manipulation by the function. This is usually termed "call by +reference". Changes made to an array parameter inside the body of a +function _are_ visible outside that function. + + NOTE: Changing an array parameter inside a function can be very + dangerous if you do not watch what you are doing. For example: + + function changeit(array, ind, nvalue) + { + array[ind] = nvalue + } + + BEGIN { + a[1] = 1; a[2] = 2; a[3] = 3 + changeit(a, 2, "two") + printf "a[1] = %s, a[2] = %s, a[3] = %s\n", + a[1], a[2], a[3] + } + + prints 'a[1] = 1, a[2] = two, a[3] = 3', because 'changeit()' + stores '"two"' in the second element of 'a'. + + Some 'awk' implementations allow you to call a function that has not +been defined. They only report a problem at runtime, when the program +actually tries to call the function. For example: + + BEGIN { + if (0) + foo() + else + bar() + } + function bar() { ... } + # note that `foo' is not defined + +Because the 'if' statement will never be true, it is not really a +problem that 'foo()' has not been defined. Usually, though, it is a +problem if a program calls an undefined function. + + If '--lint' is specified (*note Options::), 'gawk' reports calls to +undefined functions. + + Some 'awk' implementations generate a runtime error if you use either +the 'next' statement or the 'nextfile' statement (*note Next +Statement::, and *note Nextfile Statement::) inside a user-defined +function. 'gawk' does not have this limitation. + + +File: gawk.info, Node: Return Statement, Next: Dynamic Typing, Prev: Function Caveats, Up: User-defined + +9.2.4 The 'return' Statement +---------------------------- + +As seen in several earlier examples, the body of a user-defined function +can contain a 'return' statement. This statement returns control to the +calling part of the 'awk' program. It can also be used to return a +value for use in the rest of the 'awk' program. It looks like this: + + 'return' [EXPRESSION] + + The EXPRESSION part is optional. Due most likely to an oversight, +POSIX does not define what the return value is if you omit the +EXPRESSION. Technically speaking, this makes the returned value +undefined, and therefore, unpredictable. In practice, though, all +versions of 'awk' simply return the null string, which acts like zero if +used in a numeric context. + + A 'return' statement without an EXPRESSION is assumed at the end of +every function definition. So, if control reaches the end of the +function body, then technically the function returns an unpredictable +value. In practice, it returns the empty string. 'awk' does _not_ warn +you if you use the return value of such a function. + + Sometimes, you want to write a function for what it does, not for +what it returns. Such a function corresponds to a 'void' function in C, +C++, or Java, or to a 'procedure' in Ada. Thus, it may be appropriate +to not return any value; simply bear in mind that you should not be +using the return value of such a function. + + The following is an example of a user-defined function that returns a +value for the largest number among the elements of an array: + + function maxelt(vec, i, ret) + { + for (i in vec) { + if (ret == "" || vec[i] > ret) + ret = vec[i] + } + return ret + } + +You call 'maxelt()' with one argument, which is an array name. The +local variables 'i' and 'ret' are not intended to be arguments; there is +nothing to stop you from passing more than one argument to 'maxelt()' +but the results would be strange. The extra space before 'i' in the +function parameter list indicates that 'i' and 'ret' are local +variables. You should follow this convention when defining functions. + + The following program uses the 'maxelt()' function. It loads an +array, calls 'maxelt()', and then reports the maximum number in that +array: + + function maxelt(vec, i, ret) + { + for (i in vec) { + if (ret == "" || vec[i] > ret) + ret = vec[i] + } + return ret + } + + # Load all fields of each record into nums. + { + for(i = 1; i <= NF; i++) + nums[NR, i] = $i + } + + END { + print maxelt(nums) + } + + Given the following input: + + 1 5 23 8 16 + 44 3 5 2 8 26 + 256 291 1396 2962 100 + -6 467 998 1101 + 99385 11 0 225 + +the program reports (predictably) that 99,385 is the largest value in +the array. + + +File: gawk.info, Node: Dynamic Typing, Prev: Return Statement, Up: User-defined + +9.2.5 Functions and Their Effects on Variable Typing +---------------------------------------------------- + +'awk' is a very fluid language. It is possible that 'awk' can't tell if +an identifier represents a scalar variable or an array until runtime. +Here is an annotated sample program: + + function foo(a) + { + a[1] = 1 # parameter is an array + } + + BEGIN { + b = 1 + foo(b) # invalid: fatal type mismatch + + foo(x) # x uninitialized, becomes an array dynamically + x = 1 # now not allowed, runtime error + } + + In this example, the first call to 'foo()' generates a fatal error, +so 'awk' will not report the second error. If you comment out that +call, though, then 'awk' does report the second error. + + Usually, such things aren't a big issue, but it's worth being aware +of them. + + +File: gawk.info, Node: Indirect Calls, Next: Functions Summary, Prev: User-defined, Up: Functions + +9.3 Indirect Function Calls +=========================== + +This section describes an advanced, 'gawk'-specific extension. + + Often, you may wish to defer the choice of function to call until +runtime. For example, you may have different kinds of records, each of +which should be processed differently. + + Normally, you would have to use a series of 'if'-'else' statements to +decide which function to call. By using "indirect" function calls, you +can specify the name of the function to call as a string variable, and +then call the function. Let's look at an example. + + Suppose you have a file with your test scores for the classes you are +taking, and you wish to get the sum and the average of your test scores. +The first field is the class name. The following fields are the +functions to call to process the data, up to a "marker" field 'data:'. +Following the marker, to the end of the record, are the various numeric +test scores. + + Here is the initial file: + + Biology_101 sum average data: 87.0 92.4 78.5 94.9 + Chemistry_305 sum average data: 75.2 98.3 94.7 88.2 + English_401 sum average data: 100.0 95.6 87.1 93.4 + + To process the data, you might write initially: + + { + class = $1 + for (i = 2; $i != "data:"; i++) { + if ($i == "sum") + sum() # processes the whole record + else if ($i == "average") + average() + ... # and so on + } + } + +This style of programming works, but can be awkward. With "indirect" +function calls, you tell 'gawk' to use the _value_ of a variable as the +_name_ of the function to call. + + The syntax is similar to that of a regular function call: an +identifier immediately followed by an opening parenthesis, any +arguments, and then a closing parenthesis, with the addition of a +leading '@' character: + + the_func = "sum" + result = @the_func() # calls the sum() function + + Here is a full program that processes the previously shown data, +using indirect function calls: + + # indirectcall.awk --- Demonstrate indirect function calls + + # average --- return the average of the values in fields $first - $last + + function average(first, last, sum, i) + { + sum = 0; + for (i = first; i <= last; i++) + sum += $i + + return sum / (last - first + 1) + } + + # sum --- return the sum of the values in fields $first - $last + + function sum(first, last, ret, i) + { + ret = 0; + for (i = first; i <= last; i++) + ret += $i + + return ret + } + + These two functions expect to work on fields; thus, the parameters +'first' and 'last' indicate where in the fields to start and end. +Otherwise, they perform the expected computations and are not unusual: + + # For each record, print the class name and the requested statistics + { + class_name = $1 + gsub(/_/, " ", class_name) # Replace _ with spaces + + # find start + for (i = 1; i <= NF; i++) { + if ($i == "data:") { + start = i + 1 + break + } + } + + printf("%s:\n", class_name) + for (i = 2; $i != "data:"; i++) { + the_function = $i + printf("\t%s: <%s>\n", $i, @the_function(start, NF) "") + } + print "" + } + + This is the main processing for each record. It prints the class +name (with underscores replaced with spaces). It then finds the start +of the actual data, saving it in 'start'. The last part of the code +loops through each function name (from '$2' up to the marker, 'data:'), +calling the function named by the field. The indirect function call +itself occurs as a parameter in the call to 'printf'. (The 'printf' +format string uses '%s' as the format specifier so that we can use +functions that return strings, as well as numbers. Note that the result +from the indirect call is concatenated with the empty string, in order +to force it to be a string value.) + + Here is the result of running the program: + + $ gawk -f indirectcall.awk class_data1 + -| Biology 101: + -| sum: <352.8> + -| average: <88.2> + -| + -| Chemistry 305: + -| sum: <356.4> + -| average: <89.1> + -| + -| English 401: + -| sum: <376.1> + -| average: <94.025> + + The ability to use indirect function calls is more powerful than you +may think at first. The C and C++ languages provide "function +pointers," which are a mechanism for calling a function chosen at +runtime. One of the most well-known uses of this ability is the C +'qsort()' function, which sorts an array using the famous "quicksort" +algorithm (see the Wikipedia article +(http://en.wikipedia.org/wiki/Quicksort) for more information). To use +this function, you supply a pointer to a comparison function. This +mechanism allows you to sort arbitrary data in an arbitrary fashion. + + We can do something similar using 'gawk', like this: + + # quicksort.awk --- Quicksort algorithm, with user-supplied + # comparison function + + # quicksort --- C.A.R. Hoare's quicksort algorithm. See Wikipedia + # or almost any algorithms or computer science text. + + function quicksort(data, left, right, less_than, i, last) + { + if (left >= right) # do nothing if array contains fewer + return # than two elements + + quicksort_swap(data, left, int((left + right) / 2)) + last = left + for (i = left + 1; i <= right; i++) + if (@less_than(data[i], data[left])) + quicksort_swap(data, ++last, i) + quicksort_swap(data, left, last) + quicksort(data, left, last - 1, less_than) + quicksort(data, last + 1, right, less_than) + } + + # quicksort_swap --- helper function for quicksort, should really be inline + + function quicksort_swap(data, i, j, temp) + { + temp = data[i] + data[i] = data[j] + data[j] = temp + } + + The 'quicksort()' function receives the 'data' array, the starting +and ending indices to sort ('left' and 'right'), and the name of a +function that performs a "less than" comparison. It then implements the +quicksort algorithm. + + To make use of the sorting function, we return to our previous +example. The first thing to do is write some comparison functions: + + # num_lt --- do a numeric less than comparison + + function num_lt(left, right) + { + return ((left + 0) < (right + 0)) + } + + # num_ge --- do a numeric greater than or equal to comparison + + function num_ge(left, right) + { + return ((left + 0) >= (right + 0)) + } + + The 'num_ge()' function is needed to perform a descending sort; when +used to perform a "less than" test, it actually does the opposite +(greater than or equal to), which yields data sorted in descending +order. + + Next comes a sorting function. It is parameterized with the starting +and ending field numbers and the comparison function. It builds an +array with the data and calls 'quicksort()' appropriately, and then +formats the results as a single string: + + # do_sort --- sort the data according to `compare' + # and return it as a string + + function do_sort(first, last, compare, data, i, retval) + { + delete data + for (i = 1; first <= last; first++) { + data[i] = $first + i++ + } + + quicksort(data, 1, i-1, compare) + + retval = data[1] + for (i = 2; i in data; i++) + retval = retval " " data[i] + + return retval + } + + Finally, the two sorting functions call 'do_sort()', passing in the +names of the two comparison functions: + + # sort --- sort the data in ascending order and return it as a string + + function sort(first, last) + { + return do_sort(first, last, "num_lt") + } + + # rsort --- sort the data in descending order and return it as a string + + function rsort(first, last) + { + return do_sort(first, last, "num_ge") + } + + Here is an extended version of the data file: + + Biology_101 sum average sort rsort data: 87.0 92.4 78.5 94.9 + Chemistry_305 sum average sort rsort data: 75.2 98.3 94.7 88.2 + English_401 sum average sort rsort data: 100.0 95.6 87.1 93.4 + + Finally, here are the results when the enhanced program is run: + + $ gawk -f quicksort.awk -f indirectcall.awk class_data2 + -| Biology 101: + -| sum: <352.8> + -| average: <88.2> + -| sort: <78.5 87.0 92.4 94.9> + -| rsort: <94.9 92.4 87.0 78.5> + -| + -| Chemistry 305: + -| sum: <356.4> + -| average: <89.1> + -| sort: <75.2 88.2 94.7 98.3> + -| rsort: <98.3 94.7 88.2 75.2> + -| + -| English 401: + -| sum: <376.1> + -| average: <94.025> + -| sort: <87.1 93.4 95.6 100.0> + -| rsort: <100.0 95.6 93.4 87.1> + + Another example where indirect functions calls are useful can be +found in processing arrays. This is described in *note Walking +Arrays::. + + Remember that you must supply a leading '@' in front of an indirect +function call. + + Starting with version 4.1.2 of 'gawk', indirect function calls may +also be used with built-in functions and with extension functions (*note +Dynamic Extensions::). There are some limitations when calling built-in +functions indirectly, as follows. + + * You cannot pass a regular expression constant to a built-in + function through an indirect function call.(1) This applies to the + 'sub()', 'gsub()', 'gensub()', 'match()', 'split()' and + 'patsplit()' functions. + + * If calling 'sub()' or 'gsub()', you may only pass two arguments, + since those functions are unusual in that they update their third + argument. This means that '$0' will be updated. + + 'gawk' does its best to make indirect function calls efficient. For +example, in the following case: + + for (i = 1; i <= n; i++) + @the_func() + +'gawk' looks up the actual function to call only once. + + ---------- Footnotes ---------- + + (1) This may change in a future version; recheck the documentation +that comes with your version of 'gawk' to see if it has. + + +File: gawk.info, Node: Functions Summary, Prev: Indirect Calls, Up: Functions + +9.4 Summary +=========== + + * 'awk' provides built-in functions and lets you define your own + functions. + + * POSIX 'awk' provides three kinds of built-in functions: numeric, + string, and I/O. 'gawk' provides functions that sort arrays, work + with values representing time, do bit manipulation, determine + variable type (array versus scalar), and internationalize and + localize programs. 'gawk' also provides several extensions to some + of standard functions, typically in the form of additional + arguments. + + * Functions accept zero or more arguments and return a value. The + expressions that provide the argument values are completely + evaluated before the function is called. Order of evaluation is + not defined. The return value can be ignored. + + * The handling of backslash in 'sub()' and 'gsub()' is not simple. + It is more straightforward in 'gawk''s 'gensub()' function, but + that function still requires care in its use. + + * User-defined functions provide important capabilities but come with + some syntactic inelegancies. In a function call, there cannot be + any space between the function name and the opening left + parenthesis of the argument list. Also, there is no provision for + local variables, so the convention is to add extra parameters, and + to separate them visually from the real parameters by extra + whitespace. + + * User-defined functions may call other user-defined (and built-in) + functions and may call themselves recursively. Function parameters + "hide" any global variables of the same names. You cannot use the + name of a reserved variable (such as 'ARGC') as the name of a + parameter in user-defined functions. + + * Scalar values are passed to user-defined functions by value. Array + parameters are passed by reference; any changes made by the + function to array parameters are thus visible after the function + has returned. + + * Use the 'return' statement to return from a user-defined function. + An optional expression becomes the function's return value. Only + scalar values may be returned by a function. + + * If a variable that has never been used is passed to a user-defined + function, how that function treats the variable can set its nature: + either scalar or array. + + * 'gawk' provides indirect function calls using a special syntax. By + setting a variable to the name of a function, you can determine at + runtime what function will be called at that point in the program. + This is equivalent to function pointers in C and C++. + + +File: gawk.info, Node: Library Functions, Next: Sample Programs, Prev: Functions, Up: Top + +10 A Library of 'awk' Functions +******************************* + +*note User-defined:: describes how to write your own 'awk' functions. +Writing functions is important, because it allows you to encapsulate +algorithms and program tasks in a single place. It simplifies +programming, making program development more manageable and making +programs more readable. + + In their seminal 1976 book, 'Software Tools',(1) Brian Kernighan and +P.J. Plauger wrote: + + Good Programming is not learned from generalities, but by seeing + how significant programs can be made clean, easy to read, easy to + maintain and modify, human-engineered, efficient and reliable, by + the application of common sense and good programming practices. + Careful study and imitation of good programs leads to better + writing. + + In fact, they felt this idea was so important that they placed this +statement on the cover of their book. Because we believe strongly that +their statement is correct, this major node and *note Sample Programs::, +provide a good-sized body of code for you to read and, we hope, to learn +from. + + This major node presents a library of useful 'awk' functions. Many +of the sample programs presented later in this Info file use these +functions. The functions are presented here in a progression from +simple to complex. + + *note Extract Program:: presents a program that you can use to +extract the source code for these example library functions and programs +from the Texinfo source for this Info file. (This has already been done +as part of the 'gawk' distribution.) + + If you have written one or more useful, general-purpose 'awk' +functions and would like to contribute them to the 'awk' user community, +see *note How To Contribute::, for more information. + + The programs in this major node and in *note Sample Programs::, +freely use 'gawk'-specific features. Rewriting these programs for +different implementations of 'awk' is pretty straightforward: + + * Diagnostic error messages are sent to '/dev/stderr'. Use '| "cat + 1>&2"' instead of '> "/dev/stderr"' if your system does not have a + '/dev/stderr', or if you cannot use 'gawk'. + + * A number of programs use 'nextfile' (*note Nextfile Statement::) to + skip any remaining input in the input file. + + * Finally, some of the programs choose to ignore upper- and lowercase + distinctions in their input. They do so by assigning one to + 'IGNORECASE'. You can achieve almost the same effect(2) by adding + the following rule to the beginning of the program: + + # ignore case + { $0 = tolower($0) } + + Also, verify that all regexp and string constants used in + comparisons use only lowercase letters. + +* Menu: + +* Library Names:: How to best name private global variables in + library functions. +* General Functions:: Functions that are of general use. +* Data File Management:: Functions for managing command-line data + files. +* Getopt Function:: A function for processing command-line + arguments. +* Passwd Functions:: Functions for getting user information. +* Group Functions:: Functions for getting group information. +* Walking Arrays:: A function to walk arrays of arrays. +* Library Functions Summary:: Summary of library functions. +* Library Exercises:: Exercises. + + ---------- Footnotes ---------- + + (1) Sadly, over 35 years later, many of the lessons taught by this +book have yet to be learned by a vast number of practicing programmers. + + (2) The effects are not identical. Output of the transformed record +will be in all lowercase, while 'IGNORECASE' preserves the original +contents of the input record. + + +File: gawk.info, Node: Library Names, Next: General Functions, Up: Library Functions + +10.1 Naming Library Function Global Variables +============================================= + +Due to the way the 'awk' language evolved, variables are either "global" +(usable by the entire program) or "local" (usable just by a specific +function). There is no intermediate state analogous to 'static' +variables in C. + + Library functions often need to have global variables that they can +use to preserve state information between calls to the function--for +example, 'getopt()''s variable '_opti' (*note Getopt Function::). Such +variables are called "private", as the only functions that need to use +them are the ones in the library. + + When writing a library function, you should try to choose names for +your private variables that will not conflict with any variables used by +either another library function or a user's main program. For example, +a name like 'i' or 'j' is not a good choice, because user programs often +use variable names like these for their own purposes. + + The example programs shown in this major node all start the names of +their private variables with an underscore ('_'). Users generally don't +use leading underscores in their variable names, so this convention +immediately decreases the chances that the variable names will be +accidentally shared with the user's program. + + In addition, several of the library functions use a prefix that helps +indicate what function or set of functions use the variables--for +example, '_pw_byname()' in the user database routines (*note Passwd +Functions::). This convention is recommended, as it even further +decreases the chance of inadvertent conflict among variable names. Note +that this convention is used equally well for variable names and for +private function names.(1) + + As a final note on variable naming, if a function makes global +variables available for use by a main program, it is a good convention +to start those variables' names with a capital letter--for example, +'getopt()''s 'Opterr' and 'Optind' variables (*note Getopt Function::). +The leading capital letter indicates that it is global, while the fact +that the variable name is not all capital letters indicates that the +variable is not one of 'awk''s predefined variables, such as 'FS'. + + It is also important that _all_ variables in library functions that +do not need to save state are, in fact, declared local.(2) If this is +not done, the variables could accidentally be used in the user's +program, leading to bugs that are very difficult to track down: + + function lib_func(x, y, l1, l2) + { + ... + # some_var should be local but by oversight is not + USE VARIABLE some_var + ... + } + + A different convention, common in the Tcl community, is to use a +single associative array to hold the values needed by the library +function(s), or "package." This significantly decreases the number of +actual global names in use. For example, the functions described in +*note Passwd Functions:: might have used array elements +'PW_data["inited"]', 'PW_data["total"]', 'PW_data["count"]', and +'PW_data["awklib"]', instead of '_pw_inited', '_pw_awklib', '_pw_total', +and '_pw_count'. + + The conventions presented in this minor node are exactly that: +conventions. You are not required to write your programs this way--we +merely recommend that you do so. + + ---------- Footnotes ---------- + + (1) Although all the library routines could have been rewritten to +use this convention, this was not done, in order to show how our own +'awk' programming style has evolved and to provide some basis for this +discussion. + + (2) 'gawk''s '--dump-variables' command-line option is useful for +verifying this. + + +File: gawk.info, Node: General Functions, Next: Data File Management, Prev: Library Names, Up: Library Functions + +10.2 General Programming +======================== + +This minor node presents a number of functions that are of general +programming use. + +* Menu: + +* Strtonum Function:: A replacement for the built-in + 'strtonum()' function. +* Assert Function:: A function for assertions in 'awk' + programs. +* Round Function:: A function for rounding if 'sprintf()' + does not do it correctly. +* Cliff Random Function:: The Cliff Random Number Generator. +* Ordinal Functions:: Functions for using characters as numbers and + vice versa. +* Join Function:: A function to join an array into a string. +* Getlocaltime Function:: A function to get formatted times. +* Readfile Function:: A function to read an entire file at once. +* Shell Quoting:: A function to quote strings for the shell. + + +File: gawk.info, Node: Strtonum Function, Next: Assert Function, Up: General Functions + +10.2.1 Converting Strings to Numbers +------------------------------------ + +The 'strtonum()' function (*note String Functions::) is a 'gawk' +extension. The following function provides an implementation for other +versions of 'awk': + + # mystrtonum --- convert string to number + + function mystrtonum(str, ret, n, i, k, c) + { + if (str ~ /^0[0-7]*$/) { + # octal + n = length(str) + ret = 0 + for (i = 1; i <= n; i++) { + c = substr(str, i, 1) + # index() returns 0 if c not in string, + # includes c == "0" + k = index("1234567", c) + + ret = ret * 8 + k + } + } else if (str ~ /^0[xX][[:xdigit:]]+$/) { + # hexadecimal + str = substr(str, 3) # lop off leading 0x + n = length(str) + ret = 0 + for (i = 1; i <= n; i++) { + c = substr(str, i, 1) + c = tolower(c) + # index() returns 0 if c not in string, + # includes c == "0" + k = index("123456789abcdef", c) + + ret = ret * 16 + k + } + } else if (str ~ \ + /^[-+]?([0-9]+([.][0-9]*([Ee][0-9]+)?)?|([.][0-9]+([Ee][-+]?[0-9]+)?))$/) { + # decimal number, possibly floating point + ret = str + 0 + } else + ret = "NOT-A-NUMBER" + + return ret + } + + # BEGIN { # gawk test harness + # a[1] = "25" + # a[2] = ".31" + # a[3] = "0123" + # a[4] = "0xdeadBEEF" + # a[5] = "123.45" + # a[6] = "1.e3" + # a[7] = "1.32" + # a[8] = "1.32E2" + # + # for (i = 1; i in a; i++) + # print a[i], strtonum(a[i]), mystrtonum(a[i]) + # } + + The function first looks for C-style octal numbers (base 8). If the +input string matches a regular expression describing octal numbers, then +'mystrtonum()' loops through each character in the string. It sets 'k' +to the index in '"1234567"' of the current octal digit. The return +value will either be the same number as the digit, or zero if the +character is not there, which will be true for a '0'. This is safe, +because the regexp test in the 'if' ensures that only octal values are +converted. + + Similar logic applies to the code that checks for and converts a +hexadecimal value, which starts with '0x' or '0X'. The use of +'tolower()' simplifies the computation for finding the correct numeric +value for each hexadecimal digit. + + Finally, if the string matches the (rather complicated) regexp for a +regular decimal integer or floating-point number, the computation 'ret = +str + 0' lets 'awk' convert the value to a number. + + A commented-out test program is included, so that the function can be +tested with 'gawk' and the results compared to the built-in 'strtonum()' +function. + + +File: gawk.info, Node: Assert Function, Next: Round Function, Prev: Strtonum Function, Up: General Functions + +10.2.2 Assertions +----------------- + +When writing large programs, it is often useful to know that a condition +or set of conditions is true. Before proceeding with a particular +computation, you make a statement about what you believe to be the case. +Such a statement is known as an "assertion". The C language provides an +'<assert.h>' header file and corresponding 'assert()' macro that a +programmer can use to make assertions. If an assertion fails, the +'assert()' macro arranges to print a diagnostic message describing the +condition that should have been true but was not, and then it kills the +program. In C, using 'assert()' looks this: + + #include <assert.h> + + int myfunc(int a, double b) + { + assert(a <= 5 && b >= 17.1); + ... + } + + If the assertion fails, the program prints a message similar to this: + + prog.c:5: assertion failed: a <= 5 && b >= 17.1 + + The C language makes it possible to turn the condition into a string +for use in printing the diagnostic message. This is not possible in +'awk', so this 'assert()' function also requires a string version of the +condition that is being tested. Following is the function: + + # assert --- assert that a condition is true. Otherwise, exit. + + function assert(condition, string) + { + if (! condition) { + printf("%s:%d: assertion failed: %s\n", + FILENAME, FNR, string) > "/dev/stderr" + _assert_exit = 1 + exit 1 + } + } + + END { + if (_assert_exit) + exit 1 + } + + The 'assert()' function tests the 'condition' parameter. If it is +false, it prints a message to standard error, using the 'string' +parameter to describe the failed condition. It then sets the variable +'_assert_exit' to one and executes the 'exit' statement. The 'exit' +statement jumps to the 'END' rule. If the 'END' rule finds +'_assert_exit' to be true, it exits immediately. + + The purpose of the test in the 'END' rule is to keep any other 'END' +rules from running. When an assertion fails, the program should exit +immediately. If no assertions fail, then '_assert_exit' is still false +when the 'END' rule is run normally, and the rest of the program's 'END' +rules execute. For all of this to work correctly, 'assert.awk' must be +the first source file read by 'awk'. The function can be used in a +program in the following way: + + function myfunc(a, b) + { + assert(a <= 5 && b >= 17.1, "a <= 5 && b >= 17.1") + ... + } + +If the assertion fails, you see a message similar to the following: + + mydata:1357: assertion failed: a <= 5 && b >= 17.1 + + There is a small problem with this version of 'assert()'. An 'END' +rule is automatically added to the program calling 'assert()'. +Normally, if a program consists of just a 'BEGIN' rule, the input files +and/or standard input are not read. However, now that the program has +an 'END' rule, 'awk' attempts to read the input data files or standard +input (*note Using BEGIN/END::), most likely causing the program to hang +as it waits for input. + + There is a simple workaround to this: make sure that such a 'BEGIN' +rule always ends with an 'exit' statement. + + +File: gawk.info, Node: Round Function, Next: Cliff Random Function, Prev: Assert Function, Up: General Functions + +10.2.3 Rounding Numbers +----------------------- + +The way 'printf' and 'sprintf()' (*note Printf::) perform rounding often +depends upon the system's C 'sprintf()' subroutine. On many machines, +'sprintf()' rounding is "unbiased", which means it doesn't always round +a trailing .5 up, contrary to naive expectations. In unbiased rounding, +.5 rounds to even, rather than always up, so 1.5 rounds to 2 but 4.5 +rounds to 4. This means that if you are using a format that does +rounding (e.g., '"%.0f"'), you should check what your system does. The +following function does traditional rounding; it might be useful if your +'awk''s 'printf' does unbiased rounding: + + # round.awk --- do normal rounding + + function round(x, ival, aval, fraction) + { + ival = int(x) # integer part, int() truncates + + # see if fractional part + if (ival == x) # no fraction + return ival # ensure no decimals + + if (x < 0) { + aval = -x # absolute value + ival = int(aval) + fraction = aval - ival + if (fraction >= .5) + return int(x) - 1 # -2.5 --> -3 + else + return int(x) # -2.3 --> -2 + } else { + fraction = x - ival + if (fraction >= .5) + return ival + 1 + else + return ival + } + } + + # test harness + # { print $0, round($0) } + + +File: gawk.info, Node: Cliff Random Function, Next: Ordinal Functions, Prev: Round Function, Up: General Functions + +10.2.4 The Cliff Random Number Generator +---------------------------------------- + +The Cliff random number generator +(http://mathworld.wolfram.com/CliffRandomNumberGenerator.html) is a very +simple random number generator that "passes the noise sphere test for +randomness by showing no structure." It is easily programmed, in less +than 10 lines of 'awk' code: + + # cliff_rand.awk --- generate Cliff random numbers + + BEGIN { _cliff_seed = 0.1 } + + function cliff_rand() + { + _cliff_seed = (100 * log(_cliff_seed)) % 1 + if (_cliff_seed < 0) + _cliff_seed = - _cliff_seed + return _cliff_seed + } + + This algorithm requires an initial "seed" of 0.1. Each new value +uses the current seed as input for the calculation. If the built-in +'rand()' function (*note Numeric Functions::) isn't random enough, you +might try using this function instead. + + +File: gawk.info, Node: Ordinal Functions, Next: Join Function, Prev: Cliff Random Function, Up: General Functions + +10.2.5 Translating Between Characters and Numbers +------------------------------------------------- + +One commercial implementation of 'awk' supplies a built-in function, +'ord()', which takes a character and returns the numeric value for that +character in the machine's character set. If the string passed to +'ord()' has more than one character, only the first one is used. + + The inverse of this function is 'chr()' (from the function of the +same name in Pascal), which takes a number and returns the corresponding +character. Both functions are written very nicely in 'awk'; there is no +real reason to build them into the 'awk' interpreter: + + # ord.awk --- do ord and chr + + # Global identifiers: + # _ord_: numerical values indexed by characters + # _ord_init: function to initialize _ord_ + + BEGIN { _ord_init() } + + function _ord_init( low, high, i, t) + { + low = sprintf("%c", 7) # BEL is ascii 7 + if (low == "\a") { # regular ascii + low = 0 + high = 127 + } else if (sprintf("%c", 128 + 7) == "\a") { + # ascii, mark parity + low = 128 + high = 255 + } else { # ebcdic(!) + low = 0 + high = 255 + } + + for (i = low; i <= high; i++) { + t = sprintf("%c", i) + _ord_[t] = i + } + } + + Some explanation of the numbers used by '_ord_init()' is worthwhile. +The most prominent character set in use today is ASCII.(1) Although an +8-bit byte can hold 256 distinct values (from 0 to 255), ASCII only +defines characters that use the values from 0 to 127.(2) In the now +distant past, at least one minicomputer manufacturer used ASCII, but +with mark parity, meaning that the leftmost bit in the byte is always 1. +This means that on those systems, characters have numeric values from +128 to 255. Finally, large mainframe systems use the EBCDIC character +set, which uses all 256 values. There are other character sets in use +on some older systems, but they are not really worth worrying about: + + function ord(str, c) + { + # only first character is of interest + c = substr(str, 1, 1) + return _ord_[c] + } + + function chr(c) + { + # force c to be numeric by adding 0 + return sprintf("%c", c + 0) + } + + #### test code #### + # BEGIN { + # for (;;) { + # printf("enter a character: ") + # if (getline var <= 0) + # break + # printf("ord(%s) = %d\n", var, ord(var)) + # } + # } + + An obvious improvement to these functions is to move the code for the +'_ord_init' function into the body of the 'BEGIN' rule. It was written +this way initially for ease of development. There is a "test program" +in a 'BEGIN' rule, to test the function. It is commented out for +production use. + + ---------- Footnotes ---------- + + (1) This is changing; many systems use Unicode, a very large +character set that includes ASCII as a subset. On systems with full +Unicode support, a character can occupy up to 32 bits, making simple +tests such as used here prohibitively expensive. + + (2) ASCII has been extended in many countries to use the values from +128 to 255 for country-specific characters. If your system uses these +extensions, you can simplify '_ord_init()' to loop from 0 to 255. + + +File: gawk.info, Node: Join Function, Next: Getlocaltime Function, Prev: Ordinal Functions, Up: General Functions + +10.2.6 Merging an Array into a String +------------------------------------- + +When doing string processing, it is often useful to be able to join all +the strings in an array into one long string. The following function, +'join()', accomplishes this task. It is used later in several of the +application programs (*note Sample Programs::). + + Good function design is important; this function needs to be general, +but it should also have a reasonable default behavior. It is called +with an array as well as the beginning and ending indices of the +elements in the array to be merged. This assumes that the array indices +are numeric--a reasonable assumption, as the array was likely created +with 'split()' (*note String Functions::): + + # join.awk --- join an array into a string + + function join(array, start, end, sep, result, i) + { + if (sep == "") + sep = " " + else if (sep == SUBSEP) # magic value + sep = "" + result = array[start] + for (i = start + 1; i <= end; i++) + result = result sep array[i] + return result + } + + An optional additional argument is the separator to use when joining +the strings back together. If the caller supplies a nonempty value, +'join()' uses it; if it is not supplied, it has a null value. In this +case, 'join()' uses a single space as a default separator for the +strings. If the value is equal to 'SUBSEP', then 'join()' joins the +strings with no separator between them. 'SUBSEP' serves as a "magic" +value to indicate that there should be no separation between the +component strings.(1) + + ---------- Footnotes ---------- + + (1) It would be nice if 'awk' had an assignment operator for +concatenation. The lack of an explicit operator for concatenation makes +string operations more difficult than they really need to be. + + +File: gawk.info, Node: Getlocaltime Function, Next: Readfile Function, Prev: Join Function, Up: General Functions + +10.2.7 Managing the Time of Day +------------------------------- + +The 'systime()' and 'strftime()' functions described in *note Time +Functions:: provide the minimum functionality necessary for dealing with +the time of day in human-readable form. Although 'strftime()' is +extensive, the control formats are not necessarily easy to remember or +intuitively obvious when reading a program. + + The following function, 'getlocaltime()', populates a user-supplied +array with preformatted time information. It returns a string with the +current time formatted in the same way as the 'date' utility: + + # getlocaltime.awk --- get the time of day in a usable format + + # Returns a string in the format of output of date(1) + # Populates the array argument time with individual values: + # time["second"] -- seconds (0 - 59) + # time["minute"] -- minutes (0 - 59) + # time["hour"] -- hours (0 - 23) + # time["althour"] -- hours (0 - 12) + # time["monthday"] -- day of month (1 - 31) + # time["month"] -- month of year (1 - 12) + # time["monthname"] -- name of the month + # time["shortmonth"] -- short name of the month + # time["year"] -- year modulo 100 (0 - 99) + # time["fullyear"] -- full year + # time["weekday"] -- day of week (Sunday = 0) + # time["altweekday"] -- day of week (Monday = 0) + # time["dayname"] -- name of weekday + # time["shortdayname"] -- short name of weekday + # time["yearday"] -- day of year (0 - 365) + # time["timezone"] -- abbreviation of timezone name + # time["ampm"] -- AM or PM designation + # time["weeknum"] -- week number, Sunday first day + # time["altweeknum"] -- week number, Monday first day + + function getlocaltime(time, ret, now, i) + { + # get time once, avoids unnecessary system calls + now = systime() + + # return date(1)-style output + ret = strftime("%a %b %e %H:%M:%S %Z %Y", now) + + # clear out target array + delete time + + # fill in values, force numeric values to be + # numeric by adding 0 + time["second"] = strftime("%S", now) + 0 + time["minute"] = strftime("%M", now) + 0 + time["hour"] = strftime("%H", now) + 0 + time["althour"] = strftime("%I", now) + 0 + time["monthday"] = strftime("%d", now) + 0 + time["month"] = strftime("%m", now) + 0 + time["monthname"] = strftime("%B", now) + time["shortmonth"] = strftime("%b", now) + time["year"] = strftime("%y", now) + 0 + time["fullyear"] = strftime("%Y", now) + 0 + time["weekday"] = strftime("%w", now) + 0 + time["altweekday"] = strftime("%u", now) + 0 + time["dayname"] = strftime("%A", now) + time["shortdayname"] = strftime("%a", now) + time["yearday"] = strftime("%j", now) + 0 + time["timezone"] = strftime("%Z", now) + time["ampm"] = strftime("%p", now) + time["weeknum"] = strftime("%U", now) + 0 + time["altweeknum"] = strftime("%W", now) + 0 + + return ret + } + + The string indices are easier to use and read than the various +formats required by 'strftime()'. The 'alarm' program presented in +*note Alarm Program:: uses this function. A more general design for the +'getlocaltime()' function would have allowed the user to supply an +optional timestamp value to use instead of the current time. + + +File: gawk.info, Node: Readfile Function, Next: Shell Quoting, Prev: Getlocaltime Function, Up: General Functions + +10.2.8 Reading a Whole File at Once +----------------------------------- + +Often, it is convenient to have the entire contents of a file available +in memory as a single string. A straightforward but naive way to do +that might be as follows: + + function readfile(file, tmp, contents) + { + if ((getline tmp < file) < 0) + return + + contents = tmp + while (getline tmp < file) > 0) + contents = contents RT tmp + + close(file) + return contents + } + + This function reads from 'file' one record at a time, building up the +full contents of the file in the local variable 'contents'. It works, +but is not necessarily efficient. + + The following function, based on a suggestion by Denis Shirokov, +reads the entire contents of the named file in one shot: + + # readfile.awk --- read an entire file at once + + function readfile(file, tmp, save_rs) + { + save_rs = RS + RS = "^$" + getline tmp < file + close(file) + RS = save_rs + + return tmp + } + + It works by setting 'RS' to '^$', a regular expression that will +never match if the file has contents. 'gawk' reads data from the file +into 'tmp', attempting to match 'RS'. The match fails after each read, +but fails quickly, such that 'gawk' fills 'tmp' with the entire contents +of the file. (*Note Records:: for information on 'RT' and 'RS'.) + + In the case that 'file' is empty, the return value is the null +string. Thus, calling code may use something like: + + contents = readfile("/some/path") + if (length(contents) == 0) + # file was empty ... + + This tests the result to see if it is empty or not. An equivalent +test would be 'contents == ""'. + + *Note Extension Sample Readfile:: for an extension function that also +reads an entire file into memory. + + +File: gawk.info, Node: Shell Quoting, Prev: Readfile Function, Up: General Functions + +10.2.9 Quoting Strings to Pass to the Shell +------------------------------------------- + +Michael Brennan offers the following programming pattern, which he uses +frequently: + + #! /bin/sh + + awkp=' + ... + ' + + INPUT_PROGRAM | awk "$awkp" | /bin/sh + + For example, a program of his named 'flac-edit' has this form: + + $ flac-edit -song="Whoope! That's Great" file.flac + + It generates the following output, which is to be piped to the shell +('/bin/sh'): + + chmod +w file.flac + metaflac --remove-tag=TITLE file.flac + LANG=en_US.88591 metaflac --set-tag=TITLE='Whoope! That'"'"'s Great' file.flac + chmod -w file.flac + + Note the need for shell quoting. The function 'shell_quote()' does +it. 'SINGLE' is the one-character string '"'"' and 'QSINGLE' is the +three-character string '"\"'\""': + + # shell_quote --- quote an argument for passing to the shell + + function shell_quote(s, # parameter + SINGLE, QSINGLE, i, X, n, ret) # locals + { + if (s == "") + return "\"\"" + + SINGLE = "\x27" # single quote + QSINGLE = "\"\x27\"" + n = split(s, X, SINGLE) + + ret = SINGLE X[1] SINGLE + for (i = 2; i <= n; i++) + ret = ret QSINGLE SINGLE X[i] SINGLE + + return ret + } + + +File: gawk.info, Node: Data File Management, Next: Getopt Function, Prev: General Functions, Up: Library Functions + +10.3 Data file Management +========================= + +This minor node presents functions that are useful for managing +command-line data files. + +* Menu: + +* Filetrans Function:: A function for handling data file transitions. +* Rewind Function:: A function for rereading the current file. +* File Checking:: Checking that data files are readable. +* Empty Files:: Checking for zero-length files. +* Ignoring Assigns:: Treating assignments as file names. + + +File: gawk.info, Node: Filetrans Function, Next: Rewind Function, Up: Data File Management + +10.3.1 Noting Data file Boundaries +---------------------------------- + +The 'BEGIN' and 'END' rules are each executed exactly once, at the +beginning and end of your 'awk' program, respectively (*note +BEGIN/END::). We (the 'gawk' authors) once had a user who mistakenly +thought that the 'BEGIN' rules were executed at the beginning of each +data file and the 'END' rules were executed at the end of each data +file. + + When informed that this was not the case, the user requested that we +add new special patterns to 'gawk', named 'BEGIN_FILE' and 'END_FILE', +that would have the desired behavior. He even supplied us the code to +do so. + + Adding these special patterns to 'gawk' wasn't necessary; the job can +be done cleanly in 'awk' itself, as illustrated by the following library +program. It arranges to call two user-supplied functions, 'beginfile()' +and 'endfile()', at the beginning and end of each data file. Besides +solving the problem in only nine(!) lines of code, it does so +_portably_; this works with any implementation of 'awk': + + # transfile.awk + # + # Give the user a hook for filename transitions + # + # The user must supply functions beginfile() and endfile() + # that each take the name of the file being started or + # finished, respectively. + + FILENAME != _oldfilename { + if (_oldfilename != "") + endfile(_oldfilename) + _oldfilename = FILENAME + beginfile(FILENAME) + } + + END { endfile(FILENAME) } + + This file must be loaded before the user's "main" program, so that +the rule it supplies is executed first. + + This rule relies on 'awk''s 'FILENAME' variable, which automatically +changes for each new data file. The current file name is saved in a +private variable, '_oldfilename'. If 'FILENAME' does not equal +'_oldfilename', then a new data file is being processed and it is +necessary to call 'endfile()' for the old file. Because 'endfile()' +should only be called if a file has been processed, the program first +checks to make sure that '_oldfilename' is not the null string. The +program then assigns the current file name to '_oldfilename' and calls +'beginfile()' for the file. Because, like all 'awk' variables, +'_oldfilename' is initialized to the null string, this rule executes +correctly even for the first data file. + + The program also supplies an 'END' rule to do the final processing +for the last file. Because this 'END' rule comes before any 'END' rules +supplied in the "main" program, 'endfile()' is called first. Once +again, the value of multiple 'BEGIN' and 'END' rules should be clear. + + If the same data file occurs twice in a row on the command line, then +'endfile()' and 'beginfile()' are not executed at the end of the first +pass and at the beginning of the second pass. The following version +solves the problem: + + # ftrans.awk --- handle datafile transitions + # + # user supplies beginfile() and endfile() functions + + FNR == 1 { + if (_filename_ != "") + endfile(_filename_) + _filename_ = FILENAME + beginfile(FILENAME) + } + + END { endfile(_filename_) } + + *note Wc Program:: shows how this library function can be used and +how it simplifies writing the main program. + + So Why Does 'gawk' Have 'BEGINFILE' and 'ENDFILE'? + + You are probably wondering, if 'beginfile()' and 'endfile()' +functions can do the job, why does 'gawk' have 'BEGINFILE' and 'ENDFILE' +patterns? + + Good question. Normally, if 'awk' cannot open a file, this causes an +immediate fatal error. In this case, there is no way for a user-defined +function to deal with the problem, as the mechanism for calling it +relies on the file being open and at the first record. Thus, the main +reason for 'BEGINFILE' is to give you a "hook" to catch files that +cannot be processed. 'ENDFILE' exists for symmetry, and because it +provides an easy way to do per-file cleanup processing. For more +information, refer to *note BEGINFILE/ENDFILE::. + + +File: gawk.info, Node: Rewind Function, Next: File Checking, Prev: Filetrans Function, Up: Data File Management + +10.3.2 Rereading the Current File +--------------------------------- + +Another request for a new built-in function was for a function that +would make it possible to reread the current file. The requesting user +didn't want to have to use 'getline' (*note Getline::) inside a loop. + + However, as long as you are not in the 'END' rule, it is quite easy +to arrange to immediately close the current input file and then start +over with it from the top. For lack of a better name, we'll call the +function 'rewind()': + + # rewind.awk --- rewind the current file and start over + + function rewind( i) + { + # shift remaining arguments up + for (i = ARGC; i > ARGIND; i--) + ARGV[i] = ARGV[i-1] + + # make sure gawk knows to keep going + ARGC++ + + # make current file next to get done + ARGV[ARGIND+1] = FILENAME + + # do it + nextfile + } + + The 'rewind()' function relies on the 'ARGIND' variable (*note +Auto-set::), which is specific to 'gawk'. It also relies on the +'nextfile' keyword (*note Nextfile Statement::). Because of this, you +should not call it from an 'ENDFILE' rule. (This isn't necessary +anyway, because 'gawk' goes to the next file as soon as an 'ENDFILE' +rule finishes!) + + You need to be careful calling 'rewind()'. You can end up causing +infinite recursion if you don't pay attention. Here is an example use: + + $ cat data + -| a + -| b + -| c + -| d + -| e + + $ cat test.awk + -| FNR == 3 && ! rewound { + -| rewound = 1 + -| rewind() + -| } + -| + -| { print FILENAME, FNR, $0 } + + $ gawk -f rewind.awk -f test.awk data + -| data 1 a + -| data 2 b + -| data 1 a + -| data 2 b + -| data 3 c + -| data 4 d + -| data 5 e + + +File: gawk.info, Node: File Checking, Next: Empty Files, Prev: Rewind Function, Up: Data File Management + +10.3.3 Checking for Readable Data files +--------------------------------------- + +Normally, if you give 'awk' a data file that isn't readable, it stops +with a fatal error. There are times when you might want to just ignore +such files and keep going.(1) You can do this by prepending the +following program to your 'awk' program: + + # readable.awk --- library file to skip over unreadable files + + BEGIN { + for (i = 1; i < ARGC; i++) { + if (ARGV[i] ~ /^[a-zA-Z_][a-zA-Z0-9_]*=.*/ \ + || ARGV[i] == "-" || ARGV[i] == "/dev/stdin") + continue # assignment or standard input + else if ((getline junk < ARGV[i]) < 0) # unreadable + delete ARGV[i] + else + close(ARGV[i]) + } + } + + This works, because the 'getline' won't be fatal. Removing the +element from 'ARGV' with 'delete' skips the file (because it's no longer +in the list). See also *note ARGC and ARGV::. + + Because 'awk' variable names only allow the English letters, the +regular expression check purposely does not use character classes such +as '[:alpha:]' and '[:alnum:]' (*note Bracket Expressions::). + + ---------- Footnotes ---------- + + (1) The 'BEGINFILE' special pattern (*note BEGINFILE/ENDFILE::) +provides an alternative mechanism for dealing with files that can't be +opened. However, the code here provides a portable solution. + + +File: gawk.info, Node: Empty Files, Next: Ignoring Assigns, Prev: File Checking, Up: Data File Management + +10.3.4 Checking for Zero-Length Files +------------------------------------- + +All known 'awk' implementations silently skip over zero-length files. +This is a by-product of 'awk''s implicit +read-a-record-and-match-against-the-rules loop: when 'awk' tries to read +a record from an empty file, it immediately receives an end-of-file +indication, closes the file, and proceeds on to the next command-line +data file, _without_ executing any user-level 'awk' program code. + + Using 'gawk''s 'ARGIND' variable (*note Built-in Variables::), it is +possible to detect when an empty data file has been skipped. Similar to +the library file presented in *note Filetrans Function::, the following +library file calls a function named 'zerofile()' that the user must +provide. The arguments passed are the file name and the position in +'ARGV' where it was found: + + # zerofile.awk --- library file to process empty input files + + BEGIN { Argind = 0 } + + ARGIND > Argind + 1 { + for (Argind++; Argind < ARGIND; Argind++) + zerofile(ARGV[Argind], Argind) + } + + ARGIND != Argind { Argind = ARGIND } + + END { + if (ARGIND > Argind) + for (Argind++; Argind <= ARGIND; Argind++) + zerofile(ARGV[Argind], Argind) + } + + The user-level variable 'Argind' allows the 'awk' program to track +its progress through 'ARGV'. Whenever the program detects that 'ARGIND' +is greater than 'Argind + 1', it means that one or more empty files were +skipped. The action then calls 'zerofile()' for each such file, +incrementing 'Argind' along the way. + + The 'Argind != ARGIND' rule simply keeps 'Argind' up to date in the +normal case. + + Finally, the 'END' rule catches the case of any empty files at the +end of the command-line arguments. Note that the test in the condition +of the 'for' loop uses the '<=' operator, not '<'. + + +File: gawk.info, Node: Ignoring Assigns, Prev: Empty Files, Up: Data File Management + +10.3.5 Treating Assignments as File names +----------------------------------------- + +Occasionally, you might not want 'awk' to process command-line variable +assignments (*note Assignment Options::). In particular, if you have a +file name that contains an '=' character, 'awk' treats the file name as +an assignment and does not process it. + + Some users have suggested an additional command-line option for +'gawk' to disable command-line assignments. However, some simple +programming with a library file does the trick: + + # noassign.awk --- library file to avoid the need for a + # special option that disables command-line assignments + + function disable_assigns(argc, argv, i) + { + for (i = 1; i < argc; i++) + if (argv[i] ~ /^[a-zA-Z_][a-zA-Z0-9_]*=.*/) + argv[i] = ("./" argv[i]) + } + + BEGIN { + if (No_command_assign) + disable_assigns(ARGC, ARGV) + } + + You then run your program this way: + + awk -v No_command_assign=1 -f noassign.awk -f yourprog.awk * + + The function works by looping through the arguments. It prepends +'./' to any argument that matches the form of a variable assignment, +turning that argument into a file name. + + The use of 'No_command_assign' allows you to disable command-line +assignments at invocation time, by giving the variable a true value. +When not set, it is initially zero (i.e., false), so the command-line +arguments are left alone. + + +File: gawk.info, Node: Getopt Function, Next: Passwd Functions, Prev: Data File Management, Up: Library Functions + +10.4 Processing Command-Line Options +==================================== + +Most utilities on POSIX-compatible systems take options on the command +line that can be used to change the way a program behaves. 'awk' is an +example of such a program (*note Options::). Often, options take +"arguments" (i.e., data that the program needs to correctly obey the +command-line option). For example, 'awk''s '-F' option requires a +string to use as the field separator. The first occurrence on the +command line of either '--' or a string that does not begin with '-' +ends the options. + + Modern Unix systems provide a C function named 'getopt()' for +processing command-line arguments. The programmer provides a string +describing the one-letter options. If an option requires an argument, +it is followed in the string with a colon. 'getopt()' is also passed +the count and values of the command-line arguments and is called in a +loop. 'getopt()' processes the command-line arguments for option +letters. Each time around the loop, it returns a single character +representing the next option letter that it finds, or '?' if it finds an +invalid option. When it returns -1, there are no options left on the +command line. + + When using 'getopt()', options that do not take arguments can be +grouped together. Furthermore, options that take arguments require that +the argument be present. The argument can immediately follow the option +letter, or it can be a separate command-line argument. + + Given a hypothetical program that takes three command-line options, +'-a', '-b', and '-c', where '-b' requires an argument, all of the +following are valid ways of invoking the program: + + prog -a -b foo -c data1 data2 data3 + prog -ac -bfoo -- data1 data2 data3 + prog -acbfoo data1 data2 data3 + + Notice that when the argument is grouped with its option, the rest of +the argument is considered to be the option's argument. In this +example, '-acbfoo' indicates that all of the '-a', '-b', and '-c' +options were supplied, and that 'foo' is the argument to the '-b' +option. + + 'getopt()' provides four external variables that the programmer can +use: + +'optind' + The index in the argument value array ('argv') where the first + nonoption command-line argument can be found. + +'optarg' + The string value of the argument to an option. + +'opterr' + Usually 'getopt()' prints an error message when it finds an invalid + option. Setting 'opterr' to zero disables this feature. (An + application might want to print its own error message.) + +'optopt' + The letter representing the command-line option. + + The following C fragment shows how 'getopt()' might process +command-line arguments for 'awk': + + int + main(int argc, char *argv[]) + { + ... + /* print our own message */ + opterr = 0; + while ((c = getopt(argc, argv, "v:f:F:W:")) != -1) { + switch (c) { + case 'f': /* file */ + ... + break; + case 'F': /* field separator */ + ... + break; + case 'v': /* variable assignment */ + ... + break; + case 'W': /* extension */ + ... + break; + case '?': + default: + usage(); + break; + } + } + ... + } + + As a side point, 'gawk' actually uses the GNU 'getopt_long()' +function to process both normal and GNU-style long options (*note +Options::). + + The abstraction provided by 'getopt()' is very useful and is quite +handy in 'awk' programs as well. Following is an 'awk' version of +'getopt()'. This function highlights one of the greatest weaknesses in +'awk', which is that it is very poor at manipulating single characters. +Repeated calls to 'substr()' are necessary for accessing individual +characters (*note String Functions::).(1) + + The discussion that follows walks through the code a bit at a time: + + # getopt.awk --- Do C library getopt(3) function in awk + + # External variables: + # Optind -- index in ARGV of first nonoption argument + # Optarg -- string value of argument to current option + # Opterr -- if nonzero, print our own diagnostic + # Optopt -- current option letter + + # Returns: + # -1 at end of options + # "?" for unrecognized option + # <c> a character representing the current option + + # Private Data: + # _opti -- index in multiflag option, e.g., -abc + + The function starts out with comments presenting a list of the global +variables it uses, what the return values are, what they mean, and any +global variables that are "private" to this library function. Such +documentation is essential for any program, and particularly for library +functions. + + The 'getopt()' function first checks that it was indeed called with a +string of options (the 'options' parameter). If 'options' has a zero +length, 'getopt()' immediately returns -1: + + function getopt(argc, argv, options, thisopt, i) + { + if (length(options) == 0) # no options given + return -1 + + if (argv[Optind] == "--") { # all done + Optind++ + _opti = 0 + return -1 + } else if (argv[Optind] !~ /^-[^:[:space:]]/) { + _opti = 0 + return -1 + } + + The next thing to check for is the end of the options. A '--' ends +the command-line options, as does any command-line argument that does +not begin with a '-'. 'Optind' is used to step through the array of +command-line arguments; it retains its value across calls to 'getopt()', +because it is a global variable. + + The regular expression that is used, '/^-[^:[:space:]/', checks for a +'-' followed by anything that is not whitespace and not a colon. If the +current command-line argument does not match this pattern, it is not an +option, and it ends option processing. Continuing on: + + if (_opti == 0) + _opti = 2 + thisopt = substr(argv[Optind], _opti, 1) + Optopt = thisopt + i = index(options, thisopt) + if (i == 0) { + if (Opterr) + printf("%c -- invalid option\n", thisopt) > "/dev/stderr" + if (_opti >= length(argv[Optind])) { + Optind++ + _opti = 0 + } else + _opti++ + return "?" + } + + The '_opti' variable tracks the position in the current command-line +argument ('argv[Optind]'). If multiple options are grouped together +with one '-' (e.g., '-abx'), it is necessary to return them to the user +one at a time. + + If '_opti' is equal to zero, it is set to two, which is the index in +the string of the next character to look at (we skip the '-', which is +at position one). The variable 'thisopt' holds the character, obtained +with 'substr()'. It is saved in 'Optopt' for the main program to use. + + If 'thisopt' is not in the 'options' string, then it is an invalid +option. If 'Opterr' is nonzero, 'getopt()' prints an error message on +the standard error that is similar to the message from the C version of +'getopt()'. + + Because the option is invalid, it is necessary to skip it and move on +to the next option character. If '_opti' is greater than or equal to +the length of the current command-line argument, it is necessary to move +on to the next argument, so 'Optind' is incremented and '_opti' is reset +to zero. Otherwise, 'Optind' is left alone and '_opti' is merely +incremented. + + In any case, because the option is invalid, 'getopt()' returns '"?"'. +The main program can examine 'Optopt' if it needs to know what the +invalid option letter actually is. Continuing on: + + if (substr(options, i + 1, 1) == ":") { + # get option argument + if (length(substr(argv[Optind], _opti + 1)) > 0) + Optarg = substr(argv[Optind], _opti + 1) + else + Optarg = argv[++Optind] + _opti = 0 + } else + Optarg = "" + + If the option requires an argument, the option letter is followed by +a colon in the 'options' string. If there are remaining characters in +the current command-line argument ('argv[Optind]'), then the rest of +that string is assigned to 'Optarg'. Otherwise, the next command-line +argument is used ('-xFOO' versus '-x FOO'). In either case, '_opti' is +reset to zero, because there are no more characters left to examine in +the current command-line argument. Continuing: + + if (_opti == 0 || _opti >= length(argv[Optind])) { + Optind++ + _opti = 0 + } else + _opti++ + return thisopt + } + + Finally, if '_opti' is either zero or greater than the length of the +current command-line argument, it means this element in 'argv' is +through being processed, so 'Optind' is incremented to point to the next +element in 'argv'. If neither condition is true, then only '_opti' is +incremented, so that the next option letter can be processed on the next +call to 'getopt()'. + + The 'BEGIN' rule initializes both 'Opterr' and 'Optind' to one. +'Opterr' is set to one, because the default behavior is for 'getopt()' +to print a diagnostic message upon seeing an invalid option. 'Optind' +is set to one, because there's no reason to look at the program name, +which is in 'ARGV[0]': + + BEGIN { + Opterr = 1 # default is to diagnose + Optind = 1 # skip ARGV[0] + + # test program + if (_getopt_test) { + while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1) + printf("c = <%c>, Optarg = <%s>\n", + _go_c, Optarg) + printf("non-option arguments:\n") + for (; Optind < ARGC; Optind++) + printf("\tARGV[%d] = <%s>\n", + Optind, ARGV[Optind]) + } + } + + The rest of the 'BEGIN' rule is a simple test program. Here are the +results of two sample runs of the test program: + + $ awk -f getopt.awk -v _getopt_test=1 -- -a -cbARG bax -x + -| c = <a>, Optarg = <> + -| c = <c>, Optarg = <> + -| c = <b>, Optarg = <ARG> + -| non-option arguments: + -| ARGV[3] = <bax> + -| ARGV[4] = <-x> + + $ awk -f getopt.awk -v _getopt_test=1 -- -a -x -- xyz abc + -| c = <a>, Optarg = <> + error-> x -- invalid option + -| c = <?>, Optarg = <> + -| non-option arguments: + -| ARGV[4] = <xyz> + -| ARGV[5] = <abc> + + In both runs, the first '--' terminates the arguments to 'awk', so +that it does not try to interpret the '-a', etc., as its own options. + + NOTE: After 'getopt()' is through, user-level code must clear out + all the elements of 'ARGV' from 1 to 'Optind', so that 'awk' does + not try to process the command-line options as file names. + + Using '#!' with the '-E' option may help avoid conflicts between your +program's options and 'gawk''s options, as '-E' causes 'gawk' to abandon +processing of further options (*note Executable Scripts:: and *note +Options::). + + Several of the sample programs presented in *note Sample Programs::, +use 'getopt()' to process their arguments. + + ---------- Footnotes ---------- + + (1) This function was written before 'gawk' acquired the ability to +split strings into single characters using '""' as the separator. We +have left it alone, as using 'substr()' is more portable. + + +File: gawk.info, Node: Passwd Functions, Next: Group Functions, Prev: Getopt Function, Up: Library Functions + +10.5 Reading the User Database +============================== + +The 'PROCINFO' array (*note Built-in Variables::) provides access to the +current user's real and effective user and group ID numbers, and, if +available, the user's supplementary group set. However, because these +are numbers, they do not provide very useful information to the average +user. There needs to be some way to find the user information +associated with the user and group ID numbers. This minor node presents +a suite of functions for retrieving information from the user database. +*Note Group Functions:: for a similar suite that retrieves information +from the group database. + + The POSIX standard does not define the file where user information is +kept. Instead, it provides the '<pwd.h>' header file and several C +language subroutines for obtaining user information. The primary +function is 'getpwent()', for "get password entry." The "password" +comes from the original user database file, '/etc/passwd', which stores +user information along with the encrypted passwords (hence the name). + + Although an 'awk' program could simply read '/etc/passwd' directly, +this file may not contain complete information about the system's set of +users.(1) To be sure you are able to produce a readable and complete +version of the user database, it is necessary to write a small C program +that calls 'getpwent()'. 'getpwent()' is defined as returning a pointer +to a 'struct passwd'. Each time it is called, it returns the next entry +in the database. When there are no more entries, it returns 'NULL', the +null pointer. When this happens, the C program should call 'endpwent()' +to close the database. Following is 'pwcat', a C program that "cats" +the password database: + + /* + * pwcat.c + * + * Generate a printable version of the password database. + */ + #include <stdio.h> + #include <pwd.h> + + int + main(int argc, char **argv) + { + struct passwd *p; + + while ((p = getpwent()) != NULL) + printf("%s:%s:%ld:%ld:%s:%s:%s\n", + p->pw_name, p->pw_passwd, (long) p->pw_uid, + (long) p->pw_gid, p->pw_gecos, p->pw_dir, p->pw_shell); + + endpwent(); + return 0; + } + + If you don't understand C, don't worry about it. The output from +'pwcat' is the user database, in the traditional '/etc/passwd' format of +colon-separated fields. The fields are: + +Login name + The user's login name. + +Encrypted password + The user's encrypted password. This may not be available on some + systems. + +User-ID + The user's numeric user ID number. (On some systems, it's a C + 'long', and not an 'int'. Thus, we cast it to 'long' for all + cases.) + +Group-ID + The user's numeric group ID number. (Similar comments about 'long' + versus 'int' apply here.) + +Full name + The user's full name, and perhaps other information associated with + the user. + +Home directory + The user's login (or "home") directory (familiar to shell + programmers as '$HOME'). + +Login shell + The program that is run when the user logs in. This is usually a + shell, such as Bash. + + A few lines representative of 'pwcat''s output are as follows: + + $ pwcat + -| root:x:0:1:Operator:/:/bin/sh + -| nobody:*:65534:65534::/: + -| daemon:*:1:1::/: + -| sys:*:2:2::/:/bin/csh + -| bin:*:3:3::/bin: + -| arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/sh + -| miriam:yxaay:112:10:Miriam Robbins:/home/miriam:/bin/sh + -| andy:abcca2:113:10:Andy Jacobs:/home/andy:/bin/sh + ... + + With that introduction, following is a group of functions for getting +user information. There are several functions here, corresponding to +the C functions of the same names: + + # passwd.awk --- access password file information + + BEGIN { + # tailor this to suit your system + _pw_awklib = "/usr/local/libexec/awk/" + } + + function _pw_init( oldfs, oldrs, olddol0, pwcat, using_fw, using_fpat) + { + if (_pw_inited) + return + + oldfs = FS + oldrs = RS + olddol0 = $0 + using_fw = (PROCINFO["FS"] == "FIELDWIDTHS") + using_fpat = (PROCINFO["FS"] == "FPAT") + FS = ":" + RS = "\n" + + pwcat = _pw_awklib "pwcat" + while ((pwcat | getline) > 0) { + _pw_byname[$1] = $0 + _pw_byuid[$3] = $0 + _pw_bycount[++_pw_total] = $0 + } + close(pwcat) + _pw_count = 0 + _pw_inited = 1 + FS = oldfs + if (using_fw) + FIELDWIDTHS = FIELDWIDTHS + else if (using_fpat) + FPAT = FPAT + RS = oldrs + $0 = olddol0 + } + + The 'BEGIN' rule sets a private variable to the directory where +'pwcat' is stored. Because it is used to help out an 'awk' library +routine, we have chosen to put it in '/usr/local/libexec/awk'; however, +you might want it to be in a different directory on your system. + + The function '_pw_init()' fills three copies of the user information +into three associative arrays. The arrays are indexed by username +('_pw_byname'), by user ID number ('_pw_byuid'), and by order of +occurrence ('_pw_bycount'). The variable '_pw_inited' is used for +efficiency, as '_pw_init()' needs to be called only once. + + Because this function uses 'getline' to read information from +'pwcat', it first saves the values of 'FS', 'RS', and '$0'. It notes in +the variable 'using_fw' whether field splitting with 'FIELDWIDTHS' is in +effect or not. Doing so is necessary, as these functions could be +called from anywhere within a user's program, and the user may have his +or her own way of splitting records and fields. This makes it possible +to restore the correct field-splitting mechanism later. The test can +only be true for 'gawk'. It is false if using 'FS' or 'FPAT', or on +some other 'awk' implementation. + + The code that checks for using 'FPAT', using 'using_fpat' and +'PROCINFO["FS"]', is similar. + + The main part of the function uses a loop to read database lines, +split the lines into fields, and then store the lines into each array as +necessary. When the loop is done, '_pw_init()' cleans up by closing the +pipeline, setting '_pw_inited' to one, and restoring 'FS' (and +'FIELDWIDTHS' or 'FPAT' if necessary), 'RS', and '$0'. The use of +'_pw_count' is explained shortly. + + The 'getpwnam()' function takes a username as a string argument. If +that user is in the database, it returns the appropriate line. +Otherwise, it relies on the array reference to a nonexistent element to +create the element with the null string as its value: + + function getpwnam(name) + { + _pw_init() + return _pw_byname[name] + } + + Similarly, the 'getpwuid()' function takes a user ID number argument. +If that user number is in the database, it returns the appropriate line. +Otherwise, it returns the null string: + + function getpwuid(uid) + { + _pw_init() + return _pw_byuid[uid] + } + + The 'getpwent()' function simply steps through the database, one +entry at a time. It uses '_pw_count' to track its current position in +the '_pw_bycount' array: + + function getpwent() + { + _pw_init() + if (_pw_count < _pw_total) + return _pw_bycount[++_pw_count] + return "" + } + + The 'endpwent()' function resets '_pw_count' to zero, so that +subsequent calls to 'getpwent()' start over again: + + function endpwent() + { + _pw_count = 0 + } + + A conscious design decision in this suite is that each subroutine +calls '_pw_init()' to initialize the database arrays. The overhead of +running a separate process to generate the user database, and the I/O to +scan it, are only incurred if the user's main program actually calls one +of these functions. If this library file is loaded along with a user's +program, but none of the routines are ever called, then there is no +extra runtime overhead. (The alternative is move the body of +'_pw_init()' into a 'BEGIN' rule, which always runs 'pwcat'. This +simplifies the code but runs an extra process that may never be needed.) + + In turn, calling '_pw_init()' is not too expensive, because the +'_pw_inited' variable keeps the program from reading the data more than +once. If you are worried about squeezing every last cycle out of your +'awk' program, the check of '_pw_inited' could be moved out of +'_pw_init()' and duplicated in all the other functions. In practice, +this is not necessary, as most 'awk' programs are I/O-bound, and such a +change would clutter up the code. + + The 'id' program in *note Id Program:: uses these functions. + + ---------- Footnotes ---------- + + (1) It is often the case that password information is stored in a +network database. + + +File: gawk.info, Node: Group Functions, Next: Walking Arrays, Prev: Passwd Functions, Up: Library Functions + +10.6 Reading the Group Database +=============================== + +Much of the discussion presented in *note Passwd Functions:: applies to +the group database as well. Although there has traditionally been a +well-known file ('/etc/group') in a well-known format, the POSIX +standard only provides a set of C library routines ('<grp.h>' and +'getgrent()') for accessing the information. Even though this file may +exist, it may not have complete information. Therefore, as with the +user database, it is necessary to have a small C program that generates +the group database as its output. 'grcat', a C program that "cats" the +group database, is as follows: + + /* + * grcat.c + * + * Generate a printable version of the group database. + */ + #include <stdio.h> + #include <grp.h> + + int + main(int argc, char **argv) + { + struct group *g; + int i; + + while ((g = getgrent()) != NULL) { + printf("%s:%s:%ld:", g->gr_name, g->gr_passwd, + (long) g->gr_gid); + for (i = 0; g->gr_mem[i] != NULL; i++) { + printf("%s", g->gr_mem[i]); + if (g->gr_mem[i+1] != NULL) + putchar(','); + } + putchar('\n'); + } + endgrent(); + return 0; + } + + Each line in the group database represents one group. The fields are +separated with colons and represent the following information: + +Group Name + The group's name. + +Group Password + The group's encrypted password. In practice, this field is never + used; it is usually empty or set to '*'. + +Group ID Number + The group's numeric group ID number; the association of name to + number must be unique within the file. (On some systems it's a C + 'long', and not an 'int'. Thus, we cast it to 'long' for all + cases.) + +Group Member List + A comma-separated list of usernames. These users are members of + the group. Modern Unix systems allow users to be members of + several groups simultaneously. If your system does, then there are + elements '"group1"' through '"groupN"' in 'PROCINFO' for those + group ID numbers. (Note that 'PROCINFO' is a 'gawk' extension; + *note Built-in Variables::.) + + Here is what running 'grcat' might produce: + + $ grcat + -| wheel:*:0:arnold + -| nogroup:*:65534: + -| daemon:*:1: + -| kmem:*:2: + -| staff:*:10:arnold,miriam,andy + -| other:*:20: + ... + + Here are the functions for obtaining information from the group +database. There are several, modeled after the C library functions of +the same names: + + # group.awk --- functions for dealing with the group file + + BEGIN { + # Change to suit your system + _gr_awklib = "/usr/local/libexec/awk/" + } + + function _gr_init( oldfs, oldrs, olddol0, grcat, + using_fw, using_fpat, n, a, i) + { + if (_gr_inited) + return + + oldfs = FS + oldrs = RS + olddol0 = $0 + using_fw = (PROCINFO["FS"] == "FIELDWIDTHS") + using_fpat = (PROCINFO["FS"] == "FPAT") + FS = ":" + RS = "\n" + + grcat = _gr_awklib "grcat" + while ((grcat | getline) > 0) { + if ($1 in _gr_byname) + _gr_byname[$1] = _gr_byname[$1] "," $4 + else + _gr_byname[$1] = $0 + if ($3 in _gr_bygid) + _gr_bygid[$3] = _gr_bygid[$3] "," $4 + else + _gr_bygid[$3] = $0 + + n = split($4, a, "[ \t]*,[ \t]*") + for (i = 1; i <= n; i++) + if (a[i] in _gr_groupsbyuser) + _gr_groupsbyuser[a[i]] = _gr_groupsbyuser[a[i]] " " $1 + else + _gr_groupsbyuser[a[i]] = $1 + + _gr_bycount[++_gr_count] = $0 + } + close(grcat) + _gr_count = 0 + _gr_inited++ + FS = oldfs + if (using_fw) + FIELDWIDTHS = FIELDWIDTHS + else if (using_fpat) + FPAT = FPAT + RS = oldrs + $0 = olddol0 + } + + The 'BEGIN' rule sets a private variable to the directory where +'grcat' is stored. Because it is used to help out an 'awk' library +routine, we have chosen to put it in '/usr/local/libexec/awk'. You +might want it to be in a different directory on your system. + + These routines follow the same general outline as the user database +routines (*note Passwd Functions::). The '_gr_inited' variable is used +to ensure that the database is scanned no more than once. The +'_gr_init()' function first saves 'FS', 'RS', and '$0', and then sets +'FS' and 'RS' to the correct values for scanning the group information. +It also takes care to note whether 'FIELDWIDTHS' or 'FPAT' is being +used, and to restore the appropriate field-splitting mechanism. + + The group information is stored in several associative arrays. The +arrays are indexed by group name ('_gr_byname'), by group ID number +('_gr_bygid'), and by position in the database ('_gr_bycount'). There +is an additional array indexed by username ('_gr_groupsbyuser'), which +is a space-separated list of groups to which each user belongs. + + Unlike in the user database, it is possible to have multiple records +in the database for the same group. This is common when a group has a +large number of members. A pair of such entries might look like the +following: + + tvpeople:*:101:johnny,jay,arsenio + tvpeople:*:101:david,conan,tom,joan + + For this reason, '_gr_init()' looks to see if a group name or group +ID number is already seen. If so, the usernames are simply concatenated +onto the previous list of users.(1) + + Finally, '_gr_init()' closes the pipeline to 'grcat', restores 'FS' +(and 'FIELDWIDTHS' or 'FPAT', if necessary), 'RS', and '$0', initializes +'_gr_count' to zero (it is used later), and makes '_gr_inited' nonzero. + + The 'getgrnam()' function takes a group name as its argument, and if +that group exists, it is returned. Otherwise, it relies on the array +reference to a nonexistent element to create the element with the null +string as its value: + + function getgrnam(group) + { + _gr_init() + return _gr_byname[group] + } + + The 'getgrgid()' function is similar; it takes a numeric group ID and +looks up the information associated with that group ID: + + function getgrgid(gid) + { + _gr_init() + return _gr_bygid[gid] + } + + The 'getgruser()' function does not have a C counterpart. It takes a +username and returns the list of groups that have the user as a member: + + function getgruser(user) + { + _gr_init() + return _gr_groupsbyuser[user] + } + + The 'getgrent()' function steps through the database one entry at a +time. It uses '_gr_count' to track its position in the list: + + function getgrent() + { + _gr_init() + if (++_gr_count in _gr_bycount) + return _gr_bycount[_gr_count] + return "" + } + + The 'endgrent()' function resets '_gr_count' to zero so that +'getgrent()' can start over again: + + function endgrent() + { + _gr_count = 0 + } + + As with the user database routines, each function calls '_gr_init()' +to initialize the arrays. Doing so only incurs the extra overhead of +running 'grcat' if these functions are used (as opposed to moving the +body of '_gr_init()' into a 'BEGIN' rule). + + Most of the work is in scanning the database and building the various +associative arrays. The functions that the user calls are themselves +very simple, relying on 'awk''s associative arrays to do work. + + The 'id' program in *note Id Program:: uses these functions. + + ---------- Footnotes ---------- + + (1) There is a subtle problem with the code just presented. Suppose +that the first time there were no names. This code adds the names with +a leading comma. It also doesn't check that there is a '$4'. + + +File: gawk.info, Node: Walking Arrays, Next: Library Functions Summary, Prev: Group Functions, Up: Library Functions + +10.7 Traversing Arrays of Arrays +================================ + +*note Arrays of Arrays:: described how 'gawk' provides arrays of arrays. +In particular, any element of an array may be either a scalar or another +array. The 'isarray()' function (*note Type Functions::) lets you +distinguish an array from a scalar. The following function, +'walk_array()', recursively traverses an array, printing the element +indices and values. You call it with the array and a string +representing the name of the array: + + function walk_array(arr, name, i) + { + for (i in arr) { + if (isarray(arr[i])) + walk_array(arr[i], (name "[" i "]")) + else + printf("%s[%s] = %s\n", name, i, arr[i]) + } + } + +It works by looping over each element of the array. If any given +element is itself an array, the function calls itself recursively, +passing the subarray and a new string representing the current index. +Otherwise, the function simply prints the element's name, index, and +value. Here is a main program to demonstrate: + + BEGIN { + a[1] = 1 + a[2][1] = 21 + a[2][2] = 22 + a[3] = 3 + a[4][1][1] = 411 + a[4][2] = 42 + + walk_array(a, "a") + } + + When run, the program produces the following output: + + $ gawk -f walk_array.awk + -| a[1] = 1 + -| a[2][1] = 21 + -| a[2][2] = 22 + -| a[3] = 3 + -| a[4][1][1] = 411 + -| a[4][2] = 42 + + The function just presented simply prints the name and value of each +scalar array element. However, it is easy to generalize it, by passing +in the name of a function to call when walking an array. The modified +function looks like this: + + function process_array(arr, name, process, do_arrays, i, new_name) + { + for (i in arr) { + new_name = (name "[" i "]") + if (isarray(arr[i])) { + if (do_arrays) + @process(new_name, arr[i]) + process_array(arr[i], new_name, process, do_arrays) + } else + @process(new_name, arr[i]) + } + } + + The arguments are as follows: + +'arr' + The array. + +'name' + The name of the array (a string). + +'process' + The name of the function to call. + +'do_arrays' + If this is true, the function can handle elements that are + subarrays. + + If subarrays are to be processed, that is done before walking them +further. + + When run with the following scaffolding, the function produces the +same results as does the earlier version of 'walk_array()': + + BEGIN { + a[1] = 1 + a[2][1] = 21 + a[2][2] = 22 + a[3] = 3 + a[4][1][1] = 411 + a[4][2] = 42 + + process_array(a, "a", "do_print", 0) + } + + function do_print(name, element) + { + printf "%s = %s\n", name, element + } + + +File: gawk.info, Node: Library Functions Summary, Next: Library Exercises, Prev: Walking Arrays, Up: Library Functions + +10.8 Summary +============ + + * Reading programs is an excellent way to learn Good Programming. + The functions and programs provided in this major node and the next + are intended to serve that purpose. + + * When writing general-purpose library functions, put some thought + into how to name any global variables so that they won't conflict + with variables from a user's program. + + * The functions presented here fit into the following categories: + + General problems + Number-to-string conversion, testing assertions, rounding, + random number generation, converting characters to numbers, + joining strings, getting easily usable time-of-day + information, and reading a whole file in one shot + + Managing data files + Noting data file boundaries, rereading the current file, + checking for readable files, checking for zero-length files, + and treating assignments as file names + + Processing command-line options + An 'awk' version of the standard C 'getopt()' function + + Reading the user and group databases + Two sets of routines that parallel the C library versions + + Traversing arrays of arrays + Two functions that traverse an array of arrays to any depth + + +File: gawk.info, Node: Library Exercises, Prev: Library Functions Summary, Up: Library Functions + +10.9 Exercises +============== + + 1. In *note Empty Files::, we presented the 'zerofile.awk' program, + which made use of 'gawk''s 'ARGIND' variable. Can this problem be + solved without relying on 'ARGIND'? If so, how? + + 2. As a related challenge, revise that code to handle the case where + an intervening value in 'ARGV' is a variable assignment. + + +File: gawk.info, Node: Sample Programs, Next: Advanced Features, Prev: Library Functions, Up: Top + +11 Practical 'awk' Programs +*************************** + +*note Library Functions::, presents the idea that reading programs in a +language contributes to learning that language. This major node +continues that theme, presenting a potpourri of 'awk' programs for your +reading enjoyment. + + Many of these programs use library functions presented in *note +Library Functions::. + +* Menu: + +* Running Examples:: How to run these examples. +* Clones:: Clones of common utilities. +* Miscellaneous Programs:: Some interesting 'awk' programs. +* Programs Summary:: Summary of programs. +* Programs Exercises:: Exercises. + + +File: gawk.info, Node: Running Examples, Next: Clones, Up: Sample Programs + +11.1 Running the Example Programs +================================= + +To run a given program, you would typically do something like this: + + awk -f PROGRAM -- OPTIONS FILES + +Here, PROGRAM is the name of the 'awk' program (such as 'cut.awk'), +OPTIONS are any command-line options for the program that start with a +'-', and FILES are the actual data files. + + If your system supports the '#!' executable interpreter mechanism +(*note Executable Scripts::), you can instead run your program directly: + + cut.awk -c1-8 myfiles > results + + If your 'awk' is not 'gawk', you may instead need to use this: + + cut.awk -- -c1-8 myfiles > results + + +File: gawk.info, Node: Clones, Next: Miscellaneous Programs, Prev: Running Examples, Up: Sample Programs + +11.2 Reinventing Wheels for Fun and Profit +========================================== + +This minor node presents a number of POSIX utilities implemented in +'awk'. Reinventing these programs in 'awk' is often enjoyable, because +the algorithms can be very clearly expressed, and the code is usually +very concise and simple. This is true because 'awk' does so much for +you. + + It should be noted that these programs are not necessarily intended +to replace the installed versions on your system. Nor may all of these +programs be fully compliant with the most recent POSIX standard. This +is not a problem; their purpose is to illustrate 'awk' language +programming for "real-world" tasks. + + The programs are presented in alphabetical order. + +* Menu: + +* Cut Program:: The 'cut' utility. +* Egrep Program:: The 'egrep' utility. +* Id Program:: The 'id' utility. +* Split Program:: The 'split' utility. +* Tee Program:: The 'tee' utility. +* Uniq Program:: The 'uniq' utility. +* Wc Program:: The 'wc' utility. + + +File: gawk.info, Node: Cut Program, Next: Egrep Program, Up: Clones + +11.2.1 Cutting Out Fields and Columns +------------------------------------- + +The 'cut' utility selects, or "cuts," characters or fields from its +standard input and sends them to its standard output. Fields are +separated by TABs by default, but you may supply a command-line option +to change the field "delimiter" (i.e., the field-separator character). +'cut''s definition of fields is less general than 'awk''s. + + A common use of 'cut' might be to pull out just the login names of +logged-on users from the output of 'who'. For example, the following +pipeline generates a sorted, unique list of the logged-on users: + + who | cut -c1-8 | sort | uniq + + The options for 'cut' are: + +'-c LIST' + Use LIST as the list of characters to cut out. Items within the + list may be separated by commas, and ranges of characters can be + separated with dashes. The list '1-8,15,22-35' specifies + characters 1 through 8, 15, and 22 through 35. + +'-f LIST' + Use LIST as the list of fields to cut out. + +'-d DELIM' + Use DELIM as the field-separator character instead of the TAB + character. + +'-s' + Suppress printing of lines that do not contain the field delimiter. + + The 'awk' implementation of 'cut' uses the 'getopt()' library +function (*note Getopt Function::) and the 'join()' library function +(*note Join Function::). + + The program begins with a comment describing the options, the library +functions needed, and a 'usage()' function that prints out a usage +message and exits. 'usage()' is called if invalid arguments are +supplied: + + # cut.awk --- implement cut in awk + + # Options: + # -f list Cut fields + # -d c Field delimiter character + # -c list Cut characters + # + # -s Suppress lines without the delimiter + # + # Requires getopt() and join() library functions + + function usage() + { + print("usage: cut [-f list] [-d c] [-s] [files...]") > "/dev/stderr" + print("usage: cut [-c list] [files...]") > "/dev/stderr" + exit 1 + } + + Next comes a 'BEGIN' rule that parses the command-line options. It +sets 'FS' to a single TAB character, because that is 'cut''s default +field separator. The rule then sets the output field separator to be +the same as the input field separator. A loop using 'getopt()' steps +through the command-line options. Exactly one of the variables +'by_fields' or 'by_chars' is set to true, to indicate that processing +should be done by fields or by characters, respectively. When cutting +by characters, the output field separator is set to the null string: + + BEGIN { + FS = "\t" # default + OFS = FS + while ((c = getopt(ARGC, ARGV, "sf:c:d:")) != -1) { + if (c == "f") { + by_fields = 1 + fieldlist = Optarg + } else if (c == "c") { + by_chars = 1 + fieldlist = Optarg + OFS = "" + } else if (c == "d") { + if (length(Optarg) > 1) { + printf("cut: using first character of %s" \ + " for delimiter\n", Optarg) > "/dev/stderr" + Optarg = substr(Optarg, 1, 1) + } + fs = FS = Optarg + OFS = FS + if (FS == " ") # defeat awk semantics + FS = "[ ]" + } else if (c == "s") + suppress = 1 + else + usage() + } + + # Clear out options + for (i = 1; i < Optind; i++) + ARGV[i] = "" + + The code must take special care when the field delimiter is a space. +Using a single space ('" "') for the value of 'FS' is incorrect--'awk' +would separate fields with runs of spaces, TABs, and/or newlines, and we +want them to be separated with individual spaces. To this end, we save +the original space character in the variable 'fs' for later use; after +setting 'FS' to '"[ ]"' we can't use it directly to see if the field +delimiter character is in the string. + + Also remember that after 'getopt()' is through (as described in *note +Getopt Function::), we have to clear out all the elements of 'ARGV' from +1 to 'Optind', so that 'awk' does not try to process the command-line +options as file names. + + After dealing with the command-line options, the program verifies +that the options make sense. Only one or the other of '-c' and '-f' +should be used, and both require a field list. Then the program calls +either 'set_fieldlist()' or 'set_charlist()' to pull apart the list of +fields or characters: + + if (by_fields && by_chars) + usage() + + if (by_fields == 0 && by_chars == 0) + by_fields = 1 # default + + if (fieldlist == "") { + print "cut: needs list for -c or -f" > "/dev/stderr" + exit 1 + } + + if (by_fields) + set_fieldlist() + else + set_charlist() + } + + 'set_fieldlist()' splits the field list apart at the commas into an +array. Then, for each element of the array, it looks to see if the +element is actually a range, and if so, splits it apart. The function +checks the range to make sure that the first number is smaller than the +second. Each number in the list is added to the 'flist' array, which +simply lists the fields that will be printed. Normal field splitting is +used. The program lets 'awk' handle the job of doing the field +splitting: + + function set_fieldlist( n, m, i, j, k, f, g) + { + n = split(fieldlist, f, ",") + j = 1 # index in flist + for (i = 1; i <= n; i++) { + if (index(f[i], "-") != 0) { # a range + m = split(f[i], g, "-") + if (m != 2 || g[1] >= g[2]) { + printf("cut: bad field list: %s\n", + f[i]) > "/dev/stderr" + exit 1 + } + for (k = g[1]; k <= g[2]; k++) + flist[j++] = k + } else + flist[j++] = f[i] + } + nfields = j - 1 + } + + The 'set_charlist()' function is more complicated than +'set_fieldlist()'. The idea here is to use 'gawk''s 'FIELDWIDTHS' +variable (*note Constant Size::), which describes constant-width input. +When using a character list, that is exactly what we have. + + Setting up 'FIELDWIDTHS' is more complicated than simply listing the +fields that need to be printed. We have to keep track of the fields to +print and also the intervening characters that have to be skipped. For +example, suppose you wanted characters 1 through 8, 15, and 22 through +35. You would use '-c 1-8,15,22-35'. The necessary value for +'FIELDWIDTHS' is '"8 6 1 6 14"'. This yields five fields, and the +fields to print are '$1', '$3', and '$5'. The intermediate fields are +"filler", which is stuff in between the desired data. 'flist' lists the +fields to print, and 't' tracks the complete field list, including +filler fields: + + function set_charlist( field, i, j, f, g, n, m, t, + filler, last, len) + { + field = 1 # count total fields + n = split(fieldlist, f, ",") + j = 1 # index in flist + for (i = 1; i <= n; i++) { + if (index(f[i], "-") != 0) { # range + m = split(f[i], g, "-") + if (m != 2 || g[1] >= g[2]) { + printf("cut: bad character list: %s\n", + f[i]) > "/dev/stderr" + exit 1 + } + len = g[2] - g[1] + 1 + if (g[1] > 1) # compute length of filler + filler = g[1] - last - 1 + else + filler = 0 + if (filler) + t[field++] = filler + t[field++] = len # length of field + last = g[2] + flist[j++] = field - 1 + } else { + if (f[i] > 1) + filler = f[i] - last - 1 + else + filler = 0 + if (filler) + t[field++] = filler + t[field++] = 1 + last = f[i] + flist[j++] = field - 1 + } + } + FIELDWIDTHS = join(t, 1, field - 1) + nfields = j - 1 + } + + Next is the rule that processes the data. If the '-s' option is +given, then 'suppress' is true. The first 'if' statement makes sure +that the input record does have the field separator. If 'cut' is +processing fields, 'suppress' is true, and the field separator character +is not in the record, then the record is skipped. + + If the record is valid, then 'gawk' has split the data into fields, +either using the character in 'FS' or using fixed-length fields and +'FIELDWIDTHS'. The loop goes through the list of fields that should be +printed. The corresponding field is printed if it contains data. If +the next field also has data, then the separator character is written +out between the fields: + + { + if (by_fields && suppress && index($0, fs) == 0) + next + + for (i = 1; i <= nfields; i++) { + if ($flist[i] != "") { + printf "%s", $flist[i] + if (i < nfields && $flist[i+1] != "") + printf "%s", OFS + } + } + print "" + } + + This version of 'cut' relies on 'gawk''s 'FIELDWIDTHS' variable to do +the character-based cutting. It is possible in other 'awk' +implementations to use 'substr()' (*note String Functions::), but it is +also extremely painful. The 'FIELDWIDTHS' variable supplies an elegant +solution to the problem of picking the input line apart by characters. + + +File: gawk.info, Node: Egrep Program, Next: Id Program, Prev: Cut Program, Up: Clones + +11.2.2 Searching for Regular Expressions in Files +------------------------------------------------- + +The 'egrep' utility searches files for patterns. It uses regular +expressions that are almost identical to those available in 'awk' (*note +Regexp::). You invoke it as follows: + + 'egrep' [OPTIONS] ''PATTERN'' FILES ... + + The PATTERN is a regular expression. In typical usage, the regular +expression is quoted to prevent the shell from expanding any of the +special characters as file name wildcards. Normally, 'egrep' prints the +lines that matched. If multiple file names are provided on the command +line, each output line is preceded by the name of the file and a colon. + + The options to 'egrep' are as follows: + +'-c' + Print out a count of the lines that matched the pattern, instead of + the lines themselves. + +'-s' + Be silent. No output is produced and the exit value indicates + whether the pattern was matched. + +'-v' + Invert the sense of the test. 'egrep' prints the lines that do + _not_ match the pattern and exits successfully if the pattern is + not matched. + +'-i' + Ignore case distinctions in both the pattern and the input data. + +'-l' + Only print (list) the names of the files that matched, not the + lines that matched. + +'-e PATTERN' + Use PATTERN as the regexp to match. The purpose of the '-e' option + is to allow patterns that start with a '-'. + + This version uses the 'getopt()' library function (*note Getopt +Function::) and the file transition library program (*note Filetrans +Function::). + + The program begins with a descriptive comment and then a 'BEGIN' rule +that processes the command-line arguments with 'getopt()'. The '-i' +(ignore case) option is particularly easy with 'gawk'; we just use the +'IGNORECASE' predefined variable (*note Built-in Variables::): + + # egrep.awk --- simulate egrep in awk + # + # Options: + # -c count of lines + # -s silent - use exit value + # -v invert test, success if no match + # -i ignore case + # -l print filenames only + # -e argument is pattern + # + # Requires getopt and file transition library functions + + BEGIN { + while ((c = getopt(ARGC, ARGV, "ce:svil")) != -1) { + if (c == "c") + count_only++ + else if (c == "s") + no_print++ + else if (c == "v") + invert++ + else if (c == "i") + IGNORECASE = 1 + else if (c == "l") + filenames_only++ + else if (c == "e") + pattern = Optarg + else + usage() + } + + Next comes the code that handles the 'egrep'-specific behavior. If +no pattern is supplied with '-e', the first nonoption on the command +line is used. The 'awk' command-line arguments up to 'ARGV[Optind]' are +cleared, so that 'awk' won't try to process them as files. If no files +are specified, the standard input is used, and if multiple files are +specified, we make sure to note this so that the file names can precede +the matched lines in the output: + + if (pattern == "") + pattern = ARGV[Optind++] + + for (i = 1; i < Optind; i++) + ARGV[i] = "" + if (Optind >= ARGC) { + ARGV[1] = "-" + ARGC = 2 + } else if (ARGC - Optind > 1) + do_filenames++ + + # if (IGNORECASE) + # pattern = tolower(pattern) + } + + The last two lines are commented out, as they are not needed in +'gawk'. They should be uncommented if you have to use another version +of 'awk'. + + The next set of lines should be uncommented if you are not using +'gawk'. This rule translates all the characters in the input line into +lowercase if the '-i' option is specified.(1) The rule is commented out +as it is not necessary with 'gawk': + + #{ + # if (IGNORECASE) + # $0 = tolower($0) + #} + + The 'beginfile()' function is called by the rule in 'ftrans.awk' when +each new file is processed. In this case, it is very simple; all it +does is initialize a variable 'fcount' to zero. 'fcount' tracks how +many lines in the current file matched the pattern. Naming the +parameter 'junk' shows we know that 'beginfile()' is called with a +parameter, but that we're not interested in its value: + + function beginfile(junk) + { + fcount = 0 + } + + The 'endfile()' function is called after each file has been +processed. It affects the output only when the user wants a count of +the number of lines that matched. 'no_print' is true only if the exit +status is desired. 'count_only' is true if line counts are desired. +'egrep' therefore only prints line counts if printing and counting are +enabled. The output format must be adjusted depending upon the number +of files to process. Finally, 'fcount' is added to 'total', so that we +know the total number of lines that matched the pattern: + + function endfile(file) + { + if (! no_print && count_only) { + if (do_filenames) + print file ":" fcount + else + print fcount + } + + total += fcount + } + + The 'BEGINFILE' and 'ENDFILE' special patterns (*note +BEGINFILE/ENDFILE::) could be used, but then the program would be +'gawk'-specific. Additionally, this example was written before 'gawk' +acquired 'BEGINFILE' and 'ENDFILE'. + + The following rule does most of the work of matching lines. The +variable 'matches' is true if the line matched the pattern. If the user +wants lines that did not match, the sense of 'matches' is inverted using +the '!' operator. 'fcount' is incremented with the value of 'matches', +which is either one or zero, depending upon a successful or unsuccessful +match. If the line does not match, the 'next' statement just moves on +to the next record. + + A number of additional tests are made, but they are only done if we +are not counting lines. First, if the user only wants the exit status +('no_print' is true), then it is enough to know that _one_ line in this +file matched, and we can skip on to the next file with 'nextfile'. +Similarly, if we are only printing file names, we can print the file +name, and then skip to the next file with 'nextfile'. Finally, each +line is printed, with a leading file name and colon if necessary: + + { + matches = ($0 ~ pattern) + if (invert) + matches = ! matches + + fcount += matches # 1 or 0 + + if (! matches) + next + + if (! count_only) { + if (no_print) + nextfile + + if (filenames_only) { + print FILENAME + nextfile + } + + if (do_filenames) + print FILENAME ":" $0 + else + print + } + } + + The 'END' rule takes care of producing the correct exit status. If +there are no matches, the exit status is one; otherwise, it is zero: + + END { + exit (total == 0) + } + + The 'usage()' function prints a usage message in case of invalid +options, and then exits: + + function usage() + { + print("Usage: egrep [-csvil] [-e pat] [files ...]") > "/dev/stderr" + print("\n\tegrep [-csvil] pat [files ...]") > "/dev/stderr" + exit 1 + } + + ---------- Footnotes ---------- + + (1) It also introduces a subtle bug; if a match happens, we output +the translated line, not the original. + + +File: gawk.info, Node: Id Program, Next: Split Program, Prev: Egrep Program, Up: Clones + +11.2.3 Printing Out User Information +------------------------------------ + +The 'id' utility lists a user's real and effective user ID numbers, real +and effective group ID numbers, and the user's group set, if any. 'id' +only prints the effective user ID and group ID if they are different +from the real ones. If possible, 'id' also supplies the corresponding +user and group names. The output might look like this: + + $ id + -| uid=1000(arnold) gid=1000(arnold) groups=1000(arnold),4(adm),7(lp),27(sudo) + + This information is part of what is provided by 'gawk''s 'PROCINFO' +array (*note Built-in Variables::). However, the 'id' utility provides +a more palatable output than just individual numbers. + + Here is a simple version of 'id' written in 'awk'. It uses the user +database library functions (*note Passwd Functions::) and the group +database library functions (*note Group Functions::) from *note Library +Functions::. + + The program is fairly straightforward. All the work is done in the +'BEGIN' rule. The user and group ID numbers are obtained from +'PROCINFO'. The code is repetitive. The entry in the user database for +the real user ID number is split into parts at the ':'. The name is the +first field. Similar code is used for the effective user ID number and +the group numbers: + + # id.awk --- implement id in awk + # + # Requires user and group library functions + # output is: + # uid=12(foo) euid=34(bar) gid=3(baz) \ + # egid=5(blat) groups=9(nine),2(two),1(one) + + BEGIN { + uid = PROCINFO["uid"] + euid = PROCINFO["euid"] + gid = PROCINFO["gid"] + egid = PROCINFO["egid"] + + printf("uid=%d", uid) + pw = getpwuid(uid) + pr_first_field(pw) + + if (euid != uid) { + printf(" euid=%d", euid) + pw = getpwuid(euid) + pr_first_field(pw) + } + + printf(" gid=%d", gid) + pw = getgrgid(gid) + pr_first_field(pw) + + if (egid != gid) { + printf(" egid=%d", egid) + pw = getgrgid(egid) + pr_first_field(pw) + } + + for (i = 1; ("group" i) in PROCINFO; i++) { + if (i == 1) + printf(" groups=") + group = PROCINFO["group" i] + printf("%d", group) + pw = getgrgid(group) + pr_first_field(pw) + if (("group" (i+1)) in PROCINFO) + printf(",") + } + + print "" + } + + function pr_first_field(str, a) + { + if (str != "") { + split(str, a, ":") + printf("(%s)", a[1]) + } + } + + The test in the 'for' loop is worth noting. Any supplementary groups +in the 'PROCINFO' array have the indices '"group1"' through '"groupN"' +for some N (i.e., the total number of supplementary groups). However, +we don't know in advance how many of these groups there are. + + This loop works by starting at one, concatenating the value with +'"group"', and then using 'in' to see if that value is in the array +(*note Reference to Elements::). Eventually, 'i' is incremented past +the last group in the array and the loop exits. + + The loop is also correct if there are _no_ supplementary groups; then +the condition is false the first time it's tested, and the loop body +never executes. + + The 'pr_first_field()' function simply isolates out some code that is +used repeatedly, making the whole program shorter and cleaner. In +particular, moving the check for the empty string into this function +saves several lines of code. + + +File: gawk.info, Node: Split Program, Next: Tee Program, Prev: Id Program, Up: Clones + +11.2.4 Splitting a Large File into Pieces +----------------------------------------- + +The 'split' program splits large text files into smaller pieces. Usage +is as follows:(1) + + 'split' ['-COUNT'] [FILE] [PREFIX] + + By default, the output files are named 'xaa', 'xab', and so on. Each +file has 1,000 lines in it, with the likely exception of the last file. +To change the number of lines in each file, supply a number on the +command line preceded with a minus sign (e.g., '-500' for files with 500 +lines in them instead of 1,000). To change the names of the output +files to something like 'myfileaa', 'myfileab', and so on, supply an +additional argument that specifies the file name prefix. + + Here is a version of 'split' in 'awk'. It uses the 'ord()' and +'chr()' functions presented in *note Ordinal Functions::. + + The program first sets its defaults, and then tests to make sure +there are not too many arguments. It then looks at each argument in +turn. The first argument could be a minus sign followed by a number. +If it is, this happens to look like a negative number, so it is made +positive, and that is the count of lines. The data file name is skipped +over and the final argument is used as the prefix for the output file +names: + + # split.awk --- do split in awk + # + # Requires ord() and chr() library functions + # usage: split [-count] [file] [outname] + + BEGIN { + outfile = "x" # default + count = 1000 + if (ARGC > 4) + usage() + + i = 1 + if (i in ARGV && ARGV[i] ~ /^-[[:digit:]]+$/) { + count = -ARGV[i] + ARGV[i] = "" + i++ + } + # test argv in case reading from stdin instead of file + if (i in ARGV) + i++ # skip datafile name + if (i in ARGV) { + outfile = ARGV[i] + ARGV[i] = "" + } + + s1 = s2 = "a" + out = (outfile s1 s2) + } + + The next rule does most of the work. 'tcount' (temporary count) +tracks how many lines have been printed to the output file so far. If +it is greater than 'count', it is time to close the current file and +start a new one. 's1' and 's2' track the current suffixes for the file +name. If they are both 'z', the file is just too big. Otherwise, 's1' +moves to the next letter in the alphabet and 's2' starts over again at +'a': + + { + if (++tcount > count) { + close(out) + if (s2 == "z") { + if (s1 == "z") { + printf("split: %s is too large to split\n", + FILENAME) > "/dev/stderr" + exit 1 + } + s1 = chr(ord(s1) + 1) + s2 = "a" + } + else + s2 = chr(ord(s2) + 1) + out = (outfile s1 s2) + tcount = 1 + } + print > out + } + +The 'usage()' function simply prints an error message and exits: + + function usage() + { + print("usage: split [-num] [file] [outname]") > "/dev/stderr" + exit 1 + } + + This program is a bit sloppy; it relies on 'awk' to automatically +close the last file instead of doing it in an 'END' rule. It also +assumes that letters are contiguous in the character set, which isn't +true for EBCDIC systems. + + ---------- Footnotes ---------- + + (1) This is the traditional usage. The POSIX usage is different, but +not relevant for what the program aims to demonstrate. + + +File: gawk.info, Node: Tee Program, Next: Uniq Program, Prev: Split Program, Up: Clones + +11.2.5 Duplicating Output into Multiple Files +--------------------------------------------- + +The 'tee' program is known as a "pipe fitting." 'tee' copies its +standard input to its standard output and also duplicates it to the +files named on the command line. Its usage is as follows: + + 'tee' ['-a'] FILE ... + + The '-a' option tells 'tee' to append to the named files, instead of +truncating them and starting over. + + The 'BEGIN' rule first makes a copy of all the command-line arguments +into an array named 'copy'. 'ARGV[0]' is not needed, so it is not +copied. 'tee' cannot use 'ARGV' directly, because 'awk' attempts to +process each file name in 'ARGV' as input data. + + If the first argument is '-a', then the flag variable 'append' is set +to true, and both 'ARGV[1]' and 'copy[1]' are deleted. If 'ARGC' is +less than two, then no file names were supplied and 'tee' prints a usage +message and exits. Finally, 'awk' is forced to read the standard input +by setting 'ARGV[1]' to '"-"' and 'ARGC' to two: + + # tee.awk --- tee in awk + # + # Copy standard input to all named output files. + # Append content if -a option is supplied. + # + BEGIN { + for (i = 1; i < ARGC; i++) + copy[i] = ARGV[i] + + if (ARGV[1] == "-a") { + append = 1 + delete ARGV[1] + delete copy[1] + ARGC-- + } + if (ARGC < 2) { + print "usage: tee [-a] file ..." > "/dev/stderr" + exit 1 + } + ARGV[1] = "-" + ARGC = 2 + } + + The following single rule does all the work. Because there is no +pattern, it is executed for each line of input. The body of the rule +simply prints the line into each file on the command line, and then to +the standard output: + + { + # moving the if outside the loop makes it run faster + if (append) + for (i in copy) + print >> copy[i] + else + for (i in copy) + print > copy[i] + print + } + +It is also possible to write the loop this way: + + for (i in copy) + if (append) + print >> copy[i] + else + print > copy[i] + +This is more concise, but it is also less efficient. The 'if' is tested +for each record and for each output file. By duplicating the loop body, +the 'if' is only tested once for each input record. If there are N +input records and M output files, the first method only executes N 'if' +statements, while the second executes N'*'M 'if' statements. + + Finally, the 'END' rule cleans up by closing all the output files: + + END { + for (i in copy) + close(copy[i]) + } + + +File: gawk.info, Node: Uniq Program, Next: Wc Program, Prev: Tee Program, Up: Clones + +11.2.6 Printing Nonduplicated Lines of Text +------------------------------------------- + +The 'uniq' utility reads sorted lines of data on its standard input, and +by default removes duplicate lines. In other words, it only prints +unique lines--hence the name. 'uniq' has a number of options. The +usage is as follows: + + 'uniq' ['-udc' ['-N']] ['+N'] [INPUTFILE [OUTPUTFILE]] + + The options for 'uniq' are: + +'-d' + Print only repeated (duplicated) lines. + +'-u' + Print only nonrepeated (unique) lines. + +'-c' + Count lines. This option overrides '-d' and '-u'. Both repeated + and nonrepeated lines are counted. + +'-N' + Skip N fields before comparing lines. The definition of fields is + similar to 'awk''s default: nonwhitespace characters separated by + runs of spaces and/or TABs. + +'+N' + Skip N characters before comparing lines. Any fields specified + with '-N' are skipped first. + +'INPUTFILE' + Data is read from the input file named on the command line, instead + of from the standard input. + +'OUTPUTFILE' + The generated output is sent to the named output file, instead of + to the standard output. + + Normally 'uniq' behaves as if both the '-d' and '-u' options are +provided. + + 'uniq' uses the 'getopt()' library function (*note Getopt Function::) +and the 'join()' library function (*note Join Function::). + + The program begins with a 'usage()' function and then a brief outline +of the options and their meanings in comments. The 'BEGIN' rule deals +with the command-line arguments and options. It uses a trick to get +'getopt()' to handle options of the form '-25', treating such an option +as the option letter '2' with an argument of '5'. If indeed two or more +digits are supplied ('Optarg' looks like a number), 'Optarg' is +concatenated with the option digit and then the result is added to zero +to make it into a number. If there is only one digit in the option, +then 'Optarg' is not needed. In this case, 'Optind' must be decremented +so that 'getopt()' processes it next time. This code is admittedly a +bit tricky. + + If no options are supplied, then the default is taken, to print both +repeated and nonrepeated lines. The output file, if provided, is +assigned to 'outputfile'. Early on, 'outputfile' is initialized to the +standard output, '/dev/stdout': + + # uniq.awk --- do uniq in awk + # + # Requires getopt() and join() library functions + + function usage() + { + print("Usage: uniq [-udc [-n]] [+n] [ in [ out ]]") > "/dev/stderr" + exit 1 + } + + # -c count lines. overrides -d and -u + # -d only repeated lines + # -u only nonrepeated lines + # -n skip n fields + # +n skip n characters, skip fields first + + BEGIN { + count = 1 + outputfile = "/dev/stdout" + opts = "udc0:1:2:3:4:5:6:7:8:9:" + while ((c = getopt(ARGC, ARGV, opts)) != -1) { + if (c == "u") + non_repeated_only++ + else if (c == "d") + repeated_only++ + else if (c == "c") + do_count++ + else if (index("0123456789", c) != 0) { + # getopt() requires args to options + # this messes us up for things like -5 + if (Optarg ~ /^[[:digit:]]+$/) + fcount = (c Optarg) + 0 + else { + fcount = c + 0 + Optind-- + } + } else + usage() + } + + if (ARGV[Optind] ~ /^\+[[:digit:]]+$/) { + charcount = substr(ARGV[Optind], 2) + 0 + Optind++ + } + + for (i = 1; i < Optind; i++) + ARGV[i] = "" + + if (repeated_only == 0 && non_repeated_only == 0) + repeated_only = non_repeated_only = 1 + + if (ARGC - Optind == 2) { + outputfile = ARGV[ARGC - 1] + ARGV[ARGC - 1] = "" + } + } + + The following function, 'are_equal()', compares the current line, +'$0', to the previous line, 'last'. It handles skipping fields and +characters. If no field count and no character count are specified, +'are_equal()' returns one or zero depending upon the result of a simple +string comparison of 'last' and '$0'. + + Otherwise, things get more complicated. If fields have to be +skipped, each line is broken into an array using 'split()' (*note String +Functions::); the desired fields are then joined back into a line using +'join()'. The joined lines are stored in 'clast' and 'cline'. If no +fields are skipped, 'clast' and 'cline' are set to 'last' and '$0', +respectively. Finally, if characters are skipped, 'substr()' is used to +strip off the leading 'charcount' characters in 'clast' and 'cline'. +The two strings are then compared and 'are_equal()' returns the result: + + function are_equal( n, m, clast, cline, alast, aline) + { + if (fcount == 0 && charcount == 0) + return (last == $0) + + if (fcount > 0) { + n = split(last, alast) + m = split($0, aline) + clast = join(alast, fcount+1, n) + cline = join(aline, fcount+1, m) + } else { + clast = last + cline = $0 + } + if (charcount) { + clast = substr(clast, charcount + 1) + cline = substr(cline, charcount + 1) + } + + return (clast == cline) + } + + The following two rules are the body of the program. The first one +is executed only for the very first line of data. It sets 'last' equal +to '$0', so that subsequent lines of text have something to be compared +to. + + The second rule does the work. The variable 'equal' is one or zero, +depending upon the results of 'are_equal()''s comparison. If 'uniq' is +counting repeated lines, and the lines are equal, then it increments the +'count' variable. Otherwise, it prints the line and resets 'count', +because the two lines are not equal. + + If 'uniq' is not counting, and if the lines are equal, 'count' is +incremented. Nothing is printed, as the point is to remove duplicates. +Otherwise, if 'uniq' is counting repeated lines and more than one line +is seen, or if 'uniq' is counting nonrepeated lines and only one line is +seen, then the line is printed, and 'count' is reset. + + Finally, similar logic is used in the 'END' rule to print the final +line of input data: + + NR == 1 { + last = $0 + next + } + + { + equal = are_equal() + + if (do_count) { # overrides -d and -u + if (equal) + count++ + else { + printf("%4d %s\n", count, last) > outputfile + last = $0 + count = 1 # reset + } + next + } + + if (equal) + count++ + else { + if ((repeated_only && count > 1) || + (non_repeated_only && count == 1)) + print last > outputfile + last = $0 + count = 1 + } + } + + END { + if (do_count) + printf("%4d %s\n", count, last) > outputfile + else if ((repeated_only && count > 1) || + (non_repeated_only && count == 1)) + print last > outputfile + close(outputfile) + } + + +File: gawk.info, Node: Wc Program, Prev: Uniq Program, Up: Clones + +11.2.7 Counting Things +---------------------- + +The 'wc' (word count) utility counts lines, words, and characters in one +or more input files. Its usage is as follows: + + 'wc' ['-lwc'] [FILES ...] + + If no files are specified on the command line, 'wc' reads its +standard input. If there are multiple files, it also prints total +counts for all the files. The options and their meanings are as +follows: + +'-l' + Count only lines. + +'-w' + Count only words. A "word" is a contiguous sequence of + nonwhitespace characters, separated by spaces and/or TABs. + Luckily, this is the normal way 'awk' separates fields in its input + data. + +'-c' + Count only characters. + + Implementing 'wc' in 'awk' is particularly elegant, because 'awk' +does a lot of the work for us; it splits lines into words (i.e., fields) +and counts them, it counts lines (i.e., records), and it can easily tell +us how long a line is. + + This program uses the 'getopt()' library function (*note Getopt +Function::) and the file-transition functions (*note Filetrans +Function::). + + This version has one notable difference from traditional versions of +'wc': it always prints the counts in the order lines, words, and +characters. Traditional versions note the order of the '-l', '-w', and +'-c' options on the command line, and print the counts in that order. + + The 'BEGIN' rule does the argument processing. The variable +'print_total' is true if more than one file is named on the command +line: + + # wc.awk --- count lines, words, characters + + # Options: + # -l only count lines + # -w only count words + # -c only count characters + # + # Default is to count lines, words, characters + # + # Requires getopt() and file transition library functions + + BEGIN { + # let getopt() print a message about + # invalid options. we ignore them + while ((c = getopt(ARGC, ARGV, "lwc")) != -1) { + if (c == "l") + do_lines = 1 + else if (c == "w") + do_words = 1 + else if (c == "c") + do_chars = 1 + } + for (i = 1; i < Optind; i++) + ARGV[i] = "" + + # if no options, do all + if (! do_lines && ! do_words && ! do_chars) + do_lines = do_words = do_chars = 1 + + print_total = (ARGC - i > 1) + } + + The 'beginfile()' function is simple; it just resets the counts of +lines, words, and characters to zero, and saves the current file name in +'fname': + + function beginfile(file) + { + lines = words = chars = 0 + fname = FILENAME + } + + The 'endfile()' function adds the current file's numbers to the +running totals of lines, words, and characters. It then prints out +those numbers for the file that was just read. It relies on +'beginfile()' to reset the numbers for the following data file: + + function endfile(file) + { + tlines += lines + twords += words + tchars += chars + if (do_lines) + printf "\t%d", lines + if (do_words) + printf "\t%d", words + if (do_chars) + printf "\t%d", chars + printf "\t%s\n", fname + } + + There is one rule that is executed for each line. It adds the length +of the record, plus one, to 'chars'.(1) Adding one plus the record +length is needed because the newline character separating records (the +value of 'RS') is not part of the record itself, and thus not included +in its length. Next, 'lines' is incremented for each line read, and +'words' is incremented by the value of 'NF', which is the number of +"words" on this line: + + # do per line + { + chars += length($0) + 1 # get newline + lines++ + words += NF + } + + Finally, the 'END' rule simply prints the totals for all the files: + + END { + if (print_total) { + if (do_lines) + printf "\t%d", tlines + if (do_words) + printf "\t%d", twords + if (do_chars) + printf "\t%d", tchars + print "\ttotal" + } + } + + ---------- Footnotes ---------- + + (1) Because 'gawk' understands multibyte locales, this code counts +characters, not bytes. + + +File: gawk.info, Node: Miscellaneous Programs, Next: Programs Summary, Prev: Clones, Up: Sample Programs + +11.3 A Grab Bag of 'awk' Programs +================================= + +This minor node is a large "grab bag" of miscellaneous programs. We +hope you find them both interesting and enjoyable. + +* Menu: + +* Dupword Program:: Finding duplicated words in a document. +* Alarm Program:: An alarm clock. +* Translate Program:: A program similar to the 'tr' utility. +* Labels Program:: Printing mailing labels. +* Word Sorting:: A program to produce a word usage count. +* History Sorting:: Eliminating duplicate entries from a history + file. +* Extract Program:: Pulling out programs from Texinfo source + files. +* Simple Sed:: A Simple Stream Editor. +* Igawk Program:: A wrapper for 'awk' that includes + files. +* Anagram Program:: Finding anagrams from a dictionary. +* Signature Program:: People do amazing things with too much time on + their hands. + + +File: gawk.info, Node: Dupword Program, Next: Alarm Program, Up: Miscellaneous Programs + +11.3.1 Finding Duplicated Words in a Document +--------------------------------------------- + +A common error when writing large amounts of prose is to accidentally +duplicate words. Typically you will see this in text as something like +"the the program does the following..." When the text is online, often +the duplicated words occur at the end of one line and the beginning of +another, making them very difficult to spot. + + This program, 'dupword.awk', scans through a file one line at a time +and looks for adjacent occurrences of the same word. It also saves the +last word on a line (in the variable 'prev') for comparison with the +first word on the next line. + + The first two statements make sure that the line is all lowercase, so +that, for example, "The" and "the" compare equal to each other. The +next statement replaces nonalphanumeric and nonwhitespace characters +with spaces, so that punctuation does not affect the comparison either. +The characters are replaced with spaces so that formatting controls +don't create nonsense words (e.g., the Texinfo '@code{NF}' becomes +'codeNF' if punctuation is simply deleted). The record is then resplit +into fields, yielding just the actual words on the line, and ensuring +that there are no empty fields. + + If there are no fields left after removing all the punctuation, the +current record is skipped. Otherwise, the program loops through each +word, comparing it to the previous one: + + # dupword.awk --- find duplicate words in text + { + $0 = tolower($0) + gsub(/[^[:alnum:][:blank:]]/, " "); + $0 = $0 # re-split + if (NF == 0) + next + if ($1 == prev) + printf("%s:%d: duplicate %s\n", + FILENAME, FNR, $1) + for (i = 2; i <= NF; i++) + if ($i == $(i-1)) + printf("%s:%d: duplicate %s\n", + FILENAME, FNR, $i) + prev = $NF + } + + +File: gawk.info, Node: Alarm Program, Next: Translate Program, Prev: Dupword Program, Up: Miscellaneous Programs + +11.3.2 An Alarm Clock Program +----------------------------- + + Nothing cures insomnia like a ringing alarm clock. + -- _Arnold Robbins_ + Sleep is for web developers. + -- _Erik Quanstrom_ + + The following program is a simple "alarm clock" program. You give it +a time of day and an optional message. At the specified time, it prints +the message on the standard output. In addition, you can give it the +number of times to repeat the message as well as a delay between +repetitions. + + This program uses the 'getlocaltime()' function from *note +Getlocaltime Function::. + + All the work is done in the 'BEGIN' rule. The first part is argument +checking and setting of defaults: the delay, the count, and the message +to print. If the user supplied a message without the ASCII BEL +character (known as the "alert" character, '"\a"'), then it is added to +the message. (On many systems, printing the ASCII BEL generates an +audible alert. Thus, when the alarm goes off, the system calls +attention to itself in case the user is not looking at the computer.) +Just for a change, this program uses a 'switch' statement (*note Switch +Statement::), but the processing could be done with a series of +'if'-'else' statements instead. Here is the program: + + # alarm.awk --- set an alarm + # + # Requires getlocaltime() library function + # usage: alarm time [ "message" [ count [ delay ] ] ] + + BEGIN { + # Initial argument sanity checking + usage1 = "usage: alarm time ['message' [count [delay]]]" + usage2 = sprintf("\t(%s) time ::= hh:mm", ARGV[1]) + + if (ARGC < 2) { + print usage1 > "/dev/stderr" + print usage2 > "/dev/stderr" + exit 1 + } + switch (ARGC) { + case 5: + delay = ARGV[4] + 0 + # fall through + case 4: + count = ARGV[3] + 0 + # fall through + case 3: + message = ARGV[2] + break + default: + if (ARGV[1] !~ /[[:digit:]]?[[:digit:]]:[[:digit:]]{2}/) { + print usage1 > "/dev/stderr" + print usage2 > "/dev/stderr" + exit 1 + } + break + } + + # set defaults for once we reach the desired time + if (delay == 0) + delay = 180 # 3 minutes + if (count == 0) + count = 5 + if (message == "") + message = sprintf("\aIt is now %s!\a", ARGV[1]) + else if (index(message, "\a") == 0) + message = "\a" message "\a" + + The next minor node of code turns the alarm time into hours and +minutes, converts it (if necessary) to a 24-hour clock, and then turns +that time into a count of the seconds since midnight. Next it turns the +current time into a count of seconds since midnight. The difference +between the two is how long to wait before setting off the alarm: + + # split up alarm time + split(ARGV[1], atime, ":") + hour = atime[1] + 0 # force numeric + minute = atime[2] + 0 # force numeric + + # get current broken down time + getlocaltime(now) + + # if time given is 12-hour hours and it's after that + # hour, e.g., `alarm 5:30' at 9 a.m. means 5:30 p.m., + # then add 12 to real hour + if (hour < 12 && now["hour"] > hour) + hour += 12 + + # set target time in seconds since midnight + target = (hour * 60 * 60) + (minute * 60) + + # get current time in seconds since midnight + current = (now["hour"] * 60 * 60) + \ + (now["minute"] * 60) + now["second"] + + # how long to sleep for + naptime = target - current + if (naptime <= 0) { + print "alarm: time is in the past!" > "/dev/stderr" + exit 1 + } + + Finally, the program uses the 'system()' function (*note I/O +Functions::) to call the 'sleep' utility. The 'sleep' utility simply +pauses for the given number of seconds. If the exit status is not zero, +the program assumes that 'sleep' was interrupted and exits. If 'sleep' +exited with an OK status (zero), then the program prints the message in +a loop, again using 'sleep' to delay for however many seconds are +necessary: + + # zzzzzz..... go away if interrupted + if (system(sprintf("sleep %d", naptime)) != 0) + exit 1 + + # time to notify! + command = sprintf("sleep %d", delay) + for (i = 1; i <= count; i++) { + print message + # if sleep command interrupted, go away + if (system(command) != 0) + break + } + + exit 0 + } + + +File: gawk.info, Node: Translate Program, Next: Labels Program, Prev: Alarm Program, Up: Miscellaneous Programs + +11.3.3 Transliterating Characters +--------------------------------- + +The system 'tr' utility transliterates characters. For example, it is +often used to map uppercase letters into lowercase for further +processing: + + GENERATE DATA | tr 'A-Z' 'a-z' | PROCESS DATA ... + + 'tr' requires two lists of characters.(1) When processing the input, +the first character in the first list is replaced with the first +character in the second list, the second character in the first list is +replaced with the second character in the second list, and so on. If +there are more characters in the "from" list than in the "to" list, the +last character of the "to" list is used for the remaining characters in +the "from" list. + + Once upon a time, a user proposed adding a transliteration function +to 'gawk'. The following program was written to prove that character +transliteration could be done with a user-level function. This program +is not as complete as the system 'tr' utility, but it does most of the +job. + + The 'translate' program was written long before 'gawk' acquired the +ability to split each character in a string into separate array +elements. Thus, it makes repeated use of the 'substr()', 'index()', and +'gsub()' built-in functions (*note String Functions::). There are two +functions. The first, 'stranslate()', takes three arguments: + +'from' + A list of characters from which to translate + +'to' + A list of characters to which to translate + +'target' + The string on which to do the translation + + Associative arrays make the translation part fairly easy. 't_ar' +holds the "to" characters, indexed by the "from" characters. Then a +simple loop goes through 'from', one character at a time. For each +character in 'from', if the character appears in 'target', it is +replaced with the corresponding 'to' character. + + The 'translate()' function calls 'stranslate()', using '$0' as the +target. The main program sets two global variables, 'FROM' and 'TO', +from the command line, and then changes 'ARGV' so that 'awk' reads from +the standard input. + + Finally, the processing rule simply calls 'translate()' for each +record: + + # translate.awk --- do tr-like stuff + # Bugs: does not handle things like tr A-Z a-z; it has + # to be spelled out. However, if `to' is shorter than `from', + # the last character in `to' is used for the rest of `from'. + + function stranslate(from, to, target, lf, lt, ltarget, t_ar, i, c, + result) + { + lf = length(from) + lt = length(to) + ltarget = length(target) + for (i = 1; i <= lt; i++) + t_ar[substr(from, i, 1)] = substr(to, i, 1) + if (lt < lf) + for (; i <= lf; i++) + t_ar[substr(from, i, 1)] = substr(to, lt, 1) + for (i = 1; i <= ltarget; i++) { + c = substr(target, i, 1) + if (c in t_ar) + c = t_ar[c] + result = result c + } + return result + } + + function translate(from, to) + { + return $0 = stranslate(from, to, $0) + } + + # main program + BEGIN { + if (ARGC < 3) { + print "usage: translate from to" > "/dev/stderr" + exit + } + FROM = ARGV[1] + TO = ARGV[2] + ARGC = 2 + ARGV[1] = "-" + } + + { + translate(FROM, TO) + print + } + + It is possible to do character transliteration in a user-level +function, but it is not necessarily efficient, and we (the 'gawk' +developers) started to consider adding a built-in function. However, +shortly after writing this program, we learned that Brian Kernighan had +added the 'toupper()' and 'tolower()' functions to his 'awk' (*note +String Functions::). These functions handle the vast majority of the +cases where character transliteration is necessary, and so we chose to +simply add those functions to 'gawk' as well and then leave well enough +alone. + + An obvious improvement to this program would be to set up the 't_ar' +array only once, in a 'BEGIN' rule. However, this assumes that the +"from" and "to" lists will never change throughout the lifetime of the +program. + + Another obvious improvement is to enable the use of ranges, such as +'a-z', as allowed by the 'tr' utility. Look at the code for 'cut.awk' +(*note Cut Program::) for inspiration. + + ---------- Footnotes ---------- + + (1) On some older systems, including Solaris, the system version of +'tr' may require that the lists be written as range expressions enclosed +in square brackets ('[a-z]') and quoted, to prevent the shell from +attempting a file name expansion. This is not a feature. + + +File: gawk.info, Node: Labels Program, Next: Word Sorting, Prev: Translate Program, Up: Miscellaneous Programs + +11.3.4 Printing Mailing Labels +------------------------------ + +Here is a "real-world"(1) program. This script reads lists of names and +addresses and generates mailing labels. Each page of labels has 20 +labels on it, two across and 10 down. The addresses are guaranteed to +be no more than five lines of data. Each address is separated from the +next by a blank line. + + The basic idea is to read 20 labels' worth of data. Each line of +each label is stored in the 'line' array. The single rule takes care of +filling the 'line' array and printing the page when 20 labels have been +read. + + The 'BEGIN' rule simply sets 'RS' to the empty string, so that 'awk' +splits records at blank lines (*note Records::). It sets 'MAXLINES' to +100, because 100 is the maximum number of lines on the page (20 * 5 = +100). + + Most of the work is done in the 'printpage()' function. The label +lines are stored sequentially in the 'line' array. But they have to +print horizontally: 'line[1]' next to 'line[6]', 'line[2]' next to +'line[7]', and so on. Two loops accomplish this. The outer loop, +controlled by 'i', steps through every 10 lines of data; this is each +row of labels. The inner loop, controlled by 'j', goes through the +lines within the row. As 'j' goes from 0 to 4, 'i+j' is the 'j'th line +in the row, and 'i+j+5' is the entry next to it. The output ends up +looking something like this: + + line 1 line 6 + line 2 line 7 + line 3 line 8 + line 4 line 9 + line 5 line 10 + ... + +The 'printf' format string '%-41s' left-aligns the data and prints it +within a fixed-width field. + + As a final note, an extra blank line is printed at lines 21 and 61, +to keep the output lined up on the labels. This is dependent on the +particular brand of labels in use when the program was written. You +will also note that there are two blank lines at the top and two blank +lines at the bottom. + + The 'END' rule arranges to flush the final page of labels; there may +not have been an even multiple of 20 labels in the data: + + # labels.awk --- print mailing labels + + # Each label is 5 lines of data that may have blank lines. + # The label sheets have 2 blank lines at the top and 2 at + # the bottom. + + BEGIN { RS = "" ; MAXLINES = 100 } + + function printpage( i, j) + { + if (Nlines <= 0) + return + + printf "\n\n" # header + + for (i = 1; i <= Nlines; i += 10) { + if (i == 21 || i == 61) + print "" + for (j = 0; j < 5; j++) { + if (i + j > MAXLINES) + break + printf " %-41s %s\n", line[i+j], line[i+j+5] + } + print "" + } + + printf "\n\n" # footer + + delete line + } + + # main rule + { + if (Count >= 20) { + printpage() + Count = 0 + Nlines = 0 + } + n = split($0, a, "\n") + for (i = 1; i <= n; i++) + line[++Nlines] = a[i] + for (; i <= 5; i++) + line[++Nlines] = "" + Count++ + } + + END { + printpage() + } + + ---------- Footnotes ---------- + + (1) "Real world" is defined as "a program actually used to get +something done." + + +File: gawk.info, Node: Word Sorting, Next: History Sorting, Prev: Labels Program, Up: Miscellaneous Programs + +11.3.5 Generating Word-Usage Counts +----------------------------------- + +When working with large amounts of text, it can be interesting to know +how often different words appear. For example, an author may overuse +certain words, in which case he or she might wish to find synonyms to +substitute for words that appear too often. This node develops a +program for counting words and presenting the frequency information in a +useful format. + + At first glance, a program like this would seem to do the job: + + # wordfreq-first-try.awk --- print list of word frequencies + + { + for (i = 1; i <= NF; i++) + freq[$i]++ + } + + END { + for (word in freq) + printf "%s\t%d\n", word, freq[word] + } + + The program relies on 'awk''s default field-splitting mechanism to +break each line up into "words" and uses an associative array named +'freq', indexed by each word, to count the number of times the word +occurs. In the 'END' rule, it prints the counts. + + This program has several problems that prevent it from being useful +on real text files: + + * The 'awk' language considers upper- and lowercase characters to be + distinct. Therefore, "bartender" and "Bartender" are not treated + as the same word. This is undesirable, because words are + capitalized if they begin sentences in normal text, and a frequency + analyzer should not be sensitive to capitalization. + + * Words are detected using the 'awk' convention that fields are + separated just by whitespace. Other characters in the input + (except newlines) don't have any special meaning to 'awk'. This + means that punctuation characters count as part of words. + + * The output does not come out in any useful order. You're more + likely to be interested in which words occur most frequently or in + having an alphabetized table of how frequently each word occurs. + + The first problem can be solved by using 'tolower()' to remove case +distinctions. The second problem can be solved by using 'gsub()' to +remove punctuation characters. Finally, we solve the third problem by +using the system 'sort' utility to process the output of the 'awk' +script. Here is the new version of the program: + + # wordfreq.awk --- print list of word frequencies + + { + $0 = tolower($0) # remove case distinctions + # remove punctuation + gsub(/[^[:alnum:]_[:blank:]]/, "", $0) + for (i = 1; i <= NF; i++) + freq[$i]++ + } + + END { + for (word in freq) + printf "%s\t%d\n", word, freq[word] + } + + The regexp '/[^[:alnum:]_[:blank:]]/' might have been written +'/[[:punct:]]/', but then underscores would also be removed, and we want +to keep them. + + Assuming we have saved this program in a file named 'wordfreq.awk', +and that the data is in 'file1', the following pipeline: + + awk -f wordfreq.awk file1 | sort -k 2nr + +produces a table of the words appearing in 'file1' in order of +decreasing frequency. + + The 'awk' program suitably massages the data and produces a word +frequency table, which is not ordered. The 'awk' script's output is +then sorted by the 'sort' utility and printed on the screen. + + The options given to 'sort' specify a sort that uses the second field +of each input line (skipping one field), that the sort keys should be +treated as numeric quantities (otherwise '15' would come before '5'), +and that the sorting should be done in descending (reverse) order. + + The 'sort' could even be done from within the program, by changing +the 'END' action to: + + END { + sort = "sort -k 2nr" + for (word in freq) + printf "%s\t%d\n", word, freq[word] | sort + close(sort) + } + + This way of sorting must be used on systems that do not have true +pipes at the command-line (or batch-file) level. See the general +operating system documentation for more information on how to use the +'sort' program. + + +File: gawk.info, Node: History Sorting, Next: Extract Program, Prev: Word Sorting, Up: Miscellaneous Programs + +11.3.6 Removing Duplicates from Unsorted Text +--------------------------------------------- + +The 'uniq' program (*note Uniq Program::) removes duplicate lines from +_sorted_ data. + + Suppose, however, you need to remove duplicate lines from a data file +but that you want to preserve the order the lines are in. A good +example of this might be a shell history file. The history file keeps a +copy of all the commands you have entered, and it is not unusual to +repeat a command several times in a row. Occasionally you might want to +compact the history by removing duplicate entries. Yet it is desirable +to maintain the order of the original commands. + + This simple program does the job. It uses two arrays. The 'data' +array is indexed by the text of each line. For each line, 'data[$0]' is +incremented. If a particular line has not been seen before, then +'data[$0]' is zero. In this case, the text of the line is stored in +'lines[count]'. Each element of 'lines' is a unique command, and the +indices of 'lines' indicate the order in which those lines are +encountered. The 'END' rule simply prints out the lines, in order: + + # histsort.awk --- compact a shell history file + # Thanks to Byron Rakitzis for the general idea + + { + if (data[$0]++ == 0) + lines[++count] = $0 + } + + END { + for (i = 1; i <= count; i++) + print lines[i] + } + + This program also provides a foundation for generating other useful +information. For example, using the following 'print' statement in the +'END' rule indicates how often a particular command is used: + + print data[lines[i]], lines[i] + +This works because 'data[$0]' is incremented each time a line is seen. + + +File: gawk.info, Node: Extract Program, Next: Simple Sed, Prev: History Sorting, Up: Miscellaneous Programs + +11.3.7 Extracting Programs from Texinfo Source Files +---------------------------------------------------- + +The nodes *note Library Functions::, and *note Sample Programs::, are +the top level nodes for a large number of 'awk' programs. If you want +to experiment with these programs, it is tedious to type them in by +hand. Here we present a program that can extract parts of a Texinfo +input file into separate files. + + This Info file is written in Texinfo +(http://www.gnu.org/software/texinfo/), the GNU Project's document +formatting language. A single Texinfo source file can be used to +produce both printed documentation, with TeX, and online documentation. +(The Texinfo language is described fully, starting with *note (Texinfo, +texinfo,Texinfo---The GNU Documentation Format)Top::.) + + For our purposes, it is enough to know three things about Texinfo +input files: + + * The "at" symbol ('@') is special in Texinfo, much as the backslash + ('\') is in C or 'awk'. Literal '@' symbols are represented in + Texinfo source files as '@@'. + + * Comments start with either '@c' or '@comment'. The file-extraction + program works by using special comments that start at the beginning + of a line. + + * Lines containing '@group' and '@end group' commands bracket example + text that should not be split across a page boundary. + (Unfortunately, TeX isn't always smart enough to do things exactly + right, so we have to give it some help.) + + The following program, 'extract.awk', reads through a Texinfo source +file and does two things, based on the special comments. Upon seeing +'@c system ...', it runs a command, by extracting the command text from +the control line and passing it on to the 'system()' function (*note I/O +Functions::). Upon seeing '@c file FILENAME', each subsequent line is +sent to the file FILENAME, until '@c endfile' is encountered. The rules +in 'extract.awk' match either '@c' or '@comment' by letting the 'omment' +part be optional. Lines containing '@group' and '@end group' are simply +removed. 'extract.awk' uses the 'join()' library function (*note Join +Function::). + + The example programs in the online Texinfo source for 'GAWK: +Effective AWK Programming' ('gawktexi.in') have all been bracketed +inside 'file' and 'endfile' lines. The 'gawk' distribution uses a copy +of 'extract.awk' to extract the sample programs and install many of them +in a standard directory where 'gawk' can find them. The Texinfo file +looks something like this: + + ... + This program has a @code{BEGIN} rule + that prints a nice message: + + @example + @c file examples/messages.awk + BEGIN @{ print "Don't panic!" @} + @c endfile + @end example + + It also prints some final advice: + + @example + @c file examples/messages.awk + END @{ print "Always avoid bored archaeologists!" @} + @c endfile + @end example + ... + + 'extract.awk' begins by setting 'IGNORECASE' to one, so that mixed +upper- and lowercase letters in the directives won't matter. + + The first rule handles calling 'system()', checking that a command is +given ('NF' is at least three) and also checking that the command exits +with a zero exit status, signifying OK: + + # extract.awk --- extract files and run programs from Texinfo files + + BEGIN { IGNORECASE = 1 } + + /^@c(omment)?[ \t]+system/ { + if (NF < 3) { + e = ("extract: " FILENAME ":" FNR) + e = (e ": badly formed `system' line") + print e > "/dev/stderr" + next + } + $1 = "" + $2 = "" + stat = system($0) + if (stat != 0) { + e = ("extract: " FILENAME ":" FNR) + e = (e ": warning: system returned " stat) + print e > "/dev/stderr" + } + } + +The variable 'e' is used so that the rule fits nicely on the screen. + + The second rule handles moving data into files. It verifies that a +file name is given in the directive. If the file named is not the +current file, then the current file is closed. Keeping the current file +open until a new file is encountered allows the use of the '>' +redirection for printing the contents, keeping open-file management +simple. + + The 'for' loop does the work. It reads lines using 'getline' (*note +Getline::). For an unexpected end-of-file, it calls the +'unexpected_eof()' function. If the line is an "endfile" line, then it +breaks out of the loop. If the line is an '@group' or '@end group' +line, then it ignores it and goes on to the next line. Similarly, +comments within examples are also ignored. + + Most of the work is in the following few lines. If the line has no +'@' symbols, the program can print it directly. Otherwise, each leading +'@' must be stripped off. To remove the '@' symbols, the line is split +into separate elements of the array 'a', using the 'split()' function +(*note String Functions::). The '@' symbol is used as the separator +character. Each element of 'a' that is empty indicates two successive +'@' symbols in the original line. For each two empty elements ('@@' in +the original file), we have to add a single '@' symbol back in. + + When the processing of the array is finished, 'join()' is called with +the value of 'SUBSEP' (*note Multidimensional::), to rejoin the pieces +back into a single line. That line is then printed to the output file: + + /^@c(omment)?[ \t]+file/ { + if (NF != 3) { + e = ("extract: " FILENAME ":" FNR ": badly formed `file' line") + print e > "/dev/stderr" + next + } + if ($3 != curfile) { + if (curfile != "") + close(curfile) + curfile = $3 + } + + for (;;) { + if ((getline line) <= 0) + unexpected_eof() + if (line ~ /^@c(omment)?[ \t]+endfile/) + break + else if (line ~ /^@(end[ \t]+)?group/) + continue + else if (line ~ /^@c(omment+)?[ \t]+/) + continue + if (index(line, "@") == 0) { + print line > curfile + continue + } + n = split(line, a, "@") + # if a[1] == "", means leading @, + # don't add one back in. + for (i = 2; i <= n; i++) { + if (a[i] == "") { # was an @@ + a[i] = "@" + if (a[i+1] == "") + i++ + } + } + print join(a, 1, n, SUBSEP) > curfile + } + } + + An important thing to note is the use of the '>' redirection. Output +done with '>' only opens the file once; it stays open and subsequent +output is appended to the file (*note Redirection::). This makes it +easy to mix program text and explanatory prose for the same sample +source file (as has been done here!) without any hassle. The file is +only closed when a new data file name is encountered or at the end of +the input file. + + Finally, the function 'unexpected_eof()' prints an appropriate error +message and then exits. The 'END' rule handles the final cleanup, +closing the open file: + + function unexpected_eof() + { + printf("extract: %s:%d: unexpected EOF or error\n", + FILENAME, FNR) > "/dev/stderr" + exit 1 + } + + END { + if (curfile) + close(curfile) + } + + +File: gawk.info, Node: Simple Sed, Next: Igawk Program, Prev: Extract Program, Up: Miscellaneous Programs + +11.3.8 A Simple Stream Editor +----------------------------- + +The 'sed' utility is a "stream editor", a program that reads a stream of +data, makes changes to it, and passes it on. It is often used to make +global changes to a large file or to a stream of data generated by a +pipeline of commands. Although 'sed' is a complicated program in its +own right, its most common use is to perform global substitutions in the +middle of a pipeline: + + COMMAND1 < orig.data | sed 's/old/new/g' | COMMAND2 > result + + Here, 's/old/new/g' tells 'sed' to look for the regexp 'old' on each +input line and globally replace it with the text 'new' (i.e., all the +occurrences on a line). This is similar to 'awk''s 'gsub()' function +(*note String Functions::). + + The following program, 'awksed.awk', accepts at least two +command-line arguments: the pattern to look for and the text to replace +it with. Any additional arguments are treated as data file names to +process. If none are provided, the standard input is used: + + # awksed.awk --- do s/foo/bar/g using just print + # Thanks to Michael Brennan for the idea + + function usage() + { + print "usage: awksed pat repl [files...]" > "/dev/stderr" + exit 1 + } + + BEGIN { + # validate arguments + if (ARGC < 3) + usage() + + RS = ARGV[1] + ORS = ARGV[2] + + # don't use arguments as files + ARGV[1] = ARGV[2] = "" + } + + # look ma, no hands! + { + if (RT == "") + printf "%s", $0 + else + print + } + + The program relies on 'gawk''s ability to have 'RS' be a regexp, as +well as on the setting of 'RT' to the actual text that terminates the +record (*note Records::). + + The idea is to have 'RS' be the pattern to look for. 'gawk' +automatically sets '$0' to the text between matches of the pattern. +This is text that we want to keep, unmodified. Then, by setting 'ORS' +to the replacement text, a simple 'print' statement outputs the text we +want to keep, followed by the replacement text. + + There is one wrinkle to this scheme, which is what to do if the last +record doesn't end with text that matches 'RS'. Using a 'print' +statement unconditionally prints the replacement text, which is not +correct. However, if the file did not end in text that matches 'RS', +'RT' is set to the null string. In this case, we can print '$0' using +'printf' (*note Printf::). + + The 'BEGIN' rule handles the setup, checking for the right number of +arguments and calling 'usage()' if there is a problem. Then it sets +'RS' and 'ORS' from the command-line arguments and sets 'ARGV[1]' and +'ARGV[2]' to the null string, so that they are not treated as file names +(*note ARGC and ARGV::). + + The 'usage()' function prints an error message and exits. Finally, +the single rule handles the printing scheme outlined earlier, using +'print' or 'printf' as appropriate, depending upon the value of 'RT'. + + +File: gawk.info, Node: Igawk Program, Next: Anagram Program, Prev: Simple Sed, Up: Miscellaneous Programs + +11.3.9 An Easy Way to Use Library Functions +------------------------------------------- + +In *note Include Files::, we saw how 'gawk' provides a built-in +file-inclusion capability. However, this is a 'gawk' extension. This +minor node provides the motivation for making file inclusion available +for standard 'awk', and shows how to do it using a combination of shell +and 'awk' programming. + + Using library functions in 'awk' can be very beneficial. It +encourages code reuse and the writing of general functions. Programs +are smaller and therefore clearer. However, using library functions is +only easy when writing 'awk' programs; it is painful when running them, +requiring multiple '-f' options. If 'gawk' is unavailable, then so too +is the 'AWKPATH' environment variable and the ability to put 'awk' +functions into a library directory (*note Options::). It would be nice +to be able to write programs in the following manner: + + # library functions + @include getopt.awk + @include join.awk + ... + + # main program + BEGIN { + while ((c = getopt(ARGC, ARGV, "a:b:cde")) != -1) + ... + ... + } + + The following program, 'igawk.sh', provides this service. It +simulates 'gawk''s searching of the 'AWKPATH' variable and also allows +"nested" includes (i.e., a file that is included with '@include' can +contain further '@include' statements). 'igawk' makes an effort to only +include files once, so that nested includes don't accidentally include a +library function twice. + + 'igawk' should behave just like 'gawk' externally. This means it +should accept all of 'gawk''s command-line arguments, including the +ability to have multiple source files specified via '-f' and the ability +to mix command-line and library source files. + + The program is written using the POSIX Shell ('sh') command +language.(1) It works as follows: + + 1. Loop through the arguments, saving anything that doesn't represent + 'awk' source code for later, when the expanded program is run. + + 2. For any arguments that do represent 'awk' text, put the arguments + into a shell variable that will be expanded. There are two cases: + + a. Literal text, provided with '-e' or '--source'. This text is + just appended directly. + + b. Source file names, provided with '-f'. We use a neat trick + and append '@include FILENAME' to the shell variable's + contents. Because the file-inclusion program works the way + 'gawk' does, this gets the text of the file included in the + program at the correct point. + + 3. Run an 'awk' program (naturally) over the shell variable's contents + to expand '@include' statements. The expanded program is placed in + a second shell variable. + + 4. Run the expanded program with 'gawk' and any other original + command-line arguments that the user supplied (such as the data + file names). + + This program uses shell variables extensively: for storing +command-line arguments and the text of the 'awk' program that will +expand the user's program, for the user's original program, and for the +expanded program. Doing so removes some potential problems that might +arise were we to use temporary files instead, at the cost of making the +script somewhat more complicated. + + The initial part of the program turns on shell tracing if the first +argument is 'debug'. + + The next part loops through all the command-line arguments. There +are several cases of interest: + +'--' + This ends the arguments to 'igawk'. Anything else should be passed + on to the user's 'awk' program without being evaluated. + +'-W' + This indicates that the next option is specific to 'gawk'. To make + argument processing easier, the '-W' is appended to the front of + the remaining arguments and the loop continues. (This is an 'sh' + programming trick. Don't worry about it if you are not familiar + with 'sh'.) + +'-v', '-F' + These are saved and passed on to 'gawk'. + +'-f', '--file', '--file=', '-Wfile=' + The file name is appended to the shell variable 'program' with an + '@include' statement. The 'expr' utility is used to remove the + leading option part of the argument (e.g., '--file='). (Typical + 'sh' usage would be to use the 'echo' and 'sed' utilities to do + this work. Unfortunately, some versions of 'echo' evaluate escape + sequences in their arguments, possibly mangling the program text. + Using 'expr' avoids this problem.) + +'--source', '--source=', '-Wsource=' + The source text is appended to 'program'. + +'--version', '-Wversion' + 'igawk' prints its version number, runs 'gawk --version' to get the + 'gawk' version information, and then exits. + + If none of the '-f', '--file', '-Wfile', '--source', or '-Wsource' +arguments are supplied, then the first nonoption argument should be the +'awk' program. If there are no command-line arguments left, 'igawk' +prints an error message and exits. Otherwise, the first argument is +appended to 'program'. In any case, after the arguments have been +processed, the shell variable 'program' contains the complete text of +the original 'awk' program. + + The program is as follows: + + #! /bin/sh + # igawk --- like gawk but do @include processing + + if [ "$1" = debug ] + then + set -x + shift + fi + + # A literal newline, so that program text is formatted correctly + n=' + ' + + # Initialize variables to empty + program= + opts= + + while [ $# -ne 0 ] # loop over arguments + do + case $1 in + --) shift + break ;; + + -W) shift + # The ${x?'message here'} construct prints a + # diagnostic if $x is the null string + set -- -W"${@?'missing operand'}" + continue ;; + + -[vF]) opts="$opts $1 '${2?'missing operand'}'" + shift ;; + + -[vF]*) opts="$opts '$1'" ;; + + -f) program="$program$n@include ${2?'missing operand'}" + shift ;; + + -f*) f=$(expr "$1" : '-f\(.*\)') + program="$program$n@include $f" ;; + + -[W-]file=*) + f=$(expr "$1" : '-.file=\(.*\)') + program="$program$n@include $f" ;; + + -[W-]file) + program="$program$n@include ${2?'missing operand'}" + shift ;; + + -[W-]source=*) + t=$(expr "$1" : '-.source=\(.*\)') + program="$program$n$t" ;; + + -[W-]source) + program="$program$n${2?'missing operand'}" + shift ;; + + -[W-]version) + echo igawk: version 3.0 1>&2 + gawk --version + exit 0 ;; + + -[W-]*) opts="$opts '$1'" ;; + + *) break ;; + esac + shift + done + + if [ -z "$program" ] + then + program=${1?'missing program'} + shift + fi + + # At this point, `program' has the program. + + The 'awk' program to process '@include' directives is stored in the +shell variable 'expand_prog'. Doing this keeps the shell script +readable. The 'awk' program reads through the user's program, one line +at a time, using 'getline' (*note Getline::). The input file names and +'@include' statements are managed using a stack. As each '@include' is +encountered, the current file name is "pushed" onto the stack and the +file named in the '@include' directive becomes the current file name. +As each file is finished, the stack is "popped," and the previous input +file becomes the current input file again. The process is started by +making the original file the first one on the stack. + + The 'pathto()' function does the work of finding the full path to a +file. It simulates 'gawk''s behavior when searching the 'AWKPATH' +environment variable (*note AWKPATH Variable::). If a file name has a +'/' in it, no path search is done. Similarly, if the file name is +'"-"', then that string is used as-is. Otherwise, the file name is +concatenated with the name of each directory in the path, and an attempt +is made to open the generated file name. The only way to test if a file +can be read in 'awk' is to go ahead and try to read it with 'getline'; +this is what 'pathto()' does.(2) If the file can be read, it is closed +and the file name is returned: + + expand_prog=' + + function pathto(file, i, t, junk) + { + if (index(file, "/") != 0) + return file + + if (file == "-") + return file + + for (i = 1; i <= ndirs; i++) { + t = (pathlist[i] "/" file) + if ((getline junk < t) > 0) { + # found it + close(t) + return t + } + } + return "" + } + + The main program is contained inside one 'BEGIN' rule. The first +thing it does is set up the 'pathlist' array that 'pathto()' uses. +After splitting the path on ':', null elements are replaced with '"."', +which represents the current directory: + + BEGIN { + path = ENVIRON["AWKPATH"] + ndirs = split(path, pathlist, ":") + for (i = 1; i <= ndirs; i++) { + if (pathlist[i] == "") + pathlist[i] = "." + } + + The stack is initialized with 'ARGV[1]', which will be +'"/dev/stdin"'. The main loop comes next. Input lines are read in +succession. Lines that do not start with '@include' are printed +verbatim. If the line does start with '@include', the file name is in +'$2'. 'pathto()' is called to generate the full path. If it cannot, +then the program prints an error message and continues. + + The next thing to check is if the file is included already. The +'processed' array is indexed by the full file name of each included file +and it tracks this information for us. If the file is seen again, a +warning message is printed. Otherwise, the new file name is pushed onto +the stack and processing continues. + + Finally, when 'getline' encounters the end of the input file, the +file is closed and the stack is popped. When 'stackptr' is less than +zero, the program is done: + + stackptr = 0 + input[stackptr] = ARGV[1] # ARGV[1] is first file + + for (; stackptr >= 0; stackptr--) { + while ((getline < input[stackptr]) > 0) { + if (tolower($1) != "@include") { + print + continue + } + fpath = pathto($2) + if (fpath == "") { + printf("igawk: %s:%d: cannot find %s\n", + input[stackptr], FNR, $2) > "/dev/stderr" + continue + } + if (! (fpath in processed)) { + processed[fpath] = input[stackptr] + input[++stackptr] = fpath # push onto stack + } else + print $2, "included in", input[stackptr], + "already included in", + processed[fpath] > "/dev/stderr" + } + close(input[stackptr]) + } + }' # close quote ends `expand_prog' variable + + processed_program=$(gawk -- "$expand_prog" /dev/stdin << EOF + $program + EOF + ) + + The shell construct 'COMMAND << MARKER' is called a "here document". +Everything in the shell script up to the MARKER is fed to COMMAND as +input. The shell processes the contents of the here document for +variable and command substitution (and possibly other things as well, +depending upon the shell). + + The shell construct '$(...)' is called "command substitution". The +output of the command inside the parentheses is substituted into the +command line. Because the result is used in a variable assignment, it +is saved as a single string, even if the results contain whitespace. + + The expanded program is saved in the variable 'processed_program'. +It's done in these steps: + + 1. Run 'gawk' with the '@include'-processing program (the value of the + 'expand_prog' shell variable) reading standard input. + + 2. Standard input is the contents of the user's program, from the + shell variable 'program'. Feed its contents to 'gawk' via a here + document. + + 3. Save the results of this processing in the shell variable + 'processed_program' by using command substitution. + + The last step is to call 'gawk' with the expanded program, along with +the original options and command-line arguments that the user supplied: + + eval gawk $opts -- '"$processed_program"' '"$@"' + + The 'eval' command is a shell construct that reruns the shell's +parsing process. This keeps things properly quoted. + + This version of 'igawk' represents the fifth version of this program. +There are four key simplifications that make the program work better: + + * Using '@include' even for the files named with '-f' makes building + the initial collected 'awk' program much simpler; all the + '@include' processing can be done once. + + * Not trying to save the line read with 'getline' in the 'pathto()' + function when testing for the file's accessibility for use with the + main program simplifies things considerably. + + * Using a 'getline' loop in the 'BEGIN' rule does it all in one + place. It is not necessary to call out to a separate loop for + processing nested '@include' statements. + + * Instead of saving the expanded program in a temporary file, putting + it in a shell variable avoids some potential security problems. + This has the disadvantage that the script relies upon more features + of the 'sh' language, making it harder to follow for those who + aren't familiar with 'sh'. + + Also, this program illustrates that it is often worthwhile to combine +'sh' and 'awk' programming together. You can usually accomplish quite a +lot, without having to resort to low-level programming in C or C++, and +it is frequently easier to do certain kinds of string and argument +manipulation using the shell than it is in 'awk'. + + Finally, 'igawk' shows that it is not always necessary to add new +features to a program; they can often be layered on top.(3) + + ---------- Footnotes ---------- + + (1) Fully explaining the 'sh' language is beyond the scope of this +book. We provide some minimal explanations, but see a good shell +programming book if you wish to understand things in more depth. + + (2) On some very old versions of 'awk', the test 'getline junk < t' +can loop forever if the file exists but is empty. + + (3) 'gawk' does '@include' processing itself in order to support the +use of 'awk' programs as Web CGI scripts. + + +File: gawk.info, Node: Anagram Program, Next: Signature Program, Prev: Igawk Program, Up: Miscellaneous Programs + +11.3.10 Finding Anagrams from a Dictionary +------------------------------------------ + +An interesting programming challenge is to search for "anagrams" in a +word list (such as '/usr/share/dict/words' on many GNU/Linux systems). +One word is an anagram of another if both words contain the same letters +(e.g., "babbling" and "blabbing"). + + Column 2, Problem C, of Jon Bentley's 'Programming Pearls', Second +Edition, presents an elegant algorithm. The idea is to give words that +are anagrams a common signature, sort all the words together by their +signatures, and then print them. Dr. Bentley observes that taking the +letters in each word and sorting them produces those common signatures. + + The following program uses arrays of arrays to bring together words +with the same signature and array sorting to print the words in sorted +order: + + # anagram.awk --- An implementation of the anagram-finding algorithm + # from Jon Bentley's "Programming Pearls," 2nd edition. + # Addison Wesley, 2000, ISBN 0-201-65788-0. + # Column 2, Problem C, section 2.8, pp 18-20. + + /'s$/ { next } # Skip possessives + + The program starts with a header, and then a rule to skip possessives +in the dictionary file. The next rule builds up the data structure. +The first dimension of the array is indexed by the signature; the second +dimension is the word itself: + + { + key = word2key($1) # Build signature + data[key][$1] = $1 # Store word with signature + } + + The 'word2key()' function creates the signature. It splits the word +apart into individual letters, sorts the letters, and then joins them +back together: + + # word2key --- split word apart into letters, sort, and join back together + + function word2key(word, a, i, n, result) + { + n = split(word, a, "") + asort(a) + + for (i = 1; i <= n; i++) + result = result a[i] + + return result + } + + Finally, the 'END' rule traverses the array and prints out the +anagram lists. It sends the output to the system 'sort' command because +otherwise the anagrams would appear in arbitrary order: + + END { + sort = "sort" + for (key in data) { + # Sort words with same key + nwords = asorti(data[key], words) + if (nwords == 1) + continue + + # And print. Minor glitch: trailing space at end of each line + for (j = 1; j <= nwords; j++) + printf("%s ", words[j]) | sort + print "" | sort + } + close(sort) + } + + Here is some partial output when the program is run: + + $ gawk -f anagram.awk /usr/share/dict/words | grep '^b' + ... + babbled blabbed + babbler blabber brabble + babblers blabbers brabbles + babbling blabbing + babbly blabby + babel bable + babels beslab + babery yabber + ... + + +File: gawk.info, Node: Signature Program, Prev: Anagram Program, Up: Miscellaneous Programs + +11.3.11 And Now for Something Completely Different +-------------------------------------------------- + +The following program was written by Davide Brini and is published on +his website (http://backreference.org/2011/02/03/obfuscated-awk/). It +serves as his signature in the Usenet group 'comp.lang.awk'. He +supplies the following copyright terms: + + Copyright (C) 2008 Davide Brini + + Copying and distribution of the code published in this page, with + or without modification, are permitted in any medium without + royalty provided the copyright notice and this notice are + preserved. + + Here is the program: + + awk 'BEGIN{O="~"~"~";o="=="=="==";o+=+o;x=O""O;while(X++<=x+o+o)c=c"%c"; + printf c,(x-O)*(x-O),x*(x-o)-o,x*(x-O)+x-O-o,+x*(x-O)-x+o,X*(o*o+O)+x-O, + X*(X-x)-o*o,(x+X)*o*o+o,x*(X-x)-O-O,x-O+(O+o+X+x)*(o+O),X*X-X*(x-O)-x+O, + O+X*(o*(o+O)+O),+x+O+X*o,x*(x-o),(o+X+x)*o*o-(x-O-O),O+(X-x)*(X+O),x-O}' + + We leave it to you to determine what the program does. (If you are +truly desperate to understand it, see Chris Johansen's explanation, +which is embedded in the Texinfo source file for this Info file.) + + +File: gawk.info, Node: Programs Summary, Next: Programs Exercises, Prev: Miscellaneous Programs, Up: Sample Programs + +11.4 Summary +============ + + * The programs provided in this major node continue on the theme that + reading programs is an excellent way to learn Good Programming. + + * Using '#!' to make 'awk' programs directly runnable makes them + easier to use. Otherwise, invoke the program using 'awk -f ...'. + + * Reimplementing standard POSIX programs in 'awk' is a pleasant + exercise; 'awk''s expressive power lets you write such programs in + relatively few lines of code, yet they are functionally complete + and usable. + + * One of standard 'awk''s weaknesses is working with individual + characters. The ability to use 'split()' with the empty string as + the separator can considerably simplify such tasks. + + * The examples here demonstrate the usefulness of the library + functions from *note Library Functions:: for a number of real (if + small) programs. + + * Besides reinventing POSIX wheels, other programs solved a selection + of interesting problems, such as finding duplicate words in text, + printing mailing labels, and finding anagrams. + + +File: gawk.info, Node: Programs Exercises, Prev: Programs Summary, Up: Sample Programs + +11.5 Exercises +============== + + 1. Rewrite 'cut.awk' (*note Cut Program::) using 'split()' with '""' + as the separator. + + 2. In *note Egrep Program::, we mentioned that 'egrep -i' could be + simulated in versions of 'awk' without 'IGNORECASE' by using + 'tolower()' on the line and the pattern. In a footnote there, we + also mentioned that this solution has a bug: the translated line is + output, and not the original one. Fix this problem. + + 3. The POSIX version of 'id' takes options that control which + information is printed. Modify the 'awk' version (*note Id + Program::) to accept the same arguments and perform in the same + way. + + 4. The 'split.awk' program (*note Split Program::) assumes that + letters are contiguous in the character set, which isn't true for + EBCDIC systems. Fix this problem. (Hint: Consider a different way + to work through the alphabet, without relying on 'ord()' and + 'chr()'.) + + 5. In 'uniq.awk' (*note Uniq Program::, the logic for choosing which + lines to print represents a "state machine", which is "a device + that can be in one of a set number of stable conditions depending + on its previous condition and on the present values of its + inputs."(1) Brian Kernighan suggests that "an alternative approach + to state machines is to just read the input into an array, then use + indexing. It's almost always easier code, and for most inputs + where you would use this, just as fast." Rewrite the logic to + follow this suggestion. + + 6. Why can't the 'wc.awk' program (*note Wc Program::) just use the + value of 'FNR' in 'endfile()'? Hint: Examine the code in *note + Filetrans Function::. + + 7. Manipulation of individual characters in the 'translate' program + (*note Translate Program::) is painful using standard 'awk' + functions. Given that 'gawk' can split strings into individual + characters using '""' as the separator, how might you use this + feature to simplify the program? + + 8. The 'extract.awk' program (*note Extract Program::) was written + before 'gawk' had the 'gensub()' function. Use it to simplify the + code. + + 9. Compare the performance of the 'awksed.awk' program (*note Simple + Sed::) with the more straightforward: + + BEGIN { + pat = ARGV[1] + repl = ARGV[2] + ARGV[1] = ARGV[2] = "" + } + + { gsub(pat, repl); print } + + 10. What are the advantages and disadvantages of 'awksed.awk' versus + the real 'sed' utility? + + 11. In *note Igawk Program::, we mentioned that not trying to save the + line read with 'getline' in the 'pathto()' function when testing + for the file's accessibility for use with the main program + simplifies things considerably. What problem does this engender + though? + + 12. As an additional example of the idea that it is not always + necessary to add new features to a program, consider the idea of + having two files in a directory in the search path: + + 'default.awk' + This file contains a set of default library functions, such as + 'getopt()' and 'assert()'. + + 'site.awk' + This file contains library functions that are specific to a + site or installation; i.e., locally developed functions. + Having a separate file allows 'default.awk' to change with new + 'gawk' releases, without requiring the system administrator to + update it each time by adding the local functions. + + One user suggested that 'gawk' be modified to automatically read + these files upon startup. Instead, it would be very simple to + modify 'igawk' to do this. Since 'igawk' can process nested + '@include' directives, 'default.awk' could simply contain + '@include' statements for the desired library functions. Make this + change. + + 13. Modify 'anagram.awk' (*note Anagram Program::), to avoid the use + of the external 'sort' utility. + + ---------- Footnotes ---------- + + (1) This is the definition returned from entering 'define: state +machine' into Google. + + +File: gawk.info, Node: Advanced Features, Next: Internationalization, Prev: Sample Programs, Up: Top + +12 Advanced Features of 'gawk' +****************************** + + Write documentation as if whoever reads it is a violent psychopath + who knows where you live. + -- _Steve English, as quoted by Peter Langston_ + + This major node discusses advanced features in 'gawk'. It's a bit of +a "grab bag" of items that are otherwise unrelated to each other. +First, we look at a command-line option that allows 'gawk' to recognize +nondecimal numbers in input data, not just in 'awk' programs. Then, +'gawk''s special features for sorting arrays are presented. Next, +two-way I/O, discussed briefly in earlier parts of this Info file, is +described in full detail, along with the basics of TCP/IP networking. +Finally, we see how 'gawk' can "profile" an 'awk' program, making it +possible to tune it for performance. + + Additional advanced features are discussed in separate major nodes of +their own: + + * *note Internationalization::, discusses how to internationalize + your 'awk' programs, so that they can speak multiple national + languages. + + * *note Debugger::, describes 'gawk''s built-in command-line debugger + for debugging 'awk' programs. + + * *note Arbitrary Precision Arithmetic::, describes how you can use + 'gawk' to perform arbitrary-precision arithmetic. + + * *note Dynamic Extensions::, discusses the ability to dynamically + add new built-in functions to 'gawk'. + +* Menu: + +* Nondecimal Data:: Allowing nondecimal input data. +* Array Sorting:: Facilities for controlling array traversal and + sorting arrays. +* Two-way I/O:: Two-way communications with another process. +* TCP/IP Networking:: Using 'gawk' for network programming. +* Profiling:: Profiling your 'awk' programs. +* Advanced Features Summary:: Summary of advanced features. + + +File: gawk.info, Node: Nondecimal Data, Next: Array Sorting, Up: Advanced Features + +12.1 Allowing Nondecimal Input Data +=================================== + +If you run 'gawk' with the '--non-decimal-data' option, you can have +nondecimal values in your input data: + + $ echo 0123 123 0x123 | + > gawk --non-decimal-data '{ printf "%d, %d, %d\n", $1, $2, $3 }' + -| 83, 123, 291 + + For this feature to work, write your program so that 'gawk' treats +your data as numeric: + + $ echo 0123 123 0x123 | gawk '{ print $1, $2, $3 }' + -| 0123 123 0x123 + +The 'print' statement treats its expressions as strings. Although the +fields can act as numbers when necessary, they are still strings, so +'print' does not try to treat them numerically. You need to add zero to +a field to force it to be treated as a number. For example: + + $ echo 0123 123 0x123 | gawk --non-decimal-data ' + > { print $1, $2, $3 + > print $1 + 0, $2 + 0, $3 + 0 }' + -| 0123 123 0x123 + -| 83 123 291 + + Because it is common to have decimal data with leading zeros, and +because using this facility could lead to surprising results, the +default is to leave it disabled. If you want it, you must explicitly +request it. + + CAUTION: _Use of this option is not recommended._ It can break old + programs very badly. Instead, use the 'strtonum()' function to + convert your data (*note String Functions::). This makes your + programs easier to write and easier to read, and leads to less + surprising results. + + This option may disappear in a future version of 'gawk'. + + +File: gawk.info, Node: Array Sorting, Next: Two-way I/O, Prev: Nondecimal Data, Up: Advanced Features + +12.2 Controlling Array Traversal and Array Sorting +================================================== + +'gawk' lets you control the order in which a 'for (INDX in ARRAY)' loop +traverses an array. + + In addition, two built-in functions, 'asort()' and 'asorti()', let +you sort arrays based on the array values and indices, respectively. +These two functions also provide control over the sorting criteria used +to order the elements during sorting. + +* Menu: + +* Controlling Array Traversal:: How to use PROCINFO["sorted_in"]. +* Array Sorting Functions:: How to use 'asort()' and 'asorti()'. + + +File: gawk.info, Node: Controlling Array Traversal, Next: Array Sorting Functions, Up: Array Sorting + +12.2.1 Controlling Array Traversal +---------------------------------- + +By default, the order in which a 'for (INDX in ARRAY)' loop scans an +array is not defined; it is generally based upon the internal +implementation of arrays inside 'awk'. + + Often, though, it is desirable to be able to loop over the elements +in a particular order that you, the programmer, choose. 'gawk' lets you +do this. + + *note Controlling Scanning:: describes how you can assign special, +predefined values to 'PROCINFO["sorted_in"]' in order to control the +order in which 'gawk' traverses an array during a 'for' loop. + + In addition, the value of 'PROCINFO["sorted_in"]' can be a function +name.(1) This lets you traverse an array based on any custom criterion. +The array elements are ordered according to the return value of this +function. The comparison function should be defined with at least four +arguments: + + function comp_func(i1, v1, i2, v2) + { + COMPARE ELEMENTS 1 AND 2 IN SOME FASHION + RETURN < 0; 0; OR > 0 + } + + Here, 'i1' and 'i2' are the indices, and 'v1' and 'v2' are the +corresponding values of the two elements being compared. Either 'v1' or +'v2', or both, can be arrays if the array being traversed contains +subarrays as values. (*Note Arrays of Arrays:: for more information +about subarrays.) The three possible return values are interpreted as +follows: + +'comp_func(i1, v1, i2, v2) < 0' + Index 'i1' comes before index 'i2' during loop traversal. + +'comp_func(i1, v1, i2, v2) == 0' + Indices 'i1' and 'i2' come together, but the relative order with + respect to each other is undefined. + +'comp_func(i1, v1, i2, v2) > 0' + Index 'i1' comes after index 'i2' during loop traversal. + + Our first comparison function can be used to scan an array in +numerical order of the indices: + + function cmp_num_idx(i1, v1, i2, v2) + { + # numerical index comparison, ascending order + return (i1 - i2) + } + + Our second function traverses an array based on the string order of +the element values rather than by indices: + + function cmp_str_val(i1, v1, i2, v2) + { + # string value comparison, ascending order + v1 = v1 "" + v2 = v2 "" + if (v1 < v2) + return -1 + return (v1 != v2) + } + + The third comparison function makes all numbers, and numeric strings +without any leading or trailing spaces, come out first during loop +traversal: + + function cmp_num_str_val(i1, v1, i2, v2, n1, n2) + { + # numbers before string value comparison, ascending order + n1 = v1 + 0 + n2 = v2 + 0 + if (n1 == v1) + return (n2 == v2) ? (n1 - n2) : -1 + else if (n2 == v2) + return 1 + return (v1 < v2) ? -1 : (v1 != v2) + } + + Here is a main program to demonstrate how 'gawk' behaves using each +of the previous functions: + + BEGIN { + data["one"] = 10 + data["two"] = 20 + data[10] = "one" + data[100] = 100 + data[20] = "two" + + f[1] = "cmp_num_idx" + f[2] = "cmp_str_val" + f[3] = "cmp_num_str_val" + for (i = 1; i <= 3; i++) { + printf("Sort function: %s\n", f[i]) + PROCINFO["sorted_in"] = f[i] + for (j in data) + printf("\tdata[%s] = %s\n", j, data[j]) + print "" + } + } + + Here are the results when the program is run: + + $ gawk -f compdemo.awk + -| Sort function: cmp_num_idx Sort by numeric index + -| data[two] = 20 + -| data[one] = 10 Both strings are numerically zero + -| data[10] = one + -| data[20] = two + -| data[100] = 100 + -| + -| Sort function: cmp_str_val Sort by element values as strings + -| data[one] = 10 + -| data[100] = 100 String 100 is less than string 20 + -| data[two] = 20 + -| data[10] = one + -| data[20] = two + -| + -| Sort function: cmp_num_str_val Sort all numeric values before all strings + -| data[one] = 10 + -| data[two] = 20 + -| data[100] = 100 + -| data[10] = one + -| data[20] = two + + Consider sorting the entries of a GNU/Linux system password file +according to login name. The following program sorts records by a +specific field position and can be used for this purpose: + + # passwd-sort.awk --- simple program to sort by field position + # field position is specified by the global variable POS + + function cmp_field(i1, v1, i2, v2) + { + # comparison by value, as string, and ascending order + return v1[POS] < v2[POS] ? -1 : (v1[POS] != v2[POS]) + } + + { + for (i = 1; i <= NF; i++) + a[NR][i] = $i + } + + END { + PROCINFO["sorted_in"] = "cmp_field" + if (POS < 1 || POS > NF) + POS = 1 + for (i in a) { + for (j = 1; j <= NF; j++) + printf("%s%c", a[i][j], j < NF ? ":" : "") + print "" + } + } + + The first field in each entry of the password file is the user's +login name, and the fields are separated by colons. Each record defines +a subarray, with each field as an element in the subarray. Running the +program produces the following output: + + $ gawk -v POS=1 -F: -f sort.awk /etc/passwd + -| adm:x:3:4:adm:/var/adm:/sbin/nologin + -| apache:x:48:48:Apache:/var/www:/sbin/nologin + -| avahi:x:70:70:Avahi daemon:/:/sbin/nologin + ... + + The comparison should normally always return the same value when +given a specific pair of array elements as its arguments. If +inconsistent results are returned, then the order is undefined. This +behavior can be exploited to introduce random order into otherwise +seemingly ordered data: + + function cmp_randomize(i1, v1, i2, v2) + { + # random order (caution: this may never terminate!) + return (2 - 4 * rand()) + } + + As already mentioned, the order of the indices is arbitrary if two +elements compare equal. This is usually not a problem, but letting the +tied elements come out in arbitrary order can be an issue, especially +when comparing item values. The partial ordering of the equal elements +may change the next time the array is traversed, if other elements are +added to or removed from the array. One way to resolve ties when +comparing elements with otherwise equal values is to include the indices +in the comparison rules. Note that doing this may make the loop +traversal less efficient, so consider it only if necessary. The +following comparison functions force a deterministic order, and are +based on the fact that the (string) indices of two elements are never +equal: + + function cmp_numeric(i1, v1, i2, v2) + { + # numerical value (and index) comparison, descending order + return (v1 != v2) ? (v2 - v1) : (i2 - i1) + } + + function cmp_string(i1, v1, i2, v2) + { + # string value (and index) comparison, descending order + v1 = v1 i1 + v2 = v2 i2 + return (v1 > v2) ? -1 : (v1 != v2) + } + + A custom comparison function can often simplify ordered loop +traversal, and the sky is really the limit when it comes to designing +such a function. + + When string comparisons are made during a sort, either for element +values where one or both aren't numbers, or for element indices handled +as strings, the value of 'IGNORECASE' (*note Built-in Variables::) +controls whether the comparisons treat corresponding upper- and +lowercase letters as equivalent or distinct. + + Another point to keep in mind is that in the case of subarrays, the +element values can themselves be arrays; a production comparison +function should use the 'isarray()' function (*note Type Functions::) to +check for this, and choose a defined sorting order for subarrays. + + All sorting based on 'PROCINFO["sorted_in"]' is disabled in POSIX +mode, because the 'PROCINFO' array is not special in that case. + + As a side note, sorting the array indices before traversing the array +has been reported to add a 15% to 20% overhead to the execution time of +'awk' programs. For this reason, sorted array traversal is not the +default. + + ---------- Footnotes ---------- + + (1) This is why the predefined sorting orders start with an '@' +character, which cannot be part of an identifier. + + +File: gawk.info, Node: Array Sorting Functions, Prev: Controlling Array Traversal, Up: Array Sorting + +12.2.2 Sorting Array Values and Indices with 'gawk' +--------------------------------------------------- + +In most 'awk' implementations, sorting an array requires writing a +'sort()' function. This can be educational for exploring different +sorting algorithms, but usually that's not the point of the program. +'gawk' provides the built-in 'asort()' and 'asorti()' functions (*note +String Functions::) for sorting arrays. For example: + + POPULATE THE ARRAY data + n = asort(data) + for (i = 1; i <= n; i++) + DO SOMETHING WITH data[i] + + After the call to 'asort()', the array 'data' is indexed from 1 to +some number N, the total number of elements in 'data'. (This count is +'asort()''s return value.) 'data[1]' <= 'data[2]' <= 'data[3]', and so +on. The default comparison is based on the type of the elements (*note +Typing and Comparison::). All numeric values come before all string +values, which in turn come before all subarrays. + + An important side effect of calling 'asort()' is that _the array's +original indices are irrevocably lost_. As this isn't always desirable, +'asort()' accepts a second argument: + + POPULATE THE ARRAY source + n = asort(source, dest) + for (i = 1; i <= n; i++) + DO SOMETHING WITH dest[i] + + In this case, 'gawk' copies the 'source' array into the 'dest' array +and then sorts 'dest', destroying its indices. However, the 'source' +array is not affected. + + Often, what's needed is to sort on the values of the _indices_ +instead of the values of the elements. To do that, use the 'asorti()' +function. The interface and behavior are identical to that of +'asort()', except that the index values are used for sorting and become +the values of the result array: + + { source[$0] = some_func($0) } + + END { + n = asorti(source, dest) + for (i = 1; i <= n; i++) { + Work with sorted indices directly: + DO SOMETHING WITH dest[i] + ... + Access original array via sorted indices: + DO SOMETHING WITH source[dest[i]] + } + } + + So far, so good. Now it starts to get interesting. Both 'asort()' +and 'asorti()' accept a third string argument to control comparison of +array elements. When we introduced 'asort()' and 'asorti()' in *note +String Functions::, we ignored this third argument; however, now is the +time to describe how this argument affects these two functions. + + Basically, the third argument specifies how the array is to be +sorted. There are two possibilities. As with 'PROCINFO["sorted_in"]', +this argument may be one of the predefined names that 'gawk' provides +(*note Controlling Scanning::), or it may be the name of a user-defined +function (*note Controlling Array Traversal::). + + In the latter case, _the function can compare elements in any way it +chooses_, taking into account just the indices, just the values, or +both. This is extremely powerful. + + Once the array is sorted, 'asort()' takes the _values_ in their final +order and uses them to fill in the result array, whereas 'asorti()' +takes the _indices_ in their final order and uses them to fill in the +result array. + + NOTE: Copying array indices and elements isn't expensive in terms + of memory. Internally, 'gawk' maintains "reference counts" to + data. For example, when 'asort()' copies the first array to the + second one, there is only one copy of the original array elements' + data, even though both arrays use the values. + + Because 'IGNORECASE' affects string comparisons, the value of +'IGNORECASE' also affects sorting for both 'asort()' and 'asorti()'. +Note also that the locale's sorting order does _not_ come into play; +comparisons are based on character values only.(1) + + The following example demonstrates the use of a comparison function +with 'asort()'. The comparison function, 'case_fold_compare()', maps +both values to lowercase in order to compare them ignoring case. + + # case_fold_compare --- compare as strings, ignoring case + + function case_fold_compare(i1, v1, i2, v2, l, r) + { + l = tolower(v1) + r = tolower(v2) + + if (l < r) + return -1 + else if (l == r) + return 0 + else + return 1 + } + + And here is the test program for it: + + # Test program + + BEGIN { + Letters = "abcdefghijklmnopqrstuvwxyz" \ + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + split(Letters, data, "") + + asort(data, result, "case_fold_compare") + + j = length(result) + for (i = 1; i <= j; i++) { + printf("%s", result[i]) + if (i % (j/2) == 0) + printf("\n") + else + printf(" ") + } + } + + When run, we get the following: + + $ gawk -f case_fold_compare.awk + -| A a B b c C D d e E F f g G H h i I J j k K l L M m + -| n N O o p P Q q r R S s t T u U V v w W X x y Y z Z + + ---------- Footnotes ---------- + + (1) This is true because locale-based comparison occurs only when in +POSIX-compatibility mode, and because 'asort()' and 'asorti()' are +'gawk' extensions, they are not available in that case. + + +File: gawk.info, Node: Two-way I/O, Next: TCP/IP Networking, Prev: Array Sorting, Up: Advanced Features + +12.3 Two-Way Communications with Another Process +================================================ + +It is often useful to be able to send data to a separate program for +processing and then read the result. This can always be done with +temporary files: + + # Write the data for processing + tempfile = ("mydata." PROCINFO["pid"]) + while (NOT DONE WITH DATA) + print DATA | ("subprogram > " tempfile) + close("subprogram > " tempfile) + + # Read the results, remove tempfile when done + while ((getline newdata < tempfile) > 0) + PROCESS newdata APPROPRIATELY + close(tempfile) + system("rm " tempfile) + +This works, but not elegantly. Among other things, it requires that the +program be run in a directory that cannot be shared among users; for +example, '/tmp' will not do, as another user might happen to be using a +temporary file with the same name.(1) + + However, with 'gawk', it is possible to open a _two-way_ pipe to +another process. The second process is termed a "coprocess", as it runs +in parallel with 'gawk'. The two-way connection is created using the +'|&' operator (borrowed from the Korn shell, 'ksh'):(2) + + do { + print DATA |& "subprogram" + "subprogram" |& getline results + } while (DATA LEFT TO PROCESS) + close("subprogram") + + The first time an I/O operation is executed using the '|&' operator, +'gawk' creates a two-way pipeline to a child process that runs the other +program. Output created with 'print' or 'printf' is written to the +program's standard input, and output from the program's standard output +can be read by the 'gawk' program using 'getline'. As is the case with +processes started by '|', the subprogram can be any program, or pipeline +of programs, that can be started by the shell. + + There are some cautionary items to be aware of: + + * As the code inside 'gawk' currently stands, the coprocess's + standard error goes to the same place that the parent 'gawk''s + standard error goes. It is not possible to read the child's + standard error separately. + + * I/O buffering may be a problem. 'gawk' automatically flushes all + output down the pipe to the coprocess. However, if the coprocess + does not flush its output, 'gawk' may hang when doing a 'getline' + in order to read the coprocess's results. This could lead to a + situation known as "deadlock", where each process is waiting for + the other one to do something. + + It is possible to close just one end of the two-way pipe to a +coprocess, by supplying a second argument to the 'close()' function of +either '"to"' or '"from"' (*note Close Files And Pipes::). These +strings tell 'gawk' to close the end of the pipe that sends data to the +coprocess or the end that reads from it, respectively. + + This is particularly necessary in order to use the system 'sort' +utility as part of a coprocess; 'sort' must read _all_ of its input data +before it can produce any output. The 'sort' program does not receive +an end-of-file indication until 'gawk' closes the write end of the pipe. + + When you have finished writing data to the 'sort' utility, you can +close the '"to"' end of the pipe, and then start reading sorted data via +'getline'. For example: + + BEGIN { + command = "LC_ALL=C sort" + n = split("abcdefghijklmnopqrstuvwxyz", a, "") + + for (i = n; i > 0; i--) + print a[i] |& command + close(command, "to") + + while ((command |& getline line) > 0) + print "got", line + close(command) + } + + This program writes the letters of the alphabet in reverse order, one +per line, down the two-way pipe to 'sort'. It then closes the write end +of the pipe, so that 'sort' receives an end-of-file indication. This +causes 'sort' to sort the data and write the sorted data back to the +'gawk' program. Once all of the data has been read, 'gawk' terminates +the coprocess and exits. + + As a side note, the assignment 'LC_ALL=C' in the 'sort' command +ensures traditional Unix (ASCII) sorting from 'sort'. This is not +strictly necessary here, but it's good to know how to do this. + + Be careful when closing the '"from"' end of a two-way pipe; in this +case 'gawk' waits for the child process to exit, which may cause your +program to hang. (Thus, this particular feature is of much less use in +practice than being able to close the '"to"' end.) + + CAUTION: Normally, it is a fatal error to write to the '"to"' end + of a two-way pipe which has been closed, and it is also a fatal + error to read from the '"from"' end of a two-way pipe that has been + closed. + + You may set 'PROCINFO["COMMAND", "NONFATAL"]' to make such + operations become nonfatal, in which case you then need to check + 'ERRNO' after each 'print', 'printf', or 'getline'. *Note + Nonfatal::, for more information. + + You may also use pseudo-ttys (ptys) for two-way communication instead +of pipes, if your system supports them. This is done on a per-command +basis, by setting a special element in the 'PROCINFO' array (*note +Auto-set::), like so: + + command = "sort -nr" # command, save in convenience variable + PROCINFO[command, "pty"] = 1 # update PROCINFO + print ... |& command # start two-way pipe + ... + +If your system does not have ptys, or if all the system's ptys are in +use, 'gawk' automatically falls back to using regular pipes. + + Using ptys usually avoids the buffer deadlock issues described +earlier, at some loss in performance. This is because the tty driver +buffers and sends data line-by-line. On systems with the 'stdbuf' (part +of the GNU Coreutils package +(http://www.gnu.org/software/coreutils/coreutils.html)), you can use +that program instead of ptys. + + Note also that ptys are not fully transparent. Certain binary +control codes, such 'Ctrl-d' for end-of-file, are interpreted by the tty +driver and not passed through. + + CAUTION: Finally, coprocesses open up the possibility of "deadlock" + between 'gawk' and the program running in the coprocess. This can + occur if you send "too much" data to the coprocess before reading + any back; each process is blocked writing data with noone available + to read what they've already written. There is no workaround for + deadlock; careful programming and knowledge of the behavior of the + coprocess are required. + + ---------- Footnotes ---------- + + (1) Michael Brennan suggests the use of 'rand()' to generate unique +file names. This is a valid point; nevertheless, temporary files remain +more difficult to use than two-way pipes. + + (2) This is very different from the same operator in the C shell and +in Bash. + + +File: gawk.info, Node: TCP/IP Networking, Next: Profiling, Prev: Two-way I/O, Up: Advanced Features + +12.4 Using 'gawk' for Network Programming +========================================= + + 'EMRED': + A host is a host from coast to coast, + and nobody talks to a host that's close, + unless the host that isn't close + is busy, hung, or dead. + -- _Mike O'Brien (aka Mr. Protocol)_ + + In addition to being able to open a two-way pipeline to a coprocess +on the same system (*note Two-way I/O::), it is possible to make a +two-way connection to another process on another system across an IP +network connection. + + You can think of this as just a _very long_ two-way pipeline to a +coprocess. The way 'gawk' decides that you want to use TCP/IP +networking is by recognizing special file names that begin with one of +'/inet/', '/inet4/', or '/inet6/'. + + The full syntax of the special file name is +'/NET-TYPE/PROTOCOL/LOCAL-PORT/REMOTE-HOST/REMOTE-PORT'. The components +are: + +NET-TYPE + Specifies the kind of Internet connection to make. Use '/inet4/' + to force IPv4, and '/inet6/' to force IPv6. Plain '/inet/' (which + used to be the only option) uses the system default, most likely + IPv4. + +PROTOCOL + The protocol to use over IP. This must be either 'tcp', or 'udp', + for a TCP or UDP IP connection, respectively. TCP should be used + for most applications. + +LOCAL-PORT + The local TCP or UDP port number to use. Use a port number of '0' + when you want the system to pick a port. This is what you should + do when writing a TCP or UDP client. You may also use a well-known + service name, such as 'smtp' or 'http', in which case 'gawk' + attempts to determine the predefined port number using the C + 'getaddrinfo()' function. + +REMOTE-HOST + The IP address or fully qualified domain name of the Internet host + to which you want to connect. + +REMOTE-PORT + The TCP or UDP port number to use on the given REMOTE-HOST. Again, + use '0' if you don't care, or else a well-known service name. + + NOTE: Failure in opening a two-way socket will result in a nonfatal + error being returned to the calling code. The value of 'ERRNO' + indicates the error (*note Auto-set::). + + Consider the following very simple example: + + BEGIN { + Service = "/inet/tcp/0/localhost/daytime" + Service |& getline + print $0 + close(Service) + } + + This program reads the current date and time from the local system's +TCP 'daytime' server. It then prints the results and closes the +connection. + + Because this topic is extensive, the use of 'gawk' for TCP/IP +programming is documented separately. See *note (General Introduction, +gawkinet, TCP/IP Internetworking with 'gawk')Top::, for a much more +complete introduction and discussion, as well as extensive examples. + + NOTE: 'gawk' can only open direct sockets. There is currently no + way to access services available over Secure Socket Layer (SSL); + this includes any web service whose URL starts with 'https://'. + + +File: gawk.info, Node: Profiling, Next: Advanced Features Summary, Prev: TCP/IP Networking, Up: Advanced Features + +12.5 Profiling Your 'awk' Programs +================================== + +You may produce execution traces of your 'awk' programs. This is done +by passing the option '--profile' to 'gawk'. When 'gawk' has finished +running, it creates a profile of your program in a file named +'awkprof.out'. Because it is profiling, it also executes up to 45% +slower than 'gawk' normally does. + + As shown in the following example, the '--profile' option can be used +to change the name of the file where 'gawk' will write the profile: + + gawk --profile=myprog.prof -f myprog.awk data1 data2 + +In the preceding example, 'gawk' places the profile in 'myprog.prof' +instead of in 'awkprof.out'. + + Here is a sample session showing a simple 'awk' program, its input +data, and the results from running 'gawk' with the '--profile' option. +First, the 'awk' program: + + BEGIN { print "First BEGIN rule" } + + END { print "First END rule" } + + /foo/ { + print "matched /foo/, gosh" + for (i = 1; i <= 3; i++) + sing() + } + + { + if (/foo/) + print "if is true" + else + print "else is true" + } + + BEGIN { print "Second BEGIN rule" } + + END { print "Second END rule" } + + function sing( dummy) + { + print "I gotta be me!" + } + + Following is the input data: + + foo + bar + baz + foo + junk + + Here is the 'awkprof.out' that results from running the 'gawk' +profiler on this program and data (this example also illustrates that +'awk' programmers sometimes get up very early in the morning to work): + + # gawk profile, created Mon Sep 29 05:16:21 2014 + + # BEGIN rule(s) + + BEGIN { + 1 print "First BEGIN rule" + } + + BEGIN { + 1 print "Second BEGIN rule" + } + + # Rule(s) + + 5 /foo/ { # 2 + 2 print "matched /foo/, gosh" + 6 for (i = 1; i <= 3; i++) { + 6 sing() + } + } + + 5 { + 5 if (/foo/) { # 2 + 2 print "if is true" + 3 } else { + 3 print "else is true" + } + } + + # END rule(s) + + END { + 1 print "First END rule" + } + + END { + 1 print "Second END rule" + } + + + # Functions, listed alphabetically + + 6 function sing(dummy) + { + 6 print "I gotta be me!" + } + + This example illustrates many of the basic features of profiling +output. They are as follows: + + * The program is printed in the order 'BEGIN' rules, 'BEGINFILE' + rules, pattern-action rules, 'ENDFILE' rules, 'END' rules, and + functions, listed alphabetically. Multiple 'BEGIN' and 'END' rules + retain their separate identities, as do multiple 'BEGINFILE' and + 'ENDFILE' rules. + + * Pattern-action rules have two counts. The first count, to the left + of the rule, shows how many times the rule's pattern was _tested_. + The second count, to the right of the rule's opening left brace in + a comment, shows how many times the rule's action was _executed_. + The difference between the two indicates how many times the rule's + pattern evaluated to false. + + * Similarly, the count for an 'if'-'else' statement shows how many + times the condition was tested. To the right of the opening left + brace for the 'if''s body is a count showing how many times the + condition was true. The count for the 'else' indicates how many + times the test failed. + + * The count for a loop header (such as 'for' or 'while') shows how + many times the loop test was executed. (Because of this, you can't + just look at the count on the first statement in a rule to + determine how many times the rule was executed. If the first + statement is a loop, the count is misleading.) + + * For user-defined functions, the count next to the 'function' + keyword indicates how many times the function was called. The + counts next to the statements in the body show how many times those + statements were executed. + + * The layout uses "K&R" style with TABs. Braces are used everywhere, + even when the body of an 'if', 'else', or loop is only a single + statement. + + * Parentheses are used only where needed, as indicated by the + structure of the program and the precedence rules. For example, + '(3 + 5) * 4' means add three and five, then multiply the total by + four. However, '3 + 5 * 4' has no parentheses, and means '3 + (5 * + 4)'. + + * Parentheses are used around the arguments to 'print' and 'printf' + only when the 'print' or 'printf' statement is followed by a + redirection. Similarly, if the target of a redirection isn't a + scalar, it gets parenthesized. + + * 'gawk' supplies leading comments in front of the 'BEGIN' and 'END' + rules, the 'BEGINFILE' and 'ENDFILE' rules, the pattern-action + rules, and the functions. + + The profiled version of your program may not look exactly like what +you typed when you wrote it. This is because 'gawk' creates the +profiled version by "pretty-printing" its internal representation of the +program. The advantage to this is that 'gawk' can produce a standard +representation. Also, things such as: + + /foo/ + +come out as: + + /foo/ { + print $0 + } + +which is correct, but possibly unexpected. + + Besides creating profiles when a program has completed, 'gawk' can +produce a profile while it is running. This is useful if your 'awk' +program goes into an infinite loop and you want to see what has been +executed. To use this feature, run 'gawk' with the '--profile' option +in the background: + + $ gawk --profile -f myprog & + [1] 13992 + +The shell prints a job number and process ID number; in this case, +13992. Use the 'kill' command to send the 'USR1' signal to 'gawk': + + $ kill -USR1 13992 + +As usual, the profiled version of the program is written to +'awkprof.out', or to a different file if one was specified with the +'--profile' option. + + Along with the regular profile, as shown earlier, the profile file +includes a trace of any active functions: + + # Function Call Stack: + + # 3. baz + # 2. bar + # 1. foo + # -- main -- + + You may send 'gawk' the 'USR1' signal as many times as you like. +Each time, the profile and function call trace are appended to the +output profile file. + + If you use the 'HUP' signal instead of the 'USR1' signal, 'gawk' +produces the profile and the function call trace and then exits. + + When 'gawk' runs on MS-Windows systems, it uses the 'INT' and 'QUIT' +signals for producing the profile, and in the case of the 'INT' signal, +'gawk' exits. This is because these systems don't support the 'kill' +command, so the only signals you can deliver to a program are those +generated by the keyboard. The 'INT' signal is generated by the +'Ctrl-c' or 'Ctrl-BREAK' key, while the 'QUIT' signal is generated by +the 'Ctrl-\' key. + + Finally, 'gawk' also accepts another option, '--pretty-print'. When +called this way, 'gawk' "pretty-prints" the program into 'awkprof.out', +without any execution counts. + + NOTE: Once upon a time, the '--pretty-print' option would also run + your program. This is is no longer the case. + + There is a significant difference between the output created when +profiling, and that created when pretty-printing. Pretty-printed output +preserves the original comments that were in the program, although their +placement may not correspond exactly to their original locations in the +source code.(1) + + However, as a deliberate design decision, profiling output _omits_ +the original program's comments. This allows you to focus on the +execution count data and helps you avoid the temptation to use the +profiler for pretty-printing. + + Additionally, pretty-printed output does not have the leading +indentation that the profiling output does. This makes it easy to +pretty-print your code once development is completed, and then use the +result as the final version of your program. + + Because the internal representation of your program is formatted to +recreate an 'awk' program, profiling and pretty-printing automatically +disable 'gawk''s default optimizations. + + ---------- Footnotes ---------- + + (1) 'gawk' does the best it can to preserve the distinction between +comments at the end of a statement and comments on lines by themselves. +Due to implementation constraints, it does not always do so correctly, +particularly for 'switch' statements. The 'gawk' maintainers hope to +improve this in a subsequent release. + + +File: gawk.info, Node: Advanced Features Summary, Prev: Profiling, Up: Advanced Features + +12.6 Summary +============ + + * The '--non-decimal-data' option causes 'gawk' to treat octal- and + hexadecimal-looking input data as octal and hexadecimal. This + option should be used with caution or not at all; use of + 'strtonum()' is preferable. Note that this option may disappear in + a future version of 'gawk'. + + * You can take over complete control of sorting in 'for (INDX in + ARRAY)' array traversal by setting 'PROCINFO["sorted_in"]' to the + name of a user-defined function that does the comparison of array + elements based on index and value. + + * Similarly, you can supply the name of a user-defined comparison + function as the third argument to either 'asort()' or 'asorti()' to + control how those functions sort arrays. Or you may provide one of + the predefined control strings that work for + 'PROCINFO["sorted_in"]'. + + * You can use the '|&' operator to create a two-way pipe to a + coprocess. You read from the coprocess with 'getline' and write to + it with 'print' or 'printf'. Use 'close()' to close off the + coprocess completely, or optionally, close off one side of the + two-way communications. + + * By using special file names with the '|&' operator, you can open a + TCP/IP (or UDP/IP) connection to remote hosts on the Internet. + 'gawk' supports both IPv4 and IPv6. + + * You can generate statement count profiles of your program. This + can help you determine which parts of your program may be taking + the most time and let you tune them more easily. Sending the + 'USR1' signal while profiling causes 'gawk' to dump the profile and + keep going, including a function call stack. + + * You can also just "pretty-print" the program. + + +File: gawk.info, Node: Internationalization, Next: Debugger, Prev: Advanced Features, Up: Top + +13 Internationalization with 'gawk' +*********************************** + +Once upon a time, computer makers wrote software that worked only in +English. Eventually, hardware and software vendors noticed that if +their systems worked in the native languages of non-English-speaking +countries, they were able to sell more systems. As a result, +internationalization and localization of programs and software systems +became a common practice. + + For many years, the ability to provide internationalization was +largely restricted to programs written in C and C++. This major node +describes the underlying library 'gawk' uses for internationalization, +as well as how 'gawk' makes internationalization features available at +the 'awk' program level. Having internationalization available at the +'awk' level gives software developers additional flexibility--they are +no longer forced to write in C or C++ when internationalization is a +requirement. + +* Menu: + +* I18N and L10N:: Internationalization and Localization. +* Explaining gettext:: How GNU 'gettext' works. +* Programmer i18n:: Features for the programmer. +* Translator i18n:: Features for the translator. +* I18N Example:: A simple i18n example. +* Gawk I18N:: 'gawk' is also internationalized. +* I18N Summary:: Summary of I18N stuff. + + +File: gawk.info, Node: I18N and L10N, Next: Explaining gettext, Up: Internationalization + +13.1 Internationalization and Localization +========================================== + +"Internationalization" means writing (or modifying) a program once, in +such a way that it can use multiple languages without requiring further +source code changes. "Localization" means providing the data necessary +for an internationalized program to work in a particular language. Most +typically, these terms refer to features such as the language used for +printing error messages, the language used to read responses, and +information related to how numerical and monetary values are printed and +read. + + +File: gawk.info, Node: Explaining gettext, Next: Programmer i18n, Prev: I18N and L10N, Up: Internationalization + +13.2 GNU 'gettext' +================== + +'gawk' uses GNU 'gettext' to provide its internationalization features. +The facilities in GNU 'gettext' focus on messages: strings printed by a +program, either directly or via formatting with 'printf' or +'sprintf()'.(1) + + When using GNU 'gettext', each application has its own "text domain". +This is a unique name, such as 'kpilot' or 'gawk', that identifies the +application. A complete application may have multiple +components--programs written in C or C++, as well as scripts written in +'sh' or 'awk'. All of the components use the same text domain. + + To make the discussion concrete, assume we're writing an application +named 'guide'. Internationalization consists of the following steps, in +this order: + + 1. The programmer reviews the source for all of 'guide''s components + and marks each string that is a candidate for translation. For + example, '"`-F': option required"' is a good candidate for + translation. A table with strings of option names is not (e.g., + 'gawk''s '--profile' option should remain the same, no matter what + the local language). + + 2. The programmer indicates the application's text domain ('"guide"') + to the 'gettext' library, by calling the 'textdomain()' function. + + 3. Messages from the application are extracted from the source code + and collected into a portable object template file ('guide.pot'), + which lists the strings and their translations. The translations + are initially empty. The original (usually English) messages serve + as the key for lookup of the translations. + + 4. For each language with a translator, 'guide.pot' is copied to a + portable object file ('.po') and translations are created and + shipped with the application. For example, there might be a + 'fr.po' for a French translation. + + 5. Each language's '.po' file is converted into a binary message + object ('.gmo') file. A message object file contains the original + messages and their translations in a binary format that allows fast + lookup of translations at runtime. + + 6. When 'guide' is built and installed, the binary translation files + are installed in a standard place. + + 7. For testing and development, it is possible to tell 'gettext' to + use '.gmo' files in a different directory than the standard one by + using the 'bindtextdomain()' function. + + 8. At runtime, 'guide' looks up each string via a call to 'gettext()'. + The returned string is the translated string if available, or the + original string if not. + + 9. If necessary, it is possible to access messages from a different + text domain than the one belonging to the application, without + having to switch the application's default text domain back and + forth. + + In C (or C++), the string marking and dynamic translation lookup are +accomplished by wrapping each string in a call to 'gettext()': + + printf("%s", gettext("Don't Panic!\n")); + + The tools that extract messages from source code pull out all strings +enclosed in calls to 'gettext()'. + + The GNU 'gettext' developers, recognizing that typing 'gettext(...)' +over and over again is both painful and ugly to look at, use the macro +'_' (an underscore) to make things easier: + + /* In the standard header file: */ + #define _(str) gettext(str) + + /* In the program text: */ + printf("%s", _("Don't Panic!\n")); + +This reduces the typing overhead to just three extra characters per +string and is considerably easier to read as well. + + There are locale "categories" for different types of locale-related +information. The defined locale categories that 'gettext' knows about +are: + +'LC_MESSAGES' + Text messages. This is the default category for 'gettext' + operations, but it is possible to supply a different one + explicitly, if necessary. (It is almost never necessary to supply + a different category.) + +'LC_COLLATE' + Text-collation information (i.e., how different characters and/or + groups of characters sort in a given language). + +'LC_CTYPE' + Character-type information (alphabetic, digit, upper- or lowercase, + and so on) as well as character encoding. This information is + accessed via the POSIX character classes in regular expressions, + such as '/[[:alnum:]]/' (*note Bracket Expressions::). + +'LC_MONETARY' + Monetary information, such as the currency symbol, and whether the + symbol goes before or after a number. + +'LC_NUMERIC' + Numeric information, such as which characters to use for the + decimal point and the thousands separator.(2) + +'LC_TIME' + Time- and date-related information, such as 12- or 24-hour clock, + month printed before or after the day in a date, local month + abbreviations, and so on. + +'LC_ALL' + All of the above. (Not too useful in the context of 'gettext'.) + + NOTE: As described in *note Locales::, environment variables with + the same name as the locale categories ('LC_CTYPE', 'LC_ALL', etc.) + influence 'gawk''s behavior (and that of other utilities). + + Normally, these variables also affect how the 'gettext' library + finds translations. However, the 'LANGUAGE' environment variable + overrides the 'LC_XXX' variables. Many GNU/Linux systems may + define this variable without your knowledge, causing 'gawk' to not + find the correct translations. If this happens to you, look to see + if 'LANGUAGE' is defined, and if so, use the shell's 'unset' + command to remove it. + + For testing translations of 'gawk' itself, you can set the +'GAWK_LOCALE_DIR' environment variable. See the documentation for the C +'bindtextdomain()' function and also see *note Other Environment +Variables::. + + ---------- Footnotes ---------- + + (1) For some operating systems, the 'gawk' port doesn't support GNU +'gettext'. Therefore, these features are not available if you are using +one of those operating systems. Sorry. + + (2) Americans use a comma every three decimal places and a period for +the decimal point, while many Europeans do exactly the opposite: +1,234.56 versus 1.234,56. + + +File: gawk.info, Node: Programmer i18n, Next: Translator i18n, Prev: Explaining gettext, Up: Internationalization + +13.3 Internationalizing 'awk' Programs +====================================== + +'gawk' provides the following variables for internationalization: + +'TEXTDOMAIN' + This variable indicates the application's text domain. For + compatibility with GNU 'gettext', the default value is + '"messages"'. + +'_"your message here"' + String constants marked with a leading underscore are candidates + for translation at runtime. String constants without a leading + underscore are not translated. + + 'gawk' provides the following functions for internationalization: + +'dcgettext(STRING [, DOMAIN [, CATEGORY]])' + Return the translation of STRING in text domain DOMAIN for locale + category CATEGORY. The default value for DOMAIN is the current + value of 'TEXTDOMAIN'. The default value for CATEGORY is + '"LC_MESSAGES"'. + + If you supply a value for CATEGORY, it must be a string equal to + one of the known locale categories described in *note Explaining + gettext::. You must also supply a text domain. Use 'TEXTDOMAIN' + if you want to use the current domain. + + CAUTION: The order of arguments to the 'awk' version of the + 'dcgettext()' function is purposely different from the order + for the C version. The 'awk' version's order was chosen to be + simple and to allow for reasonable 'awk'-style default + arguments. + +'dcngettext(STRING1, STRING2, NUMBER [, DOMAIN [, CATEGORY]])' + Return the plural form used for NUMBER of the translation of + STRING1 and STRING2 in text domain DOMAIN for locale category + CATEGORY. STRING1 is the English singular variant of a message, + and STRING2 is the English plural variant of the same message. The + default value for DOMAIN is the current value of 'TEXTDOMAIN'. The + default value for CATEGORY is '"LC_MESSAGES"'. + + The same remarks about argument order as for the 'dcgettext()' + function apply. + +'bindtextdomain(DIRECTORY [, DOMAIN ])' + Change the directory in which 'gettext' looks for '.gmo' files, in + case they will not or cannot be placed in the standard locations + (e.g., during testing). Return the directory in which DOMAIN is + "bound." + + The default DOMAIN is the value of 'TEXTDOMAIN'. If DIRECTORY is + the null string ('""'), then 'bindtextdomain()' returns the current + binding for the given DOMAIN. + + To use these facilities in your 'awk' program, follow these steps: + + 1. Set the variable 'TEXTDOMAIN' to the text domain of your program. + This is best done in a 'BEGIN' rule (*note BEGIN/END::), or it can + also be done via the '-v' command-line option (*note Options::): + + BEGIN { + TEXTDOMAIN = "guide" + ... + } + + 2. Mark all translatable strings with a leading underscore ('_') + character. It _must_ be adjacent to the opening quote of the + string. For example: + + print _"hello, world" + x = _"you goofed" + printf(_"Number of users is %d\n", nusers) + + 3. If you are creating strings dynamically, you can still translate + them, using the 'dcgettext()' built-in function:(1) + + if (groggy) + message = dcgettext("%d customers disturbing me\n", "adminprog") + else + message = dcgettext("enjoying %d customers\n", "adminprog") + printf(message, ncustomers) + + Here, the call to 'dcgettext()' supplies a different text domain + ('"adminprog"') in which to find the message, but it uses the + default '"LC_MESSAGES"' category. + + The previous example only works if 'ncustomers' is greater than + one. This example would be better done with 'dcngettext()': + + if (groggy) + message = dcngettext("%d customer disturbing me\n", + "%d customers disturbing me\n", "adminprog") + else + message = dcngettext("enjoying %d customer\n", + "enjoying %d customers\n", "adminprog") + printf(message, ncustomers) + + 4. During development, you might want to put the '.gmo' file in a + private directory for testing. This is done with the + 'bindtextdomain()' built-in function: + + BEGIN { + TEXTDOMAIN = "guide" # our text domain + if (Testing) { + # where to find our files + bindtextdomain("testdir") + # joe is in charge of adminprog + bindtextdomain("../joe/testdir", "adminprog") + } + ... + } + + *Note I18N Example:: for an example program showing the steps to +create and use translations from 'awk'. + + ---------- Footnotes ---------- + + (1) Thanks to Bruno Haible for this example. + + +File: gawk.info, Node: Translator i18n, Next: I18N Example, Prev: Programmer i18n, Up: Internationalization + +13.4 Translating 'awk' Programs +=============================== + +Once a program's translatable strings have been marked, they must be +extracted to create the initial '.pot' file. As part of translation, it +is often helpful to rearrange the order in which arguments to 'printf' +are output. + + 'gawk''s '--gen-pot' command-line option extracts the messages and is +discussed next. After that, 'printf''s ability to rearrange the order +for 'printf' arguments at runtime is covered. + +* Menu: + +* String Extraction:: Extracting marked strings. +* Printf Ordering:: Rearranging 'printf' arguments. +* I18N Portability:: 'awk'-level portability issues. + + +File: gawk.info, Node: String Extraction, Next: Printf Ordering, Up: Translator i18n + +13.4.1 Extracting Marked Strings +-------------------------------- + +Once your 'awk' program is working, and all the strings have been marked +and you've set (and perhaps bound) the text domain, it is time to +produce translations. First, use the '--gen-pot' command-line option to +create the initial '.pot' file: + + gawk --gen-pot -f guide.awk > guide.pot + + When run with '--gen-pot', 'gawk' does not execute your program. +Instead, it parses it as usual and prints all marked strings to standard +output in the format of a GNU 'gettext' Portable Object file. Also +included in the output are any constant strings that appear as the first +argument to 'dcgettext()' or as the first and second argument to +'dcngettext()'.(1) You should distribute the generated '.pot' file with +your 'awk' program; translators will eventually use it to provide you +translations that you can also then distribute. *Note I18N Example:: +for the full list of steps to go through to create and test translations +for 'guide'. + + ---------- Footnotes ---------- + + (1) The 'xgettext' utility that comes with GNU 'gettext' can handle +'.awk' files. + + +File: gawk.info, Node: Printf Ordering, Next: I18N Portability, Prev: String Extraction, Up: Translator i18n + +13.4.2 Rearranging 'printf' Arguments +------------------------------------- + +Format strings for 'printf' and 'sprintf()' (*note Printf::) present a +special problem for translation. Consider the following:(1) + + printf(_"String `%s' has %d characters\n", + string, length(string))) + + A possible German translation for this might be: + + "%d Zeichen lang ist die Zeichenkette `%s'\n" + + The problem should be obvious: the order of the format specifications +is different from the original! Even though 'gettext()' can return the +translated string at runtime, it cannot change the argument order in the +call to 'printf'. + + To solve this problem, 'printf' format specifiers may have an +additional optional element, which we call a "positional specifier". +For example: + + "%2$d Zeichen lang ist die Zeichenkette `%1$s'\n" + + Here, the positional specifier consists of an integer count, which +indicates which argument to use, and a '$'. Counts are one-based, and +the format string itself is _not_ included. Thus, in the following +example, 'string' is the first argument and 'length(string)' is the +second: + + $ gawk 'BEGIN { + > string = "Don\47t Panic" + > printf "%2$d characters live in \"%1$s\"\n", + > string, length(string) + > }' + -| 11 characters live in "Don't Panic" + + If present, positional specifiers come first in the format +specification, before the flags, the field width, and/or the precision. + + Positional specifiers can be used with the dynamic field width and +precision capability: + + $ gawk 'BEGIN { + > printf("%*.*s\n", 10, 20, "hello") + > printf("%3$*2$.*1$s\n", 20, 10, "hello") + > }' + -| hello + -| hello + + NOTE: When using '*' with a positional specifier, the '*' comes + first, then the integer position, and then the '$'. This is + somewhat counterintuitive. + + 'gawk' does not allow you to mix regular format specifiers and those +with positional specifiers in the same string: + + $ gawk 'BEGIN { printf "%d %3$s\n", 1, 2, "hi" }' + error-> gawk: cmd. line:1: fatal: must use `count$' on all formats or none + + NOTE: There are some pathological cases that 'gawk' may fail to + diagnose. In such cases, the output may not be what you expect. + It's still a bad idea to try mixing them, even if 'gawk' doesn't + detect it. + + Although positional specifiers can be used directly in 'awk' +programs, their primary purpose is to help in producing correct +translations of format strings into languages different from the one in +which the program is first written. + + ---------- Footnotes ---------- + + (1) This example is borrowed from the GNU 'gettext' manual. + + +File: gawk.info, Node: I18N Portability, Prev: Printf Ordering, Up: Translator i18n + +13.4.3 'awk' Portability Issues +------------------------------- + +'gawk''s internationalization features were purposely chosen to have as +little impact as possible on the portability of 'awk' programs that use +them to other versions of 'awk'. Consider this program: + + BEGIN { + TEXTDOMAIN = "guide" + if (Test_Guide) # set with -v + bindtextdomain("/test/guide/messages") + print _"don't panic!" + } + +As written, it won't work on other versions of 'awk'. However, it is +actually almost portable, requiring very little change: + + * Assignments to 'TEXTDOMAIN' won't have any effect, because + 'TEXTDOMAIN' is not special in other 'awk' implementations. + + * Non-GNU versions of 'awk' treat marked strings as the concatenation + of a variable named '_' with the string following it.(1) + Typically, the variable '_' has the null string ('""') as its + value, leaving the original string constant as the result. + + * By defining "dummy" functions to replace 'dcgettext()', + 'dcngettext()', and 'bindtextdomain()', the 'awk' program can be + made to run, but all the messages are output in the original + language. For example: + + function bindtextdomain(dir, domain) + { + return dir + } + + function dcgettext(string, domain, category) + { + return string + } + + function dcngettext(string1, string2, number, domain, category) + { + return (number == 1 ? string1 : string2) + } + + * The use of positional specifications in 'printf' or 'sprintf()' is + _not_ portable. To support 'gettext()' at the C level, many + systems' C versions of 'sprintf()' do support positional + specifiers. But it works only if enough arguments are supplied in + the function call. Many versions of 'awk' pass 'printf' formats + and arguments unchanged to the underlying C library version of + 'sprintf()', but only one format and argument at a time. What + happens if a positional specification is used is anybody's guess. + However, because the positional specifications are primarily for + use in _translated_ format strings, and because non-GNU 'awk's + never retrieve the translated string, this should not be a problem + in practice. + + ---------- Footnotes ---------- + + (1) This is good fodder for an "Obfuscated 'awk'" contest. + + +File: gawk.info, Node: I18N Example, Next: Gawk I18N, Prev: Translator i18n, Up: Internationalization + +13.5 A Simple Internationalization Example +========================================== + +Now let's look at a step-by-step example of how to internationalize and +localize a simple 'awk' program, using 'guide.awk' as our original +source: + + BEGIN { + TEXTDOMAIN = "guide" + bindtextdomain(".") # for testing + print _"Don't Panic" + print _"The Answer Is", 42 + print "Pardon me, Zaphod who?" + } + +Run 'gawk --gen-pot' to create the '.pot' file: + + $ gawk --gen-pot -f guide.awk > guide.pot + +This produces: + + #: guide.awk:4 + msgid "Don't Panic" + msgstr "" + + #: guide.awk:5 + msgid "The Answer Is" + msgstr "" + + + This original portable object template file is saved and reused for +each language into which the application is translated. The 'msgid' is +the original string and the 'msgstr' is the translation. + + NOTE: Strings not marked with a leading underscore do not appear in + the 'guide.pot' file. + + Next, the messages must be translated. Here is a translation to a +hypothetical dialect of English, called "Mellow":(1) + + $ cp guide.pot guide-mellow.po + ADD TRANSLATIONS TO guide-mellow.po ... + +Following are the translations: + + #: guide.awk:4 + msgid "Don't Panic" + msgstr "Hey man, relax!" + + #: guide.awk:5 + msgid "The Answer Is" + msgstr "Like, the scoop is" + + + The next step is to make the directory to hold the binary message +object file and then to create the 'guide.mo' file. We pretend that our +file is to be used in the 'en_US.UTF-8' locale, because we have to use a +locale name known to the C 'gettext' routines. The directory layout +shown here is standard for GNU 'gettext' on GNU/Linux systems. Other +versions of 'gettext' may use a different layout: + + $ mkdir en_US.UTF-8 en_US.UTF-8/LC_MESSAGES + + The 'msgfmt' utility does the conversion from human-readable '.po' +file to machine-readable '.mo' file. By default, 'msgfmt' creates a +file named 'messages'. This file must be renamed and placed in the +proper directory (using the '-o' option) so that 'gawk' can find it: + + $ msgfmt guide-mellow.po -o en_US.UTF-8/LC_MESSAGES/guide.mo + + Finally, we run the program to test it: + + $ gawk -f guide.awk + -| Hey man, relax! + -| Like, the scoop is 42 + -| Pardon me, Zaphod who? + + If the three replacement functions for 'dcgettext()', 'dcngettext()', +and 'bindtextdomain()' (*note I18N Portability::) are in a file named +'libintl.awk', then we can run 'guide.awk' unchanged as follows: + + $ gawk --posix -f guide.awk -f libintl.awk + -| Don't Panic + -| The Answer Is 42 + -| Pardon me, Zaphod who? + + ---------- Footnotes ---------- + + (1) Perhaps it would be better if it were called "Hippy." Ah, well. + + +File: gawk.info, Node: Gawk I18N, Next: I18N Summary, Prev: I18N Example, Up: Internationalization + +13.6 'gawk' Can Speak Your Language +=================================== + +'gawk' itself has been internationalized using the GNU 'gettext' +package. (GNU 'gettext' is described in complete detail in *note (GNU +'gettext' utilities, gettext, GNU 'gettext' utilities)Top::.) As of +this writing, the latest version of GNU 'gettext' is version 0.19.4 +(ftp://ftp.gnu.org/gnu/gettext/gettext-0.19.4.tar.gz). + + If a translation of 'gawk''s messages exists, then 'gawk' produces +usage messages, warnings, and fatal errors in the local language. + + +File: gawk.info, Node: I18N Summary, Prev: Gawk I18N, Up: Internationalization + +13.7 Summary +============ + + * Internationalization means writing a program such that it can use + multiple languages without requiring source code changes. + Localization means providing the data necessary for an + internationalized program to work in a particular language. + + * 'gawk' uses GNU 'gettext' to let you internationalize and localize + 'awk' programs. A program's text domain identifies the program for + grouping all messages and other data together. + + * You mark a program's strings for translation by preceding them with + an underscore. Once that is done, the strings are extracted into a + '.pot' file. This file is copied for each language into a '.po' + file, and the '.po' files are compiled into '.gmo' files for use at + runtime. + + * You can use positional specifications with 'sprintf()' and 'printf' + to rearrange the placement of argument values in formatted strings + and output. This is useful for the translation of format control + strings. + + * The internationalization features have been designed so that they + can be easily worked around in a standard 'awk'. + + * 'gawk' itself has been internationalized and ships with a number of + translations for its messages. + + +File: gawk.info, Node: Debugger, Next: Arbitrary Precision Arithmetic, Prev: Internationalization, Up: Top + +14 Debugging 'awk' Programs +*************************** + +It would be nice if computer programs worked perfectly the first time +they were run, but in real life, this rarely happens for programs of any +complexity. Thus, most programming languages have facilities available +for "debugging" programs, and now 'awk' is no exception. + + The 'gawk' debugger is purposely modeled after the GNU Debugger (GDB) +(http://www.gnu.org/software/gdb/) command-line debugger. If you are +familiar with GDB, learning how to use 'gawk' for debugging your program +is easy. + +* Menu: + +* Debugging:: Introduction to 'gawk' debugger. +* Sample Debugging Session:: Sample debugging session. +* List of Debugger Commands:: Main debugger commands. +* Readline Support:: Readline support. +* Limitations:: Limitations and future plans. +* Debugging Summary:: Debugging summary. + + +File: gawk.info, Node: Debugging, Next: Sample Debugging Session, Up: Debugger + +14.1 Introduction to the 'gawk' Debugger +======================================== + +This minor node introduces debugging in general and begins the +discussion of debugging in 'gawk'. + +* Menu: + +* Debugging Concepts:: Debugging in General. +* Debugging Terms:: Additional Debugging Concepts. +* Awk Debugging:: Awk Debugging. + + +File: gawk.info, Node: Debugging Concepts, Next: Debugging Terms, Up: Debugging + +14.1.1 Debugging in General +--------------------------- + +(If you have used debuggers in other languages, you may want to skip +ahead to *note Awk Debugging::.) + + Of course, a debugging program cannot remove bugs for you, because it +has no way of knowing what you or your users consider a "bug" versus a +"feature." (Sometimes, we humans have a hard time with this ourselves.) +In that case, what can you expect from such a tool? The answer to that +depends on the language being debugged, but in general, you can expect +at least the following: + + * The ability to watch a program execute its instructions one by one, + giving you, the programmer, the opportunity to think about what is + happening on a time scale of seconds, minutes, or hours, rather + than the nanosecond time scale at which the code usually runs. + + * The opportunity to not only passively observe the operation of your + program, but to control it and try different paths of execution, + without having to change your source files. + + * The chance to see the values of data in the program at any point in + execution, and also to change that data on the fly, to see how that + affects what happens afterward. (This often includes the ability + to look at internal data structures besides the variables you + actually defined in your code.) + + * The ability to obtain additional information about your program's + state or even its internal structure. + + All of these tools provide a great amount of help in using your own +skills and understanding of the goals of your program to find where it +is going wrong (or, for that matter, to better comprehend a perfectly +functional program that you or someone else wrote). + + +File: gawk.info, Node: Debugging Terms, Next: Awk Debugging, Prev: Debugging Concepts, Up: Debugging + +14.1.2 Debugging Concepts +------------------------- + +Before diving in to the details, we need to introduce several important +concepts that apply to just about all debuggers. The following list +defines terms used throughout the rest of this major node: + +"Stack frame" + Programs generally call functions during the course of their + execution. One function can call another, or a function can call + itself (recursion). You can view the chain of called functions + (main program calls A, which calls B, which calls C), as a stack of + executing functions: the currently running function is the topmost + one on the stack, and when it finishes (returns), the next one down + then becomes the active function. Such a stack is termed a "call + stack". + + For each function on the call stack, the system maintains a data + area that contains the function's parameters, local variables, and + return value, as well as any other "bookkeeping" information needed + to manage the call stack. This data area is termed a "stack + frame". + + 'gawk' also follows this model, and gives you access to the call + stack and to each stack frame. You can see the call stack, as well + as from where each function on the stack was invoked. Commands + that print the call stack print information about each stack frame + (as detailed later on). + +"Breakpoint" + During debugging, you often wish to let the program run until it + reaches a certain point, and then continue execution from there one + statement (or instruction) at a time. The way to do this is to set + a "breakpoint" within the program. A breakpoint is where the + execution of the program should break off (stop), so that you can + take over control of the program's execution. You can add and + remove as many breakpoints as you like. + +"Watchpoint" + A watchpoint is similar to a breakpoint. The difference is that + breakpoints are oriented around the code: stop when a certain point + in the code is reached. A watchpoint, however, specifies that + program execution should stop when a _data value_ is changed. This + is useful, as sometimes it happens that a variable receives an + erroneous value, and it's hard to track down where this happens + just by looking at the code. By using a watchpoint, you can stop + whenever a variable is assigned to, and usually find the errant + code quite quickly. + + +File: gawk.info, Node: Awk Debugging, Prev: Debugging Terms, Up: Debugging + +14.1.3 'awk' Debugging +---------------------- + +Debugging an 'awk' program has some specific aspects that are not shared +with programs written in other languages. + + First of all, the fact that 'awk' programs usually take input line by +line from a file or files and operate on those lines using specific +rules makes it especially useful to organize viewing the execution of +the program in terms of these rules. As we will see, each 'awk' rule is +treated almost like a function call, with its own specific block of +instructions. + + In addition, because 'awk' is by design a very concise language, it +is easy to lose sight of everything that is going on "inside" each line +of 'awk' code. The debugger provides the opportunity to look at the +individual primitive instructions carried out by the higher-level 'awk' +commands. + + +File: gawk.info, Node: Sample Debugging Session, Next: List of Debugger Commands, Prev: Debugging, Up: Debugger + +14.2 Sample 'gawk' Debugging Session +==================================== + +In order to illustrate the use of 'gawk' as a debugger, let's look at a +sample debugging session. We will use the 'awk' implementation of the +POSIX 'uniq' command described earlier (*note Uniq Program::) as our +example. + +* Menu: + +* Debugger Invocation:: How to Start the Debugger. +* Finding The Bug:: Finding the Bug. + + +File: gawk.info, Node: Debugger Invocation, Next: Finding The Bug, Up: Sample Debugging Session + +14.2.1 How to Start the Debugger +-------------------------------- + +Starting the debugger is almost exactly like running 'gawk' normally, +except you have to pass an additional option, '--debug', or the +corresponding short option, '-D'. The file(s) containing the program +and any supporting code are given on the command line as arguments to +one or more '-f' options. ('gawk' is not designed to debug command-line +programs, only programs contained in files.) In our case, we invoke the +debugger like this: + + $ gawk -D -f getopt.awk -f join.awk -f uniq.awk -1 inputfile + +where both 'getopt.awk' and 'uniq.awk' are in '$AWKPATH'. (Experienced +users of GDB or similar debuggers should note that this syntax is +slightly different from what you are used to. With the 'gawk' debugger, +you give the arguments for running the program in the command line to +the debugger rather than as part of the 'run' command at the debugger +prompt.) The '-1' is an option to 'uniq.awk'. + + Instead of immediately running the program on 'inputfile', as 'gawk' +would ordinarily do, the debugger merely loads all the program source +files, compiles them internally, and then gives us a prompt: + + gawk> + +from which we can issue commands to the debugger. At this point, no +code has been executed. + + +File: gawk.info, Node: Finding The Bug, Prev: Debugger Invocation, Up: Sample Debugging Session + +14.2.2 Finding the Bug +---------------------- + +Let's say that we are having a problem using (a faulty version of) +'uniq.awk' in the "field-skipping" mode, and it doesn't seem to be +catching lines which should be identical when skipping the first field, +such as: + + awk is a wonderful program! + gawk is a wonderful program! + + This could happen if we were thinking (C-like) of the fields in a +record as being numbered in a zero-based fashion, so instead of the +lines: + + clast = join(alast, fcount+1, n) + cline = join(aline, fcount+1, m) + +we wrote: + + clast = join(alast, fcount, n) + cline = join(aline, fcount, m) + + The first thing we usually want to do when trying to investigate a +problem like this is to put a breakpoint in the program so that we can +watch it at work and catch what it is doing wrong. A reasonable spot +for a breakpoint in 'uniq.awk' is at the beginning of the function +'are_equal()', which compares the current line with the previous one. +To set the breakpoint, use the 'b' (breakpoint) command: + + gawk> b are_equal + -| Breakpoint 1 set at file `awklib/eg/prog/uniq.awk', line 63 + + The debugger tells us the file and line number where the breakpoint +is. Now type 'r' or 'run' and the program runs until it hits the +breakpoint for the first time: + + gawk> r + -| Starting program: + -| Stopping in Rule ... + -| Breakpoint 1, are_equal(n, m, clast, cline, alast, aline) + at `awklib/eg/prog/uniq.awk':63 + -| 63 if (fcount == 0 && charcount == 0) + gawk> + + Now we can look at what's going on inside our program. First of all, +let's see how we got to where we are. At the prompt, we type 'bt' +(short for "backtrace"), and the debugger responds with a listing of the +current stack frames: + + gawk> bt + -| #0 are_equal(n, m, clast, cline, alast, aline) + at `awklib/eg/prog/uniq.awk':68 + -| #1 in main() at `awklib/eg/prog/uniq.awk':88 + + This tells us that 'are_equal()' was called by the main program at +line 88 of 'uniq.awk'. (This is not a big surprise, because this is the +only call to 'are_equal()' in the program, but in more complex programs, +knowing who called a function and with what parameters can be the key to +finding the source of the problem.) + + Now that we're in 'are_equal()', we can start looking at the values +of some variables. Let's say we type 'p n' ('p' is short for "print"). +We would expect to see the value of 'n', a parameter to 'are_equal()'. +Actually, the debugger gives us: + + gawk> p n + -| n = untyped variable + +In this case, 'n' is an uninitialized local variable, because the +function was called without arguments (*note Function Calls::). + + A more useful variable to display might be the current record: + + gawk> p $0 + -| $0 = "gawk is a wonderful program!" + +This might be a bit puzzling at first, as this is the second line of our +test input. Let's look at 'NR': + + gawk> p NR + -| NR = 2 + +So we can see that 'are_equal()' was only called for the second record +of the file. Of course, this is because our program contains a rule for +'NR == 1': + + NR == 1 { + last = $0 + next + } + + OK, let's just check that that rule worked correctly: + + gawk> p last + -| last = "awk is a wonderful program!" + + Everything we have done so far has verified that the program has +worked as planned, up to and including the call to 'are_equal()', so the +problem must be inside this function. To investigate further, we must +begin "stepping through" the lines of 'are_equal()'. We start by typing +'n' (for "next"): + + gawk> n + -| 66 if (fcount > 0) { + + This tells us that 'gawk' is now ready to execute line 66, which +decides whether to give the lines the special "field-skipping" treatment +indicated by the '-1' command-line option. (Notice that we skipped from +where we were before, at line 63, to here, because the condition in line +63, 'if (fcount == 0 && charcount == 0)', was false.) + + Continuing to step, we now get to the splitting of the current and +last records: + + gawk> n + -| 67 n = split(last, alast) + gawk> n + -| 68 m = split($0, aline) + + At this point, we should be curious to see what our records were +split into, so we try to look: + + gawk> p n m alast aline + -| n = 5 + -| m = untyped variable + -| alast = array, 5 elements + -| aline = untyped variable + +(The 'p' command can take more than one argument, similar to 'awk''s +'print' statement.) + + This is kind of disappointing, though. All we found out is that +there are five elements in 'alast'; 'm' and 'aline' don't have values +because we are at line 68 but haven't executed it yet. This information +is useful enough (we now know that none of the words were accidentally +left out), but what if we want to see inside the array? + + The first choice would be to use subscripts: + + gawk> p alast[0] + -| "0" not in array `alast' + +Oops! + + gawk> p alast[1] + -| alast["1"] = "awk" + + This would be kind of slow for a 100-member array, though, so 'gawk' +provides a shortcut (reminiscent of another language not to be +mentioned): + + gawk> p @alast + -| alast["1"] = "awk" + -| alast["2"] = "is" + -| alast["3"] = "a" + -| alast["4"] = "wonderful" + -| alast["5"] = "program!" + + It looks like we got this far OK. Let's take another step or two: + + gawk> n + -| 69 clast = join(alast, fcount, n) + gawk> n + -| 70 cline = join(aline, fcount, m) + + Well, here we are at our error (sorry to spoil the suspense). What +we had in mind was to join the fields starting from the second one to +make the virtual record to compare, and if the first field were numbered +zero, this would work. Let's look at what we've got: + + gawk> p cline clast + -| cline = "gawk is a wonderful program!" + -| clast = "awk is a wonderful program!" + + Hey, those look pretty familiar! They're just our original, +unaltered input records. A little thinking (the human brain is still +the best debugging tool), and we realize that we were off by one! + + We get out of the debugger: + + gawk> q + -| The program is running. Exit anyway (y/n)? y + +Then we get into an editor: + + clast = join(alast, fcount+1, n) + cline = join(aline, fcount+1, m) + +and problem solved! + + +File: gawk.info, Node: List of Debugger Commands, Next: Readline Support, Prev: Sample Debugging Session, Up: Debugger + +14.3 Main Debugger Commands +=========================== + +The 'gawk' debugger command set can be divided into the following +categories: + + * Breakpoint control + + * Execution control + + * Viewing and changing data + + * Working with the stack + + * Getting information + + * Miscellaneous + + Each of these are discussed in the following subsections. In the +following descriptions, commands that may be abbreviated show the +abbreviation on a second description line. A debugger command name may +also be truncated if that partial name is unambiguous. The debugger has +the built-in capability to automatically repeat the previous command +just by hitting 'Enter'. This works for the commands 'list', 'next', +'nexti', 'step', 'stepi', and 'continue' executed without any argument. + +* Menu: + +* Breakpoint Control:: Control of Breakpoints. +* Debugger Execution Control:: Control of Execution. +* Viewing And Changing Data:: Viewing and Changing Data. +* Execution Stack:: Dealing with the Stack. +* Debugger Info:: Obtaining Information about the Program and + the Debugger State. +* Miscellaneous Debugger Commands:: Miscellaneous Commands. + + +File: gawk.info, Node: Breakpoint Control, Next: Debugger Execution Control, Up: List of Debugger Commands + +14.3.1 Control of Breakpoints +----------------------------- + +As we saw earlier, the first thing you probably want to do in a +debugging session is to get your breakpoints set up, because your +program will otherwise just run as if it was not under the debugger. +The commands for controlling breakpoints are: + +'break' [[FILENAME':']N | FUNCTION] ['"EXPRESSION"'] +'b' [[FILENAME':']N | FUNCTION] ['"EXPRESSION"'] + Without any argument, set a breakpoint at the next instruction to + be executed in the selected stack frame. Arguments can be one of + the following: + + N + Set a breakpoint at line number N in the current source file. + + FILENAME':'N + Set a breakpoint at line number N in source file FILENAME. + + FUNCTION + Set a breakpoint at entry to (the first instruction of) + function FUNCTION. + + Each breakpoint is assigned a number that can be used to delete it + from the breakpoint list using the 'delete' command. + + With a breakpoint, you may also supply a condition. This is an + 'awk' expression (enclosed in double quotes) that the debugger + evaluates whenever the breakpoint is reached. If the condition is + true, then the debugger stops execution and prompts for a command. + Otherwise, it continues executing the program. + +'clear' [[FILENAME':']N | FUNCTION] + Without any argument, delete any breakpoint at the next instruction + to be executed in the selected stack frame. If the program stops + at a breakpoint, this deletes that breakpoint so that the program + does not stop at that location again. Arguments can be one of the + following: + + N + Delete breakpoint(s) set at line number N in the current + source file. + + FILENAME':'N + Delete breakpoint(s) set at line number N in source file + FILENAME. + + FUNCTION + Delete breakpoint(s) set at entry to function FUNCTION. + +'condition' N '"EXPRESSION"' + Add a condition to existing breakpoint or watchpoint N. The + condition is an 'awk' expression _enclosed in double quotes_ that + the debugger evaluates whenever the breakpoint or watchpoint is + reached. If the condition is true, then the debugger stops + execution and prompts for a command. Otherwise, the debugger + continues executing the program. If the condition expression is + not specified, any existing condition is removed (i.e., the + breakpoint or watchpoint is made unconditional). + +'delete' [N1 N2 ...] [N-M] +'d' [N1 N2 ...] [N-M] + Delete specified breakpoints or a range of breakpoints. Delete all + defined breakpoints if no argument is supplied. + +'disable' [N1 N2 ... | N-M] + Disable specified breakpoints or a range of breakpoints. Without + any argument, disable all breakpoints. + +'enable' ['del' | 'once'] [N1 N2 ...] [N-M] +'e' ['del' | 'once'] [N1 N2 ...] [N-M] + Enable specified breakpoints or a range of breakpoints. Without + any argument, enable all breakpoints. Optionally, you can specify + how to enable the breakpoints: + + 'del' + Enable the breakpoints temporarily, then delete each one when + the program stops at it. + + 'once' + Enable the breakpoints temporarily, then disable each one when + the program stops at it. + +'ignore' N COUNT + Ignore breakpoint number N the next COUNT times it is hit. + +'tbreak' [[FILENAME':']N | FUNCTION] +'t' [[FILENAME':']N | FUNCTION] + Set a temporary breakpoint (enabled for only one stop). The + arguments are the same as for 'break'. + + +File: gawk.info, Node: Debugger Execution Control, Next: Viewing And Changing Data, Prev: Breakpoint Control, Up: List of Debugger Commands + +14.3.2 Control of Execution +--------------------------- + +Now that your breakpoints are ready, you can start running the program +and observing its behavior. There are more commands for controlling +execution of the program than we saw in our earlier example: + +'commands' [N] +'silent' +... +'end' + Set a list of commands to be executed upon stopping at a breakpoint + or watchpoint. N is the breakpoint or watchpoint number. Without + a number, the last one set is used. The actual commands follow, + starting on the next line, and terminated by the 'end' command. If + the command 'silent' is in the list, the usual messages about + stopping at a breakpoint and the source line are not printed. Any + command in the list that resumes execution (e.g., 'continue') + terminates the list (an implicit 'end'), and subsequent commands + are ignored. For example: + + gawk> commands + > silent + > printf "A silent breakpoint; i = %d\n", i + > info locals + > set i = 10 + > continue + > end + gawk> + +'continue' [COUNT] +'c' [COUNT] + Resume program execution. If continued from a breakpoint and COUNT + is specified, ignore the breakpoint at that location the next COUNT + times before stopping. + +'finish' + Execute until the selected stack frame returns. Print the returned + value. + +'next' [COUNT] +'n' [COUNT] + Continue execution to the next source line, stepping over function + calls. The argument COUNT controls how many times to repeat the + action, as in 'step'. + +'nexti' [COUNT] +'ni' [COUNT] + Execute one (or COUNT) instruction(s), stepping over function + calls. + +'return' [VALUE] + Cancel execution of a function call. If VALUE (either a string or + a number) is specified, it is used as the function's return value. + If used in a frame other than the innermost one (the currently + executing function; i.e., frame number 0), discard all inner frames + in addition to the selected one, and the caller of that frame + becomes the innermost frame. + +'run' +'r' + Start/restart execution of the program. When restarting, the + debugger retains the current breakpoints, watchpoints, command + history, automatic display variables, and debugger options. + +'step' [COUNT] +'s' [COUNT] + Continue execution until control reaches a different source line in + the current stack frame, stepping inside any function called within + the line. If the argument COUNT is supplied, steps that many times + before stopping, unless it encounters a breakpoint or watchpoint. + +'stepi' [COUNT] +'si' [COUNT] + Execute one (or COUNT) instruction(s), stepping inside function + calls. (For illustration of what is meant by an "instruction" in + 'gawk', see the output shown under 'dump' in *note Miscellaneous + Debugger Commands::.) + +'until' [[FILENAME':']N | FUNCTION] +'u' [[FILENAME':']N | FUNCTION] + Without any argument, continue execution until a line past the + current line in the current stack frame is reached. With an + argument, continue execution until the specified location is + reached, or the current stack frame returns. + + +File: gawk.info, Node: Viewing And Changing Data, Next: Execution Stack, Prev: Debugger Execution Control, Up: List of Debugger Commands + +14.3.3 Viewing and Changing Data +-------------------------------- + +The commands for viewing and changing variables inside of 'gawk' are: + +'display' [VAR | '$'N] + Add variable VAR (or field '$N') to the display list. The value of + the variable or field is displayed each time the program stops. + Each variable added to the list is identified by a unique number: + + gawk> display x + -| 10: x = 1 + + This displays the assigned item number, the variable name, and its + current value. If the display variable refers to a function + parameter, it is silently deleted from the list as soon as the + execution reaches a context where no such variable of the given + name exists. Without argument, 'display' displays the current + values of items on the list. + +'eval "AWK STATEMENTS"' + Evaluate AWK STATEMENTS in the context of the running program. You + can do anything that an 'awk' program would do: assign values to + variables, call functions, and so on. + +'eval' PARAM, ... +AWK STATEMENTS +'end' + This form of 'eval' is similar, but it allows you to define "local + variables" that exist in the context of the AWK STATEMENTS, instead + of using variables or function parameters defined by the program. + +'print' VAR1[',' VAR2 ...] +'p' VAR1[',' VAR2 ...] + Print the value of a 'gawk' variable or field. Fields must be + referenced by constants: + + gawk> print $3 + + This prints the third field in the input record (if the specified + field does not exist, it prints 'Null field'). A variable can be + an array element, with the subscripts being constant string values. + To print the contents of an array, prefix the name of the array + with the '@' symbol: + + gawk> print @a + + This prints the indices and the corresponding values for all + elements in the array 'a'. + +'printf' FORMAT [',' ARG ...] + Print formatted text. The FORMAT may include escape sequences, + such as '\n' (*note Escape Sequences::). No newline is printed + unless one is specified. + +'set' VAR'='VALUE + Assign a constant (number or string) value to an 'awk' variable or + field. String values must be enclosed between double quotes + ('"'...'"'). + + You can also set special 'awk' variables, such as 'FS', 'NF', 'NR', + and so on. + +'watch' VAR | '$'N ['"EXPRESSION"'] +'w' VAR | '$'N ['"EXPRESSION"'] + Add variable VAR (or field '$N') to the watch list. The debugger + then stops whenever the value of the variable or field changes. + Each watched item is assigned a number that can be used to delete + it from the watch list using the 'unwatch' command. + + With a watchpoint, you may also supply a condition. This is an + 'awk' expression (enclosed in double quotes) that the debugger + evaluates whenever the watchpoint is reached. If the condition is + true, then the debugger stops execution and prompts for a command. + Otherwise, 'gawk' continues executing the program. + +'undisplay' [N] + Remove item number N (or all items, if no argument) from the + automatic display list. + +'unwatch' [N] + Remove item number N (or all items, if no argument) from the watch + list. + + +File: gawk.info, Node: Execution Stack, Next: Debugger Info, Prev: Viewing And Changing Data, Up: List of Debugger Commands + +14.3.4 Working with the Stack +----------------------------- + +Whenever you run a program that contains any function calls, 'gawk' +maintains a stack of all of the function calls leading up to where the +program is right now. You can see how you got to where you are, and +also move around in the stack to see what the state of things was in the +functions that called the one you are in. The commands for doing this +are: + +'backtrace' [COUNT] +'bt' [COUNT] +'where' [COUNT] + Print a backtrace of all function calls (stack frames), or + innermost COUNT frames if COUNT > 0. Print the outermost COUNT + frames if COUNT < 0. The backtrace displays the name and arguments + to each function, the source file name, and the line number. The + alias 'where' for 'backtrace' is provided for longtime GDB users + who may be used to that command. + +'down' [COUNT] + Move COUNT (default 1) frames down the stack toward the innermost + frame. Then select and print the frame. + +'frame' [N] +'f' [N] + Select and print stack frame N. Frame 0 is the currently + executing, or "innermost", frame (function call); frame 1 is the + frame that called the innermost one. The highest-numbered frame is + the one for the main program. The printed information consists of + the frame number, function and argument names, source file, and the + source line. + +'up' [COUNT] + Move COUNT (default 1) frames up the stack toward the outermost + frame. Then select and print the frame. + + +File: gawk.info, Node: Debugger Info, Next: Miscellaneous Debugger Commands, Prev: Execution Stack, Up: List of Debugger Commands + +14.3.5 Obtaining Information About the Program and the Debugger State +--------------------------------------------------------------------- + +Besides looking at the values of variables, there is often a need to get +other sorts of information about the state of your program and of the +debugging environment itself. The 'gawk' debugger has one command that +provides this information, appropriately called 'info'. 'info' is used +with one of a number of arguments that tell it exactly what you want to +know: + +'info' WHAT +'i' WHAT + The value for WHAT should be one of the following: + + 'args' + List arguments of the selected frame. + + 'break' + List all currently set breakpoints. + + 'display' + List all items in the automatic display list. + + 'frame' + Give a description of the selected stack frame. + + 'functions' + List all function definitions including source file names and + line numbers. + + 'locals' + List local variables of the selected frame. + + 'source' + Print the name of the current source file. Each time the + program stops, the current source file is the file containing + the current instruction. When the debugger first starts, the + current source file is the first file included via the '-f' + option. The 'list FILENAME:LINENO' command can be used at any + time to change the current source. + + 'sources' + List all program sources. + + 'variables' + List all global variables. + + 'watch' + List all items in the watch list. + + Additional commands give you control over the debugger, the ability +to save the debugger's state, and the ability to run debugger commands +from a file. The commands are: + +'option' [NAME['='VALUE]] +'o' [NAME['='VALUE]] + Without an argument, display the available debugger options and + their current values. 'option NAME' shows the current value of the + named option. 'option NAME=VALUE' assigns a new value to the named + option. The available options are: + + 'history_size' + Set the maximum number of lines to keep in the history file + './.gawk_history'. The default is 100. + + 'listsize' + Specify the number of lines that 'list' prints. The default + is 15. + + 'outfile' + Send 'gawk' output to a file; debugger output still goes to + standard output. An empty string ('""') resets output to + standard output. + + 'prompt' + Change the debugger prompt. The default is 'gawk> '. + + 'save_history' ['on' | 'off'] + Save command history to file './.gawk_history'. The default + is 'on'. + + 'save_options' ['on' | 'off'] + Save current options to file './.gawkrc' upon exit. The + default is 'on'. Options are read back into the next session + upon startup. + + 'trace' ['on' | 'off'] + Turn instruction tracing on or off. The default is 'off'. + +'save' FILENAME + Save the commands from the current session to the given file name, + so that they can be replayed using the 'source' command. + +'source' FILENAME + Run command(s) from a file; an error in any command does not + terminate execution of subsequent commands. Comments (lines + starting with '#') are allowed in a command file. Empty lines are + ignored; they do _not_ repeat the last command. You can't restart + the program by having more than one 'run' command in the file. + Also, the list of commands may include additional 'source' + commands; however, the 'gawk' debugger will not source the same + file more than once in order to avoid infinite recursion. + + In addition to, or instead of, the 'source' command, you can use + the '-D FILE' or '--debug=FILE' command-line options to execute + commands from a file non-interactively (*note Options::). + + +File: gawk.info, Node: Miscellaneous Debugger Commands, Prev: Debugger Info, Up: List of Debugger Commands + +14.3.6 Miscellaneous Commands +----------------------------- + +There are a few more commands that do not fit into the previous +categories, as follows: + +'dump' [FILENAME] + Dump byte code of the program to standard output or to the file + named in FILENAME. This prints a representation of the internal + instructions that 'gawk' executes to implement the 'awk' commands + in a program. This can be very enlightening, as the following + partial dump of Davide Brini's obfuscated code (*note Signature + Program::) demonstrates: + + gawk> dump + -| # BEGIN + -| + -| [ 1:0xfcd340] Op_rule : [in_rule = BEGIN] [source_file = brini.awk] + -| [ 1:0xfcc240] Op_push_i : "~" [MALLOC|STRING|STRCUR] + -| [ 1:0xfcc2a0] Op_push_i : "~" [MALLOC|STRING|STRCUR] + -| [ 1:0xfcc280] Op_match : + -| [ 1:0xfcc1e0] Op_store_var : O + -| [ 1:0xfcc2e0] Op_push_i : "==" [MALLOC|STRING|STRCUR] + -| [ 1:0xfcc340] Op_push_i : "==" [MALLOC|STRING|STRCUR] + -| [ 1:0xfcc320] Op_equal : + -| [ 1:0xfcc200] Op_store_var : o + -| [ 1:0xfcc380] Op_push : o + -| [ 1:0xfcc360] Op_plus_i : 0 [MALLOC|NUMCUR|NUMBER] + -| [ 1:0xfcc220] Op_push_lhs : o [do_reference = true] + -| [ 1:0xfcc300] Op_assign_plus : + -| [ :0xfcc2c0] Op_pop : + -| [ 1:0xfcc400] Op_push : O + -| [ 1:0xfcc420] Op_push_i : "" [MALLOC|STRING|STRCUR] + -| [ :0xfcc4a0] Op_no_op : + -| [ 1:0xfcc480] Op_push : O + -| [ :0xfcc4c0] Op_concat : [expr_count = 3] [concat_flag = 0] + -| [ 1:0xfcc3c0] Op_store_var : x + -| [ 1:0xfcc440] Op_push_lhs : X [do_reference = true] + -| [ 1:0xfcc3a0] Op_postincrement : + -| [ 1:0xfcc4e0] Op_push : x + -| [ 1:0xfcc540] Op_push : o + -| [ 1:0xfcc500] Op_plus : + -| [ 1:0xfcc580] Op_push : o + -| [ 1:0xfcc560] Op_plus : + -| [ 1:0xfcc460] Op_leq : + -| [ :0xfcc5c0] Op_jmp_false : [target_jmp = 0xfcc5e0] + -| [ 1:0xfcc600] Op_push_i : "%c" [MALLOC|STRING|STRCUR] + -| [ :0xfcc660] Op_no_op : + -| [ 1:0xfcc520] Op_assign_concat : c + -| [ :0xfcc620] Op_jmp : [target_jmp = 0xfcc440] + -| + ... + -| + -| [ 2:0xfcc5a0] Op_K_printf : [expr_count = 17] [redir_type = ""] + -| [ :0xfcc140] Op_no_op : + -| [ :0xfcc1c0] Op_atexit : + -| [ :0xfcc640] Op_stop : + -| [ :0xfcc180] Op_no_op : + -| [ :0xfcd150] Op_after_beginfile : + -| [ :0xfcc160] Op_no_op : + -| [ :0xfcc1a0] Op_after_endfile : + gawk> + +'exit' + Exit the debugger. See the entry for 'quit', later in this list. + +'help' +'h' + Print a list of all of the 'gawk' debugger commands with a short + summary of their usage. 'help COMMAND' prints the information + about the command COMMAND. + +'list' ['-' | '+' | N | FILENAME':'N | N-M | FUNCTION] +'l' ['-' | '+' | N | FILENAME':'N | N-M | FUNCTION] + Print the specified lines (default 15) from the current source file + or the file named FILENAME. The possible arguments to 'list' are + as follows: + + '-' (Minus) + Print lines before the lines last printed. + + '+' + Print lines after the lines last printed. 'list' without any + argument does the same thing. + + N + Print lines centered around line number N. + + N-M + Print lines from N to M. + + FILENAME':'N + Print lines centered around line number N in source file + FILENAME. This command may change the current source file. + + FUNCTION + Print lines centered around the beginning of the function + FUNCTION. This command may change the current source file. + +'quit' +'q' + Exit the debugger. Debugging is great fun, but sometimes we all + have to tend to other obligations in life, and sometimes we find + the bug and are free to go on to the next one! As we saw earlier, + if you are running a program, the debugger warns you when you type + 'q' or 'quit', to make sure you really want to quit. + +'trace' ['on' | 'off'] + Turn on or off continuous printing of the instructions that are + about to be executed, along with the 'awk' lines they implement. + The default is 'off'. + + It is to be hoped that most of the "opcodes" in these instructions + are fairly self-explanatory, and using 'stepi' and 'nexti' while + 'trace' is on will make them into familiar friends. + + +File: gawk.info, Node: Readline Support, Next: Limitations, Prev: List of Debugger Commands, Up: Debugger + +14.4 Readline Support +===================== + +If 'gawk' is compiled with the GNU Readline library +(http://cnswww.cns.cwru.edu/php/chet/readline/readline.html), you can +take advantage of that library's command completion and history +expansion features. The following types of completion are available: + +Command completion + Command names. + +Source file name completion + Source file names. Relevant commands are 'break', 'clear', 'list', + 'tbreak', and 'until'. + +Argument completion + Non-numeric arguments to a command. Relevant commands are 'enable' + and 'info'. + +Variable name completion + Global variable names, and function arguments in the current + context if the program is running. Relevant commands are + 'display', 'print', 'set', and 'watch'. + + +File: gawk.info, Node: Limitations, Next: Debugging Summary, Prev: Readline Support, Up: Debugger + +14.5 Limitations +================ + +We hope you find the 'gawk' debugger useful and enjoyable to work with, +but as with any program, especially in its early releases, it still has +some limitations. A few that it's worth being aware of are: + + * At this point, the debugger does not give a detailed explanation of + what you did wrong when you type in something it doesn't like. + Rather, it just responds 'syntax error'. When you do figure out + what your mistake was, though, you'll feel like a real guru. + + * If you perused the dump of opcodes in *note Miscellaneous Debugger + Commands:: (or if you are already familiar with 'gawk' internals), + you will realize that much of the internal manipulation of data in + 'gawk', as in many interpreters, is done on a stack. 'Op_push', + 'Op_pop', and the like are the "bread and butter" of most 'gawk' + code. + + Unfortunately, as of now, the 'gawk' debugger does not allow you to + examine the stack's contents. That is, the intermediate results of + expression evaluation are on the stack, but cannot be printed. + Rather, only variables that are defined in the program can be + printed. Of course, a workaround for this is to use more explicit + variables at the debugging stage and then change back to obscure, + perhaps more optimal code later. + + * There is no way to look "inside" the process of compiling regular + expressions to see if you got it right. As an 'awk' programmer, + you are expected to know the meaning of '/[^[:alnum:][:blank:]]/'. + + * The 'gawk' debugger is designed to be used by running a program + (with all its parameters) on the command line, as described in + *note Debugger Invocation::. There is no way (as of now) to attach + or "break into" a running program. This seems reasonable for a + language that is used mainly for quickly executing, short programs. + + * The 'gawk' debugger only accepts source code supplied with the '-f' + option. + + One other point is worth discussing. Conventional debuggers run in a +separate process (and thus address space) from the programs that they +debug (the "debuggee", if you will). + + The 'gawk' debugger is different; it is an integrated part of 'gawk' +itself. This makes it possible, in rare cases, for 'gawk' to become an +excellent demonstrator of Heisenberg Uncertainty physics, where the mere +act of observing something can change it. Consider the following:(1) + + $ cat test.awk + -| { print typeof($1), typeof($2) } + $ cat test.data + -| abc 123 + $ gawk -f test.awk test.data + -| strnum strnum + + This is all as expected: field data has the STRNUM attribute (*note +Variable Typing::). Now watch what happens when we run this program +under the debugger: + + $ gawk -D -f test.awk test.data + gawk> w $1 Set watchpoint on $1 + -| Watchpoint 1: $1 + gawk> w $2 Set watchpoint on $2 + -| Watchpoint 2: $2 + gawk> r Start the program + -| Starting program: + -| Stopping in Rule ... + -| Watchpoint 1: $1 Watchpoint fires + -| Old value: "" + -| New value: "abc" + -| main() at `test.awk':1 + -| 1 { print typeof($1), typeof($2) } + gawk> n Keep going ... + -| Watchpoint 2: $2 Watchpoint fires + -| Old value: "" + -| New value: "123" + -| main() at `test.awk':1 + -| 1 { print typeof($1), typeof($2) } + gawk> n Get result from typeof() + -| strnum number Result for $2 isn't right + -| Program exited normally with exit value: 0 + gawk> quit + + In this case, the act of comparing the new value of '$2' with the old +one caused 'gawk' to evaluate it and determine that it is indeed a +number, and this is reflected in the result of 'typeof()'. + + Cases like this where the debugger is not transparent to the +program's execution should be rare. If you encounter one, please report +it (*note Bugs::). + + ---------- Footnotes ---------- + + (1) Thanks to Hermann Peifer for this example. + + +File: gawk.info, Node: Debugging Summary, Prev: Limitations, Up: Debugger + +14.6 Summary +============ + + * Programs rarely work correctly the first time. Finding bugs is + called debugging, and a program that helps you find bugs is a + debugger. 'gawk' has a built-in debugger that works very similarly + to the GNU Debugger, GDB. + + * Debuggers let you step through your program one statement at a + time, examine and change variable and array values, and do a number + of other things that let you understand what your program is + actually doing (as opposed to what it is supposed to do). + + * Like most debuggers, the 'gawk' debugger works in terms of stack + frames, and lets you set both breakpoints (stop at a point in the + code) and watchpoints (stop when a data value changes). + + * The debugger command set is fairly complete, providing control over + breakpoints, execution, viewing and changing data, working with the + stack, getting information, and other tasks. + + * If the GNU Readline library is available when 'gawk' is compiled, + it is used by the debugger to provide command-line history and + editing. + + * Usually, the debugger does not not affect the program being + debugged, but occasionally it can. + + +File: gawk.info, Node: Arbitrary Precision Arithmetic, Next: Dynamic Extensions, Prev: Debugger, Up: Top + +15 Arithmetic and Arbitrary-Precision Arithmetic with 'gawk' +************************************************************ + +This major node introduces some basic concepts relating to how computers +do arithmetic and defines some important terms. It then proceeds to +describe floating-point arithmetic, which is what 'awk' uses for all its +computations, including a discussion of arbitrary-precision +floating-point arithmetic, which is a feature available only in 'gawk'. +It continues on to present arbitrary-precision integers, and concludes +with a description of some points where 'gawk' and the POSIX standard +are not quite in agreement. + + NOTE: Most users of 'gawk' can safely skip this chapter. But if + you want to do scientific calculations with 'gawk', this is the + place to be. + +* Menu: + +* Computer Arithmetic:: A quick intro to computer math. +* Math Definitions:: Defining terms used. +* MPFR features:: The MPFR features in 'gawk'. +* FP Math Caution:: Things to know. +* Arbitrary Precision Integers:: Arbitrary Precision Integer Arithmetic with + 'gawk'. +* POSIX Floating Point Problems:: Standards Versus Existing Practice. +* Floating point summary:: Summary of floating point discussion. + + +File: gawk.info, Node: Computer Arithmetic, Next: Math Definitions, Up: Arbitrary Precision Arithmetic + +15.1 A General Description of Computer Arithmetic +================================================= + +Until now, we have worked with data as either numbers or strings. +Ultimately, however, computers represent everything in terms of "binary +digits", or "bits". A decimal digit can take on any of 10 values: zero +through nine. A binary digit can take on any of two values, zero or +one. Using binary, computers (and computer software) can represent and +manipulate numerical and character data. In general, the more bits you +can use to represent a particular thing, the greater the range of +possible values it can take on. + + Modern computers support at least two, and often more, ways to do +arithmetic. Each kind of arithmetic uses a different representation +(organization of the bits) for the numbers. The kinds of arithmetic +that interest us are: + +Decimal arithmetic + This is the kind of arithmetic you learned in elementary school, + using paper and pencil (and/or a calculator). In theory, numbers + can have an arbitrary number of digits on either side (or both + sides) of the decimal point, and the results of a computation are + always exact. + + Some modern systems can do decimal arithmetic in hardware, but + usually you need a special software library to provide access to + these instructions. There are also libraries that do decimal + arithmetic entirely in software. + + Despite the fact that some users expect 'gawk' to be performing + decimal arithmetic,(1) it does not do so. + +Integer arithmetic + In school, integer values were referred to as "whole" numbers--that + is, numbers without any fractional part, such as 1, 42, or -17. + The advantage to integer numbers is that they represent values + exactly. The disadvantage is that their range is limited. + + In computers, integer values come in two flavors: "signed" and + "unsigned". Signed values may be negative or positive, whereas + unsigned values are always greater than or equal to zero. + + In computer systems, integer arithmetic is exact, but the possible + range of values is limited. Integer arithmetic is generally faster + than floating-point arithmetic. + +Floating-point arithmetic + Floating-point numbers represent what were called in school "real" + numbers (i.e., those that have a fractional part, such as + 3.1415927). The advantage to floating-point numbers is that they + can represent a much larger range of values than can integers. The + disadvantage is that there are numbers that they cannot represent + exactly. + + Modern systems support floating-point arithmetic in hardware, with + a limited range of values. There are software libraries that allow + the use of arbitrary-precision floating-point calculations. + + POSIX 'awk' uses "double-precision" floating-point numbers, which + can hold more digits than "single-precision" floating-point + numbers. 'gawk' has facilities for performing arbitrary-precision + floating-point arithmetic, which we describe in more detail + shortly. + + Computers work with integer and floating-point values of different +ranges. Integer values are usually either 32 or 64 bits in size. +Single-precision floating-point values occupy 32 bits, whereas +double-precision floating-point values occupy 64 bits. Floating-point +values are always signed. The possible ranges of values are shown in +*note Table 15.1: table-numeric-ranges. + +Numeric representation Minimum value Maximum value +--------------------------------------------------------------------------- +32-bit signed integer -2,147,483,648 2,147,483,647 +32-bit unsigned 0 4,294,967,295 +integer +64-bit signed integer -9,223,372,036,854,775,8089,223,372,036,854,775,807 +64-bit unsigned 0 18,446,744,073,709,551,615 +integer +Single-precision 1.175494e-38 3.402823e38 +floating point +(approximate) +Double-precision 2.225074e-308 1.797693e308 +floating point +(approximate) + +Table 15.1: Value ranges for different numeric representations + + ---------- Footnotes ---------- + + (1) We don't know why they expect this, but they do. + + +File: gawk.info, Node: Math Definitions, Next: MPFR features, Prev: Computer Arithmetic, Up: Arbitrary Precision Arithmetic + +15.2 Other Stuff to Know +======================== + +The rest of this major node uses a number of terms. Here are some +informal definitions that should help you work your way through the +material here: + +"Accuracy" + A floating-point calculation's accuracy is how close it comes to + the real (paper and pencil) value. + +"Error" + The difference between what the result of a computation "should be" + and what it actually is. It is best to minimize error as much as + possible. + +"Exponent" + The order of magnitude of a value; some number of bits in a + floating-point value store the exponent. + +"Inf" + A special value representing infinity. Operations involving + another number and infinity produce infinity. + +"NaN" + "Not a number."(1) A special value that results from attempting a + calculation that has no answer as a real number. In such a case, + programs can either receive a floating-point exception, or get + 'NaN' back as the result. The IEEE 754 standard recommends that + systems return 'NaN'. Some examples: + + 'sqrt(-1)' + This makes sense in the range of complex numbers, but not in + the range of real numbers, so the result is 'NaN'. + + 'log(-8)' + -8 is out of the domain of 'log()', so the result is 'NaN'. + +"Normalized" + How the significand (see later in this list) is usually stored. + The value is adjusted so that the first bit is one, and then that + leading one is assumed instead of physically stored. This provides + one extra bit of precision. + +"Precision" + The number of bits used to represent a floating-point number. The + more bits, the more digits you can represent. Binary and decimal + precisions are related approximately, according to the formula: + + PREC = 3.322 * DPS + + Here, _prec_ denotes the binary precision (measured in bits) and + _dps_ (short for decimal places) is the decimal digits. + +"Rounding mode" + How numbers are rounded up or down when necessary. More details + are provided later. + +"Significand" + A floating-point value consists of the significand multiplied by 10 + to the power of the exponent. For example, in '1.2345e67', the + significand is '1.2345'. + +"Stability" + From the Wikipedia article on numerical stability + (http://en.wikipedia.org/wiki/Numerical_stability): "Calculations + that can be proven not to magnify approximation errors are called + "numerically stable"." + + See the Wikipedia article on accuracy and precision +(http://en.wikipedia.org/wiki/Accuracy_and_precision) for more +information on some of those terms. + + On modern systems, floating-point hardware uses the representation +and operations defined by the IEEE 754 standard. Three of the standard +IEEE 754 types are 32-bit single precision, 64-bit double precision, and +128-bit quadruple precision. The standard also specifies extended +precision formats to allow greater precisions and larger exponent +ranges. ('awk' uses only the 64-bit double-precision format.) + + *note Table 15.2: table-ieee-formats. lists the precision and +exponent field values for the basic IEEE 754 binary formats. + +Name Total bits Precision Minimum Maximum + exponent exponent +--------------------------------------------------------------------------- +Single 32 24 -126 +127 +Double 64 53 -1022 +1023 +Quadruple 128 113 -16382 +16383 + +Table 15.2: Basic IEEE format values + + NOTE: The precision numbers include the implied leading one that + gives them one extra bit of significand. + + ---------- Footnotes ---------- + + (1) Thanks to Michael Brennan for this description, which we have +paraphrased, and for the examples. + + +File: gawk.info, Node: MPFR features, Next: FP Math Caution, Prev: Math Definitions, Up: Arbitrary Precision Arithmetic + +15.3 Arbitrary-Precision Arithmetic Features in 'gawk' +====================================================== + +By default, 'gawk' uses the double-precision floating-point values +supplied by the hardware of the system it runs on. However, if it was +compiled to do so, and the '-M' command-line option is supplied, 'gawk' +uses the GNU MPFR (http://www.mpfr.org) and GNU MP (http://gmplib.org) +(GMP) libraries for arbitrary-precision arithmetic on numbers. You can +see if MPFR support is available like so: + + $ gawk --version + -| GNU Awk 4.1.2, API: 1.1 (GNU MPFR 3.1.0-p3, GNU MP 5.0.2) + -| Copyright (C) 1989, 1991-2015 Free Software Foundation. + ... + +(You may see different version numbers than what's shown here. That's +OK; what's important is to see that GNU MPFR and GNU MP are listed in +the output.) + + Additionally, there are a few elements available in the 'PROCINFO' +array to provide information about the MPFR and GMP libraries (*note +Auto-set::). + + The MPFR library provides precise control over precisions and +rounding modes, and gives correctly rounded, reproducible, +platform-independent results. With the '-M' command-line option, all +floating-point arithmetic operators and numeric functions can yield +results to any desired precision level supported by MPFR. + + Two predefined variables, 'PREC' and 'ROUNDMODE', provide control +over the working precision and the rounding mode. The precision and the +rounding mode are set globally for every operation to follow. *Note +Setting precision:: and *note Setting the rounding mode:: for more +information. + + +File: gawk.info, Node: FP Math Caution, Next: Arbitrary Precision Integers, Prev: MPFR features, Up: Arbitrary Precision Arithmetic + +15.4 Floating-Point Arithmetic: Caveat Emptor! +============================================== + + Math class is tough! + -- _Teen Talk Barbie, July 1992_ + + This minor node provides a high-level overview of the issues involved +when doing lots of floating-point arithmetic.(1) The discussion applies +to both hardware and arbitrary-precision floating-point arithmetic. + + CAUTION: The material here is purposely general. If you need to do + serious computer arithmetic, you should do some research first, and + not rely just on what we tell you. + +* Menu: + +* Inexactness of computations:: Floating point math is not exact. +* Getting Accuracy:: Getting more accuracy takes some work. +* Try To Round:: Add digits and round. +* Setting precision:: How to set the precision. +* Setting the rounding mode:: How to set the rounding mode. + + ---------- Footnotes ---------- + + (1) There is a very nice paper on floating-point arithmetic +(http://www.validlab.com/goldberg/paper.pdf) by David Goldberg, "What +Every Computer Scientist Should Know About Floating-Point Arithmetic," +'ACM Computing Surveys' *23*, 1 (1991-03): 5-48. This is worth reading +if you are interested in the details, but it does require a background +in computer science. + + +File: gawk.info, Node: Inexactness of computations, Next: Getting Accuracy, Up: FP Math Caution + +15.4.1 Floating-Point Arithmetic Is Not Exact +--------------------------------------------- + +Binary floating-point representations and arithmetic are inexact. +Simple values like 0.1 cannot be precisely represented using binary +floating-point numbers, and the limited precision of floating-point +numbers means that slight changes in the order of operations or the +precision of intermediate storage can change the result. To make +matters worse, with arbitrary-precision floating-point arithmetic, you +can set the precision before starting a computation, but then you cannot +be sure of the number of significant decimal places in the final result. + +* Menu: + +* Inexact representation:: Numbers are not exactly represented. +* Comparing FP Values:: How to compare floating point values. +* Errors accumulate:: Errors get bigger as they go. + + +File: gawk.info, Node: Inexact representation, Next: Comparing FP Values, Up: Inexactness of computations + +15.4.1.1 Many Numbers Cannot Be Represented Exactly +................................................... + +So, before you start to write any code, you should think about what you +really want and what's really happening. Consider the two numbers in +the following example: + + x = 0.875 # 1/2 + 1/4 + 1/8 + y = 0.425 + + Unlike the number in 'y', the number stored in 'x' is exactly +representable in binary because it can be written as a finite sum of one +or more fractions whose denominators are all powers of two. When 'gawk' +reads a floating-point number from program source, it automatically +rounds that number to whatever precision your machine supports. If you +try to print the numeric content of a variable using an output format +string of '"%.17g"', it may not produce the same number as you assigned +to it: + + $ gawk 'BEGIN { x = 0.875; y = 0.425 + > printf("%0.17g, %0.17g\n", x, y) }' + -| 0.875, 0.42499999999999999 + + Often the error is so small you do not even notice it, and if you do, +you can always specify how much precision you would like in your output. +Usually this is a format string like '"%.15g"', which, when used in the +previous example, produces an output identical to the input. + + +File: gawk.info, Node: Comparing FP Values, Next: Errors accumulate, Prev: Inexact representation, Up: Inexactness of computations + +15.4.1.2 Be Careful Comparing Values +.................................... + +Because the underlying representation can be a little bit off from the +exact value, comparing floating-point values to see if they are exactly +equal is generally a bad idea. Here is an example where it does not +work like you would expect: + + $ gawk 'BEGIN { print (0.1 + 12.2 == 12.3) }' + -| 0 + + The general wisdom when comparing floating-point values is to see if +they are within some small range of each other (called a "delta", or +"tolerance"). You have to decide how small a delta is important to you. +Code to do this looks something like the following: + + delta = 0.00001 # for example + difference = abs(a) - abs(b) # subtract the two values + if (difference < delta) + # all ok + else + # not ok + +(We assume that you have a simple absolute value function named 'abs()' +defined elsewhere in your program.) + + +File: gawk.info, Node: Errors accumulate, Prev: Comparing FP Values, Up: Inexactness of computations + +15.4.1.3 Errors Accumulate +.......................... + +The loss of accuracy during a single computation with floating-point +numbers usually isn't enough to worry about. However, if you compute a +value that is the result of a sequence of floating-point operations, the +error can accumulate and greatly affect the computation itself. Here is +an attempt to compute the value of pi using one of its many series +representations: + + BEGIN { + x = 1.0 / sqrt(3.0) + n = 6 + for (i = 1; i < 30; i++) { + n = n * 2.0 + x = (sqrt(x * x + 1) - 1) / x + printf("%.15f\n", n * x) + } + } + + When run, the early errors propagate through later computations, +causing the loop to terminate prematurely after attempting to divide by +zero: + + $ gawk -f pi.awk + -| 3.215390309173475 + -| 3.159659942097510 + -| 3.146086215131467 + -| 3.142714599645573 + ... + -| 3.224515243534819 + -| 2.791117213058638 + -| 0.000000000000000 + error-> gawk: pi.awk:6: fatal: division by zero attempted + + Here is an additional example where the inaccuracies in internal +representations yield an unexpected result: + + $ gawk 'BEGIN { + > for (d = 1.1; d <= 1.5; d += 0.1) # loop five times (?) + > i++ + > print i + > }' + -| 4 + + +File: gawk.info, Node: Getting Accuracy, Next: Try To Round, Prev: Inexactness of computations, Up: FP Math Caution + +15.4.2 Getting the Accuracy You Need +------------------------------------ + +Can arbitrary-precision arithmetic give exact results? There are no +easy answers. The standard rules of algebra often do not apply when +using floating-point arithmetic. Among other things, the distributive +and associative laws do not hold completely, and order of operation may +be important for your computation. Rounding error, cumulative precision +loss, and underflow are often troublesome. + + When 'gawk' tests the expressions '0.1 + 12.2' and '12.3' for +equality using the machine double-precision arithmetic, it decides that +they are not equal! (*Note Comparing FP Values::.) You can get the +result you want by increasing the precision; 56 bits in this case does +the job: + + $ gawk -M -v PREC=56 'BEGIN { print (0.1 + 12.2 == 12.3) }' + -| 1 + + If adding more bits is good, perhaps adding even more bits of +precision is better? Here is what happens if we use an even larger +value of 'PREC': + + $ gawk -M -v PREC=201 'BEGIN { print (0.1 + 12.2 == 12.3) }' + -| 0 + + This is not a bug in 'gawk' or in the MPFR library. It is easy to +forget that the finite number of bits used to store the value is often +just an approximation after proper rounding. The test for equality +succeeds if and only if _all_ bits in the two operands are exactly the +same. Because this is not necessarily true after floating-point +computations with a particular precision and effective rounding mode, a +straight test for equality may not work. Instead, compare the two +numbers to see if they are within the desirable delta of each other. + + In applications where 15 or fewer decimal places suffice, hardware +double-precision arithmetic can be adequate, and is usually much faster. +But you need to keep in mind that every floating-point operation can +suffer a new rounding error with catastrophic consequences, as +illustrated by our earlier attempt to compute the value of pi. Extra +precision can greatly enhance the stability and the accuracy of your +computation in such cases. + + Additionally, you should understand that repeated addition is not +necessarily equivalent to multiplication in floating-point arithmetic. +In the example in *note Errors accumulate::: + + $ gawk 'BEGIN { + > for (d = 1.1; d <= 1.5; d += 0.1) # loop five times (?) + > i++ + > print i + > }' + -| 4 + +you may or may not succeed in getting the correct result by choosing an +arbitrarily large value for 'PREC'. Reformulation of the problem at +hand is often the correct approach in such situations. + + +File: gawk.info, Node: Try To Round, Next: Setting precision, Prev: Getting Accuracy, Up: FP Math Caution + +15.4.3 Try a Few Extra Bits of Precision and Rounding +----------------------------------------------------- + +Instead of arbitrary-precision floating-point arithmetic, often all you +need is an adjustment of your logic or a different order for the +operations in your calculation. The stability and the accuracy of the +computation of pi in the earlier example can be enhanced by using the +following simple algebraic transformation: + + (sqrt(x * x + 1) - 1) / x == x / (sqrt(x * x + 1) + 1) + +After making this change, the program converges to pi in under 30 +iterations: + + $ gawk -f pi2.awk + -| 3.215390309173473 + -| 3.159659942097501 + -| 3.146086215131436 + -| 3.142714599645370 + -| 3.141873049979825 + ... + -| 3.141592653589797 + -| 3.141592653589797 + + +File: gawk.info, Node: Setting precision, Next: Setting the rounding mode, Prev: Try To Round, Up: FP Math Caution + +15.4.4 Setting the Precision +---------------------------- + +'gawk' uses a global working precision; it does not keep track of the +precision or accuracy of individual numbers. Performing an arithmetic +operation or calling a built-in function rounds the result to the +current working precision. The default working precision is 53 bits, +which you can modify using the predefined variable 'PREC'. You can also +set the value to one of the predefined case-insensitive strings shown in +*note Table 15.3: table-predefined-precision-strings, to emulate an IEEE +754 binary format. + +'PREC' IEEE 754 binary format +--------------------------------------------------- +'"half"' 16-bit half-precision +'"single"' Basic 32-bit single precision +'"double"' Basic 64-bit double precision +'"quad"' Basic 128-bit quadruple precision +'"oct"' 256-bit octuple precision + +Table 15.3: Predefined precision strings for 'PREC' + + The following example illustrates the effects of changing precision +on arithmetic operations: + + $ gawk -M -v PREC=100 'BEGIN { x = 1.0e-400; print x + 0 + > PREC = "double"; print x + 0 }' + -| 1e-400 + -| 0 + + CAUTION: Be wary of floating-point constants! When reading a + floating-point constant from program source code, 'gawk' uses the + default precision (that of a C 'double'), unless overridden by an + assignment to the special variable 'PREC' on the command line, to + store it internally as an MPFR number. Changing the precision + using 'PREC' in the program text does _not_ change the precision of + a constant. + + If you need to represent a floating-point constant at a higher + precision than the default and cannot use a command-line assignment + to 'PREC', you should either specify the constant as a string, or + as a rational number, whenever possible. The following example + illustrates the differences among various ways to print a + floating-point constant: + + $ gawk -M 'BEGIN { PREC = 113; printf("%0.25f\n", 0.1) }' + -| 0.1000000000000000055511151 + $ gawk -M -v PREC=113 'BEGIN { printf("%0.25f\n", 0.1) }' + -| 0.1000000000000000000000000 + $ gawk -M 'BEGIN { PREC = 113; printf("%0.25f\n", "0.1") }' + -| 0.1000000000000000000000000 + $ gawk -M 'BEGIN { PREC = 113; printf("%0.25f\n", 1/10) }' + -| 0.1000000000000000000000000 + + +File: gawk.info, Node: Setting the rounding mode, Prev: Setting precision, Up: FP Math Caution + +15.4.5 Setting the Rounding Mode +-------------------------------- + +The 'ROUNDMODE' variable provides program-level control over the +rounding mode. The correspondence between 'ROUNDMODE' and the IEEE +rounding modes is shown in *note Table 15.4: table-gawk-rounding-modes. + +Rounding mode IEEE name 'ROUNDMODE' +--------------------------------------------------------------------------- +Round to nearest, ties to even 'roundTiesToEven' '"N"' or '"n"' +Round toward positive infinity 'roundTowardPositive' '"U"' or '"u"' +Round toward negative infinity 'roundTowardNegative' '"D"' or '"d"' +Round toward zero 'roundTowardZero' '"Z"' or '"z"' +Round to nearest, ties away 'roundTiesToAway' '"A"' or '"a"' +from zero + +Table 15.4: 'gawk' rounding modes + + 'ROUNDMODE' has the default value '"N"', which selects the IEEE 754 +rounding mode 'roundTiesToEven'. In *note Table 15.4: +table-gawk-rounding-modes, the value '"A"' selects 'roundTiesToAway'. +This is only available if your version of the MPFR library supports it; +otherwise, setting 'ROUNDMODE' to '"A"' has no effect. + + The default mode 'roundTiesToEven' is the most preferred, but the +least intuitive. This method does the obvious thing for most values, by +rounding them up or down to the nearest digit. For example, rounding +1.132 to two digits yields 1.13, and rounding 1.157 yields 1.16. + + However, when it comes to rounding a value that is exactly halfway +between, things do not work the way you probably learned in school. In +this case, the number is rounded to the nearest even digit. So rounding +0.125 to two digits rounds down to 0.12, but rounding 0.6875 to three +digits rounds up to 0.688. You probably have already encountered this +rounding mode when using 'printf' to format floating-point numbers. For +example: + + BEGIN { + x = -4.5 + for (i = 1; i < 10; i++) { + x += 1.0 + printf("%4.1f => %2.0f\n", x, x) + } + } + +produces the following output when run on the author's system:(1) + + -3.5 => -4 + -2.5 => -2 + -1.5 => -2 + -0.5 => 0 + 0.5 => 0 + 1.5 => 2 + 2.5 => 2 + 3.5 => 4 + 4.5 => 4 + + The theory behind 'roundTiesToEven' is that it more or less evenly +distributes upward and downward rounds of exact halves, which might +cause any accumulating round-off error to cancel itself out. This is +the default rounding mode for IEEE 754 computing functions and +operators. + + The other rounding modes are rarely used. Rounding toward positive +infinity ('roundTowardPositive') and toward negative infinity +('roundTowardNegative') are often used to implement interval arithmetic, +where you adjust the rounding mode to calculate upper and lower bounds +for the range of output. The 'roundTowardZero' mode can be used for +converting floating-point numbers to integers. The rounding mode +'roundTiesToAway' rounds the result to the nearest number and selects +the number with the larger magnitude if a tie occurs. + + Some numerical analysts will tell you that your choice of rounding +style has tremendous impact on the final outcome, and advise you to wait +until final output for any rounding. Instead, you can often avoid +round-off error problems by setting the precision initially to some +value sufficiently larger than the final desired precision, so that the +accumulation of round-off error does not influence the outcome. If you +suspect that results from your computation are sensitive to accumulation +of round-off error, look for a significant difference in output when you +change the rounding mode to be sure. + + ---------- Footnotes ---------- + + (1) It is possible for the output to be completely different if the C +library in your system does not use the IEEE 754 even-rounding rule to +round halfway cases for 'printf'. + + +File: gawk.info, Node: Arbitrary Precision Integers, Next: POSIX Floating Point Problems, Prev: FP Math Caution, Up: Arbitrary Precision Arithmetic + +15.5 Arbitrary-Precision Integer Arithmetic with 'gawk' +======================================================= + +When given the '-M' option, 'gawk' performs all integer arithmetic using +GMP arbitrary-precision integers. Any number that looks like an integer +in a source or data file is stored as an arbitrary-precision integer. +The size of the integer is limited only by the available memory. For +example, the following computes 5^4^3^2, the result of which is beyond +the limits of ordinary hardware double-precision floating-point values: + + $ gawk -M 'BEGIN { + > x = 5^4^3^2 + > print "number of digits =", length(x) + > print substr(x, 1, 20), "...", substr(x, length(x) - 19, 20) + > }' + -| number of digits = 183231 + -| 62060698786608744707 ... 92256259918212890625 + + If instead you were to compute the same value using +arbitrary-precision floating-point values, the precision needed for +correct output (using the formula 'prec = 3.322 * dps') would be 3.322 x +183231, or 608693. + + The result from an arithmetic operation with an integer and a +floating-point value is a floating-point value with a precision equal to +the working precision. The following program calculates the eighth term +in Sylvester's sequence(1) using a recurrence: + + $ gawk -M 'BEGIN { + > s = 2.0 + > for (i = 1; i <= 7; i++) + > s = s * (s - 1) + 1 + > print s + > }' + -| 113423713055421845118910464 + + The output differs from the actual number, +113,423,713,055,421,844,361,000,443, because the default precision of 53 +bits is not enough to represent the floating-point results exactly. You +can either increase the precision (100 bits is enough in this case), or +replace the floating-point constant '2.0' with an integer, to perform +all computations using integer arithmetic to get the correct output. + + Sometimes 'gawk' must implicitly convert an arbitrary-precision +integer into an arbitrary-precision floating-point value. This is +primarily because the MPFR library does not always provide the relevant +interface to process arbitrary-precision integers or mixed-mode numbers +as needed by an operation or function. In such a case, the precision is +set to the minimum value necessary for exact conversion, and the working +precision is not used for this purpose. If this is not what you need or +want, you can employ a subterfuge and convert the integer to floating +point first, like this: + + gawk -M 'BEGIN { n = 13; print (n + 0.0) % 2.0 }' + + You can avoid this issue altogether by specifying the number as a +floating-point value to begin with: + + gawk -M 'BEGIN { n = 13.0; print n % 2.0 }' + + Note that for this particular example, it is likely best to just use +the following: + + gawk -M 'BEGIN { n = 13; print n % 2 }' + + When dividing two arbitrary precision integers with either '/' or +'%', the result is typically an arbitrary precision floating point value +(unless the denominator evenly divides into the numerator). In order to +do integer division or remainder with arbitrary precision integers, use +the built-in 'intdiv()' function (*note Numeric Functions::). + + You can simulate the 'intdiv()' function in standard 'awk' using this +user-defined function: + + # intdiv --- do integer division + + function intdiv(numerator, denominator, result) + { + split("", result) + + numerator = int(numerator) + denominator = int(denominator) + result["quotient"] = int(numerator / denominator) + result["remainder"] = int(numerator % denominator) + + return 0.0 + } + + The following example program, contributed by Katie Wasserman, uses +'intdiv()' to compute the digits of pi to as many places as you choose +to set: + + # pi.awk --- compute the digits of pi + + BEGIN { + digits = 100000 + two = 2 * 10 ^ digits + pi = two + for (m = digits * 4; m > 0; --m) { + d = m * 2 + 1 + x = pi * m + intdiv(x, d, result) + pi = result["quotient"] + pi = pi + two + } + print pi + } + + When asked about the algorithm used, Katie replied: + + It's not that well known but it's not that obscure either. It's + Euler's modification to Newton's method for calculating pi. Take a + look at lines (23) - (25) here: + <http://mathworld.wolfram.com/PiFormulas.html>. + + The algorithm I wrote simply expands the multiply by 2 and works + from the innermost expression outwards. I used this to program HP + calculators because it's quite easy to modify for tiny memory + devices with smallish word sizes. See + <http://www.hpmuseum.org/cgi-sys/cgiwrap/hpmuseum/articles.cgi?read=899>. + + ---------- Footnotes ---------- + + (1) Weisstein, Eric W. 'Sylvester's Sequence'. From MathWorld--A +Wolfram Web Resource +(<http://mathworld.wolfram.com/SylvestersSequence.html>). + + +File: gawk.info, Node: POSIX Floating Point Problems, Next: Floating point summary, Prev: Arbitrary Precision Integers, Up: Arbitrary Precision Arithmetic + +15.6 Standards Versus Existing Practice +======================================= + +Historically, 'awk' has converted any nonnumeric-looking string to the +numeric value zero, when required. Furthermore, the original definition +of the language and the original POSIX standards specified that 'awk' +only understands decimal numbers (base 10), and not octal (base 8) or +hexadecimal numbers (base 16). + + Changes in the language of the 2001 and 2004 POSIX standards can be +interpreted to imply that 'awk' should support additional features. +These features are: + + * Interpretation of floating-point data values specified in + hexadecimal notation (e.g., '0xDEADBEEF'). (Note: data values, + _not_ source code constants.) + + * Support for the special IEEE 754 floating-point values "not a + number" (NaN), positive infinity ("inf"), and negative infinity + ("-inf"). In particular, the format for these values is as + specified by the ISO 1999 C standard, which ignores case and can + allow implementation-dependent additional characters after the + 'nan' and allow either 'inf' or 'infinity'. + + The first problem is that both of these are clear changes to +historical practice: + + * The 'gawk' maintainer feels that supporting hexadecimal + floating-point values, in particular, is ugly, and was never + intended by the original designers to be part of the language. + + * Allowing completely alphabetic strings to have valid numeric values + is also a very severe departure from historical practice. + + The second problem is that the 'gawk' maintainer feels that this +interpretation of the standard, which required a certain amount of +"language lawyering" to arrive at in the first place, was not even +intended by the standard developers. In other words, "We see how you +got where you are, but we don't think that that's where you want to be." + + Recognizing these issues, but attempting to provide compatibility +with the earlier versions of the standard, the 2008 POSIX standard added +explicit wording to allow, but not require, that 'awk' support +hexadecimal floating-point values and special values for "not a number" +and infinity. + + Although the 'gawk' maintainer continues to feel that providing those +features is inadvisable, nevertheless, on systems that support IEEE +floating point, it seems reasonable to provide _some_ way to support NaN +and infinity values. The solution implemented in 'gawk' is as follows: + + * With the '--posix' command-line option, 'gawk' becomes "hands off." + String values are passed directly to the system library's + 'strtod()' function, and if it successfully returns a numeric + value, that is what's used.(1) By definition, the results are not + portable across different systems. They are also a little + surprising: + + $ echo nanny | gawk --posix '{ print $1 + 0 }' + -| nan + $ echo 0xDeadBeef | gawk --posix '{ print $1 + 0 }' + -| 3735928559 + + * Without '--posix', 'gawk' interprets the four string values '+inf', + '-inf', '+nan', and '-nan' specially, producing the corresponding + special numeric values. The leading sign acts a signal to 'gawk' + (and the user) that the value is really numeric. Hexadecimal + floating point is not supported (unless you also use + '--non-decimal-data', which is _not_ recommended). For example: + + $ echo nanny | gawk '{ print $1 + 0 }' + -| 0 + $ echo +nan | gawk '{ print $1 + 0 }' + -| nan + $ echo 0xDeadBeef | gawk '{ print $1 + 0 }' + -| 0 + + 'gawk' ignores case in the four special values. Thus, '+nan' and + '+NaN' are the same. + + ---------- Footnotes ---------- + + (1) You asked for it, you got it. + + +File: gawk.info, Node: Floating point summary, Prev: POSIX Floating Point Problems, Up: Arbitrary Precision Arithmetic + +15.7 Summary +============ + + * Most computer arithmetic is done using either integers or + floating-point values. Standard 'awk' uses double-precision + floating-point values. + + * In the early 1990s Barbie mistakenly said, "Math class is tough!" + Although math isn't tough, floating-point arithmetic isn't the same + as pencil-and-paper math, and care must be taken: + + - Not all numbers can be represented exactly. + + - Comparing values should use a delta, instead of being done + directly with '==' and '!='. + + - Errors accumulate. + + - Operations are not always truly associative or distributive. + + * Increasing the accuracy can help, but it is not a panacea. + + * Often, increasing the accuracy and then rounding to the desired + number of digits produces reasonable results. + + * Use '-M' (or '--bignum') to enable MPFR arithmetic. Use 'PREC' to + set the precision in bits, and 'ROUNDMODE' to set the IEEE 754 + rounding mode. + + * With '-M', 'gawk' performs arbitrary-precision integer arithmetic + using the GMP library. This is faster and more space-efficient + than using MPFR for the same calculations. + + * There are several areas with respect to floating-point numbers + where 'gawk' disagrees with the POSIX standard. It pays to be + aware of them. + + * Overall, there is no need to be unduly suspicious about the results + from floating-point arithmetic. The lesson to remember is that + floating-point arithmetic is always more complex than arithmetic + using pencil and paper. In order to take advantage of the power of + floating-point arithmetic, you need to know its limitations and + work within them. For most casual use of floating-point + arithmetic, you will often get the expected result if you simply + round the display of your final results to the correct number of + significant decimal digits. + + * As general advice, avoid presenting numerical data in a manner that + implies better precision than is actually the case. + + +File: gawk.info, Node: Dynamic Extensions, Next: Language History, Prev: Arbitrary Precision Arithmetic, Up: Top + +16 Writing Extensions for 'gawk' +******************************** + +It is possible to add new functions written in C or C++ to 'gawk' using +dynamically loaded libraries. This facility is available on systems +that support the C 'dlopen()' and 'dlsym()' functions. This major node +describes how to create extensions using code written in C or C++. + + If you don't know anything about C programming, you can safely skip +this major node, although you may wish to review the documentation on +the extensions that come with 'gawk' (*note Extension Samples::), and +the information on the 'gawkextlib' project (*note gawkextlib::). The +sample extensions are automatically built and installed when 'gawk' is. + + NOTE: When '--sandbox' is specified, extensions are disabled (*note + Options::). + +* Menu: + +* Extension Intro:: What is an extension. +* Plugin License:: A note about licensing. +* Extension Mechanism Outline:: An outline of how it works. +* Extension API Description:: A full description of the API. +* Finding Extensions:: How 'gawk' finds compiled extensions. +* Extension Example:: Example C code for an extension. +* Extension Samples:: The sample extensions that ship with + 'gawk'. +* gawkextlib:: The 'gawkextlib' project. +* Extension summary:: Extension summary. +* Extension Exercises:: Exercises. + + +File: gawk.info, Node: Extension Intro, Next: Plugin License, Up: Dynamic Extensions + +16.1 Introduction +================= + +An "extension" (sometimes called a "plug-in") is a piece of external +compiled code that 'gawk' can load at runtime to provide additional +functionality, over and above the built-in capabilities described in the +rest of this Info file. + + Extensions are useful because they allow you (of course) to extend +'gawk''s functionality. For example, they can provide access to system +calls (such as 'chdir()' to change directory) and to other C library +routines that could be of use. As with most software, "the sky is the +limit"; if you can imagine something that you might want to do and can +write in C or C++, you can write an extension to do it! + + Extensions are written in C or C++, using the "application +programming interface" (API) defined for this purpose by the 'gawk' +developers. The rest of this major node explains the facilities that +the API provides and how to use them, and presents a small example +extension. In addition, it documents the sample extensions included in +the 'gawk' distribution and describes the 'gawkextlib' project. *Note +Extension Design::, for a discussion of the extension mechanism goals +and design. + + +File: gawk.info, Node: Plugin License, Next: Extension Mechanism Outline, Prev: Extension Intro, Up: Dynamic Extensions + +16.2 Extension Licensing +======================== + +Every dynamic extension must be distributed under a license that is +compatible with the GNU GPL (*note Copying::). + + In order for the extension to tell 'gawk' that it is properly +licensed, the extension must define the global symbol +'plugin_is_GPL_compatible'. If this symbol does not exist, 'gawk' emits +a fatal error and exits when it tries to load your extension. + + The declared type of the symbol should be 'int'. It does not need to +be in any allocated section, though. The code merely asserts that the +symbol exists in the global scope. Something like this is enough: + + int plugin_is_GPL_compatible; + + +File: gawk.info, Node: Extension Mechanism Outline, Next: Extension API Description, Prev: Plugin License, Up: Dynamic Extensions + +16.3 How It Works at a High Level +================================= + +Communication between 'gawk' and an extension is two-way. First, when +an extension is loaded, 'gawk' passes it a pointer to a 'struct' whose +fields are function pointers. This is shown in *note Figure 16.1: +figure-load-extension. + + + Struct + +---+ + | | + +---+ + +---------------| | + | +---+ dl_load(api_p, id); + | | | ___________________ + | +---+ | + | +---------| | __________________ | + | | +---+ || + | | | | || + | | +---+ || + | | +---| | || + | | | +---+ \\ || / + | | | \\ / + v v v \\/ ++-------+-+---+-+---+-+------------------+--------------------+ +| |x| |x| |x| |OOOOOOOOOOOOOOOOOOOO| +| |x| |x| |x| |OOOOOOOOOOOOOOOOOOOO| +| |x| |x| |x| |OOOOOOOOOOOOOOOOOOOO| ++-------+-+---+-+---+-+------------------+--------------------+ + + gawk Main Program Address Space Extension" + +Figure 16.1: Loading the extension + + The extension can call functions inside 'gawk' through these function +pointers, at runtime, without needing (link-time) access to 'gawk''s +symbols. One of these function pointers is to a function for +"registering" new functions. This is shown in *note Figure 16.2: +figure-register-new-function. + + + + +--------------------------------------------+ + | | + V | ++-------+-+---+-+---+-+------------------+--------------+-+---+ +| |x| |x| |x| |OOOOOOOOOOOOOO|X|OOO| +| |x| |x| |x| |OOOOOOOOOOOOOO|X|OOO| +| |x| |x| |x| |OOOOOOOOOOOOOO|X|OOO| ++-------+-+---+-+---+-+------------------+--------------+-+---+ + + gawk Main Program Address Space Extension" + +Figure 16.2: Registering a new function + + In the other direction, the extension registers its new functions +with 'gawk' by passing function pointers to the functions that provide +the new feature ('do_chdir()', for example). 'gawk' associates the +function pointer with a name and can then call it, using a defined +calling convention. This is shown in *note Figure 16.3: +figure-call-new-function. + + + chdir(\"/path\") (*fnptr)(1); + } + +--------------------------------------------+ + | | + | V ++-------+-+---+-+---+-+------------------+--------------+-+---+ +| |x| |x| |x| |OOOOOOOOOOOOOO|X|OOO| +| |x| |x| |x| |OOOOOOOOOOOOOO|X|OOO| +| |x| |x| |x| |OOOOOOOOOOOOOO|X|OOO| ++-------+-+---+-+---+-+------------------+--------------+-+---+ + + gawk Main Program Address Space Extension" + +Figure 16.3: Calling the new function + + The 'do_XXX()' function, in turn, then uses the function pointers in +the API 'struct' to do its work, such as updating variables or arrays, +printing messages, setting 'ERRNO', and so on. + + Convenience macros make calling through the function pointers look +like regular function calls so that extension code is quite readable and +understandable. + + Although all of this sounds somewhat complicated, the result is that +extension code is quite straightforward to write and to read. You can +see this in the sample extension 'filefuncs.c' (*note Extension +Example::) and also in the 'testext.c' code for testing the APIs. + + Some other bits and pieces: + + * The API provides access to 'gawk''s 'do_XXX' values, reflecting + command-line options, like 'do_lint', 'do_profiling', and so on + (*note Extension API Variables::). These are informational: an + extension cannot affect their values inside 'gawk'. In addition, + attempting to assign to them produces a compile-time error. + + * The API also provides major and minor version numbers, so that an + extension can check if the 'gawk' it is loaded with supports the + facilities it was compiled with. (Version mismatches "shouldn't" + happen, but we all know how _that_ goes.) *Note Extension + Versioning:: for details. + + +File: gawk.info, Node: Extension API Description, Next: Finding Extensions, Prev: Extension Mechanism Outline, Up: Dynamic Extensions + +16.4 API Description +==================== + +C or C++ code for an extension must include the header file 'gawkapi.h', +which declares the functions and defines the data types used to +communicate with 'gawk'. This (rather large) minor node describes the +API in detail. + +* Menu: + +* Extension API Functions Introduction:: Introduction to the API functions. +* General Data Types:: The data types. +* Memory Allocation Functions:: Functions for allocating memory. +* Constructor Functions:: Functions for creating values. +* Registration Functions:: Functions to register things with + 'gawk'. +* Printing Messages:: Functions for printing messages. +* Updating ERRNO:: Functions for updating 'ERRNO'. +* Requesting Values:: How to get a value. +* Accessing Parameters:: Functions for accessing parameters. +* Symbol Table Access:: Functions for accessing global + variables. +* Array Manipulation:: Functions for working with arrays. +* Redirection API:: How to access and manipulate redirections. +* Extension API Variables:: Variables provided by the API. +* Extension API Boilerplate:: Boilerplate code for using the API. + + +File: gawk.info, Node: Extension API Functions Introduction, Next: General Data Types, Up: Extension API Description + +16.4.1 Introduction +------------------- + +Access to facilities within 'gawk' is achieved by calling through +function pointers passed into your extension. + + API function pointers are provided for the following kinds of +operations: + + * Allocating, reallocating, and releasing memory. + + * Registration functions. You may register: + + - Extension functions + - Exit callbacks + - A version string + - Input parsers + - Output wrappers + - Two-way processors + + All of these are discussed in detail later in this major node. + + * Printing fatal, warning, and "lint" warning messages. + + * Updating 'ERRNO', or unsetting it. + + * Accessing parameters, including converting an undefined parameter + into an array. + + * Symbol table access: retrieving a global variable, creating one, or + changing one. + + * Creating and releasing cached values; this provides an efficient + way to use values for multiple variables and can be a big + performance win. + + * Manipulating arrays: + + - Retrieving, adding, deleting, and modifying elements + + - Getting the count of elements in an array + + - Creating a new array + + - Clearing an array + + - Flattening an array for easy C-style looping over all its + indices and elements + + * Accessing and manipulating redirections. + + Some points about using the API: + + * The following types, macros, and/or functions are referenced in + 'gawkapi.h'. For correct use, you must therefore include the + corresponding standard header file _before_ including 'gawkapi.h': + + C entity Header file + ------------------------------------------- + 'EOF' '<stdio.h>' + Values for 'errno' '<errno.h>' + 'FILE' '<stdio.h>' + 'NULL' '<stddef.h>' + 'memcpy()' '<string.h>' + 'memset()' '<string.h>' + 'size_t' '<sys/types.h>' + 'struct stat' '<sys/stat.h>' + + Due to portability concerns, especially to systems that are not + fully standards-compliant, it is your responsibility to include the + correct files in the correct way. This requirement is necessary in + order to keep 'gawkapi.h' clean, instead of becoming a portability + hodge-podge as can be seen in some parts of the 'gawk' source code. + + * The 'gawkapi.h' file may be included more than once without ill + effect. Doing so, however, is poor coding practice. + + * Although the API only uses ISO C 90 features, there is an + exception; the "constructor" functions use the 'inline' keyword. + If your compiler does not support this keyword, you should either + place '-Dinline=''' on your command line or use the GNU Autotools + and include a 'config.h' file in your extensions. + + * All pointers filled in by 'gawk' point to memory managed by 'gawk' + and should be treated by the extension as read-only. Memory for + _all_ strings passed into 'gawk' from the extension _must_ come + from calling one of 'gawk_malloc()', 'gawk_calloc()', or + 'gawk_realloc()', and is managed by 'gawk' from then on. + + * The API defines several simple 'struct's that map values as seen + from 'awk'. A value can be a 'double', a string, or an array (as + in multidimensional arrays, or when creating a new array). String + values maintain both pointer and length, because embedded NUL + characters are allowed. + + NOTE: By intent, strings are maintained using the current + multibyte encoding (as defined by 'LC_XXX' environment + variables) and not using wide characters. This matches how + 'gawk' stores strings internally and also how characters are + likely to be input into and output from files. + + * When retrieving a value (such as a parameter or that of a global + variable or array element), the extension requests a specific type + (number, string, scalar, value cookie, array, or "undefined"). + When the request is "undefined," the returned value will have the + real underlying type. + + However, if the request and actual type don't match, the access + function returns "false" and fills in the type of the actual value + that is there, so that the extension can, e.g., print an error + message (such as "scalar passed where array expected"). + + You may call the API functions by using the function pointers +directly, but the interface is not so pretty. To make extension code +look more like regular code, the 'gawkapi.h' header file defines several +macros that you should use in your code. This minor node presents the +macros as if they were functions. + + +File: gawk.info, Node: General Data Types, Next: Memory Allocation Functions, Prev: Extension API Functions Introduction, Up: Extension API Description + +16.4.2 General-Purpose Data Types +--------------------------------- + + I have a true love/hate relationship with unions. + -- _Arnold Robbins_ + + That's the thing about unions: the compiler will arrange things so + they can accommodate both love and hate. + -- _Chet Ramey_ + + The extension API defines a number of simple types and structures for +general-purpose use. Additional, more specialized, data structures are +introduced in subsequent minor nodes, together with the functions that +use them. + + The general-purpose types and structures are as follows: + +'typedef void *awk_ext_id_t;' + A value of this type is received from 'gawk' when an extension is + loaded. That value must then be passed back to 'gawk' as the first + parameter of each API function. + +'#define awk_const ...' + This macro expands to 'const' when compiling an extension, and to + nothing when compiling 'gawk' itself. This makes certain fields in + the API data structures unwritable from extension code, while + allowing 'gawk' to use them as it needs to. + +'typedef enum awk_bool {' +' awk_false = 0,' +' awk_true' +'} awk_bool_t;' + A simple Boolean type. + +'typedef struct awk_string {' +' char *str; /* data */' +' size_t len; /* length thereof, in chars */' +'} awk_string_t;' + This represents a mutable string. 'gawk' owns the memory pointed + to if it supplied the value. Otherwise, it takes ownership of the + memory pointed to. _Such memory must come from calling one of the + 'gawk_malloc()', 'gawk_calloc()', or 'gawk_realloc()' functions!_ + + As mentioned earlier, strings are maintained using the current + multibyte encoding. + +'typedef enum {' +' AWK_UNDEFINED,' +' AWK_NUMBER,' +' AWK_STRING,' +' AWK_ARRAY,' +' AWK_SCALAR, /* opaque access to a variable */' +' AWK_VALUE_COOKIE /* for updating a previously created value */' +'} awk_valtype_t;' + This 'enum' indicates the type of a value. It is used in the + following 'struct'. + +'typedef struct awk_value {' +' awk_valtype_t val_type;' +' union {' +' awk_string_t s;' +' double d;' +' awk_array_t a;' +' awk_scalar_t scl;' +' awk_value_cookie_t vc;' +' } u;' +'} awk_value_t;' + An "'awk' value." The 'val_type' member indicates what kind of + value the 'union' holds, and each member is of the appropriate + type. + +'#define str_value u.s' +'#define num_value u.d' +'#define array_cookie u.a' +'#define scalar_cookie u.scl' +'#define value_cookie u.vc' + Using these macros makes accessing the fields of the 'awk_value_t' + more readable. + +'typedef void *awk_scalar_t;' + Scalars can be represented as an opaque type. These values are + obtained from 'gawk' and then passed back into it. This is + discussed in a general fashion in the text following this list, and + in more detail in *note Symbol table by cookie::. + +'typedef void *awk_value_cookie_t;' + A "value cookie" is an opaque type representing a cached value. + This is also discussed in a general fashion in the text following + this list, and in more detail in *note Cached values::. + + Scalar values in 'awk' are either numbers or strings. The +'awk_value_t' struct represents values. The 'val_type' member indicates +what is in the 'union'. + + Representing numbers is easy--the API uses a C 'double'. Strings +require more work. Because 'gawk' allows embedded NUL bytes in string +values, a string must be represented as a pair containing a data pointer +and length. This is the 'awk_string_t' type. + + Identifiers (i.e., the names of global variables) can be associated +with either scalar values or with arrays. In addition, 'gawk' provides +true arrays of arrays, where any given array element can itself be an +array. Discussion of arrays is delayed until *note Array +Manipulation::. + + The various macros listed earlier make it easier to use the elements +of the 'union' as if they were fields in a 'struct'; this is a common +coding practice in C. Such code is easier to write and to read, but it +remains _your_ responsibility to make sure that the 'val_type' member +correctly reflects the type of the value in the 'awk_value_t' struct. + + Conceptually, the first three members of the 'union' (number, string, +and array) are all that is needed for working with 'awk' values. +However, because the API provides routines for accessing and changing +the value of a global scalar variable only by using the variable's name, +there is a performance penalty: 'gawk' must find the variable each time +it is accessed and changed. This turns out to be a real issue, not just +a theoretical one. + + Thus, if you know that your extension will spend considerable time +reading and/or changing the value of one or more scalar variables, you +can obtain a "scalar cookie"(1) object for that variable, and then use +the cookie for getting the variable's value or for changing the +variable's value. The 'awk_scalar_t' type holds a scalar cookie, and +the 'scalar_cookie' macro provides access to the value of that type in +the 'awk_value_t' struct. Given a scalar cookie, 'gawk' can directly +retrieve or modify the value, as required, without having to find it +first. + + The 'awk_value_cookie_t' type and 'value_cookie' macro are similar. +If you know that you wish to use the same numeric or string _value_ for +one or more variables, you can create the value once, retaining a "value +cookie" for it, and then pass in that value cookie whenever you wish to +set the value of a variable. This saves storage space within the +running 'gawk' process and reduces the time needed to create the value. + + ---------- Footnotes ---------- + + (1) See the "cookie" entry in the Jargon file +(http://catb.org/jargon/html/C/cookie.html) for a definition of +"cookie", and the "magic cookie" entry in the Jargon file +(http://catb.org/jargon/html/M/magic-cookie.html) for a nice example. +See also the entry for "Cookie" in the *note Glossary::. + + +File: gawk.info, Node: Memory Allocation Functions, Next: Constructor Functions, Prev: General Data Types, Up: Extension API Description + +16.4.3 Memory Allocation Functions and Convenience Macros +--------------------------------------------------------- + +The API provides a number of "memory allocation" functions for +allocating memory that can be passed to 'gawk', as well as a number of +convenience macros. This node presents them all as function prototypes, +in the way that extension code would use them: + +'void *gawk_malloc(size_t size);' + Call the correct version of 'malloc()' to allocate storage that may + be passed to 'gawk'. + +'void *gawk_calloc(size_t nmemb, size_t size);' + Call the correct version of 'calloc()' to allocate storage that may + be passed to 'gawk'. + +'void *gawk_realloc(void *ptr, size_t size);' + Call the correct version of 'realloc()' to allocate storage that + may be passed to 'gawk'. + +'void gawk_free(void *ptr);' + Call the correct version of 'free()' to release storage that was + allocated with 'gawk_malloc()', 'gawk_calloc()', or + 'gawk_realloc()'. + + The API has to provide these functions because it is possible for an +extension to be compiled and linked against a different version of the C +library than was used for the 'gawk' executable.(1) If 'gawk' were to +use its version of 'free()' when the memory came from an unrelated +version of 'malloc()', unexpected behavior would likely result. + + Two convenience macros may be used for allocating storage from +'gawk_malloc()' and 'gawk_realloc()'. If the allocation fails, they +cause 'gawk' to exit with a fatal error message. They should be used as +if they were procedure calls that do not return a value: + +'#define emalloc(pointer, type, size, message) ...' + The arguments to this macro are as follows: + + 'pointer' + The pointer variable to point at the allocated storage. + + 'type' + The type of the pointer variable. This is used to create a + cast for the call to 'gawk_malloc()'. + + 'size' + The total number of bytes to be allocated. + + 'message' + A message to be prefixed to the fatal error message. + Typically this is the name of the function using the macro. + + For example, you might allocate a string value like so: + + awk_value_t result; + char *message; + const char greet[] = "Don't Panic!"; + + emalloc(message, char *, sizeof(greet), "myfunc"); + strcpy(message, greet); + make_malloced_string(message, strlen(message), & result); + +'#define erealloc(pointer, type, size, message) ...' + This is like 'emalloc()', but it calls 'gawk_realloc()' instead of + 'gawk_malloc()'. The arguments are the same as for the 'emalloc()' + macro. + + ---------- Footnotes ---------- + + (1) This is more common on MS-Windows systems, but it can happen on +Unix-like systems as well. + + +File: gawk.info, Node: Constructor Functions, Next: Registration Functions, Prev: Memory Allocation Functions, Up: Extension API Description + +16.4.4 Constructor Functions +---------------------------- + +The API provides a number of "constructor" functions for creating string +and numeric values, as well as a number of convenience macros. This +node presents them all as function prototypes, in the way that extension +code would use them: + +'static inline awk_value_t *' +'make_const_string(const char *string, size_t length, awk_value_t *result);' + This function creates a string value in the 'awk_value_t' variable + pointed to by 'result'. It expects 'string' to be a C string + constant (or other string data), and automatically creates a _copy_ + of the data for storage in 'result'. It returns 'result'. + +'static inline awk_value_t *' +'make_malloced_string(const char *string, size_t length, awk_value_t *result);' + This function creates a string value in the 'awk_value_t' variable + pointed to by 'result'. It expects 'string' to be a 'char *' value + pointing to data previously obtained from 'gawk_malloc()', + 'gawk_calloc()', or 'gawk_realloc()'. The idea here is that the + data is passed directly to 'gawk', which assumes responsibility for + it. It returns 'result'. + +'static inline awk_value_t *' +'make_null_string(awk_value_t *result);' + This specialized function creates a null string (the "undefined" + value) in the 'awk_value_t' variable pointed to by 'result'. It + returns 'result'. + +'static inline awk_value_t *' +'make_number(double num, awk_value_t *result);' + This function simply creates a numeric value in the 'awk_value_t' + variable pointed to by 'result'. + + +File: gawk.info, Node: Registration Functions, Next: Printing Messages, Prev: Constructor Functions, Up: Extension API Description + +16.4.5 Registration Functions +----------------------------- + +This minor node describes the API functions for registering parts of +your extension with 'gawk'. + +* Menu: + +* Extension Functions:: Registering extension functions. +* Exit Callback Functions:: Registering an exit callback. +* Extension Version String:: Registering a version string. +* Input Parsers:: Registering an input parser. +* Output Wrappers:: Registering an output wrapper. +* Two-way processors:: Registering a two-way processor. + + +File: gawk.info, Node: Extension Functions, Next: Exit Callback Functions, Up: Registration Functions + +16.4.5.1 Registering An Extension Function +.......................................... + +Extension functions are described by the following record: + + typedef struct awk_ext_func { + const char *name; + awk_value_t *(*function)(int num_actual_args, awk_value_t *result); + size_t max_expected_args; + } awk_ext_func_t; + + The fields are: + +'const char *name;' + The name of the new function. 'awk'-level code calls the function + by this name. This is a regular C string. + + Function names must obey the rules for 'awk' identifiers. That is, + they must begin with either an English letter or an underscore, + which may be followed by any number of letters, digits, and + underscores. Letter case in function names is significant. + +'awk_value_t *(*function)(int num_actual_args, awk_value_t *result);' + This is a pointer to the C function that provides the extension's + functionality. The function must fill in '*result' with either a + number or a string. 'gawk' takes ownership of any string memory. + As mentioned earlier, string memory _must_ come from one of + 'gawk_malloc()', 'gawk_calloc()', or 'gawk_realloc()'. + + The 'num_actual_args' argument tells the C function how many actual + parameters were passed from the calling 'awk' code. + + The function must return the value of 'result'. This is for the + convenience of the calling code inside 'gawk'. + +'size_t max_expected_args;' + This is the maximum number of arguments the function expects to + receive. Each extension function may decide what to do if the + number of arguments isn't what it expected. As with real 'awk' + functions, it is likely OK to ignore extra arguments. This value + does not affect actual program execution. + + Extension functions should compare this value to the number of + actual arguments passed and possibly issue a lint warning if there + is an undesirable mismatch. Of course, if '--lint=fatal' is used, + this would cause the program to exit. + + Once you have a record representing your extension function, you +register it with 'gawk' using this API function: + +'awk_bool_t add_ext_func(const char *namespace, const awk_ext_func_t *func);' + This function returns true upon success, false otherwise. The + 'namespace' parameter is currently not used; you should pass in an + empty string ('""'). The 'func' pointer is the address of a + 'struct' representing your function, as just described. + + +File: gawk.info, Node: Exit Callback Functions, Next: Extension Version String, Prev: Extension Functions, Up: Registration Functions + +16.4.5.2 Registering An Exit Callback Function +.............................................. + +An "exit callback" function is a function that 'gawk' calls before it +exits. Such functions are useful if you have general "cleanup" tasks +that should be performed in your extension (such as closing database +connections or other resource deallocations). You can register such a +function with 'gawk' using the following function: + +'void awk_atexit(void (*funcp)(void *data, int exit_status),' +' void *arg0);' + The parameters are: + + 'funcp' + A pointer to the function to be called before 'gawk' exits. + The 'data' parameter will be the original value of 'arg0'. + The 'exit_status' parameter is the exit status value that + 'gawk' intends to pass to the 'exit()' system call. + + 'arg0' + A pointer to private data that 'gawk' saves in order to pass + to the function pointed to by 'funcp'. + + Exit callback functions are called in last-in, first-out (LIFO) +order--that is, in the reverse order in which they are registered with +'gawk'. + + +File: gawk.info, Node: Extension Version String, Next: Input Parsers, Prev: Exit Callback Functions, Up: Registration Functions + +16.4.5.3 Registering An Extension Version String +................................................ + +You can register a version string that indicates the name and version of +your extension with 'gawk', as follows: + +'void register_ext_version(const char *version);' + Register the string pointed to by 'version' with 'gawk'. Note that + 'gawk' does _not_ copy the 'version' string, so it should not be + changed. + + 'gawk' prints all registered extension version strings when it is +invoked with the '--version' option. + + +File: gawk.info, Node: Input Parsers, Next: Output Wrappers, Prev: Extension Version String, Up: Registration Functions + +16.4.5.4 Customized Input Parsers +................................. + +By default, 'gawk' reads text files as its input. It uses the value of +'RS' to find the end of the record, and then uses 'FS' (or 'FIELDWIDTHS' +or 'FPAT') to split it into fields (*note Reading Files::). +Additionally, it sets the value of 'RT' (*note Built-in Variables::). + + If you want, you can provide your own custom input parser. An input +parser's job is to return a record to the 'gawk' record-processing code, +along with indicators for the value and length of the data to be used +for 'RT', if any. + + To provide an input parser, you must first provide two functions +(where XXX is a prefix name for your extension): + +'awk_bool_t XXX_can_take_file(const awk_input_buf_t *iobuf);' + This function examines the information available in 'iobuf' (which + we discuss shortly). Based on the information there, it decides if + the input parser should be used for this file. If so, it should + return true. Otherwise, it should return false. It should not + change any state (variable values, etc.) within 'gawk'. + +'awk_bool_t XXX_take_control_of(awk_input_buf_t *iobuf);' + When 'gawk' decides to hand control of the file over to the input + parser, it calls this function. This function in turn must fill in + certain fields in the 'awk_input_buf_t' structure and ensure that + certain conditions are true. It should then return true. If an + error of some kind occurs, it should not fill in any fields and + should return false; then 'gawk' will not use the input parser. + The details are presented shortly. + + Your extension should package these functions inside an +'awk_input_parser_t', which looks like this: + + typedef struct awk_input_parser { + const char *name; /* name of parser */ + awk_bool_t (*can_take_file)(const awk_input_buf_t *iobuf); + awk_bool_t (*take_control_of)(awk_input_buf_t *iobuf); + awk_const struct awk_input_parser *awk_const next; /* for gawk */ + } awk_input_parser_t; + + The fields are: + +'const char *name;' + The name of the input parser. This is a regular C string. + +'awk_bool_t (*can_take_file)(const awk_input_buf_t *iobuf);' + A pointer to your 'XXX_can_take_file()' function. + +'awk_bool_t (*take_control_of)(awk_input_buf_t *iobuf);' + A pointer to your 'XXX_take_control_of()' function. + +'awk_const struct input_parser *awk_const next;' + This is for use by 'gawk'; therefore it is marked 'awk_const' so + that the extension cannot modify it. + + The steps are as follows: + + 1. Create a 'static awk_input_parser_t' variable and initialize it + appropriately. + + 2. When your extension is loaded, register your input parser with + 'gawk' using the 'register_input_parser()' API function (described + next). + + An 'awk_input_buf_t' looks like this: + + typedef struct awk_input { + const char *name; /* filename */ + int fd; /* file descriptor */ + #define INVALID_HANDLE (-1) + void *opaque; /* private data for input parsers */ + int (*get_record)(char **out, struct awk_input *iobuf, + int *errcode, char **rt_start, size_t *rt_len); + ssize_t (*read_func)(); + void (*close_func)(struct awk_input *iobuf); + struct stat sbuf; /* stat buf */ + } awk_input_buf_t; + + The fields can be divided into two categories: those for use +(initially, at least) by 'XXX_can_take_file()', and those for use by +'XXX_take_control_of()'. The first group of fields and their uses are +as follows: + +'const char *name;' + The name of the file. + +'int fd;' + A file descriptor for the file. If 'gawk' was able to open the + file, then 'fd' will _not_ be equal to 'INVALID_HANDLE'. + Otherwise, it will. + +'struct stat sbuf;' + If the file descriptor is valid, then 'gawk' will have filled in + this structure via a call to the 'fstat()' system call. + + The 'XXX_can_take_file()' function should examine these fields and +decide if the input parser should be used for the file. The decision +can be made based upon 'gawk' state (the value of a variable defined +previously by the extension and set by 'awk' code), the name of the +file, whether or not the file descriptor is valid, the information in +the 'struct stat', or any combination of these factors. + + Once 'XXX_can_take_file()' has returned true, and 'gawk' has decided +to use your input parser, it calls 'XXX_take_control_of()'. That +function then fills either the 'get_record' field or the 'read_func' +field in the 'awk_input_buf_t'. It must also ensure that 'fd' is _not_ +set to 'INVALID_HANDLE'. The following list describes the fields that +may be filled by 'XXX_take_control_of()': + +'void *opaque;' + This is used to hold any state information needed by the input + parser for this file. It is "opaque" to 'gawk'. The input parser + is not required to use this pointer. + +'int (*get_record)(char **out,' +' struct awk_input *iobuf,' +' int *errcode,' +' char **rt_start,' +' size_t *rt_len);' + This function pointer should point to a function that creates the + input records. Said function is the core of the input parser. Its + behavior is described in the text following this list. + +'ssize_t (*read_func)();' + This function pointer should point to a function that has the same + behavior as the standard POSIX 'read()' system call. It is an + alternative to the 'get_record' pointer. Its behavior is also + described in the text following this list. + +'void (*close_func)(struct awk_input *iobuf);' + This function pointer should point to a function that does the + "teardown." It should release any resources allocated by + 'XXX_take_control_of()'. It may also close the file. If it does + so, it should set the 'fd' field to 'INVALID_HANDLE'. + + If 'fd' is still not 'INVALID_HANDLE' after the call to this + function, 'gawk' calls the regular 'close()' system call. + + Having a "teardown" function is optional. If your input parser + does not need it, do not set this field. Then, 'gawk' calls the + regular 'close()' system call on the file descriptor, so it should + be valid. + + The 'XXX_get_record()' function does the work of creating input +records. The parameters are as follows: + +'char **out' + This is a pointer to a 'char *' variable that is set to point to + the record. 'gawk' makes its own copy of the data, so the + extension must manage this storage. + +'struct awk_input *iobuf' + This is the 'awk_input_buf_t' for the file. The fields should be + used for reading data ('fd') and for managing private state + ('opaque'), if any. + +'int *errcode' + If an error occurs, '*errcode' should be set to an appropriate code + from '<errno.h>'. + +'char **rt_start' +'size_t *rt_len' + If the concept of a "record terminator" makes sense, then + '*rt_start' should be set to point to the data to be used for 'RT', + and '*rt_len' should be set to the length of the data. Otherwise, + '*rt_len' should be set to zero. 'gawk' makes its own copy of this + data, so the extension must manage this storage. + + The return value is the length of the buffer pointed to by '*out', or +'EOF' if end-of-file was reached or an error occurred. + + It is guaranteed that 'errcode' is a valid pointer, so there is no +need to test for a 'NULL' value. 'gawk' sets '*errcode' to zero, so +there is no need to set it unless an error occurs. + + If an error does occur, the function should return 'EOF' and set +'*errcode' to a value greater than zero. In that case, if '*errcode' +does not equal zero, 'gawk' automatically updates the 'ERRNO' variable +based on the value of '*errcode'. (In general, setting '*errcode = +errno' should do the right thing.) + + As an alternative to supplying a function that returns an input +record, you may instead supply a function that simply reads bytes, and +let 'gawk' parse the data into records. If you do so, the data should +be returned in the multibyte encoding of the current locale. Such a +function should follow the same behavior as the 'read()' system call, +and you fill in the 'read_func' pointer with its address in the +'awk_input_buf_t' structure. + + By default, 'gawk' sets the 'read_func' pointer to point to the +'read()' system call. So your extension need not set this field +explicitly. + + NOTE: You must choose one method or the other: either a function + that returns a record, or one that returns raw data. In + particular, if you supply a function to get a record, 'gawk' will + call it, and will never call the raw read function. + + 'gawk' ships with a sample extension that reads directories, +returning records for each entry in a directory (*note Extension Sample +Readdir::). You may wish to use that code as a guide for writing your +own input parser. + + When writing an input parser, you should think about (and document) +how it is expected to interact with 'awk' code. You may want it to +always be called, and to take effect as appropriate (as the 'readdir' +extension does). Or you may want it to take effect based upon the value +of an 'awk' variable, as the XML extension from the 'gawkextlib' project +does (*note gawkextlib::). In the latter case, code in a 'BEGINFILE' +rule can look at 'FILENAME' and 'ERRNO' to decide whether or not to +activate an input parser (*note BEGINFILE/ENDFILE::). + + You register your input parser with the following function: + +'void register_input_parser(awk_input_parser_t *input_parser);' + Register the input parser pointed to by 'input_parser' with 'gawk'. + + +File: gawk.info, Node: Output Wrappers, Next: Two-way processors, Prev: Input Parsers, Up: Registration Functions + +16.4.5.5 Customized Output Wrappers +................................... + +An "output wrapper" is the mirror image of an input parser. It allows +an extension to take over the output to a file opened with the '>' or +'>>' I/O redirection operators (*note Redirection::). + + The output wrapper is very similar to the input parser structure: + + typedef struct awk_output_wrapper { + const char *name; /* name of the wrapper */ + awk_bool_t (*can_take_file)(const awk_output_buf_t *outbuf); + awk_bool_t (*take_control_of)(awk_output_buf_t *outbuf); + awk_const struct awk_output_wrapper *awk_const next; /* for gawk */ + } awk_output_wrapper_t; + + The members are as follows: + +'const char *name;' + This is the name of the output wrapper. + +'awk_bool_t (*can_take_file)(const awk_output_buf_t *outbuf);' + This points to a function that examines the information in the + 'awk_output_buf_t' structure pointed to by 'outbuf'. It should + return true if the output wrapper wants to take over the file, and + false otherwise. It should not change any state (variable values, + etc.) within 'gawk'. + +'awk_bool_t (*take_control_of)(awk_output_buf_t *outbuf);' + The function pointed to by this field is called when 'gawk' decides + to let the output wrapper take control of the file. It should fill + in appropriate members of the 'awk_output_buf_t' structure, as + described next, and return true if successful, false otherwise. + +'awk_const struct output_wrapper *awk_const next;' + This is for use by 'gawk'; therefore it is marked 'awk_const' so + that the extension cannot modify it. + + The 'awk_output_buf_t' structure looks like this: + + typedef struct awk_output_buf { + const char *name; /* name of output file */ + const char *mode; /* mode argument to fopen */ + FILE *fp; /* stdio file pointer */ + awk_bool_t redirected; /* true if a wrapper is active */ + void *opaque; /* for use by output wrapper */ + size_t (*gawk_fwrite)(const void *buf, size_t size, size_t count, + FILE *fp, void *opaque); + int (*gawk_fflush)(FILE *fp, void *opaque); + int (*gawk_ferror)(FILE *fp, void *opaque); + int (*gawk_fclose)(FILE *fp, void *opaque); + } awk_output_buf_t; + + Here too, your extension will define 'XXX_can_take_file()' and +'XXX_take_control_of()' functions that examine and update data members +in the 'awk_output_buf_t'. The data members are as follows: + +'const char *name;' + The name of the output file. + +'const char *mode;' + The mode string (as would be used in the second argument to + 'fopen()') with which the file was opened. + +'FILE *fp;' + The 'FILE' pointer from '<stdio.h>'. 'gawk' opens the file before + attempting to find an output wrapper. + +'awk_bool_t redirected;' + This field must be set to true by the 'XXX_take_control_of()' + function. + +'void *opaque;' + This pointer is opaque to 'gawk'. The extension should use it to + store a pointer to any private data associated with the file. + +'size_t (*gawk_fwrite)(const void *buf, size_t size, size_t count,' +' FILE *fp, void *opaque);' +'int (*gawk_fflush)(FILE *fp, void *opaque);' +'int (*gawk_ferror)(FILE *fp, void *opaque);' +'int (*gawk_fclose)(FILE *fp, void *opaque);' + These pointers should be set to point to functions that perform the + equivalent function as the '<stdio.h>' functions do, if + appropriate. 'gawk' uses these function pointers for all output. + 'gawk' initializes the pointers to point to internal "pass-through" + functions that just call the regular '<stdio.h>' functions, so an + extension only needs to redefine those functions that are + appropriate for what it does. + + The 'XXX_can_take_file()' function should make a decision based upon +the 'name' and 'mode' fields, and any additional state (such as 'awk' +variable values) that is appropriate. + + When 'gawk' calls 'XXX_take_control_of()', that function should fill +in the other fields as appropriate, except for 'fp', which it should +just use normally. + + You register your output wrapper with the following function: + +'void register_output_wrapper(awk_output_wrapper_t *output_wrapper);' + Register the output wrapper pointed to by 'output_wrapper' with + 'gawk'. + + +File: gawk.info, Node: Two-way processors, Prev: Output Wrappers, Up: Registration Functions + +16.4.5.6 Customized Two-way Processors +...................................... + +A "two-way processor" combines an input parser and an output wrapper for +two-way I/O with the '|&' operator (*note Redirection::). It makes +identical use of the 'awk_input_parser_t' and 'awk_output_buf_t' +structures as described earlier. + + A two-way processor is represented by the following structure: + + typedef struct awk_two_way_processor { + const char *name; /* name of the two-way processor */ + awk_bool_t (*can_take_two_way)(const char *name); + awk_bool_t (*take_control_of)(const char *name, + awk_input_buf_t *inbuf, + awk_output_buf_t *outbuf); + awk_const struct awk_two_way_processor *awk_const next; /* for gawk */ + } awk_two_way_processor_t; + + The fields are as follows: + +'const char *name;' + The name of the two-way processor. + +'awk_bool_t (*can_take_two_way)(const char *name);' + The function pointed to by this field should return true if it + wants to take over two-way I/O for this file name. It should not + change any state (variable values, etc.) within 'gawk'. + +'awk_bool_t (*take_control_of)(const char *name,' +' awk_input_buf_t *inbuf,' +' awk_output_buf_t *outbuf);' + The function pointed to by this field should fill in the + 'awk_input_buf_t' and 'awk_output_buf_t' structures pointed to by + 'inbuf' and 'outbuf', respectively. These structures were + described earlier. + +'awk_const struct two_way_processor *awk_const next;' + This is for use by 'gawk'; therefore it is marked 'awk_const' so + that the extension cannot modify it. + + As with the input parser and output processor, you provide "yes I can +take this" and "take over for this" functions, 'XXX_can_take_two_way()' +and 'XXX_take_control_of()'. + + You register your two-way processor with the following function: + +'void register_two_way_processor(awk_two_way_processor_t *two_way_processor);' + Register the two-way processor pointed to by 'two_way_processor' + with 'gawk'. + + +File: gawk.info, Node: Printing Messages, Next: Updating ERRNO, Prev: Registration Functions, Up: Extension API Description + +16.4.6 Printing Messages +------------------------ + +You can print different kinds of warning messages from your extension, +as described here. Note that for these functions, you must pass in the +extension ID received from 'gawk' when the extension was loaded:(1) + +'void fatal(awk_ext_id_t id, const char *format, ...);' + Print a message and then cause 'gawk' to exit immediately. + +'void nonfatal(awk_ext_id_t id, const char *format, ...);' + Print a nonfatal error message. + +'void warning(awk_ext_id_t id, const char *format, ...);' + Print a warning message. + +'void lintwarn(awk_ext_id_t id, const char *format, ...);' + Print a "lint warning." Normally this is the same as printing a + warning message, but if 'gawk' was invoked with '--lint=fatal', + then lint warnings become fatal error messages. + + All of these functions are otherwise like the C 'printf()' family of +functions, where the 'format' parameter is a string with literal +characters and formatting codes intermixed. + + ---------- Footnotes ---------- + + (1) Because the API uses only ISO C 90 features, it cannot make use +of the ISO C 99 variadic macro feature to hide that parameter. More's +the pity. + + +File: gawk.info, Node: Updating ERRNO, Next: Requesting Values, Prev: Printing Messages, Up: Extension API Description + +16.4.7 Updating 'ERRNO' +----------------------- + +The following functions allow you to update the 'ERRNO' variable: + +'void update_ERRNO_int(int errno_val);' + Set 'ERRNO' to the string equivalent of the error code in + 'errno_val'. The value should be one of the defined error codes in + '<errno.h>', and 'gawk' turns it into a (possibly translated) + string using the C 'strerror()' function. + +'void update_ERRNO_string(const char *string);' + Set 'ERRNO' directly to the string value of 'ERRNO'. 'gawk' makes + a copy of the value of 'string'. + +'void unset_ERRNO(void);' + Unset 'ERRNO'. + + +File: gawk.info, Node: Requesting Values, Next: Accessing Parameters, Prev: Updating ERRNO, Up: Extension API Description + +16.4.8 Requesting Values +------------------------ + +All of the functions that return values from 'gawk' work in the same +way. You pass in an 'awk_valtype_t' value to indicate what kind of +value you expect. If the actual value matches what you requested, the +function returns true and fills in the 'awk_value_t' result. Otherwise, +the function returns false, and the 'val_type' member indicates the type +of the actual value. You may then print an error message or reissue the +request for the actual value type, as appropriate. This behavior is +summarized in *note Table 16.1: table-value-types-returned. + + Type of Actual Value +-------------------------------------------------------------------------- + String Number Array Undefined +------------------------------------------------------------------------------ + String String String False False + Number Number if Number False False + can be + converted, + else false +Type Array False False Array False +Requested Scalar Scalar Scalar False False + Undefined String Number Array Undefined + Value False False False False + cookie + +Table 16.1: API value types returned + + +File: gawk.info, Node: Accessing Parameters, Next: Symbol Table Access, Prev: Requesting Values, Up: Extension API Description + +16.4.9 Accessing and Updating Parameters +---------------------------------------- + +Two functions give you access to the arguments (parameters) passed to +your extension function. They are: + +'awk_bool_t get_argument(size_t count,' +' awk_valtype_t wanted,' +' awk_value_t *result);' + Fill in the 'awk_value_t' structure pointed to by 'result' with the + 'count'th argument. Return true if the actual type matches + 'wanted', and false otherwise. In the latter case, + 'result->val_type' indicates the actual type (*note Table 16.1: + table-value-types-returned.). Counts are zero-based--the first + argument is numbered zero, the second one, and so on. 'wanted' + indicates the type of value expected. + +'awk_bool_t set_argument(size_t count, awk_array_t array);' + Convert a parameter that was undefined into an array; this provides + call by reference for arrays. Return false if 'count' is too big, + or if the argument's type is not undefined. *Note Array + Manipulation:: for more information on creating arrays. + + +File: gawk.info, Node: Symbol Table Access, Next: Array Manipulation, Prev: Accessing Parameters, Up: Extension API Description + +16.4.10 Symbol Table Access +--------------------------- + +Two sets of routines provide access to global variables, and one set +allows you to create and release cached values. + +* Menu: + +* Symbol table by name:: Accessing variables by name. +* Symbol table by cookie:: Accessing variables by "cookie". +* Cached values:: Creating and using cached values. + + +File: gawk.info, Node: Symbol table by name, Next: Symbol table by cookie, Up: Symbol Table Access + +16.4.10.1 Variable Access and Update by Name +............................................ + +The following routines provide the ability to access and update global +'awk'-level variables by name. In compiler terminology, identifiers of +different kinds are termed "symbols", thus the "sym" in the routines' +names. The data structure that stores information about symbols is +termed a "symbol table". The functions are as follows: + +'awk_bool_t sym_lookup(const char *name,' +' awk_valtype_t wanted,' +' awk_value_t *result);' + Fill in the 'awk_value_t' structure pointed to by 'result' with the + value of the variable named by the string 'name', which is a + regular C string. 'wanted' indicates the type of value expected. + Return true if the actual type matches 'wanted', and false + otherwise. In the latter case, 'result->val_type' indicates the + actual type (*note Table 16.1: table-value-types-returned.). + +'awk_bool_t sym_update(const char *name, awk_value_t *value);' + Update the variable named by the string 'name', which is a regular + C string. The variable is added to 'gawk''s symbol table if it is + not there. Return true if everything worked, and false otherwise. + + Changing types (scalar to array or vice versa) of an existing + variable is _not_ allowed, nor may this routine be used to update + an array. This routine cannot be used to update any of the + predefined variables (such as 'ARGC' or 'NF'). + + An extension can look up the value of 'gawk''s special variables. +However, with the exception of the 'PROCINFO' array, an extension cannot +change any of those variables. + + CAUTION: It is possible for the lookup of 'PROCINFO' to fail. This + happens if the 'awk' program being run does not reference + 'PROCINFO'; in this case, 'gawk' doesn't bother to create the array + and populate it. + + +File: gawk.info, Node: Symbol table by cookie, Next: Cached values, Prev: Symbol table by name, Up: Symbol Table Access + +16.4.10.2 Variable Access and Update by Cookie +.............................................. + +A "scalar cookie" is an opaque handle that provides access to a global +variable or array. It is an optimization that avoids looking up +variables in 'gawk''s symbol table every time access is needed. This +was discussed earlier, in *note General Data Types::. + + The following functions let you work with scalar cookies: + +'awk_bool_t sym_lookup_scalar(awk_scalar_t cookie,' +' awk_valtype_t wanted,' +' awk_value_t *result);' + Retrieve the current value of a scalar cookie. Once you have + obtained a scalar cookie using 'sym_lookup()', you can use this + function to get its value more efficiently. Return false if the + value cannot be retrieved. + +'awk_bool_t sym_update_scalar(awk_scalar_t cookie, awk_value_t *value);' + Update the value associated with a scalar cookie. Return false if + the new value is not of type 'AWK_STRING' or 'AWK_NUMBER'. Here + too, the predefined variables may not be updated. + + It is not obvious at first glance how to work with scalar cookies or +what their raison d'e^tre really is. In theory, the 'sym_lookup()' and +'sym_update()' routines are all you really need to work with variables. +For example, you might have code that looks up the value of a variable, +evaluates a condition, and then possibly changes the value of the +variable based on the result of that evaluation, like so: + + /* do_magic --- do something really great */ + + static awk_value_t * + do_magic(int nargs, awk_value_t *result) + { + awk_value_t value; + + if ( sym_lookup("MAGIC_VAR", AWK_NUMBER, & value) + && some_condition(value.num_value)) { + value.num_value += 42; + sym_update("MAGIC_VAR", & value); + } + + return make_number(0.0, result); + } + +This code looks (and is) simple and straightforward. So what's the +problem? + + Well, consider what happens if 'awk'-level code associated with your +extension calls the 'magic()' function (implemented in C by +'do_magic()'), once per record, while processing hundreds of thousands +or millions of records. The 'MAGIC_VAR' variable is looked up in the +symbol table once or twice per function call! + + The symbol table lookup is really pure overhead; it is considerably +more efficient to get a cookie that represents the variable, and use +that to get the variable's value and update it as needed.(1) + + Thus, the way to use cookies is as follows. First, install your +extension's variable in 'gawk''s symbol table using 'sym_update()', as +usual. Then get a scalar cookie for the variable using 'sym_lookup()': + + static awk_scalar_t magic_var_cookie; /* cookie for MAGIC_VAR */ + + static void + my_extension_init() + { + awk_value_t value; + + /* install initial value */ + sym_update("MAGIC_VAR", make_number(42.0, & value)); + + /* get the cookie */ + sym_lookup("MAGIC_VAR", AWK_SCALAR, & value); + + /* save the cookie */ + magic_var_cookie = value.scalar_cookie; + ... + } + + Next, use the routines in this minor node for retrieving and updating +the value through the cookie. Thus, 'do_magic()' now becomes something +like this: + + /* do_magic --- do something really great */ + + static awk_value_t * + do_magic(int nargs, awk_value_t *result) + { + awk_value_t value; + + if ( sym_lookup_scalar(magic_var_cookie, AWK_NUMBER, & value) + && some_condition(value.num_value)) { + value.num_value += 42; + sym_update_scalar(magic_var_cookie, & value); + } + ... + + return make_number(0.0, result); + } + + NOTE: The previous code omitted error checking for presentation + purposes. Your extension code should be more robust and carefully + check the return values from the API functions. + + ---------- Footnotes ---------- + + (1) The difference is measurable and quite real. Trust us. + + +File: gawk.info, Node: Cached values, Prev: Symbol table by cookie, Up: Symbol Table Access + +16.4.10.3 Creating and Using Cached Values +.......................................... + +The routines in this minor node allow you to create and release cached +values. Like scalar cookies, in theory, cached values are not +necessary. You can create numbers and strings using the functions in +*note Constructor Functions::. You can then assign those values to +variables using 'sym_update()' or 'sym_update_scalar()', as you like. + + However, you can understand the point of cached values if you +remember that _every_ string value's storage _must_ come from +'gawk_malloc()', 'gawk_calloc()', or 'gawk_realloc()'. If you have 20 +variables, all of which have the same string value, you must create 20 +identical copies of the string.(1) + + It is clearly more efficient, if possible, to create a value once, +and then tell 'gawk' to reuse the value for multiple variables. That is +what the routines in this minor node let you do. The functions are as +follows: + +'awk_bool_t create_value(awk_value_t *value, awk_value_cookie_t *result);' + Create a cached string or numeric value from 'value' for efficient + later assignment. Only values of type 'AWK_NUMBER' and + 'AWK_STRING' are allowed. Any other type is rejected. + 'AWK_UNDEFINED' could be allowed, but doing so would result in + inferior performance. + +'awk_bool_t release_value(awk_value_cookie_t vc);' + Release the memory associated with a value cookie obtained from + 'create_value()'. + + You use value cookies in a fashion similar to the way you use scalar +cookies. In the extension initialization routine, you create the value +cookie: + + static awk_value_cookie_t answer_cookie; /* static value cookie */ + + static void + my_extension_init() + { + awk_value_t value; + char *long_string; + size_t long_string_len; + + /* code from earlier */ + ... + /* ... fill in long_string and long_string_len ... */ + make_malloced_string(long_string, long_string_len, & value); + create_value(& value, & answer_cookie); /* create cookie */ + ... + } + + Once the value is created, you can use it as the value of any number +of variables: + + static awk_value_t * + do_magic(int nargs, awk_value_t *result) + { + awk_value_t new_value; + + ... /* as earlier */ + + value.val_type = AWK_VALUE_COOKIE; + value.value_cookie = answer_cookie; + sym_update("VAR1", & value); + sym_update("VAR2", & value); + ... + sym_update("VAR100", & value); + ... + } + +Using value cookies in this way saves considerable storage, as all of +'VAR1' through 'VAR100' share the same value. + + You might be wondering, "Is this sharing problematic? What happens +if 'awk' code assigns a new value to 'VAR1'; are all the others changed +too?" + + That's a great question. The answer is that no, it's not a problem. +Internally, 'gawk' uses "reference-counted strings". This means that +many variables can share the same string value, and 'gawk' keeps track +of the usage. When a variable's value changes, 'gawk' simply decrements +the reference count on the old value and updates the variable to use the +new value. + + Finally, as part of your cleanup action (*note Exit Callback +Functions::) you should release any cached values that you created, +using 'release_value()'. + + ---------- Footnotes ---------- + + (1) Numeric values are clearly less problematic, requiring only a C +'double' to store. + + +File: gawk.info, Node: Array Manipulation, Next: Redirection API, Prev: Symbol Table Access, Up: Extension API Description + +16.4.11 Array Manipulation +-------------------------- + +The primary data structure(1) in 'awk' is the associative array (*note +Arrays::). Extensions need to be able to manipulate 'awk' arrays. The +API provides a number of data structures for working with arrays, +functions for working with individual elements, and functions for +working with arrays as a whole. This includes the ability to "flatten" +an array so that it is easy for C code to traverse every element in an +array. The array data structures integrate nicely with the data +structures for values to make it easy to both work with and create true +arrays of arrays (*note General Data Types::). + +* Menu: + +* Array Data Types:: Data types for working with arrays. +* Array Functions:: Functions for working with arrays. +* Flattening Arrays:: How to flatten arrays. +* Creating Arrays:: How to create and populate arrays. + + ---------- Footnotes ---------- + + (1) OK, the only data structure. + + +File: gawk.info, Node: Array Data Types, Next: Array Functions, Up: Array Manipulation + +16.4.11.1 Array Data Types +.......................... + +The data types associated with arrays are as follows: + +'typedef void *awk_array_t;' + If you request the value of an array variable, you get back an + 'awk_array_t' value. This value is opaque(1) to the extension; it + uniquely identifies the array but can only be used by passing it + into API functions or receiving it from API functions. This is + very similar to way 'FILE *' values are used with the '<stdio.h>' + library routines. + +'typedef struct awk_element {' +' /* convenience linked list pointer, not used by gawk */' +' struct awk_element *next;' +' enum {' +' AWK_ELEMENT_DEFAULT = 0, /* set by gawk */' +' AWK_ELEMENT_DELETE = 1 /* set by extension */' +' } flags;' +' awk_value_t index;' +' awk_value_t value;' +'} awk_element_t;' + The 'awk_element_t' is a "flattened" array element. 'awk' produces + an array of these inside the 'awk_flat_array_t' (see the next + item). Individual elements may be marked for deletion. New + elements must be added individually, one at a time, using the + separate API for that purpose. The fields are as follows: + + 'struct awk_element *next;' + This pointer is for the convenience of extension writers. It + allows an extension to create a linked list of new elements + that can then be added to an array in a loop that traverses + the list. + + 'enum { ... } flags;' + A set of flag values that convey information between the + extension and 'gawk'. Currently there is only one: + 'AWK_ELEMENT_DELETE'. Setting it causes 'gawk' to delete the + element from the original array upon release of the flattened + array. + + 'index' + 'value' + The index and value of the element, respectively. _All_ + memory pointed to by 'index' and 'value' belongs to 'gawk'. + +'typedef struct awk_flat_array {' +' awk_const void *awk_const opaque1; /* for use by gawk */' +' awk_const void *awk_const opaque2; /* for use by gawk */' +' awk_const size_t count; /* how many elements */' +' awk_element_t elements[1]; /* will be extended */' +'} awk_flat_array_t;' + This is a flattened array. When an extension gets one of these + from 'gawk', the 'elements' array is of actual size 'count'. The + 'opaque1' and 'opaque2' pointers are for use by 'gawk'; therefore + they are marked 'awk_const' so that the extension cannot modify + them. + + ---------- Footnotes ---------- + + (1) It is also a "cookie," but the 'gawk' developers did not wish to +overuse this term. + + +File: gawk.info, Node: Array Functions, Next: Flattening Arrays, Prev: Array Data Types, Up: Array Manipulation + +16.4.11.2 Array Functions +......................... + +The following functions relate to individual array elements: + +'awk_bool_t get_element_count(awk_array_t a_cookie, size_t *count);' + For the array represented by 'a_cookie', place in '*count' the + number of elements it contains. A subarray counts as a single + element. Return false if there is an error. + +'awk_bool_t get_array_element(awk_array_t a_cookie,' +' const awk_value_t *const index,' +' awk_valtype_t wanted,' +' awk_value_t *result);' + For the array represented by 'a_cookie', return in '*result' the + value of the element whose index is 'index'. 'wanted' specifies + the type of value you wish to retrieve. Return false if 'wanted' + does not match the actual type or if 'index' is not in the array + (*note Table 16.1: table-value-types-returned.). + + The value for 'index' can be numeric, in which case 'gawk' converts + it to a string. Using nonintegral values is possible, but requires + that you understand how such values are converted to strings (*note + Conversion::); thus, using integral values is safest. + + As with _all_ strings passed into 'gawk' from an extension, the + string value of 'index' must come from 'gawk_malloc()', + 'gawk_calloc()', or 'gawk_realloc()', and 'gawk' releases the + storage. + +'awk_bool_t set_array_element(awk_array_t a_cookie,' +' const awk_value_t *const index,' +' const awk_value_t *const value);' + In the array represented by 'a_cookie', create or modify the + element whose index is given by 'index'. The 'ARGV' and 'ENVIRON' + arrays may not be changed, although the 'PROCINFO' array can be. + +'awk_bool_t set_array_element_by_elem(awk_array_t a_cookie,' +' awk_element_t element);' + Like 'set_array_element()', but take the 'index' and 'value' from + 'element'. This is a convenience macro. + +'awk_bool_t del_array_element(awk_array_t a_cookie,' +' const awk_value_t* const index);' + Remove the element with the given index from the array represented + by 'a_cookie'. Return true if the element was removed, or false if + the element did not exist in the array. + + The following functions relate to arrays as a whole: + +'awk_array_t create_array(void);' + Create a new array to which elements may be added. *Note Creating + Arrays:: for a discussion of how to create a new array and add + elements to it. + +'awk_bool_t clear_array(awk_array_t a_cookie);' + Clear the array represented by 'a_cookie'. Return false if there + was some kind of problem, true otherwise. The array remains an + array, but after calling this function, it has no elements. This + is equivalent to using the 'delete' statement (*note Delete::). + +'awk_bool_t flatten_array(awk_array_t a_cookie, awk_flat_array_t **data);' + For the array represented by 'a_cookie', create an + 'awk_flat_array_t' structure and fill it in. Set the pointer whose + address is passed as 'data' to point to this structure. Return + true upon success, or false otherwise. *Note Flattening Arrays::, + for a discussion of how to flatten an array and work with it. + +'awk_bool_t release_flattened_array(awk_array_t a_cookie,' +' awk_flat_array_t *data);' + When done with a flattened array, release the storage using this + function. You must pass in both the original array cookie and the + address of the created 'awk_flat_array_t' structure. The function + returns true upon success, false otherwise. + + +File: gawk.info, Node: Flattening Arrays, Next: Creating Arrays, Prev: Array Functions, Up: Array Manipulation + +16.4.11.3 Working With All The Elements of an Array +................................................... + +To "flatten" an array is to create a structure that represents the full +array in a fashion that makes it easy for C code to traverse the entire +array. Some of the code in 'extension/testext.c' does this, and also +serves as a nice example showing how to use the APIs. + + We walk through that part of the code one step at a time. First, the +'gawk' script that drives the test extension: + + @load "testext" + BEGIN { + n = split("blacky rusty sophie raincloud lucky", pets) + printf("pets has %d elements\n", length(pets)) + ret = dump_array_and_delete("pets", "3") + printf("dump_array_and_delete(pets) returned %d\n", ret) + if ("3" in pets) + printf("dump_array_and_delete() did NOT remove index \"3\"!\n") + else + printf("dump_array_and_delete() did remove index \"3\"!\n") + print "" + } + +This code creates an array with 'split()' (*note String Functions::) and +then calls 'dump_array_and_delete()'. That function looks up the array +whose name is passed as the first argument, and deletes the element at +the index passed in the second argument. The 'awk' code then prints the +return value and checks if the element was indeed deleted. Here is the +C code that implements 'dump_array_and_delete()'. It has been edited +slightly for presentation. + + The first part declares variables, sets up the default return value +in 'result', and checks that the function was called with the correct +number of arguments: + + static awk_value_t * + dump_array_and_delete(int nargs, awk_value_t *result) + { + awk_value_t value, value2, value3; + awk_flat_array_t *flat_array; + size_t count; + char *name; + int i; + + assert(result != NULL); + make_number(0.0, result); + + if (nargs != 2) { + printf("dump_array_and_delete: nargs not right " + "(%d should be 2)\n", nargs); + goto out; + } + + The function then proceeds in steps, as follows. First, retrieve the +name of the array, passed as the first argument, followed by the array +itself. If either operation fails, print an error message and return: + + /* get argument named array as flat array and print it */ + if (get_argument(0, AWK_STRING, & value)) { + name = value.str_value.str; + if (sym_lookup(name, AWK_ARRAY, & value2)) + printf("dump_array_and_delete: sym_lookup of %s passed\n", + name); + else { + printf("dump_array_and_delete: sym_lookup of %s failed\n", + name); + goto out; + } + } else { + printf("dump_array_and_delete: get_argument(0) failed\n"); + goto out; + } + + For testing purposes and to make sure that the C code sees the same +number of elements as the 'awk' code, the second step is to get the +count of elements in the array and print it: + + if (! get_element_count(value2.array_cookie, & count)) { + printf("dump_array_and_delete: get_element_count failed\n"); + goto out; + } + + printf("dump_array_and_delete: incoming size is %lu\n", + (unsigned long) count); + + The third step is to actually flatten the array, and then to +double-check that the count in the 'awk_flat_array_t' is the same as the +count just retrieved: + + if (! flatten_array(value2.array_cookie, & flat_array)) { + printf("dump_array_and_delete: could not flatten array\n"); + goto out; + } + + if (flat_array->count != count) { + printf("dump_array_and_delete: flat_array->count (%lu)" + " != count (%lu)\n", + (unsigned long) flat_array->count, + (unsigned long) count); + goto out; + } + + The fourth step is to retrieve the index of the element to be +deleted, which was passed as the second argument. Remember that +argument counts passed to 'get_argument()' are zero-based, and thus the +second argument is numbered one: + + if (! get_argument(1, AWK_STRING, & value3)) { + printf("dump_array_and_delete: get_argument(1) failed\n"); + goto out; + } + + The fifth step is where the "real work" is done. The function loops +over every element in the array, printing the index and element values. +In addition, upon finding the element with the index that is supposed to +be deleted, the function sets the 'AWK_ELEMENT_DELETE' bit in the +'flags' field of the element. When the array is released, 'gawk' +traverses the flattened array, and deletes any elements that have this +flag bit set: + + for (i = 0; i < flat_array->count; i++) { + printf("\t%s[\"%.*s\"] = %s\n", + name, + (int) flat_array->elements[i].index.str_value.len, + flat_array->elements[i].index.str_value.str, + valrep2str(& flat_array->elements[i].value)); + + if (strcmp(value3.str_value.str, + flat_array->elements[i].index.str_value.str) == 0) { + flat_array->elements[i].flags |= AWK_ELEMENT_DELETE; + printf("dump_array_and_delete: marking element \"%s\" " + "for deletion\n", + flat_array->elements[i].index.str_value.str); + } + } + + The sixth step is to release the flattened array. This tells 'gawk' +that the extension is no longer using the array, and that it should +delete any elements marked for deletion. 'gawk' also frees any storage +that was allocated, so you should not use the pointer ('flat_array' in +this code) once you have called 'release_flattened_array()': + + if (! release_flattened_array(value2.array_cookie, flat_array)) { + printf("dump_array_and_delete: could not release flattened array\n"); + goto out; + } + + Finally, because everything was successful, the function sets the +return value to success, and returns: + + make_number(1.0, result); + out: + return result; + } + + Here is the output from running this part of the test: + + pets has 5 elements + dump_array_and_delete: sym_lookup of pets passed + dump_array_and_delete: incoming size is 5 + pets["1"] = "blacky" + pets["2"] = "rusty" + pets["3"] = "sophie" + dump_array_and_delete: marking element "3" for deletion + pets["4"] = "raincloud" + pets["5"] = "lucky" + dump_array_and_delete(pets) returned 1 + dump_array_and_delete() did remove index "3"! + + +File: gawk.info, Node: Creating Arrays, Prev: Flattening Arrays, Up: Array Manipulation + +16.4.11.4 How To Create and Populate Arrays +........................................... + +Besides working with arrays created by 'awk' code, you can create arrays +and populate them as you see fit, and then 'awk' code can access them +and manipulate them. + + There are two important points about creating arrays from extension +code: + + * You must install a new array into 'gawk''s symbol table immediately + upon creating it. Once you have done so, you can then populate the + array. + + Similarly, if installing a new array as a subarray of an existing + array, you must add the new array to its parent before adding any + elements to it. + + Thus, the correct way to build an array is to work "top down." + Create the array, and immediately install it in 'gawk''s symbol + table using 'sym_update()', or install it as an element in a + previously existing array using 'set_array_element()'. We show + example code shortly. + + * Due to 'gawk' internals, after using 'sym_update()' to install an + array into 'gawk', you have to retrieve the array cookie from the + value passed in to 'sym_update()' before doing anything else with + it, like so: + + awk_value_t val; + awk_array_t new_array; + + new_array = create_array(); + val.val_type = AWK_ARRAY; + val.array_cookie = new_array; + + /* install array in the symbol table */ + sym_update("array", & val); + + new_array = val.array_cookie; /* YOU MUST DO THIS */ + + If installing an array as a subarray, you must also retrieve the + value of the array cookie after the call to 'set_element()'. + + The following C code is a simple test extension to create an array +with two regular elements and with a subarray. The leading '#include' +directives and boilerplate variable declarations (*note Extension API +Boilerplate::) are omitted for brevity. The first step is to create a +new array and then install it in the symbol table: + + /* create_new_array --- create a named array */ + + static void + create_new_array() + { + awk_array_t a_cookie; + awk_array_t subarray; + awk_value_t index, value; + + a_cookie = create_array(); + value.val_type = AWK_ARRAY; + value.array_cookie = a_cookie; + + if (! sym_update("new_array", & value)) + printf("create_new_array: sym_update(\"new_array\") failed!\n"); + a_cookie = value.array_cookie; + +Note how 'a_cookie' is reset from the 'array_cookie' field in the +'value' structure. + + The second step is to install two regular values into 'new_array': + + (void) make_const_string("hello", 5, & index); + (void) make_const_string("world", 5, & value); + if (! set_array_element(a_cookie, & index, & value)) { + printf("fill_in_array: set_array_element failed\n"); + return; + } + + (void) make_const_string("answer", 6, & index); + (void) make_number(42.0, & value); + if (! set_array_element(a_cookie, & index, & value)) { + printf("fill_in_array: set_array_element failed\n"); + return; + } + + The third step is to create the subarray and install it: + + (void) make_const_string("subarray", 8, & index); + subarray = create_array(); + value.val_type = AWK_ARRAY; + value.array_cookie = subarray; + if (! set_array_element(a_cookie, & index, & value)) { + printf("fill_in_array: set_array_element failed\n"); + return; + } + subarray = value.array_cookie; + + The final step is to populate the subarray with its own element: + + (void) make_const_string("foo", 3, & index); + (void) make_const_string("bar", 3, & value); + if (! set_array_element(subarray, & index, & value)) { + printf("fill_in_array: set_array_element failed\n"); + return; + } + } + + Here is a sample script that loads the extension and then dumps the +array: + + @load "subarray" + + function dumparray(name, array, i) + { + for (i in array) + if (isarray(array[i])) + dumparray(name "[\"" i "\"]", array[i]) + else + printf("%s[\"%s\"] = %s\n", name, i, array[i]) + } + + BEGIN { + dumparray("new_array", new_array); + } + + Here is the result of running the script: + + $ AWKLIBPATH=$PWD ./gawk -f subarray.awk + -| new_array["subarray"]["foo"] = bar + -| new_array["hello"] = world + -| new_array["answer"] = 42 + +(*Note Finding Extensions:: for more information on the 'AWKLIBPATH' +environment variable.) + + +File: gawk.info, Node: Redirection API, Next: Extension API Variables, Prev: Array Manipulation, Up: Extension API Description + +16.4.12 Accessing and Manipulating Redirections +----------------------------------------------- + +The following function allows extensions to access and manipulate +redirections. + +'awk_bool_t get_file(const char *name,' +' size_t name_len,' +' const char *filetype,' +' int fd,' +' const awk_input_buf_t **ibufp,' +' const awk_output_buf_t **obufp);' + Look up a file in 'gawk''s internal redirection table. If 'name' + is 'NULL' or 'name_len' is zero, return data for the currently open + input file corresponding to 'FILENAME'. (This does not access the + 'filetype' argument, so that may be undefined). If the file is not + already open, attempt to open it. The 'filetype' argument must be + zero-terminated and should be one of: + + '">"' + A file opened for output. + + '">>"' + A file opened for append. + + '"<"' + A file opened for input. + + '"|>"' + A pipe opened for output. + + '"|<"' + A pipe opened for input. + + '"|&"' + A two-way coprocess. + + On error, return a 'false' value. Otherwise, return 'true', and + return additional information about the redirection in the 'ibufp' + and 'obufp' pointers. For input redirections, the '*ibufp' value + should be non-'NULL', and '*obufp' should be 'NULL'. For output + redirections, the '*obufp' value should be non-'NULL', and '*ibufp' + should be 'NULL'. For two-way coprocesses, both values should be + non-'NULL'. + + In the usual case, the extension is interested in '(*ibufp)->fd' + and/or 'fileno((*obufp)->fp)'. If the file is not already open, + and the 'fd' argument is non-negative, 'gawk' will use that file + descriptor instead of opening the file in the usual way. If 'fd' + is non-negative, but the file exists already, 'gawk' ignores 'fd' + and returns the existing file. It is the caller's responsibility + to notice that neither the 'fd' in the returned 'awk_input_buf_t' + nor the 'fd' in the returned 'awk_output_buf_t' matches the + requested value. + + Note that supplying a file descriptor is currently _not_ supported + for pipes. However, supplying a file descriptor should work for + input, output, append, and two-way (coprocess) sockets. If + 'filetype' is two-way, 'gawk' assumes that it is a socket! Note + that in the two-way case, the input and output file descriptors may + differ. To check for success, you must check whether either + matches. + + It is anticipated that this API function will be used to implement +I/O multiplexing and a socket library. + + +File: gawk.info, Node: Extension API Variables, Next: Extension API Boilerplate, Prev: Redirection API, Up: Extension API Description + +16.4.13 API Variables +--------------------- + +The API provides two sets of variables. The first provides information +about the version of the API (both with which the extension was +compiled, and with which 'gawk' was compiled). The second provides +information about how 'gawk' was invoked. + +* Menu: + +* Extension Versioning:: API Version information. +* Extension API Informational Variables:: Variables providing information about + 'gawk''s invocation. + + +File: gawk.info, Node: Extension Versioning, Next: Extension API Informational Variables, Up: Extension API Variables + +16.4.13.1 API Version Constants and Variables +............................................. + +The API provides both a "major" and a "minor" version number. The API +versions are available at compile time as C preprocessor defines to +support conditional compilation, and as enum constants to facilitate +debugging: + +API Version C preprocessor define enum constant +--------------------------------------------------------------------------- +Major gawk_api_major_version GAWK_API_MAJOR_VERSION +Minor gawk_api_minor_version GAWK_API_MINOR_VERSION + +Table 16.2: gawk API version constants + + The minor version increases when new functions are added to the API. +Such new functions are always added to the end of the API 'struct'. + + The major version increases (and the minor version is reset to zero) +if any of the data types change size or member order, or if any of the +existing functions change signature. + + It could happen that an extension may be compiled against one version +of the API but loaded by a version of 'gawk' using a different version. +For this reason, the major and minor API versions of the running 'gawk' +are included in the API 'struct' as read-only constant integers: + +'api->major_version' + The major version of the running 'gawk' + +'api->minor_version' + The minor version of the running 'gawk' + + It is up to the extension to decide if there are API +incompatibilities. Typically, a check like this is enough: + + if (api->major_version != GAWK_API_MAJOR_VERSION + || api->minor_version < GAWK_API_MINOR_VERSION) { + fprintf(stderr, "foo_extension: version mismatch with gawk!\n"); + fprintf(stderr, "\tmy version (%d, %d), gawk version (%d, %d)\n", + GAWK_API_MAJOR_VERSION, GAWK_API_MINOR_VERSION, + api->major_version, api->minor_version); + exit(1); + } + + Such code is included in the boilerplate 'dl_load_func()' macro +provided in 'gawkapi.h' (discussed in *note Extension API +Boilerplate::). + + +File: gawk.info, Node: Extension API Informational Variables, Prev: Extension Versioning, Up: Extension API Variables + +16.4.13.2 Informational Variables +................................. + +The API provides access to several variables that describe whether the +corresponding command-line options were enabled when 'gawk' was invoked. +The variables are: + +'do_debug' + This variable is true if 'gawk' was invoked with '--debug' option. + +'do_lint' + This variable is true if 'gawk' was invoked with '--lint' option. + +'do_mpfr' + This variable is true if 'gawk' was invoked with '--bignum' option. + +'do_profile' + This variable is true if 'gawk' was invoked with '--profile' + option. + +'do_sandbox' + This variable is true if 'gawk' was invoked with '--sandbox' + option. + +'do_traditional' + This variable is true if 'gawk' was invoked with '--traditional' + option. + + The value of 'do_lint' can change if 'awk' code modifies the 'LINT' +predefined variable (*note Built-in Variables::). The others should not +change during execution. + + +File: gawk.info, Node: Extension API Boilerplate, Prev: Extension API Variables, Up: Extension API Description + +16.4.14 Boilerplate Code +------------------------ + +As mentioned earlier (*note Extension Mechanism Outline::), the function +definitions as presented are really macros. To use these macros, your +extension must provide a small amount of boilerplate code (variables and +functions) toward the top of your source file, using predefined names as +described here. The boilerplate needed is also provided in comments in +the 'gawkapi.h' header file: + + /* Boilerplate code: */ + int plugin_is_GPL_compatible; + + static gawk_api_t *const api; + static awk_ext_id_t ext_id; + static const char *ext_version = NULL; /* or ... = "some string" */ + + static awk_ext_func_t func_table[] = { + { "name", do_name, 1 }, + /* ... */ + }; + + /* EITHER: */ + + static awk_bool_t (*init_func)(void) = NULL; + + /* OR: */ + + static awk_bool_t + init_my_extension(void) + { + ... + } + + static awk_bool_t (*init_func)(void) = init_my_extension; + + dl_load_func(func_table, some_name, "name_space_in_quotes") + + These variables and functions are as follows: + +'int plugin_is_GPL_compatible;' + This asserts that the extension is compatible with the GNU GPL + (*note Copying::). If your extension does not have this, 'gawk' + will not load it (*note Plugin License::). + +'static gawk_api_t *const api;' + This global 'static' variable should be set to point to the + 'gawk_api_t' pointer that 'gawk' passes to your 'dl_load()' + function. This variable is used by all of the macros. + +'static awk_ext_id_t ext_id;' + This global static variable should be set to the 'awk_ext_id_t' + value that 'gawk' passes to your 'dl_load()' function. This + variable is used by all of the macros. + +'static const char *ext_version = NULL; /* or ... = "some string" */' + This global 'static' variable should be set either to 'NULL', or to + point to a string giving the name and version of your extension. + +'static awk_ext_func_t func_table[] = { ... };' + This is an array of one or more 'awk_ext_func_t' structures, as + described earlier (*note Extension Functions::). It can then be + looped over for multiple calls to 'add_ext_func()'. + +'static awk_bool_t (*init_func)(void) = NULL;' +' OR' +'static awk_bool_t init_my_extension(void) { ... }' +'static awk_bool_t (*init_func)(void) = init_my_extension;' + If you need to do some initialization work, you should define a + function that does it (creates variables, opens files, etc.) and + then define the 'init_func' pointer to point to your function. The + function should return 'awk_false' upon failure, or 'awk_true' if + everything goes well. + + If you don't need to do any initialization, define the pointer and + initialize it to 'NULL'. + +'dl_load_func(func_table, some_name, "name_space_in_quotes")' + This macro expands to a 'dl_load()' function that performs all the + necessary initializations. + + The point of all the variables and arrays is to let the 'dl_load()' +function (from the 'dl_load_func()' macro) do all the standard work. It +does the following: + + 1. Check the API versions. If the extension major version does not + match 'gawk''s, or if the extension minor version is greater than + 'gawk''s, it prints a fatal error message and exits. + + 2. Load the functions defined in 'func_table'. If any of them fails + to load, it prints a warning message but continues on. + + 3. If the 'init_func' pointer is not 'NULL', call the function it + points to. If it returns 'awk_false', print a warning message. + + 4. If 'ext_version' is not 'NULL', register the version string with + 'gawk'. + + +File: gawk.info, Node: Finding Extensions, Next: Extension Example, Prev: Extension API Description, Up: Dynamic Extensions + +16.5 How 'gawk' Finds Extensions +================================ + +Compiled extensions have to be installed in a directory where 'gawk' can +find them. If 'gawk' is configured and built in the default fashion, +the directory in which to find extensions is '/usr/local/lib/gawk'. You +can also specify a search path with a list of directories to search for +compiled extensions. *Note AWKLIBPATH Variable:: for more information. + + +File: gawk.info, Node: Extension Example, Next: Extension Samples, Prev: Finding Extensions, Up: Dynamic Extensions + +16.6 Example: Some File Functions +================================= + + No matter where you go, there you are. + -- _Buckaroo Banzai_ + + Two useful functions that are not in 'awk' are 'chdir()' (so that an +'awk' program can change its directory) and 'stat()' (so that an 'awk' +program can gather information about a file). In order to illustrate +the API in action, this minor node implements these functions for 'gawk' +in an extension. + +* Menu: + +* Internal File Description:: What the new functions will do. +* Internal File Ops:: The code for internal file operations. +* Using Internal File Ops:: How to use an external extension. + + +File: gawk.info, Node: Internal File Description, Next: Internal File Ops, Up: Extension Example + +16.6.1 Using 'chdir()' and 'stat()' +----------------------------------- + +This minor node shows how to use the new functions at the 'awk' level +once they've been integrated into the running 'gawk' interpreter. Using +'chdir()' is very straightforward. It takes one argument, the new +directory to change to: + + @load "filefuncs" + ... + newdir = "/home/arnold/funstuff" + ret = chdir(newdir) + if (ret < 0) { + printf("could not change to %s: %s\n", newdir, ERRNO) > "/dev/stderr" + exit 1 + } + ... + + The return value is negative if the 'chdir()' failed, and 'ERRNO' +(*note Built-in Variables::) is set to a string indicating the error. + + Using 'stat()' is a bit more complicated. The C 'stat()' function +fills in a structure that has a fair amount of information. The right +way to model this in 'awk' is to fill in an associative array with the +appropriate information: + + file = "/home/arnold/.profile" + ret = stat(file, fdata) + if (ret < 0) { + printf("could not stat %s: %s\n", + file, ERRNO) > "/dev/stderr" + exit 1 + } + printf("size of %s is %d bytes\n", file, fdata["size"]) + + The 'stat()' function always clears the data array, even if the +'stat()' fails. It fills in the following elements: + +'"name"' + The name of the file that was 'stat()'ed. + +'"dev"' +'"ino"' + The file's device and inode numbers, respectively. + +'"mode"' + The file's mode, as a numeric value. This includes both the file's + type and its permissions. + +'"nlink"' + The number of hard links (directory entries) the file has. + +'"uid"' +'"gid"' + The numeric user and group ID numbers of the file's owner. + +'"size"' + The size in bytes of the file. + +'"blocks"' + The number of disk blocks the file actually occupies. This may not + be a function of the file's size if the file has holes. + +'"atime"' +'"mtime"' +'"ctime"' + The file's last access, modification, and inode update times, + respectively. These are numeric timestamps, suitable for + formatting with 'strftime()' (*note Time Functions::). + +'"pmode"' + The file's "printable mode." This is a string representation of + the file's type and permissions, such as is produced by 'ls + -l'--for example, '"drwxr-xr-x"'. + +'"type"' + A printable string representation of the file's type. The value is + one of the following: + + '"blockdev"' + '"chardev"' + The file is a block or character device ("special file"). + + '"directory"' + The file is a directory. + + '"fifo"' + The file is a named pipe (also known as a FIFO). + + '"file"' + The file is just a regular file. + + '"socket"' + The file is an 'AF_UNIX' ("Unix domain") socket in the + filesystem. + + '"symlink"' + The file is a symbolic link. + +'"devbsize"' + The size of a block for the element indexed by '"blocks"'. This + information is derived from either the 'DEV_BSIZE' constant defined + in '<sys/param.h>' on most systems, or the 'S_BLKSIZE' constant in + '<sys/stat.h>' on BSD systems. For some other systems, "a priori" + knowledge is used to provide a value. Where no value can be + determined, it defaults to 512. + + Several additional elements may be present, depending upon the +operating system and the type of the file. You can test for them in +your 'awk' program by using the 'in' operator (*note Reference to +Elements::): + +'"blksize"' + The preferred block size for I/O to the file. This field is not + present on all POSIX-like systems in the C 'stat' structure. + +'"linkval"' + If the file is a symbolic link, this element is the name of the + file the link points to (i.e., the value of the link). + +'"rdev"' +'"major"' +'"minor"' + If the file is a block or character device file, then these values + represent the numeric device number and the major and minor + components of that number, respectively. + + +File: gawk.info, Node: Internal File Ops, Next: Using Internal File Ops, Prev: Internal File Description, Up: Extension Example + +16.6.2 C Code for 'chdir()' and 'stat()' +---------------------------------------- + +Here is the C code for these extensions.(1) + + The file includes a number of standard header files, and then +includes the 'gawkapi.h' header file, which provides the API +definitions. Those are followed by the necessary variable declarations +to make use of the API macros and boilerplate code (*note Extension API +Boilerplate::): + + #ifdef HAVE_CONFIG_H + #include <config.h> + #endif + + #include <stdio.h> + #include <assert.h> + #include <errno.h> + #include <stdlib.h> + #include <string.h> + #include <unistd.h> + + #include <sys/types.h> + #include <sys/stat.h> + + #include "gawkapi.h" + + #include "gettext.h" + #define _(msgid) gettext(msgid) + #define N_(msgid) msgid + + #include "gawkfts.h" + #include "stack.h" + + static const gawk_api_t *api; /* for convenience macros to work */ + static awk_ext_id_t *ext_id; + static awk_bool_t init_filefuncs(void); + static awk_bool_t (*init_func)(void) = init_filefuncs; + static const char *ext_version = "filefuncs extension: version 1.0"; + + int plugin_is_GPL_compatible; + + By convention, for an 'awk' function 'foo()', the C function that +implements it is called 'do_foo()'. The function should have two +arguments. The first is an 'int', usually called 'nargs', that +represents the number of actual arguments for the function. The second +is a pointer to an 'awk_value_t' structure, usually named 'result': + + /* do_chdir --- provide dynamically loaded chdir() function for gawk */ + + static awk_value_t * + do_chdir(int nargs, awk_value_t *result) + { + awk_value_t newdir; + int ret = -1; + + assert(result != NULL); + + if (do_lint && nargs != 1) + lintwarn(ext_id, + _("chdir: called with incorrect number of arguments, " + "expecting 1")); + + The 'newdir' variable represents the new directory to change to, +which is retrieved with 'get_argument()'. Note that the first argument +is numbered zero. + + If the argument is retrieved successfully, the function calls the +'chdir()' system call. If the 'chdir()' fails, 'ERRNO' is updated: + + if (get_argument(0, AWK_STRING, & newdir)) { + ret = chdir(newdir.str_value.str); + if (ret < 0) + update_ERRNO_int(errno); + } + + Finally, the function returns the return value to the 'awk' level: + + return make_number(ret, result); + } + + The 'stat()' extension is more involved. First comes a function that +turns a numeric mode into a printable representation (e.g., octal '0644' +becomes '-rw-r--r--'). This is omitted here for brevity: + + /* format_mode --- turn a stat mode field into something readable */ + + static char * + format_mode(unsigned long fmode) + { + ... + } + + Next comes a function for reading symbolic links, which is also +omitted here for brevity: + + /* read_symlink --- read a symbolic link into an allocated buffer. + ... */ + + static char * + read_symlink(const char *fname, size_t bufsize, ssize_t *linksize) + { + ... + } + + Two helper functions simplify entering values in the array that will +contain the result of the 'stat()': + + /* array_set --- set an array element */ + + static void + array_set(awk_array_t array, const char *sub, awk_value_t *value) + { + awk_value_t index; + + set_array_element(array, + make_const_string(sub, strlen(sub), & index), + value); + + } + + /* array_set_numeric --- set an array element with a number */ + + static void + array_set_numeric(awk_array_t array, const char *sub, double num) + { + awk_value_t tmp; + + array_set(array, sub, make_number(num, & tmp)); + } + + The following function does most of the work to fill in the +'awk_array_t' result array with values obtained from a valid 'struct +stat'. This work is done in a separate function to support the 'stat()' +function for 'gawk' and also to support the 'fts()' extension, which is +included in the same file but whose code is not shown here (*note +Extension Sample File Functions::). + + The first part of the function is variable declarations, including a +table to map file types to strings: + + /* fill_stat_array --- do the work to fill an array with stat info */ + + static int + fill_stat_array(const char *name, awk_array_t array, struct stat *sbuf) + { + char *pmode; /* printable mode */ + const char *type = "unknown"; + awk_value_t tmp; + static struct ftype_map { + unsigned int mask; + const char *type; + } ftype_map[] = { + { S_IFREG, "file" }, + { S_IFBLK, "blockdev" }, + { S_IFCHR, "chardev" }, + { S_IFDIR, "directory" }, + #ifdef S_IFSOCK + { S_IFSOCK, "socket" }, + #endif + #ifdef S_IFIFO + { S_IFIFO, "fifo" }, + #endif + #ifdef S_IFLNK + { S_IFLNK, "symlink" }, + #endif + #ifdef S_IFDOOR /* Solaris weirdness */ + { S_IFDOOR, "door" }, + #endif /* S_IFDOOR */ + }; + int j, k; + + The destination array is cleared, and then code fills in various +elements based on values in the 'struct stat': + + /* empty out the array */ + clear_array(array); + + /* fill in the array */ + array_set(array, "name", make_const_string(name, strlen(name), + & tmp)); + array_set_numeric(array, "dev", sbuf->st_dev); + array_set_numeric(array, "ino", sbuf->st_ino); + array_set_numeric(array, "mode", sbuf->st_mode); + array_set_numeric(array, "nlink", sbuf->st_nlink); + array_set_numeric(array, "uid", sbuf->st_uid); + array_set_numeric(array, "gid", sbuf->st_gid); + array_set_numeric(array, "size", sbuf->st_size); + array_set_numeric(array, "blocks", sbuf->st_blocks); + array_set_numeric(array, "atime", sbuf->st_atime); + array_set_numeric(array, "mtime", sbuf->st_mtime); + array_set_numeric(array, "ctime", sbuf->st_ctime); + + /* for block and character devices, add rdev, + major and minor numbers */ + if (S_ISBLK(sbuf->st_mode) || S_ISCHR(sbuf->st_mode)) { + array_set_numeric(array, "rdev", sbuf->st_rdev); + array_set_numeric(array, "major", major(sbuf->st_rdev)); + array_set_numeric(array, "minor", minor(sbuf->st_rdev)); + } + +The latter part of the function makes selective additions to the +destination array, depending upon the availability of certain members +and/or the type of the file. It then returns zero, for success: + + #ifdef HAVE_STRUCT_STAT_ST_BLKSIZE + array_set_numeric(array, "blksize", sbuf->st_blksize); + #endif /* HAVE_STRUCT_STAT_ST_BLKSIZE */ + + pmode = format_mode(sbuf->st_mode); + array_set(array, "pmode", make_const_string(pmode, strlen(pmode), + & tmp)); + + /* for symbolic links, add a linkval field */ + if (S_ISLNK(sbuf->st_mode)) { + char *buf; + ssize_t linksize; + + if ((buf = read_symlink(name, sbuf->st_size, + & linksize)) != NULL) + array_set(array, "linkval", + make_malloced_string(buf, linksize, & tmp)); + else + warning(ext_id, _("stat: unable to read symbolic link `%s'"), + name); + } + + /* add a type field */ + type = "unknown"; /* shouldn't happen */ + for (j = 0, k = sizeof(ftype_map)/sizeof(ftype_map[0]); j < k; j++) { + if ((sbuf->st_mode & S_IFMT) == ftype_map[j].mask) { + type = ftype_map[j].type; + break; + } + } + + array_set(array, "type", make_const_string(type, strlen(type), & tmp)); + + return 0; + } + + The third argument to 'stat()' was not discussed previously. This +argument is optional. If present, it causes 'do_stat()' to use the +'stat()' system call instead of the 'lstat()' system call. This is done +by using a function pointer: 'statfunc'. 'statfunc' is initialized to +point to 'lstat()' (instead of 'stat()') to get the file information, in +case the file is a symbolic link. However, if the third argument is +included, 'statfunc' is set to point to 'stat()', instead. + + Here is the 'do_stat()' function, which starts with variable +declarations and argument checking: + + /* do_stat --- provide a stat() function for gawk */ + + static awk_value_t * + do_stat(int nargs, awk_value_t *result) + { + awk_value_t file_param, array_param; + char *name; + awk_array_t array; + int ret; + struct stat sbuf; + /* default is lstat() */ + int (*statfunc)(const char *path, struct stat *sbuf) = lstat; + + assert(result != NULL); + + if (nargs != 2 && nargs != 3) { + if (do_lint) + lintwarn(ext_id, + _("stat: called with wrong number of arguments")); + return make_number(-1, result); + } + + Then comes the actual work. First, the function gets the arguments. +Next, it gets the information for the file. If the called function +('lstat()' or 'stat()') returns an error, the code sets 'ERRNO' and +returns: + + /* file is first arg, array to hold results is second */ + if ( ! get_argument(0, AWK_STRING, & file_param) + || ! get_argument(1, AWK_ARRAY, & array_param)) { + warning(ext_id, _("stat: bad parameters")); + return make_number(-1, result); + } + + if (nargs == 3) { + statfunc = stat; + } + + name = file_param.str_value.str; + array = array_param.array_cookie; + + /* always empty out the array */ + clear_array(array); + + /* stat the file; if error, set ERRNO and return */ + ret = statfunc(name, & sbuf); + if (ret < 0) { + update_ERRNO_int(errno); + return make_number(ret, result); + } + + The tedious work is done by 'fill_stat_array()', shown earlier. When +done, the function returns the result from 'fill_stat_array()': + + ret = fill_stat_array(name, array, & sbuf); + + return make_number(ret, result); + } + + Finally, it's necessary to provide the "glue" that loads the new +function(s) into 'gawk'. + + The 'filefuncs' extension also provides an 'fts()' function, which we +omit here (*note Extension Sample File Functions::). For its sake, +there is an initialization function: + + /* init_filefuncs --- initialization routine */ + + static awk_bool_t + init_filefuncs(void) + { + ... + } + + We are almost done. We need an array of 'awk_ext_func_t' structures +for loading each function into 'gawk': + + static awk_ext_func_t func_table[] = { + { "chdir", do_chdir, 1 }, + { "stat", do_stat, 2 }, + #ifndef __MINGW32__ + { "fts", do_fts, 3 }, + #endif + }; + + Each extension must have a routine named 'dl_load()' to load +everything that needs to be loaded. It is simplest to use the +'dl_load_func()' macro in 'gawkapi.h': + + /* define the dl_load() function using the boilerplate macro */ + + dl_load_func(func_table, filefuncs, "") + + And that's it! + + ---------- Footnotes ---------- + + (1) This version is edited slightly for presentation. See +'extension/filefuncs.c' in the 'gawk' distribution for the complete +version. + + +File: gawk.info, Node: Using Internal File Ops, Prev: Internal File Ops, Up: Extension Example + +16.6.3 Integrating the Extensions +--------------------------------- + +Now that the code is written, it must be possible to add it at runtime +to the running 'gawk' interpreter. First, the code must be compiled. +Assuming that the functions are in a file named 'filefuncs.c', and IDIR +is the location of the 'gawkapi.h' header file, the following steps(1) +create a GNU/Linux shared library: + + $ gcc -fPIC -shared -DHAVE_CONFIG_H -c -O -g -IIDIR filefuncs.c + $ gcc -o filefuncs.so -shared filefuncs.o + + Once the library exists, it is loaded by using the '@load' keyword: + + # file testff.awk + @load "filefuncs" + + BEGIN { + "pwd" | getline curdir # save current directory + close("pwd") + + chdir("/tmp") + system("pwd") # test it + chdir(curdir) # go back + + print "Info for testff.awk" + ret = stat("testff.awk", data) + print "ret =", ret + for (i in data) + printf "data[\"%s\"] = %s\n", i, data[i] + print "testff.awk modified:", + strftime("%m %d %Y %H:%M:%S", data["mtime"]) + + print "\nInfo for JUNK" + ret = stat("JUNK", data) + print "ret =", ret + for (i in data) + printf "data[\"%s\"] = %s\n", i, data[i] + print "JUNK modified:", strftime("%m %d %Y %H:%M:%S", data["mtime"]) + } + + The 'AWKLIBPATH' environment variable tells 'gawk' where to find +extensions (*note Finding Extensions::). We set it to the current +directory and run the program: + + $ AWKLIBPATH=$PWD gawk -f testff.awk + -| /tmp + -| Info for testff.awk + -| ret = 0 + -| data["blksize"] = 4096 + -| data["devbsize"] = 512 + -| data["mtime"] = 1412004710 + -| data["mode"] = 33204 + -| data["type"] = file + -| data["dev"] = 2053 + -| data["gid"] = 1000 + -| data["ino"] = 10358899 + -| data["ctime"] = 1412004710 + -| data["blocks"] = 8 + -| data["nlink"] = 1 + -| data["name"] = testff.awk + -| data["atime"] = 1412004716 + -| data["pmode"] = -rw-rw-r-- + -| data["size"] = 666 + -| data["uid"] = 1000 + -| testff.awk modified: 09 29 2014 18:31:50 + -| + -| Info for JUNK + -| ret = -1 + -| JUNK modified: 01 01 1970 02:00:00 + + ---------- Footnotes ---------- + + (1) In practice, you would probably want to use the GNU Autotools +(Automake, Autoconf, Libtool, and 'gettext') to configure and build your +libraries. Instructions for doing so are beyond the scope of this Info +file. *Note gawkextlib:: for Internet links to the tools. + + +File: gawk.info, Node: Extension Samples, Next: gawkextlib, Prev: Extension Example, Up: Dynamic Extensions + +16.7 The Sample Extensions in the 'gawk' Distribution +===================================================== + +This minor node provides a brief overview of the sample extensions that +come in the 'gawk' distribution. Some of them are intended for +production use (e.g., the 'filefuncs', 'readdir', and 'inplace' +extensions). Others mainly provide example code that shows how to use +the extension API. + +* Menu: + +* Extension Sample File Functions:: The file functions sample. +* Extension Sample Fnmatch:: An interface to 'fnmatch()'. +* Extension Sample Fork:: An interface to 'fork()' and other + process functions. +* Extension Sample Inplace:: Enabling in-place file editing. +* Extension Sample Ord:: Character to value to character + conversions. +* Extension Sample Readdir:: An interface to 'readdir()'. +* Extension Sample Revout:: Reversing output sample output wrapper. +* Extension Sample Rev2way:: Reversing data sample two-way processor. +* Extension Sample Read write array:: Serializing an array to a file. +* Extension Sample Readfile:: Reading an entire file into a string. +* Extension Sample Time:: An interface to 'gettimeofday()' + and 'sleep()'. +* Extension Sample API Tests:: Tests for the API. + + +File: gawk.info, Node: Extension Sample File Functions, Next: Extension Sample Fnmatch, Up: Extension Samples + +16.7.1 File-Related Functions +----------------------------- + +The 'filefuncs' extension provides three different functions, as +follows. The usage is: + +'@load "filefuncs"' + This is how you load the extension. + +'result = chdir("/some/directory")' + The 'chdir()' function is a direct hook to the 'chdir()' system + call to change the current directory. It returns zero upon success + or a value less than zero upon error. In the latter case, it + updates 'ERRNO'. + +'result = stat("/some/path", statdata' [', follow']')' + The 'stat()' function provides a hook into the 'stat()' system + call. It returns zero upon success or a value less than zero upon + error. In the latter case, it updates 'ERRNO'. + + By default, it uses the 'lstat()' system call. However, if passed + a third argument, it uses 'stat()' instead. + + In all cases, it clears the 'statdata' array. When the call is + successful, 'stat()' fills the 'statdata' array with information + retrieved from the filesystem, as follows: + + Subscript Field in 'struct stat' File type + ---------------------------------------------------------------- + '"name"' The file name All + '"dev"' 'st_dev' All + '"ino"' 'st_ino' All + '"mode"' 'st_mode' All + '"nlink"' 'st_nlink' All + '"uid"' 'st_uid' All + '"gid"' 'st_gid' All + '"size"' 'st_size' All + '"atime"' 'st_atime' All + '"mtime"' 'st_mtime' All + '"ctime"' 'st_ctime' All + '"rdev"' 'st_rdev' Device files + '"major"' 'st_major' Device files + '"minor"' 'st_minor' Device files + '"blksize"' 'st_blksize' All + '"pmode"' A human-readable version of the All + mode value, like that printed by + 'ls' (for example, '"-rwxr-xr-x"') + '"linkval"' The value of the symbolic link Symbolic + links + '"type"' The type of the file as a All + string--one of '"file"', + '"blockdev"', '"chardev"', + '"directory"', '"socket"', + '"fifo"', '"symlink"', '"door"', + or '"unknown"' (not all systems + support all file types) + +'flags = or(FTS_PHYSICAL, ...)' +'result = fts(pathlist, flags, filedata)' + Walk the file trees provided in 'pathlist' and fill in the + 'filedata' array, as described next. 'flags' is the bitwise OR of + several predefined values, also described in a moment. Return zero + if there were no errors, otherwise return -1. + + The 'fts()' function provides a hook to the C library 'fts()' +routines for traversing file hierarchies. Instead of returning data +about one file at a time in a stream, it fills in a multidimensional +array with data about each file and directory encountered in the +requested hierarchies. + + The arguments are as follows: + +'pathlist' + An array of file names. The element values are used; the index + values are ignored. + +'flags' + This should be the bitwise OR of one or more of the following + predefined constant flag values. At least one of 'FTS_LOGICAL' or + 'FTS_PHYSICAL' must be provided; otherwise 'fts()' returns an error + value and sets 'ERRNO'. The flags are: + + 'FTS_LOGICAL' + Do a "logical" file traversal, where the information returned + for a symbolic link refers to the linked-to file, and not to + the symbolic link itself. This flag is mutually exclusive + with 'FTS_PHYSICAL'. + + 'FTS_PHYSICAL' + Do a "physical" file traversal, where the information returned + for a symbolic link refers to the symbolic link itself. This + flag is mutually exclusive with 'FTS_LOGICAL'. + + 'FTS_NOCHDIR' + As a performance optimization, the C library 'fts()' routines + change directory as they traverse a file hierarchy. This flag + disables that optimization. + + 'FTS_COMFOLLOW' + Immediately follow a symbolic link named in 'pathlist', + whether or not 'FTS_LOGICAL' is set. + + 'FTS_SEEDOT' + By default, the C library 'fts()' routines do not return + entries for '.' (dot) and '..' (dot-dot). This option causes + entries for dot-dot to also be included. (The extension + always includes an entry for dot; more on this in a moment.) + + 'FTS_XDEV' + During a traversal, do not cross onto a different mounted + filesystem. + +'filedata' + The 'filedata' array holds the results. 'fts()' first clears it. + Then it creates an element in 'filedata' for every element in + 'pathlist'. The index is the name of the directory or file given + in 'pathlist'. The element for this index is itself an array. + There are two cases: + + _The path is a file_ + In this case, the array contains two or three elements: + + '"path"' + The full path to this file, starting from the "root" that + was given in the 'pathlist' array. + + '"stat"' + This element is itself an array, containing the same + information as provided by the 'stat()' function + described earlier for its 'statdata' argument. The + element may not be present if the 'stat()' system call + for the file failed. + + '"error"' + If some kind of error was encountered, the array will + also contain an element named '"error"', which is a + string describing the error. + + _The path is a directory_ + In this case, the array contains one element for each entry in + the directory. If an entry is a file, that element is the + same as for files, just described. If the entry is a + directory, that element is (recursively) an array describing + the subdirectory. If 'FTS_SEEDOT' was provided in the flags, + then there will also be an element named '".."'. This element + will be an array containing the data as provided by 'stat()'. + + In addition, there will be an element whose index is '"."'. + This element is an array containing the same two or three + elements as for a file: '"path"', '"stat"', and '"error"'. + + The 'fts()' function returns zero if there were no errors. +Otherwise, it returns -1. + + NOTE: The 'fts()' extension does not exactly mimic the interface of + the C library 'fts()' routines, choosing instead to provide an + interface that is based on associative arrays, which is more + comfortable to use from an 'awk' program. This includes the lack + of a comparison function, because 'gawk' already provides powerful + array sorting facilities. Although an 'fts_read()'-like interface + could have been provided, this felt less natural than simply + creating a multidimensional array to represent the file hierarchy + and its information. + + See 'test/fts.awk' in the 'gawk' distribution for an example use of +the 'fts()' extension function. + + +File: gawk.info, Node: Extension Sample Fnmatch, Next: Extension Sample Fork, Prev: Extension Sample File Functions, Up: Extension Samples + +16.7.2 Interface to 'fnmatch()' +------------------------------- + +This extension provides an interface to the C library 'fnmatch()' +function. The usage is: + +'@load "fnmatch"' + This is how you load the extension. + +'result = fnmatch(pattern, string, flags)' + The return value is zero on success, 'FNM_NOMATCH' if the string + did not match the pattern, or a different nonzero value if an error + occurred. + + In addition to the 'fnmatch()' function, the 'fnmatch' extension adds +one constant ('FNM_NOMATCH'), and an array of flag values named 'FNM'. + + The arguments to 'fnmatch()' are: + +'pattern' + The file name wildcard to match + +'string' + The file name string + +'flag' + Either zero, or the bitwise OR of one or more of the flags in the + 'FNM' array + + The flags are as follows: + +Array element Corresponding flag defined by 'fnmatch()' +-------------------------------------------------------------------------- +'FNM["CASEFOLD"]' 'FNM_CASEFOLD' +'FNM["FILE_NAME"]' 'FNM_FILE_NAME' +'FNM["LEADING_DIR"]''FNM_LEADING_DIR' +'FNM["NOESCAPE"]' 'FNM_NOESCAPE' +'FNM["PATHNAME"]' 'FNM_PATHNAME' +'FNM["PERIOD"]' 'FNM_PERIOD' + + Here is an example: + + @load "fnmatch" + ... + flags = or(FNM["PERIOD"], FNM["NOESCAPE"]) + if (fnmatch("*.a", "foo.c", flags) == FNM_NOMATCH) + print "no match" + + +File: gawk.info, Node: Extension Sample Fork, Next: Extension Sample Inplace, Prev: Extension Sample Fnmatch, Up: Extension Samples + +16.7.3 Interface to 'fork()', 'wait()', and 'waitpid()' +------------------------------------------------------- + +The 'fork' extension adds three functions, as follows: + +'@load "fork"' + This is how you load the extension. + +'pid = fork()' + This function creates a new process. The return value is zero in + the child and the process ID number of the child in the parent, or + -1 upon error. In the latter case, 'ERRNO' indicates the problem. + In the child, 'PROCINFO["pid"]' and 'PROCINFO["ppid"]' are updated + to reflect the correct values. + +'ret = waitpid(pid)' + This function takes a numeric argument, which is the process ID to + wait for. The return value is that of the 'waitpid()' system call. + +'ret = wait()' + This function waits for the first child to die. The return value + is that of the 'wait()' system call. + + There is no corresponding 'exec()' function. + + Here is an example: + + @load "fork" + ... + if ((pid = fork()) == 0) + print "hello from the child" + else + print "hello from the parent" + + +File: gawk.info, Node: Extension Sample Inplace, Next: Extension Sample Ord, Prev: Extension Sample Fork, Up: Extension Samples + +16.7.4 Enabling In-Place File Editing +------------------------------------- + +The 'inplace' extension emulates GNU 'sed''s '-i' option, which performs +"in-place" editing of each input file. It uses the bundled +'inplace.awk' include file to invoke the extension properly: + + # inplace --- load and invoke the inplace extension. + + @load "inplace" + + # Please set INPLACE_SUFFIX to make a backup copy. For example, you may + # want to set INPLACE_SUFFIX to .bak on the command line or in a BEGIN rule. + + # By default, each filename on the command line will be edited inplace. + # But you can selectively disable this by adding an inplace=0 argument + # prior to files that you do not want to process this way. You can then + # reenable it later on the commandline by putting inplace=1 before files + # that you wish to be subject to inplace editing. + + # N.B. We call inplace_end() in the BEGINFILE and END rules so that any + # actions in an ENDFILE rule will be redirected as expected. + + BEGIN { + inplace = 1 # enabled by default + } + + BEGINFILE { + if (_inplace_filename != "") + inplace_end(_inplace_filename, INPLACE_SUFFIX) + if (inplace) + inplace_begin(_inplace_filename = FILENAME, INPLACE_SUFFIX) + else + _inplace_filename = "" + } + + END { + if (_inplace_filename != "") + inplace_end(_inplace_filename, INPLACE_SUFFIX) + } + + For each regular file that is processed, the extension redirects +standard output to a temporary file configured to have the same owner +and permissions as the original. After the file has been processed, the +extension restores standard output to its original destination. If +'INPLACE_SUFFIX' is not an empty string, the original file is linked to +a backup file name created by appending that suffix. Finally, the +temporary file is renamed to the original file name. + + Note that the use of this feature can be controlled by placing +'inplace=0' on the command-line prior to listing files that should not +be processed this way. You can reenable inplace editing by adding an +'inplace=1' argument prior to files that should be subject to inplace +editing. + + The '_inplace_filename' variable serves to keep track of the current +filename so as to not invoke 'inplace_end()' before processing the first +file. + + If any error occurs, the extension issues a fatal error to terminate +processing immediately without damaging the original file. + + Here are some simple examples: + + $ gawk -i inplace '{ gsub(/foo/, "bar") }; { print }' file1 file2 file3 + + To keep a backup copy of the original files, try this: + + $ gawk -i inplace -v INPLACE_SUFFIX=.bak '{ gsub(/foo/, "bar") } + > { print }' file1 file2 file3 + + Please note that, while the extension does attempt to preserve +ownership and permissions, it makes no attempt to copy the ACLs from the +original file. + + If the program dies prematurely, as might happen if an unhandled +signal is received, a temporary file may be left behind. + + +File: gawk.info, Node: Extension Sample Ord, Next: Extension Sample Readdir, Prev: Extension Sample Inplace, Up: Extension Samples + +16.7.5 Character and Numeric values: 'ord()' and 'chr()' +-------------------------------------------------------- + +The 'ordchr' extension adds two functions, named 'ord()' and 'chr()', as +follows: + +'@load "ordchr"' + This is how you load the extension. + +'number = ord(string)' + Return the numeric value of the first character in 'string'. + +'char = chr(number)' + Return a string whose first character is that represented by + 'number'. + + These functions are inspired by the Pascal language functions of the +same name. Here is an example: + + @load "ordchr" + ... + printf("The numeric value of 'A' is %d\n", ord("A")) + printf("The string value of 65 is %s\n", chr(65)) + + +File: gawk.info, Node: Extension Sample Readdir, Next: Extension Sample Revout, Prev: Extension Sample Ord, Up: Extension Samples + +16.7.6 Reading Directories +-------------------------- + +The 'readdir' extension adds an input parser for directories. The usage +is as follows: + + @load "readdir" + + When this extension is in use, instead of skipping directories named +on the command line (or with 'getline'), they are read, with each entry +returned as a record. + + The record consists of three fields. The first two are the inode +number and the file name, separated by a forward slash character. On +systems where the directory entry contains the file type, the record has +a third field (also separated by a slash), which is a single letter +indicating the type of the file. The letters and their corresponding +file types are shown in *note Table 16.3: table-readdir-file-types. + +Letter File type +-------------------------------------------------------------------------- +'b' Block device +'c' Character device +'d' Directory +'f' Regular file +'l' Symbolic link +'p' Named pipe (FIFO) +'s' Socket +'u' Anything else (unknown) + +Table 16.3: File types returned by the 'readdir' extension + + On systems without the file type information, the third field is +always 'u'. + + NOTE: On GNU/Linux systems, there are filesystems that don't + support the 'd_type' entry (see the readdir(3) manual page), and so + the file type is always 'u'. You can use the 'filefuncs' extension + to call 'stat()' in order to get correct type information. + + Here is an example: + + @load "readdir" + ... + BEGIN { FS = "/" } + { print "file name is", $2 } + + +File: gawk.info, Node: Extension Sample Revout, Next: Extension Sample Rev2way, Prev: Extension Sample Readdir, Up: Extension Samples + +16.7.7 Reversing Output +----------------------- + +The 'revoutput' extension adds a simple output wrapper that reverses the +characters in each output line. Its main purpose is to show how to +write an output wrapper, although it may be mildly amusing for the +unwary. Here is an example: + + @load "revoutput" + + BEGIN { + REVOUT = 1 + print "don't panic" > "/dev/stdout" + } + + The output from this program is 'cinap t'nod'. + + +File: gawk.info, Node: Extension Sample Rev2way, Next: Extension Sample Read write array, Prev: Extension Sample Revout, Up: Extension Samples + +16.7.8 Two-Way I/O Example +-------------------------- + +The 'revtwoway' extension adds a simple two-way processor that reverses +the characters in each line sent to it for reading back by the 'awk' +program. Its main purpose is to show how to write a two-way processor, +although it may also be mildly amusing. The following example shows how +to use it: + + @load "revtwoway" + + BEGIN { + cmd = "/magic/mirror" + print "don't panic" |& cmd + cmd |& getline result + print result + close(cmd) + } + + The output from this program is: 'cinap t'nod'. + + +File: gawk.info, Node: Extension Sample Read write array, Next: Extension Sample Readfile, Prev: Extension Sample Rev2way, Up: Extension Samples + +16.7.9 Dumping and Restoring an Array +------------------------------------- + +The 'rwarray' extension adds two functions, named 'writea()' and +'reada()', as follows: + +'@load "rwarray"' + This is how you load the extension. + +'ret = writea(file, array)' + This function takes a string argument, which is the name of the + file to which to dump the array, and the array itself as the second + argument. 'writea()' understands arrays of arrays. It returns one + on success, or zero upon failure. + +'ret = reada(file, array)' + 'reada()' is the inverse of 'writea()'; it reads the file named as + its first argument, filling in the array named as the second + argument. It clears the array first. Here too, the return value + is one on success, or zero upon failure. + + The array created by 'reada()' is identical to that written by +'writea()' in the sense that the contents are the same. However, due to +implementation issues, the array traversal order of the re-created array +is likely to be different from that of the original array. As array +traversal order in 'awk' is by default undefined, this is (technically) +not a problem. If you need to guarantee a particular traversal order, +use the array sorting features in 'gawk' to do so (*note Array +Sorting::). + + The file contains binary data. All integral values are written in +network byte order. However, double-precision floating-point values are +written as native binary data. Thus, arrays containing only string data +can theoretically be dumped on systems with one byte order and restored +on systems with a different one, but this has not been tried. + + Here is an example: + + @load "rwarray" + ... + ret = writea("arraydump.bin", array) + ... + ret = reada("arraydump.bin", array) + + +File: gawk.info, Node: Extension Sample Readfile, Next: Extension Sample Time, Prev: Extension Sample Read write array, Up: Extension Samples + +16.7.10 Reading an Entire File +------------------------------ + +The 'readfile' extension adds a single function named 'readfile()', and +an input parser: + +'@load "readfile"' + This is how you load the extension. + +'result = readfile("/some/path")' + The argument is the name of the file to read. The return value is + a string containing the entire contents of the requested file. + Upon error, the function returns the empty string and sets 'ERRNO'. + +'BEGIN { PROCINFO["readfile"] = 1 }' + In addition, the extension adds an input parser that is activated + if 'PROCINFO["readfile"]' exists. When activated, each input file + is returned in its entirety as '$0'. 'RT' is set to the null + string. + + Here is an example: + + @load "readfile" + ... + contents = readfile("/path/to/file"); + if (contents == "" && ERRNO != "") { + print("problem reading file", ERRNO) > "/dev/stderr" + ... + } + + +File: gawk.info, Node: Extension Sample Time, Next: Extension Sample API Tests, Prev: Extension Sample Readfile, Up: Extension Samples + +16.7.11 Extension Time Functions +-------------------------------- + +The 'time' extension adds two functions, named 'gettimeofday()' and +'sleep()', as follows: + +'@load "time"' + This is how you load the extension. + +'the_time = gettimeofday()' + Return the time in seconds that has elapsed since 1970-01-01 UTC as + a floating-point value. If the time is unavailable on this + platform, return -1 and set 'ERRNO'. The returned time should have + sub-second precision, but the actual precision may vary based on + the platform. If the standard C 'gettimeofday()' system call is + available on this platform, then it simply returns the value. + Otherwise, if on MS-Windows, it tries to use + 'GetSystemTimeAsFileTime()'. + +'result = sleep(SECONDS)' + Attempt to sleep for SECONDS seconds. If SECONDS is negative, or + the attempt to sleep fails, return -1 and set 'ERRNO'. Otherwise, + return zero after sleeping for the indicated amount of time. Note + that SECONDS may be a floating-point (nonintegral) value. + Implementation details: depending on platform availability, this + function tries to use 'nanosleep()' or 'select()' to implement the + delay. + + +File: gawk.info, Node: Extension Sample API Tests, Prev: Extension Sample Time, Up: Extension Samples + +16.7.12 API Tests +----------------- + +The 'testext' extension exercises parts of the extension API that are +not tested by the other samples. The 'extension/testext.c' file +contains both the C code for the extension and 'awk' test code inside C +comments that run the tests. The testing framework extracts the 'awk' +code and runs the tests. See the source file for more information. + + +File: gawk.info, Node: gawkextlib, Next: Extension summary, Prev: Extension Samples, Up: Dynamic Extensions + +16.8 The 'gawkextlib' Project +============================= + +The 'gawkextlib' (http://sourceforge.net/projects/gawkextlib/) project +provides a number of 'gawk' extensions, including one for processing XML +files. This is the evolution of the original 'xgawk' (XML 'gawk') +project. + + As of this writing, there are seven extensions: + + * 'errno' extension + + * GD graphics library extension + + * MPFR library extension (this provides access to a number of MPFR + functions that 'gawk''s native MPFR support does not) + + * PDF extension + + * PostgreSQL extension + + * Redis extension + + * Select extension + + * XML parser extension, using the Expat + (http://expat.sourceforge.net) XML parsing library + + You can check out the code for the 'gawkextlib' project using the Git +(http://git-scm.com) distributed source code control system. The +command is as follows: + + git clone git://git.code.sf.net/p/gawkextlib/code gawkextlib-code + + You will need to have the Expat (http://expat.sourceforge.net) XML +parser library installed in order to build and use the XML extension. + + In addition, you must have the GNU Autotools installed (Autoconf +(http://www.gnu.org/software/autoconf), Automake +(http://www.gnu.org/software/automake), Libtool +(http://www.gnu.org/software/libtool), and GNU 'gettext' +(http://www.gnu.org/software/gettext)). + + The simple recipe for building and testing 'gawkextlib' is as +follows. First, build and install 'gawk': + + cd .../path/to/gawk/code + ./configure --prefix=/tmp/newgawk Install in /tmp/newgawk for now + make && make check Build and check that all is OK + make install Install gawk + + Next, go to <http://sourceforge.net/projects/gawkextlib/files> to +download 'gawkextlib' and any extensions that you would like to build. +The 'README' file at that site explains how to build the code. If you +installed 'gawk' in a non-standard location, you will need to specify +'./configure --with-gawk=/PATH/TO/GAWK' to find it. You may need to use +the 'sudo' utility to install both 'gawk' and 'gawkextlib', depending +upon how your system works. + + If you write an extension that you wish to share with other 'gawk' +users, consider doing so through the 'gawkextlib' project. See the +project's website for more information. + + +File: gawk.info, Node: Extension summary, Next: Extension Exercises, Prev: gawkextlib, Up: Dynamic Extensions + +16.9 Summary +============ + + * You can write extensions (sometimes called plug-ins) for 'gawk' in + C or C++ using the application programming interface (API) defined + by the 'gawk' developers. + + * Extensions must have a license compatible with the GNU General + Public License (GPL), and they must assert that fact by declaring a + variable named 'plugin_is_GPL_compatible'. + + * Communication between 'gawk' and an extension is two-way. 'gawk' + passes a 'struct' to the extension that contains various data + fields and function pointers. The extension can then call into + 'gawk' via the supplied function pointers to accomplish certain + tasks. + + * One of these tasks is to "register" the name and implementation of + new 'awk'-level functions with 'gawk'. The implementation takes + the form of a C function pointer with a defined signature. By + convention, implementation functions are named 'do_XXXX()' for some + 'awk'-level function 'XXXX()'. + + * The API is defined in a header file named 'gawkapi.h'. You must + include a number of standard header files _before_ including it in + your source file. + + * API function pointers are provided for the following kinds of + operations: + + * Allocating, reallocating, and releasing memory + + * Registration functions (you may register extension functions, + exit callbacks, a version string, input parsers, output + wrappers, and two-way processors) + + * Printing fatal, nonfatal, warning, and "lint" warning messages + + * Updating 'ERRNO', or unsetting it + + * Accessing parameters, including converting an undefined + parameter into an array + + * Symbol table access (retrieving a global variable, creating + one, or changing one) + + * Creating and releasing cached values; this provides an + efficient way to use values for multiple variables and can be + a big performance win + + * Manipulating arrays (retrieving, adding, deleting, and + modifying elements; getting the count of elements in an array; + creating a new array; clearing an array; and flattening an + array for easy C-style looping over all its indices and + elements) + + * The API defines a number of standard data types for representing + 'awk' values, array elements, and arrays. + + * The API provides convenience functions for constructing values. It + also provides memory management functions to ensure compatibility + between memory allocated by 'gawk' and memory allocated by an + extension. + + * _All_ memory passed from 'gawk' to an extension must be treated as + read-only by the extension. + + * _All_ memory passed from an extension to 'gawk' must come from the + API's memory allocation functions. 'gawk' takes responsibility for + the memory and releases it when appropriate. + + * The API provides information about the running version of 'gawk' so + that an extension can make sure it is compatible with the 'gawk' + that loaded it. + + * It is easiest to start a new extension by copying the boilerplate + code described in this major node. Macros in the 'gawkapi.h' + header file make this easier to do. + + * The 'gawk' distribution includes a number of small but useful + sample extensions. The 'gawkextlib' project includes several more + (larger) extensions. If you wish to write an extension and + contribute it to the community of 'gawk' users, the 'gawkextlib' + project is the place to do so. + + +File: gawk.info, Node: Extension Exercises, Prev: Extension summary, Up: Dynamic Extensions + +16.10 Exercises +=============== + + 1. Add functions to implement system calls such as 'chown()', + 'chmod()', and 'umask()' to the file operations extension presented + in *note Internal File Ops::. + + 2. Write an input parser that prints a prompt if the input is a from a + "terminal" device. You can use the 'isatty()' function to tell if + the input file is a terminal. (Hint: this function is usually + expensive to call; try to call it just once.) The content of the + prompt should come from a variable settable by 'awk'-level code. + You can write the prompt to standard error. However, for best + results, open a new file descriptor (or file pointer) on '/dev/tty' + and print the prompt there, in case standard error has been + redirected. + + Why is standard error a better choice than standard output for + writing the prompt? Which reading mechanism should you replace, + the one to get a record, or the one to read raw bytes? + + 3. (Hard.) How would you provide namespaces in 'gawk', so that the + names of functions in different extensions don't conflict with each + other? If you come up with a really good scheme, contact the + 'gawk' maintainer to tell him about it. + + 4. Write a wrapper script that provides an interface similar to 'sed + -i' for the "inplace" extension presented in *note Extension Sample + Inplace::. + + +File: gawk.info, Node: Language History, Next: Installation, Prev: Dynamic Extensions, Up: Top + +Appendix A The Evolution of the 'awk' Language +********************************************** + +This Info file describes the GNU implementation of 'awk', which follows +the POSIX specification. Many longtime 'awk' users learned 'awk' +programming with the original 'awk' implementation in Version 7 Unix. +(This implementation was the basis for 'awk' in Berkeley Unix, through +4.3-Reno. Subsequent versions of Berkeley Unix, and, for a while, some +systems derived from 4.4BSD-Lite, used various versions of 'gawk' for +their 'awk'.) This major node briefly describes the evolution of the +'awk' language, with cross-references to other parts of the Info file +where you can find more information. + +* Menu: + +* V7/SVR3.1:: The major changes between V7 and System V + Release 3.1. +* SVR4:: Minor changes between System V Releases 3.1 + and 4. +* POSIX:: New features from the POSIX standard. +* BTL:: New features from Brian Kernighan's version of + 'awk'. +* POSIX/GNU:: The extensions in 'gawk' not in POSIX + 'awk'. +* Feature History:: The history of the features in 'gawk'. +* Common Extensions:: Common Extensions Summary. +* Ranges and Locales:: How locales used to affect regexp ranges. +* Contributors:: The major contributors to 'gawk'. +* History summary:: History summary. + + +File: gawk.info, Node: V7/SVR3.1, Next: SVR4, Up: Language History + +A.1 Major Changes Between V7 and SVR3.1 +======================================= + +The 'awk' language evolved considerably between the release of Version 7 +Unix (1978) and the new version that was first made generally available +in System V Release 3.1 (1987). This minor node summarizes the changes, +with cross-references to further details: + + * The requirement for ';' to separate rules on a line (*note + Statements/Lines::) + + * User-defined functions and the 'return' statement (*note + User-defined::) + + * The 'delete' statement (*note Delete::) + + * The 'do'-'while' statement (*note Do Statement::) + + * The built-in functions 'atan2()', 'cos()', 'sin()', 'rand()', and + 'srand()' (*note Numeric Functions::) + + * The built-in functions 'gsub()', 'sub()', and 'match()' (*note + String Functions::) + + * The built-in functions 'close()' and 'system()' (*note I/O + Functions::) + + * The 'ARGC', 'ARGV', 'FNR', 'RLENGTH', 'RSTART', and 'SUBSEP' + predefined variables (*note Built-in Variables::) + + * Assignable '$0' (*note Changing Fields::) + + * The conditional expression using the ternary operator '?:' (*note + Conditional Exp::) + + * The expression 'INDX in ARRAY' outside of 'for' statements (*note + Reference to Elements::) + + * The exponentiation operator '^' (*note Arithmetic Ops::) and its + assignment operator form '^=' (*note Assignment Ops::) + + * C-compatible operator precedence, which breaks some old 'awk' + programs (*note Precedence::) + + * Regexps as the value of 'FS' (*note Field Separators::) and as the + third argument to the 'split()' function (*note String + Functions::), rather than using only the first character of 'FS' + + * Dynamic regexps as operands of the '~' and '!~' operators (*note + Computed Regexps::) + + * The escape sequences '\b', '\f', and '\r' (*note Escape + Sequences::) + + * Redirection of input for the 'getline' function (*note Getline::) + + * Multiple 'BEGIN' and 'END' rules (*note BEGIN/END::) + + * Multidimensional arrays (*note Multidimensional::) + + +File: gawk.info, Node: SVR4, Next: POSIX, Prev: V7/SVR3.1, Up: Language History + +A.2 Changes Between SVR3.1 and SVR4 +=================================== + +The System V Release 4 (1989) version of Unix 'awk' added these features +(some of which originated in 'gawk'): + + * The 'ENVIRON' array (*note Built-in Variables::) + + * Multiple '-f' options on the command line (*note Options::) + + * The '-v' option for assigning variables before program execution + begins (*note Options::) + + * The '--' signal for terminating command-line options + + * The '\a', '\v', and '\x' escape sequences (*note Escape + Sequences::) + + * A defined return value for the 'srand()' built-in function (*note + Numeric Functions::) + + * The 'toupper()' and 'tolower()' built-in string functions for case + translation (*note String Functions::) + + * A cleaner specification for the '%c' format-control letter in the + 'printf' function (*note Control Letters::) + + * The ability to dynamically pass the field width and precision + ('"%*.*d"') in the argument list of 'printf' and 'sprintf()' (*note + Control Letters::) + + * The use of regexp constants, such as '/foo/', as expressions, where + they are equivalent to using the matching operator, as in '$0 ~ + /foo/' (*note Using Constant Regexps::) + + * Processing of escape sequences inside command-line variable + assignments (*note Assignment Options::) + + +File: gawk.info, Node: POSIX, Next: BTL, Prev: SVR4, Up: Language History + +A.3 Changes Between SVR4 and POSIX 'awk' +======================================== + +The POSIX Command Language and Utilities standard for 'awk' (1992) +introduced the following changes into the language: + + * The use of '-W' for implementation-specific options (*note + Options::) + + * The use of 'CONVFMT' for controlling the conversion of numbers to + strings (*note Conversion::) + + * The concept of a numeric string and tighter comparison rules to go + with it (*note Typing and Comparison::) + + * The use of predefined variables as function parameter names is + forbidden (*note Definition Syntax::) + + * More complete documentation of many of the previously undocumented + features of the language + + In 2012, a number of extensions that had been commonly available for +many years were finally added to POSIX. They are: + + * The 'fflush()' built-in function for flushing buffered output + (*note I/O Functions::) + + * The 'nextfile' statement (*note Nextfile Statement::) + + * The ability to delete all of an array at once with 'delete ARRAY' + (*note Delete::) + + *Note Common Extensions:: for a list of common extensions not +permitted by the POSIX standard. + + The 2008 POSIX standard can be found online at +<http://www.opengroup.org/onlinepubs/9699919799/>. + + +File: gawk.info, Node: BTL, Next: POSIX/GNU, Prev: POSIX, Up: Language History + +A.4 Extensions in Brian Kernighan's 'awk' +========================================= + +Brian Kernighan has made his version available via his home page (*note +Other Versions::). + + This minor node describes common extensions that originally appeared +in his version of 'awk': + + * The '**' and '**=' operators (*note Arithmetic Ops:: and *note + Assignment Ops::) + + * The use of 'func' as an abbreviation for 'function' (*note + Definition Syntax::) + + * The 'fflush()' built-in function for flushing buffered output + (*note I/O Functions::) + + *Note Common Extensions:: for a full list of the extensions available +in his 'awk'. + + +File: gawk.info, Node: POSIX/GNU, Next: Feature History, Prev: BTL, Up: Language History + +A.5 Extensions in 'gawk' Not in POSIX 'awk' +=========================================== + +The GNU implementation, 'gawk', adds a large number of features. They +can all be disabled with either the '--traditional' or '--posix' options +(*note Options::). + + A number of features have come and gone over the years. This minor +node summarizes the additional features over POSIX 'awk' that are in the +current version of 'gawk'. + + * Additional predefined variables: + + - The 'ARGIND', 'BINMODE', 'ERRNO', 'FIELDWIDTHS', 'FPAT', + 'IGNORECASE', 'LINT', 'PROCINFO', 'RT', and 'TEXTDOMAIN' + variables (*note Built-in Variables::) + + * Special files in I/O redirections: + + - The '/dev/stdin', '/dev/stdout', '/dev/stderr', and + '/dev/fd/N' special file names (*note Special Files::) + + - The '/inet', '/inet4', and '/inet6' special files for TCP/IP + networking using '|&' to specify which version of the IP + protocol to use (*note TCP/IP Networking::) + + * Changes and/or additions to the language: + + - The '\x' escape sequence (*note Escape Sequences::) + + - Full support for both POSIX and GNU regexps (*note Regexp::) + + - The ability for 'FS' and for the third argument to 'split()' + to be null strings (*note Single Character Fields::) + + - The ability for 'RS' to be a regexp (*note Records::) + + - The ability to use octal and hexadecimal constants in 'awk' + program source code (*note Nondecimal-numbers::) + + - The '|&' operator for two-way I/O to a coprocess (*note + Two-way I/O::) + + - Indirect function calls (*note Indirect Calls::) + + - Directories on the command line produce a warning and are + skipped (*note Command-line directories::) + + - Output with 'print' and 'printf' need not be fatal (*note + Nonfatal::) + + * New keywords: + + - The 'BEGINFILE' and 'ENDFILE' special patterns (*note + BEGINFILE/ENDFILE::) + + - The 'switch' statement (*note Switch Statement::) + + * Changes to standard 'awk' functions: + + - The optional second argument to 'close()' that allows closing + one end of a two-way pipe to a coprocess (*note Two-way I/O::) + + - POSIX compliance for 'gsub()' and 'sub()' with '--posix' + + - The 'length()' function accepts an array argument and returns + the number of elements in the array (*note String Functions::) + + - The optional third argument to the 'match()' function for + capturing text-matching subexpressions within a regexp (*note + String Functions::) + + - Positional specifiers in 'printf' formats for making + translations easier (*note Printf Ordering::) + + - The 'split()' function's additional optional fourth argument, + which is an array to hold the text of the field separators + (*note String Functions::) + + * Additional functions only in 'gawk': + + - The 'gensub()', 'patsplit()', and 'strtonum()' functions for + more powerful text manipulation (*note String Functions::) + + - The 'asort()' and 'asorti()' functions for sorting arrays + (*note Array Sorting::) + + - The 'mktime()', 'systime()', and 'strftime()' functions for + working with timestamps (*note Time Functions::) + + - The 'and()', 'compl()', 'lshift()', 'or()', 'rshift()', and + 'xor()' functions for bit manipulation (*note Bitwise + Functions::) + + - The 'isarray()' function to check if a variable is an array or + not (*note Type Functions::) + + - The 'bindtextdomain()', 'dcgettext()', and 'dcngettext()' + functions for internationalization (*note Programmer i18n::) + + - The 'intdiv()' function for doing integer division and + remainder (*note Numeric Functions::) + + * Changes and/or additions in the command-line options: + + - The 'AWKPATH' environment variable for specifying a path + search for the '-f' command-line option (*note Options::) + + - The 'AWKLIBPATH' environment variable for specifying a path + search for the '-l' command-line option (*note Options::) + + - The '-b', '-c', '-C', '-d', '-D', '-e', '-E', '-g', '-h', + '-i', '-l', '-L', '-M', '-n', '-N', '-o', '-O', '-p', '-P', + '-r', '-s', '-S', '-t', and '-V' short options. Also, the + ability to use GNU-style long-named options that start with + '--', and the '--assign', '--bignum', '--characters-as-bytes', + '--copyright', '--debug', '--dump-variables', '--exec', + '--field-separator', '--file', '--gen-pot', '--help', + '--include', '--lint', '--lint-old', '--load', + '--non-decimal-data', '--optimize', '--no-optimize', + '--posix', '--pretty-print', '--profile', '--re-interval', + '--sandbox', '--source', '--traditional', '--use-lc-numeric', + and '--version' long options (*note Options::). + + * Support for the following obsolete systems was removed from the + code and the documentation for 'gawk' version 4.0: + + - Amiga + + - Atari + + - BeOS + + - Cray + + - MIPS RiscOS + + - MS-DOS with the Microsoft Compiler + + - MS-Windows with the Microsoft Compiler + + - NeXT + + - SunOS 3.x, Sun 386 (Road Runner) + + - Tandem (non-POSIX) + + - Prestandard VAX C compiler for VAX/VMS + + - GCC for VAX and Alpha has not been tested for a while. + + * Support for the following obsolete system was removed from the code + for 'gawk' version 4.1: + + - Ultrix + + * Support for the following systems was removed from the code for + 'gawk' version 4.2: + + - MirBSD + + +File: gawk.info, Node: Feature History, Next: Common Extensions, Prev: POSIX/GNU, Up: Language History + +A.6 History of 'gawk' Features +============================== + +This minor node describes the features in 'gawk' over and above those in +POSIX 'awk', in the order they were added to 'gawk'. + + Version 2.10 of 'gawk' introduced the following features: + + * The 'AWKPATH' environment variable for specifying a path search for + the '-f' command-line option (*note Options::). + + * The 'IGNORECASE' variable and its effects (*note + Case-sensitivity::). + + * The '/dev/stdin', '/dev/stdout', '/dev/stderr' and '/dev/fd/N' + special file names (*note Special Files::). + + Version 2.13 of 'gawk' introduced the following features: + + * The 'FIELDWIDTHS' variable and its effects (*note Constant Size::). + + * The 'systime()' and 'strftime()' built-in functions for obtaining + and printing timestamps (*note Time Functions::). + + * Additional command-line options (*note Options::): + + - The '-W lint' option to provide error and portability checking + for both the source code and at runtime. + + - The '-W compat' option to turn off the GNU extensions. + + - The '-W posix' option for full POSIX compliance. + + Version 2.14 of 'gawk' introduced the following feature: + + * The 'next file' statement for skipping to the next data file (*note + Nextfile Statement::). + + Version 2.15 of 'gawk' introduced the following features: + + * New variables (*note Built-in Variables::): + + - 'ARGIND', which tracks the movement of 'FILENAME' through + 'ARGV'. + + - 'ERRNO', which contains the system error message when + 'getline' returns -1 or 'close()' fails. + + * The '/dev/pid', '/dev/ppid', '/dev/pgrpid', and '/dev/user' special + file names. These have since been removed. + + * The ability to delete all of an array at once with 'delete ARRAY' + (*note Delete::). + + * Command-line option changes (*note Options::): + + - The ability to use GNU-style long-named options that start + with '--'. + + - The '--source' option for mixing command-line and library-file + source code. + + Version 3.0 of 'gawk' introduced the following features: + + * New or changed variables: + + - 'IGNORECASE' changed, now applying to string comparison as + well as regexp operations (*note Case-sensitivity::). + + - 'RT', which contains the input text that matched 'RS' (*note + Records::). + + * Full support for both POSIX and GNU regexps (*note Regexp::). + + * The 'gensub()' function for more powerful text manipulation (*note + String Functions::). + + * The 'strftime()' function acquired a default time format, allowing + it to be called with no arguments (*note Time Functions::). + + * The ability for 'FS' and for the third argument to 'split()' to be + null strings (*note Single Character Fields::). + + * The ability for 'RS' to be a regexp (*note Records::). + + * The 'next file' statement became 'nextfile' (*note Nextfile + Statement::). + + * The 'fflush()' function from BWK 'awk' (then at Bell Laboratories; + *note I/O Functions::). + + * New command-line options: + + - The '--lint-old' option to warn about constructs that are not + available in the original Version 7 Unix version of 'awk' + (*note V7/SVR3.1::). + + - The '-m' option from BWK 'awk'. (Brian was still at Bell + Laboratories at the time.) This was later removed from both + his 'awk' and from 'gawk'. + + - The '--re-interval' option to provide interval expressions in + regexps (*note Regexp Operators::). + + - The '--traditional' option was added as a better name for + '--compat' (*note Options::). + + * The use of GNU Autoconf to control the configuration process (*note + Quick Installation::). + + * Amiga support. This has since been removed. + + Version 3.1 of 'gawk' introduced the following features: + + * New variables (*note Built-in Variables::): + + - 'BINMODE', for non-POSIX systems, which allows binary I/O for + input and/or output files (*note PC Using::). + + - 'LINT', which dynamically controls lint warnings. + + - 'PROCINFO', an array for providing process-related + information. + + - 'TEXTDOMAIN', for setting an application's + internationalization text domain (*note + Internationalization::). + + * The ability to use octal and hexadecimal constants in 'awk' program + source code (*note Nondecimal-numbers::). + + * The '|&' operator for two-way I/O to a coprocess (*note Two-way + I/O::). + + * The '/inet' special files for TCP/IP networking using '|&' (*note + TCP/IP Networking::). + + * The optional second argument to 'close()' that allows closing one + end of a two-way pipe to a coprocess (*note Two-way I/O::). + + * The optional third argument to the 'match()' function for capturing + text-matching subexpressions within a regexp (*note String + Functions::). + + * Positional specifiers in 'printf' formats for making translations + easier (*note Printf Ordering::). + + * A number of new built-in functions: + + - The 'asort()' and 'asorti()' functions for sorting arrays + (*note Array Sorting::). + + - The 'bindtextdomain()', 'dcgettext()' and 'dcngettext()' + functions for internationalization (*note Programmer i18n::). + + - The 'extension()' function and the ability to add new built-in + functions dynamically (*note Dynamic Extensions::). + + - The 'mktime()' function for creating timestamps (*note Time + Functions::). + + - The 'and()', 'or()', 'xor()', 'compl()', 'lshift()', + 'rshift()', and 'strtonum()' functions (*note Bitwise + Functions::). + + * The support for 'next file' as two words was removed completely + (*note Nextfile Statement::). + + * Additional command-line options (*note Options::): + + - The '--dump-variables' option to print a list of all global + variables. + + - The '--exec' option, for use in CGI scripts. + + - The '--gen-po' command-line option and the use of a leading + underscore to mark strings that should be translated (*note + String Extraction::). + + - The '--non-decimal-data' option to allow non-decimal input + data (*note Nondecimal Data::). + + - The '--profile' option and 'pgawk', the profiling version of + 'gawk', for producing execution profiles of 'awk' programs + (*note Profiling::). + + - The '--use-lc-numeric' option to force 'gawk' to use the + locale's decimal point for parsing input data (*note + Conversion::). + + * The use of GNU Automake to help in standardizing the configuration + process (*note Quick Installation::). + + * The use of GNU 'gettext' for 'gawk''s own message output (*note + Gawk I18N::). + + * BeOS support. This was later removed. + + * Tandem support. This was later removed. + + * The Atari port became officially unsupported and was later removed + entirely. + + * The source code changed to use ISO C standard-style function + definitions. + + * POSIX compliance for 'sub()' and 'gsub()' (*note Gory Details::). + + * The 'length()' function was extended to accept an array argument + and return the number of elements in the array (*note String + Functions::). + + * The 'strftime()' function acquired a third argument to enable + printing times as UTC (*note Time Functions::). + + Version 4.0 of 'gawk' introduced the following features: + + * Variable additions: + + - 'FPAT', which allows you to specify a regexp that matches the + fields, instead of matching the field separator (*note + Splitting By Content::). + + - If 'PROCINFO["sorted_in"]' exists, 'for(iggy in foo)' loops + sort the indices before looping over them. The value of this + element provides control over how the indices are sorted + before the loop traversal starts (*note Controlling + Scanning::). + + - 'PROCINFO["strftime"]', which holds the default format for + 'strftime()' (*note Time Functions::). + + * The special files '/dev/pid', '/dev/ppid', '/dev/pgrpid' and + '/dev/user' were removed. + + * Support for IPv6 was added via the '/inet6' special file. '/inet4' + forces IPv4 and '/inet' chooses the system default, which is + probably IPv4 (*note TCP/IP Networking::). + + * The use of '\s' and '\S' escape sequences in regular expressions + (*note GNU Regexp Operators::). + + * Interval expressions became part of default regular expressions + (*note Regexp Operators::). + + * POSIX character classes work even with '--traditional' (*note + Regexp Operators::). + + * 'break' and 'continue' became invalid outside a loop, even with + '--traditional' (*note Break Statement::, and also see *note + Continue Statement::). + + * 'fflush()', 'nextfile', and 'delete ARRAY' are allowed if '--posix' + or '--traditional', since they are all now part of POSIX. + + * An optional third argument to 'asort()' and 'asorti()', specifying + how to sort (*note String Functions::). + + * The behavior of 'fflush()' changed to match BWK 'awk' and for + POSIX; now both 'fflush()' and 'fflush("")' flush all open output + redirections (*note I/O Functions::). + + * The 'isarray()' function which distinguishes if an item is an array + or not, to make it possible to traverse arrays of arrays (*note + Type Functions::). + + * The 'patsplit()' function which gives the same capability as + 'FPAT', for splitting (*note String Functions::). + + * An optional fourth argument to the 'split()' function, which is an + array to hold the values of the separators (*note String + Functions::). + + * Arrays of arrays (*note Arrays of Arrays::). + + * The 'BEGINFILE' and 'ENDFILE' special patterns (*note + BEGINFILE/ENDFILE::). + + * Indirect function calls (*note Indirect Calls::). + + * 'switch' / 'case' are enabled by default (*note Switch + Statement::). + + * Command-line option changes (*note Options::): + + - The '-b' and '--characters-as-bytes' options which prevent + 'gawk' from treating input as a multibyte string. + + - The redundant '--compat', '--copyleft', and '--usage' long + options were removed. + + - The '--gen-po' option was finally renamed to the correct + '--gen-pot'. + + - The '--sandbox' option which disables certain features. + + - All long options acquired corresponding short options, for use + in '#!' scripts. + + * Directories named on the command line now produce a warning, not a + fatal error, unless '--posix' or '--traditional' are used (*note + Command-line directories::). + + * The 'gawk' internals were rewritten, bringing the 'dgawk' debugger + and possibly improved performance (*note Debugger::). + + * Per the GNU Coding Standards, dynamic extensions must now define a + global symbol indicating that they are GPL-compatible (*note Plugin + License::). + + * In POSIX mode, string comparisons use 'strcoll()' / 'wcscoll()' + (*note POSIX String Comparison::). + + * The option for raw sockets was removed, since it was never + implemented (*note TCP/IP Networking::). + + * Ranges of the form '[d-h]' are treated as if they were in the C + locale, no matter what kind of regexp is being used, and even if + '--posix' (*note Ranges and Locales::). + + * Support was removed for the following systems: + + - Atari + + - Amiga + + - BeOS + + - Cray + + - MIPS RiscOS + + - MS-DOS with Microsoft Compiler + + - MS-Windows with Microsoft Compiler + + - NeXT + + - SunOS 3.x, Sun 386 (Road Runner) + + - Tandem (non-POSIX) + + - Prestandard VAX C compiler for VAX/VMS + + Version 4.1 of 'gawk' introduced the following features: + + * Three new arrays: 'SYMTAB', 'FUNCTAB', and + 'PROCINFO["identifiers"]' (*note Auto-set::). + + * The three executables 'gawk', 'pgawk', and 'dgawk', were merged + into one, named just 'gawk'. As a result the command-line options + changed. + + * Command-line option changes (*note Options::): + + - The '-D' option invokes the debugger. + + - The '-i' and '--include' options load 'awk' library files. + + - The '-l' and '--load' options load compiled dynamic + extensions. + + - The '-M' and '--bignum' options enable MPFR. + + - The '-o' option only does pretty-printing. + + - The '-p' option is used for profiling. + + - The '-R' option was removed. + + * Support for high precision arithmetic with MPFR (*note Arbitrary + Precision Arithmetic::). + + * The 'and()', 'or()' and 'xor()' functions changed to allow any + number of arguments, with a minimum of two (*note Bitwise + Functions::). + + * The dynamic extension interface was completely redone (*note + Dynamic Extensions::). + + * Redirected 'getline' became allowed inside 'BEGINFILE' and + 'ENDFILE' (*note BEGINFILE/ENDFILE::). + + * The 'where' command was added to the debugger (*note Execution + Stack::). + + * Support for Ultrix was removed. + + Version 4.2 introduced the following changes: + + * Changes to 'ENVIRON' are reflected into 'gawk''s environment and + that of programs that it runs. *Note Auto-set::. + + * The '--pretty-print' option no longer runs the 'awk' program too. + *Note Options::. + + * The 'igawk' program and its manual page are no longer installed + when 'gawk' is built. *Note Igawk Program::. + + * The 'intdiv()' function. *Note Numeric Functions::. + + * The maximum number of hexadecimal digits in '\x' escapes is now + two. *Note Escape Sequences::. + + * Nonfatal output with 'print' and 'printf'. *Note Nonfatal::. + + * For many years, POSIX specified that default field splitting only + allowed spaces and tabs to separate fields, and this was how 'gawk' + behaved with '--posix'. As of 2013, the standard restored + historical behavior, and now default field splitting with '--posix' + also allows newlines to separate fields. + + * Support for MirBSD was removed. + + * Support for GNU/Linux on Alpha was removed. + + +File: gawk.info, Node: Common Extensions, Next: Ranges and Locales, Prev: Feature History, Up: Language History + +A.7 Common Extensions Summary +============================= + +The following table summarizes the common extensions supported by +'gawk', Brian Kernighan's 'awk', and 'mawk', the three most widely used +freely available versions of 'awk' (*note Other Versions::). + +Feature BWK 'awk' 'mawk' 'gawk' Now standard +-------------------------------------------------------------------------- +'\x' escape sequence X X X +'FS' as null string X X X +'/dev/stdin' special file X X X +'/dev/stdout' special file X X X +'/dev/stderr' special file X X X +'delete' without subscript X X X X +'fflush()' function X X X X +'length()' of an array X X X +'nextfile' statement X X X X +'**' and '**=' operators X X +'func' keyword X X +'BINMODE' variable X X +'RS' as regexp X X +Time-related functions X X + + +File: gawk.info, Node: Ranges and Locales, Next: Contributors, Prev: Common Extensions, Up: Language History + +A.8 Regexp Ranges and Locales: A Long Sad Story +=============================================== + +This minor node describes the confusing history of ranges within regular +expressions and their interactions with locales, and how this affected +different versions of 'gawk'. + + The original Unix tools that worked with regular expressions defined +character ranges (such as '[a-z]') to match any character between the +first character in the range and the last character in the range, +inclusive. Ordering was based on the numeric value of each character in +the machine's native character set. Thus, on ASCII-based systems, +'[a-z]' matched all the lowercase letters, and only the lowercase +letters, as the numeric values for the letters from 'a' through 'z' were +contiguous. (On an EBCDIC system, the range '[a-z]' includes additional +nonalphabetic characters as well.) + + Almost all introductory Unix literature explained range expressions +as working in this fashion, and in particular, would teach that the +"correct" way to match lowercase letters was with '[a-z]', and that +'[A-Z]' was the "correct" way to match uppercase letters. And indeed, +this was true.(1) + + The 1992 POSIX standard introduced the idea of locales (*note +Locales::). Because many locales include other letters besides the +plain 26 letters of the English alphabet, the POSIX standard added +character classes (*note Bracket Expressions::) as a way to match +different kinds of characters besides the traditional ones in the ASCII +character set. + + However, the standard _changed_ the interpretation of range +expressions. In the '"C"' and '"POSIX"' locales, a range expression +like '[a-dx-z]' is still equivalent to '[abcdxyz]', as in ASCII. But +outside those locales, the ordering was defined to be based on +"collation order". + + What does that mean? In many locales, 'A' and 'a' are both less than +'B'. In other words, these locales sort characters in dictionary order, +and '[a-dx-z]' is typically not equivalent to '[abcdxyz]'; instead, it +might be equivalent to '[ABCXYabcdxyz]', for example. + + This point needs to be emphasized: much literature teaches that you +should use '[a-z]' to match a lowercase character. But on systems with +non-ASCII locales, this also matches all of the uppercase characters +except 'A' or 'Z'! This was a continuous cause of confusion, even well +into the twenty-first century. + + To demonstrate these issues, the following example uses the 'sub()' +function, which does text replacement (*note String Functions::). Here, +the intent is to remove trailing uppercase characters: + + $ echo something1234abc | gawk-3.1.8 '{ sub("[A-Z]*$", ""); print }' + -| something1234a + +This output is unexpected, as the 'bc' at the end of 'something1234abc' +should not normally match '[A-Z]*'. This result is due to the locale +setting (and thus you may not see it on your system). + + Similar considerations apply to other ranges. For example, '["-/]' +is perfectly valid in ASCII, but is not valid in many Unicode locales, +such as 'en_US.UTF-8'. + + Early versions of 'gawk' used regexp matching code that was not +locale-aware, so ranges had their traditional interpretation. + + When 'gawk' switched to using locale-aware regexp matchers, the +problems began; especially as both GNU/Linux and commercial Unix vendors +started implementing non-ASCII locales, _and making them the default_. +Perhaps the most frequently asked question became something like, "Why +does '[A-Z]' match lowercase letters?!?" + + This situation existed for close to 10 years, if not more, and the +'gawk' maintainer grew weary of trying to explain that 'gawk' was being +nicely standards-compliant, and that the issue was in the user's locale. +During the development of version 4.0, he modified 'gawk' to always +treat ranges in the original, pre-POSIX fashion, unless '--posix' was +used (*note Options::).(2) + + Fortunately, shortly before the final release of 'gawk' 4.0, the +maintainer learned that the 2008 standard had changed the definition of +ranges, such that outside the '"C"' and '"POSIX"' locales, the meaning +of range expressions was _undefined_.(3) + + By using this lovely technical term, the standard gives license to +implementers to implement ranges in whatever way they choose. The +'gawk' maintainer chose to apply the pre-POSIX meaning both with the +default regexp matching and when '--traditional' or '--posix' are used. +In all cases 'gawk' remains POSIX-compliant. + + ---------- Footnotes ---------- + + (1) And Life was good. + + (2) And thus was born the Campaign for Rational Range Interpretation +(or RRI). A number of GNU tools have already implemented this change, or +will soon. Thanks to Karl Berry for coining the phrase "Rational Range +Interpretation." + + (3) See the standard +(http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_05) +and its rationale +(http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap09.html#tag_21_09_03_05). + + +File: gawk.info, Node: Contributors, Next: History summary, Prev: Ranges and Locales, Up: Language History + +A.9 Major Contributors to 'gawk' +================================ + + Always give credit where credit is due. + -- _Anonymous_ + + This minor node names the major contributors to 'gawk' and/or this +Info file, in approximate chronological order: + + * Dr. Alfred V. Aho, Dr. Peter J. Weinberger, and Dr. Brian W. + Kernighan, all of Bell Laboratories, designed and implemented Unix + 'awk', from which 'gawk' gets the majority of its feature set. + + * Paul Rubin did the initial design and implementation in 1986, and + wrote the first draft (around 40 pages) of this Info file. + + * Jay Fenlason finished the initial implementation. + + * Diane Close revised the first draft of this Info file, bringing it + to around 90 pages. + + * Richard Stallman helped finish the implementation and the initial + draft of this Info file. He is also the founder of the FSF and the + GNU Project. + + * John Woods contributed parts of the code (mostly fixes) in the + initial version of 'gawk'. + + * In 1988, David Trueman took over primary maintenance of 'gawk', + making it compatible with "new" 'awk', and greatly improving its + performance. + + * Conrad Kwok, Scott Garfinkle, and Kent Williams did the initial + ports to MS-DOS with various versions of MSC. + + * Pat Rankin provided the VMS port and its documentation. + + * Hal Peterson provided help in porting 'gawk' to Cray systems. + (This is no longer supported.) + + * Kai Uwe Rommel provided the initial port to OS/2 and its + documentation. + + * Michal Jaegermann provided the port to Atari systems and its + documentation. (This port is no longer supported.) He continues + to provide portability checking, and has done a lot of work to make + sure 'gawk' works on non-32-bit systems. + + * Fred Fish provided the port to Amiga systems and its documentation. + (With Fred's sad passing, this is no longer supported.) + + * Scott Deifik maintained the MS-DOS port using DJGPP. + + * Eli Zaretskii currently maintains the MS-Windows port using MinGW. + + * Juan Grigera provided a port to Windows32 systems. (This is no + longer supported.) + + * For many years, Dr. Darrel Hankerson acted as coordinator for the + various ports to different PC platforms and created binary + distributions for various PC operating systems. He was also + instrumental in keeping the documentation up to date for the + various PC platforms. + + * Christos Zoulas provided the 'extension()' built-in function for + dynamically adding new functions. (This was obsoleted at 'gawk' + 4.1.) + + * Ju"rgen Kahrs contributed the initial version of the TCP/IP + networking code and documentation, and motivated the inclusion of + the '|&' operator. + + * Stephen Davies provided the initial port to Tandem systems and its + documentation. (However, this is no longer supported.) He was + also instrumental in the initial work to integrate the byte-code + internals into the 'gawk' code base. + + * Matthew Woehlke provided improvements for Tandem's POSIX-compliant + systems. + + * Martin Brown provided the port to BeOS and its documentation. + (This is no longer supported.) + + * Arno Peters did the initial work to convert 'gawk' to use GNU + Automake and GNU 'gettext'. + + * Alan J. Broder provided the initial version of the 'asort()' + function as well as the code for the optional third argument to the + 'match()' function. + + * Andreas Buening updated the 'gawk' port for OS/2. + + * Isamu Hasegawa, of IBM in Japan, contributed support for multibyte + characters. + + * Michael Benzinger contributed the initial code for 'switch' + statements. + + * Patrick T.J. McPhee contributed the code for dynamic loading in + Windows32 environments. (This is no longer supported.) + + * Anders Wallin helped keep the VMS port going for several years. + + * Assaf Gordon contributed the code to implement the '--sandbox' + option. + + * John Haque made the following contributions: + + - The modifications to convert 'gawk' into a byte-code + interpreter, including the debugger + + - The addition of true arrays of arrays + + - The additional modifications for support of + arbitrary-precision arithmetic + + - The initial text of *note Arbitrary Precision Arithmetic:: + + - The work to merge the three versions of 'gawk' into one, for + the 4.1 release + + - Improved array internals for arrays indexed by integers + + - The improved array sorting features were also driven by John, + together with Pat Rankin + + * Panos Papadopoulos contributed the original text for *note Include + Files::. + + * Efraim Yawitz contributed the original text for *note Debugger::. + + * The development of the extension API first released with 'gawk' 4.1 + was driven primarily by Arnold Robbins and Andrew Schorr, with + notable contributions from the rest of the development team. + + * John Malmberg contributed significant improvements to the OpenVMS + port and the related documentation. + + * Antonio Giovanni Colombo rewrote a number of examples in the early + chapters that were severely dated, for which I am incredibly + grateful. + + * Arnold Robbins has been working on 'gawk' since 1988, at first + helping David Trueman, and as the primary maintainer since around + 1994. + + +File: gawk.info, Node: History summary, Prev: Contributors, Up: Language History + +A.10 Summary +============ + + * The 'awk' language has evolved over time. The first release was + with V7 Unix, circa 1978. In 1987, for System V Release 3.1, major + additions, including user-defined functions, were made to the + language. Additional changes were made for System V Release 4, in + 1989. Since then, further minor changes have happened under the + auspices of the POSIX standard. + + * Brian Kernighan's 'awk' provides a small number of extensions that + are implemented in common with other versions of 'awk'. + + * 'gawk' provides a large number of extensions over POSIX 'awk'. + They can be disabled with either the '--traditional' or '--posix' + options. + + * The interaction of POSIX locales and regexp matching in 'gawk' has + been confusing over the years. Today, 'gawk' implements Rational + Range Interpretation, where ranges of the form '[a-z]' match _only_ + the characters numerically between 'a' through 'z' in the machine's + native character set. Usually this is ASCII, but it can be EBCDIC + on IBM S/390 systems. + + * Many people have contributed to 'gawk' development over the years. + We hope that the list provided in this major node is complete and + gives the appropriate credit where credit is due. + + +File: gawk.info, Node: Installation, Next: Notes, Prev: Language History, Up: Top + +Appendix B Installing 'gawk' +**************************** + +This appendix provides instructions for installing 'gawk' on the various +platforms that are supported by the developers. The primary developer +supports GNU/Linux (and Unix), whereas the other ports are contributed. +*Note Bugs:: for the email addresses of the people who maintain the +respective ports. + +* Menu: + +* Gawk Distribution:: What is in the 'gawk' distribution. +* Unix Installation:: Installing 'gawk' under various + versions of Unix. +* Non-Unix Installation:: Installation on Other Operating Systems. +* Bugs:: Reporting Problems and Bugs. +* Other Versions:: Other freely available 'awk' + implementations. +* Installation summary:: Summary of installation. + + +File: gawk.info, Node: Gawk Distribution, Next: Unix Installation, Up: Installation + +B.1 The 'gawk' Distribution +=========================== + +This minor node describes how to get the 'gawk' distribution, how to +extract it, and then what is in the various files and subdirectories. + +* Menu: + +* Getting:: How to get the distribution. +* Extracting:: How to extract the distribution. +* Distribution contents:: What is in the distribution. + + +File: gawk.info, Node: Getting, Next: Extracting, Up: Gawk Distribution + +B.1.1 Getting the 'gawk' Distribution +------------------------------------- + +There are two ways to get GNU software: + + * Copy it from someone else who already has it. + + * Retrieve 'gawk' from the Internet host 'ftp.gnu.org', in the + directory '/gnu/gawk'. Both anonymous 'ftp' and 'http' access are + supported. If you have the 'wget' program, you can use a command + like the following: + + wget http://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.gz + + The GNU software archive is mirrored around the world. The +up-to-date list of mirror sites is available from the main FSF website +(http://www.gnu.org/order/ftp.html). Try to use one of the mirrors; +they will be less busy, and you can usually find one closer to your +site. + + You may also retrieve the 'gawk' source code from the official Git +repository; for more information see *note Accessing The Source::. + + +File: gawk.info, Node: Extracting, Next: Distribution contents, Prev: Getting, Up: Gawk Distribution + +B.1.2 Extracting the Distribution +--------------------------------- + +'gawk' is distributed as several 'tar' files compressed with different +compression programs: 'gzip', 'bzip2', and 'xz'. For simplicity, the +rest of these instructions assume you are using the one compressed with +the GNU Gzip program ('gzip'). + + Once you have the distribution (e.g., 'gawk-4.1.4.tar.gz'), use +'gzip' to expand the file and then use 'tar' to extract it. You can use +the following pipeline to produce the 'gawk' distribution: + + gzip -d -c gawk-4.1.4.tar.gz | tar -xvpf - + + On a system with GNU 'tar', you can let 'tar' do the decompression +for you: + + tar -xvpzf gawk-4.1.4.tar.gz + +Extracting the archive creates a directory named 'gawk-4.1.4' in the +current directory. + + The distribution file name is of the form 'gawk-V.R.P.tar.gz'. The V +represents the major version of 'gawk', the R represents the current +release of version V, and the P represents a "patch level", meaning that +minor bugs have been fixed in the release. The current patch level is +4, but when retrieving distributions, you should get the version with +the highest version, release, and patch level. (Note, however, that +patch levels greater than or equal to 70 denote "beta" or nonproduction +software; you might not want to retrieve such a version unless you don't +mind experimenting.) If you are not on a Unix or GNU/Linux system, you +need to make other arrangements for getting and extracting the 'gawk' +distribution. You should consult a local expert. + + +File: gawk.info, Node: Distribution contents, Prev: Extracting, Up: Gawk Distribution + +B.1.3 Contents of the 'gawk' Distribution +----------------------------------------- + +The 'gawk' distribution has a number of C source files, documentation +files, subdirectories, and files related to the configuration process +(*note Unix Installation::), as well as several subdirectories related +to different non-Unix operating systems: + +Various '.c', '.y', and '.h' files + These files contain the actual 'gawk' source code. + +'ABOUT-NLS' + A file containing information about GNU 'gettext' and translations. + +'AUTHORS' + A file with some information about the authorship of 'gawk'. It + exists only to satisfy the pedants at the Free Software Foundation. + +'README' +'README_d/README.*' + Descriptive files: 'README' for 'gawk' under Unix and the rest for + the various hardware and software combinations. + +'INSTALL' + A file providing an overview of the configuration and installation + process. + +'ChangeLog' + A detailed list of source code changes as bugs are fixed or + improvements made. + +'ChangeLog.0' + An older list of source code changes. + +'NEWS' + A list of changes to 'gawk' since the last release or patch. + +'NEWS.0' + An older list of changes to 'gawk'. + +'COPYING' + The GNU General Public License. + +'POSIX.STD' + A description of behaviors in the POSIX standard for 'awk' that are + left undefined, or where 'gawk' may not comply fully, as well as a + list of things that the POSIX standard should describe but does + not. + +'doc/awkforai.txt' + Pointers to the original draft of a short article describing why + 'gawk' is a good language for artificial intelligence (AI) + programming. + +'doc/bc_notes' + A brief description of 'gawk''s "byte code" internals. + +'doc/README.card' +'doc/ad.block' +'doc/awkcard.in' +'doc/cardfonts' +'doc/colors' +'doc/macros' +'doc/no.colors' +'doc/setter.outline' + The 'troff' source for a five-color 'awk' reference card. A modern + version of 'troff' such as GNU 'troff' ('groff') is needed to + produce the color version. See the file 'README.card' for + instructions if you have an older 'troff'. + +'doc/gawk.1' + The 'troff' source for a manual page describing 'gawk'. This is + distributed for the convenience of Unix users. + +'doc/gawktexi.in' +'doc/sidebar.awk' + The Texinfo source file for this Info file. It should be processed + by 'doc/sidebar.awk' before processing with 'texi2dvi' or + 'texi2pdf' to produce a printed document, and with 'makeinfo' to + produce an Info or HTML file. The 'Makefile' takes care of this + processing and produces printable output via 'texi2dvi' or + 'texi2pdf'. + +'doc/gawk.texi' + The file produced after processing 'gawktexi.in' with + 'sidebar.awk'. + +'doc/gawk.info' + The generated Info file for this Info file. + +'doc/gawkinet.texi' + The Texinfo source file for *note (General Introduction, gawkinet, + TCP/IP Internetworking with 'gawk')Top::. It should be processed + with TeX (via 'texi2dvi' or 'texi2pdf') to produce a printed + document and with 'makeinfo' to produce an Info or HTML file. + +'doc/gawkinet.info' + The generated Info file for 'TCP/IP Internetworking with 'gawk''. + +'doc/igawk.1' + The 'troff' source for a manual page describing the 'igawk' program + presented in *note Igawk Program::. (Since 'gawk' can do its own + '@include' processing, neither 'igawk' nor 'igawk.1' are + installed.) + +'doc/Makefile.in' + The input file used during the configuration process to generate + the actual 'Makefile' for creating the documentation. + +'Makefile.am' +'*/Makefile.am' + Files used by the GNU Automake software for generating the + 'Makefile.in' files used by Autoconf and 'configure'. + +'Makefile.in' +'aclocal.m4' +'bisonfix.awk' +'config.guess' +'configh.in' +'configure.ac' +'configure' +'custom.h' +'depcomp' +'install-sh' +'missing_d/*' +'mkinstalldirs' +'m4/*' + These files and subdirectories are used when configuring and + compiling 'gawk' for various Unix systems. Most of them are + explained in *note Unix Installation::. The rest are there to + support the main infrastructure. + +'po/*' + The 'po' library contains message translations. + +'awklib/extract.awk' +'awklib/Makefile.am' +'awklib/Makefile.in' +'awklib/eg/*' + The 'awklib' directory contains a copy of 'extract.awk' (*note + Extract Program::), which can be used to extract the sample + programs from the Texinfo source file for this Info file. It also + contains a 'Makefile.in' file, which 'configure' uses to generate a + 'Makefile'. 'Makefile.am' is used by GNU Automake to create + 'Makefile.in'. The library functions from *note Library + Functions::, are included as ready-to-use files in the 'gawk' + distribution. They are installed as part of the installation + process. The rest of the programs in this Info file are available + in appropriate subdirectories of 'awklib/eg'. + +'extension/*' + The source code, manual pages, and infrastructure files for the + sample extensions included with 'gawk'. *Note Dynamic + Extensions::, for more information. + +'extras/*' + Additional non-essential files. Currently, this directory contains + some shell startup files to be installed in '/etc/profile.d' to aid + in manipulating the 'AWKPATH' and 'AWKLIBPATH' environment + variables. *Note Shell Startup Files::, for more information. + +'posix/*' + Files needed for building 'gawk' on POSIX-compliant systems. + +'pc/*' + Files needed for building 'gawk' under MS-Windows (*note PC + Installation:: for details). + +'vms/*' + Files needed for building 'gawk' under Vax/VMS and OpenVMS (*note + VMS Installation:: for details). + +'test/*' + A test suite for 'gawk'. You can use 'make check' from the + top-level 'gawk' directory to run your version of 'gawk' against + the test suite. If 'gawk' successfully passes 'make check', then + you can be confident of a successful port. + + +File: gawk.info, Node: Unix Installation, Next: Non-Unix Installation, Prev: Gawk Distribution, Up: Installation + +B.2 Compiling and Installing 'gawk' on Unix-Like Systems +======================================================== + +Usually, you can compile and install 'gawk' by typing only two commands. +However, if you use an unusual system, you may need to configure 'gawk' +for your system yourself. + +* Menu: + +* Quick Installation:: Compiling 'gawk' under Unix. +* Shell Startup Files:: Shell convenience functions. +* Additional Configuration Options:: Other compile-time options. +* Configuration Philosophy:: How it's all supposed to work. + + +File: gawk.info, Node: Quick Installation, Next: Shell Startup Files, Up: Unix Installation + +B.2.1 Compiling 'gawk' for Unix-Like Systems +-------------------------------------------- + +The normal installation steps should work on all modern commercial +Unix-derived systems, GNU/Linux, BSD-based systems, and the Cygwin +environment for MS-Windows. + + After you have extracted the 'gawk' distribution, 'cd' to +'gawk-4.1.4'. As with most GNU software, you configure 'gawk' for your +system by running the 'configure' program. This program is a Bourne +shell script that is generated automatically using GNU Autoconf. (The +Autoconf software is described fully starting with *note (Autoconf, +autoconf,Autoconf---Generating Automatic Configuration Scripts)Top::.) + + To configure 'gawk', simply run 'configure': + + sh ./configure + + This produces a 'Makefile' and 'config.h' tailored to your system. +The 'config.h' file describes various facts about your system. You +might want to edit the 'Makefile' to change the 'CFLAGS' variable, which +controls the command-line options that are passed to the C compiler +(such as optimization levels or compiling for debugging). + + Alternatively, you can add your own values for most 'make' variables +on the command line, such as 'CC' and 'CFLAGS', when running +'configure': + + CC=cc CFLAGS=-g sh ./configure + +See the file 'INSTALL' in the 'gawk' distribution for all the details. + + After you have run 'configure' and possibly edited the 'Makefile', +type: + + make + +Shortly thereafter, you should have an executable version of 'gawk'. +That's all there is to it! To verify that 'gawk' is working properly, +run 'make check'. All of the tests should succeed. If these steps do +not work, or if any of the tests fail, check the files in the 'README_d' +directory to see if you've found a known problem. If the failure is not +described there, send in a bug report (*note Bugs::). + + Of course, once you've built 'gawk', it is likely that you will wish +to install it. To do so, you need to run the command 'make install', as +a user with the appropriate permissions. How to do this varies by +system, but on many systems you can use the 'sudo' command to do so. +The command then becomes 'sudo make install'. It is likely that you +will be asked for your password, and you will have to have been set up +previously as a user who is allowed to run the 'sudo' command. + + +File: gawk.info, Node: Shell Startup Files, Next: Additional Configuration Options, Prev: Quick Installation, Up: Unix Installation + +B.2.2 Shell Startup Files +------------------------- + +The distribution contains shell startup files 'gawk.sh' and 'gawk.csh' +containing functions to aid in manipulating the 'AWKPATH' and +'AWKLIBPATH' environment variables. On a Fedora system, these files +should be installed in '/etc/profile.d'; on other platforms, the +appropriate location may be different. + +'gawkpath_default' + Reset the 'AWKPATH' environment variable to its default value. + +'gawkpath_prepend' + Add the argument to the front of the 'AWKPATH' environment + variable. + +'gawkpath_append' + Add the argument to the end of the 'AWKPATH' environment variable. + +'gawklibpath_default' + Reset the 'AWKLIBPATH' environment variable to its default value. + +'gawklibpath_prepend' + Add the argument to the front of the 'AWKLIBPATH' environment + variable. + +'gawklibpath_append' + Add the argument to the end of the 'AWKLIBPATH' environment + variable. + + +File: gawk.info, Node: Additional Configuration Options, Next: Configuration Philosophy, Prev: Shell Startup Files, Up: Unix Installation + +B.2.3 Additional Configuration Options +-------------------------------------- + +There are several additional options you may use on the 'configure' +command line when compiling 'gawk' from scratch, including: + +'--disable-extensions' + Disable configuring and building the sample extensions in the + 'extension' directory. This is useful for cross-compiling. The + default action is to dynamically check if the extensions can be + configured and compiled. + +'--disable-lint' + Disable all lint checking within 'gawk'. The '--lint' and + '--lint-old' options (*note Options::) are accepted, but silently + do nothing. Similarly, setting the 'LINT' variable (*note + User-modified::) has no effect on the running 'awk' program. + + When used with the GNU Compiler Collection's (GCC's) automatic + dead-code-elimination, this option cuts almost 23K bytes off the + size of the 'gawk' executable on GNU/Linux x86_64 systems. Results + on other systems and with other compilers are likely to vary. + Using this option may bring you some slight performance + improvement. + + CAUTION: Using this option will cause some of the tests in the + test suite to fail. This option may be removed at a later + date. + +'--disable-nls' + Disable all message-translation facilities. This is usually not + desirable, but it may bring you some slight performance + improvement. + +'--with-whiny-user-strftime' + Force use of the included version of the C 'strftime()' function + for deficient systems. + + Use the command './configure --help' to see the full list of options +supplied by 'configure'. + + +File: gawk.info, Node: Configuration Philosophy, Prev: Additional Configuration Options, Up: Unix Installation + +B.2.4 The Configuration Process +------------------------------- + +This minor node is of interest only if you know something about using +the C language and Unix-like operating systems. + + The source code for 'gawk' generally attempts to adhere to formal +standards wherever possible. This means that 'gawk' uses library +routines that are specified by the ISO C standard and by the POSIX +operating system interface standard. The 'gawk' source code requires +using an ISO C compiler (the 1990 standard). + + Many Unix systems do not support all of either the ISO or the POSIX +standards. The 'missing_d' subdirectory in the 'gawk' distribution +contains replacement versions of those functions that are most likely to +be missing. + + The 'config.h' file that 'configure' creates contains definitions +that describe features of the particular operating system where you are +attempting to compile 'gawk'. The three things described by this file +are: what header files are available, so that they can be correctly +included, what (supposedly) standard functions are actually available in +your C libraries, and various miscellaneous facts about your operating +system. For example, there may not be an 'st_blksize' element in the +'stat' structure. In this case, 'HAVE_STRUCT_STAT_ST_BLKSIZE' is +undefined. + + It is possible for your C compiler to lie to 'configure'. It may do +so by not exiting with an error when a library function is not +available. To get around this, edit the 'custom.h' file. Use an +'#ifdef' that is appropriate for your system, and either '#define' any +constants that 'configure' should have defined but didn't, or '#undef' +any constants that 'configure' defined and should not have. The +'custom.h' file is automatically included by the 'config.h' file. + + It is also possible that the 'configure' program generated by +Autoconf will not work on your system in some other fashion. If you do +have a problem, the 'configure.ac' file is the input for Autoconf. You +may be able to change this file and generate a new version of +'configure' that works on your system (*note Bugs:: for information on +how to report problems in configuring 'gawk'). The same mechanism may +be used to send in updates to 'configure.ac' and/or 'custom.h'. + + +File: gawk.info, Node: Non-Unix Installation, Next: Bugs, Prev: Unix Installation, Up: Installation + +B.3 Installation on Other Operating Systems +=========================================== + +This minor node describes how to install 'gawk' on various non-Unix +systems. + +* Menu: + +* PC Installation:: Installing and Compiling 'gawk' on + Microsoft Windows. +* VMS Installation:: Installing 'gawk' on VMS. + + +File: gawk.info, Node: PC Installation, Next: VMS Installation, Up: Non-Unix Installation + +B.3.1 Installation on MS-Windows +-------------------------------- + +This minor node covers installation and usage of 'gawk' on Intel +architecture machines running any version of MS-Windows. In this minor +node, the term "Windows32" refers to any of Microsoft Windows +95/98/ME/NT/2000/XP/Vista/7/8/10. + + See also the 'README_d/README.pc' file in the distribution. + +* Menu: + +* PC Binary Installation:: Installing a prepared distribution. +* PC Compiling:: Compiling 'gawk' for Windows32. +* PC Using:: Running 'gawk' on Windows32. +* Cygwin:: Building and running 'gawk' for + Cygwin. +* MSYS:: Using 'gawk' In The MSYS Environment. + + +File: gawk.info, Node: PC Binary Installation, Next: PC Compiling, Up: PC Installation + +B.3.1.1 Installing a Prepared Distribution for MS-Windows Systems +................................................................. + +The only supported binary distribution for MS-Windows systems is that +provided by Eli Zaretskii's "ezwinports" +(https://sourceforge.net/projects/ezwinports/) project. Install the +compiled 'gawk' from there. + + +File: gawk.info, Node: PC Compiling, Next: PC Using, Prev: PC Binary Installation, Up: PC Installation + +B.3.1.2 Compiling 'gawk' for PC Operating Systems +................................................. + +'gawk' can be compiled for Windows32 using MinGW (Windows32). The file +'README_d/README.pc' in the 'gawk' distribution contains additional +notes, and 'pc/Makefile' contains important information on compilation +options. + + To build 'gawk' for Windows32, copy the files in the 'pc' directory +(_except_ for 'ChangeLog') to the directory with the rest of the 'gawk' +sources, then invoke 'make' with the appropriate target name as an +argument to build 'gawk'. The 'Makefile' copied from the 'pc' directory +contains a configuration section with comments and may need to be edited +in order to work with your 'make' utility. + + The 'Makefile' supports a number of targets for building various +MS-DOS and Windows32 versions. A list of targets is printed if the +'make' command is given without a target. As an example, to build a +native MS-Windows binary of 'gawk' using the MinGW tools, type 'make +mingw32'. + + +File: gawk.info, Node: PC Using, Next: Cygwin, Prev: PC Compiling, Up: PC Installation + +B.3.1.3 Using 'gawk' on PC Operating Systems +............................................ + +Under MS-Windows, the Cygwin and MinGW environments support both the +'|&' operator and TCP/IP networking (*note TCP/IP Networking::). + + The MS-Windows version of 'gawk' searches for program files as +described in *note AWKPATH Variable::. However, semicolons (rather than +colons) separate elements in the 'AWKPATH' variable. If 'AWKPATH' is +not set or is empty, then the default search path is +'.;c:/lib/awk;c:/gnu/lib/awk'. + + Under MS-Windows, 'gawk' (and many other text programs) silently +translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on +output. A special 'BINMODE' variable (c.e.) allows control over these +translations and is interpreted as follows: + + * If 'BINMODE' is '"r"' or one, then binary mode is set on read + (i.e., no translations on reads). + + * If 'BINMODE' is '"w"' or two, then binary mode is set on write + (i.e., no translations on writes). + + * If 'BINMODE' is '"rw"' or '"wr"' or three, binary mode is set for + both read and write. + + * 'BINMODE=NON-NULL-STRING' is the same as 'BINMODE=3' (i.e., no + translations on reads or writes). However, 'gawk' issues a warning + message if the string is not one of '"rw"' or '"wr"'. + +The modes for standard input and standard output are set one time only +(after the command line is read, but before processing any of the 'awk' +program). Setting 'BINMODE' for standard input or standard output is +accomplished by using an appropriate '-v BINMODE=N' option on the +command line. 'BINMODE' is set at the time a file or pipe is opened and +cannot be changed midstream. + + The name 'BINMODE' was chosen to match 'mawk' (*note Other +Versions::). 'mawk' and 'gawk' handle 'BINMODE' similarly; however, +'mawk' adds a '-W BINMODE=N' option and an environment variable that can +set 'BINMODE', 'RS', and 'ORS'. The files 'binmode[1-3].awk' (under +'gnu/lib/awk' in some of the prepared binary distributions) have been +chosen to match 'mawk''s '-W BINMODE=N' option. These can be changed or +discarded; in particular, the setting of 'RS' giving the fewest +"surprises" is open to debate. 'mawk' uses 'RS = "\r\n"' if binary mode +is set on read, which is appropriate for files with the MS-DOS-style +end-of-line. + + To illustrate, the following examples set binary mode on writes for +standard output and other files, and set 'ORS' as the "usual" +MS-DOS-style end-of-line: + + gawk -v BINMODE=2 -v ORS="\r\n" ... + +or: + + gawk -v BINMODE=w -f binmode2.awk ... + +These give the same result as the '-W BINMODE=2' option in 'mawk'. The +following changes the record separator to '"\r\n"' and sets binary mode +on reads, but does not affect the mode on standard input: + + gawk -v RS="\r\n" -e "BEGIN { BINMODE = 1 }" ... + +or: + + gawk -f binmode1.awk ... + +With proper quoting, in the first example the setting of 'RS' can be +moved into the 'BEGIN' rule. + + +File: gawk.info, Node: Cygwin, Next: MSYS, Prev: PC Using, Up: PC Installation + +B.3.1.4 Using 'gawk' In The Cygwin Environment +.............................................. + +'gawk' can be built and used "out of the box" under MS-Windows if you +are using the Cygwin environment (http://www.cygwin.com). This +environment provides an excellent simulation of GNU/Linux, using Bash, +GCC, GNU Make, and other GNU programs. Compilation and installation for +Cygwin is the same as for a Unix system: + + tar -xvpzf gawk-4.1.4.tar.gz + cd gawk-4.1.4 + ./configure + make && make check + + When compared to GNU/Linux on the same system, the 'configure' step +on Cygwin takes considerably longer. However, it does finish, and then +the 'make' proceeds as usual. + + +File: gawk.info, Node: MSYS, Prev: Cygwin, Up: PC Installation + +B.3.1.5 Using 'gawk' In The MSYS Environment +............................................ + +In the MSYS environment under MS-Windows, 'gawk' automatically uses +binary mode for reading and writing files. Thus, there is no need to +use the 'BINMODE' variable. + + This can cause problems with other Unix-like components that have +been ported to MS-Windows that expect 'gawk' to do automatic translation +of '"\r\n"', because it won't. + + +File: gawk.info, Node: VMS Installation, Prev: PC Installation, Up: Non-Unix Installation + +B.3.2 Compiling and Installing 'gawk' on Vax/VMS and OpenVMS +------------------------------------------------------------ + +This node describes how to compile and install 'gawk' under VMS. The +older designation "VMS" is used throughout to refer to OpenVMS. + +* Menu: + +* VMS Compilation:: How to compile 'gawk' under VMS. +* VMS Dynamic Extensions:: Compiling 'gawk' dynamic extensions on + VMS. +* VMS Installation Details:: How to install 'gawk' under VMS. +* VMS Running:: How to run 'gawk' under VMS. +* VMS GNV:: The VMS GNV Project. +* VMS Old Gawk:: An old version comes with some VMS systems. + + +File: gawk.info, Node: VMS Compilation, Next: VMS Dynamic Extensions, Up: VMS Installation + +B.3.2.1 Compiling 'gawk' on VMS +............................... + +To compile 'gawk' under VMS, there is a 'DCL' command procedure that +issues all the necessary 'CC' and 'LINK' commands. There is also a +'Makefile' for use with the 'MMS' and 'MMK' utilities. From the source +directory, use either: + + $ @[.vms]vmsbuild.com + +or: + + $ MMS/DESCRIPTION=[.vms]descrip.mms gawk + +or: + + $ MMK/DESCRIPTION=[.vms]descrip.mms gawk + + 'MMK' is an open source, free, near-clone of 'MMS' and can better +handle ODS-5 volumes with upper- and lowercase file names. 'MMK' is +available from <https://github.com/endlesssoftware/mmk>. + + With ODS-5 volumes and extended parsing enabled, the case of the +target parameter may need to be exact. + + 'gawk' has been tested under VAX/VMS 7.3 and Alpha/VMS 7.3-1 using +Compaq C V6.4, and under Alpha/VMS 7.3, Alpha/VMS 7.3-2, and IA64/VMS +8.3. The most recent builds used HP C V7.3 on Alpha VMS 8.3 and both +Alpha and IA64 VMS 8.4 used HP C 7.3.(1) + + *Note VMS GNV:: for information on building 'gawk' as a PCSI kit that +is compatible with the GNV product. + + ---------- Footnotes ---------- + + (1) The IA64 architecture is also known as "Itanium." + + +File: gawk.info, Node: VMS Dynamic Extensions, Next: VMS Installation Details, Prev: VMS Compilation, Up: VMS Installation + +B.3.2.2 Compiling 'gawk' Dynamic Extensions on VMS +.................................................. + +The extensions that have been ported to VMS can be built using one of +the following commands: + + $ MMS/DESCRIPTION=[.vms]descrip.mms extensions + +or: + + $ MMK/DESCRIPTION=[.vms]descrip.mms extensions + + 'gawk' uses 'AWKLIBPATH' as either an environment variable or a +logical name to find the dynamic extensions. + + Dynamic extensions need to be compiled with the same compiler options +for floating-point, pointer size, and symbol name handling as were used +to compile 'gawk' itself. Alpha and Itanium should use IEEE floating +point. The pointer size is 32 bits, and the symbol name handling should +be exact case with CRC shortening for symbols longer than 32 bits. + + For Alpha and Itanium: + + /name=(as_is,short) + /float=ieee/ieee_mode=denorm_results + + For VAX: + + /name=(as_is,short) + + Compile-time macros need to be defined before the first VMS-supplied +header file is included, as follows: + + #if (__CRTL_VER >= 70200000) && !defined (__VAX) + #define _LARGEFILE 1 + #endif + + #ifndef __VAX + #ifdef __CRTL_VER + #if __CRTL_VER >= 80200000 + #define _USE_STD_STAT 1 + #endif + #endif + #endif + + If you are writing your own extensions to run on VMS, you must supply +these definitions yourself. The 'config.h' file created when building +'gawk' on VMS does this for you; if instead you use that file or a +similar one, then you must remember to include it before any +VMS-supplied header files. + + +File: gawk.info, Node: VMS Installation Details, Next: VMS Running, Prev: VMS Dynamic Extensions, Up: VMS Installation + +B.3.2.3 Installing 'gawk' on VMS +................................ + +To use 'gawk', all you need is a "foreign" command, which is a 'DCL' +symbol whose value begins with a dollar sign. For example: + + $ GAWK :== $disk1:[gnubin]gawk + +Substitute the actual location of 'gawk.exe' for '$disk1:[gnubin]'. The +symbol should be placed in the 'login.com' of any user who wants to run +'gawk', so that it is defined every time the user logs on. +Alternatively, the symbol may be placed in the system-wide 'sylogin.com' +procedure, which allows all users to run 'gawk'. + + If your 'gawk' was installed by a PCSI kit into the 'GNV$GNU:' +directory tree, the program will be known as 'GNV$GNU:[bin]gnv$gawk.exe' +and the help file will be 'GNV$GNU:[vms_help]gawk.hlp'. + + The PCSI kit also installs a 'GNV$GNU:[vms_bin]gawk_verb.cld' file +that can be used to add 'gawk' and 'awk' as DCL commands. + + For just the current process you can use: + + $ set command gnv$gnu:[vms_bin]gawk_verb.cld + + Or the system manager can use 'GNV$GNU:[vms_bin]gawk_verb.cld' to add +the 'gawk' and 'awk' to the system-wide 'DCLTABLES'. + + The DCL syntax is documented in the 'gawk.hlp' file. + + Optionally, the 'gawk.hlp' entry can be loaded into a VMS help +library: + + $ LIBRARY/HELP sys$help:helplib [.vms]gawk.hlp + +(You may want to substitute a site-specific help library rather than the +standard VMS library 'HELPLIB'.) After loading the help text, the +command: + + $ HELP GAWK + +provides information about both the 'gawk' implementation and the 'awk' +programming language. + + The logical name 'AWK_LIBRARY' can designate a default location for +'awk' program files. For the '-f' option, if the specified file name +has no device or directory path information in it, 'gawk' looks in the +current directory first, then in the directory specified by the +translation of 'AWK_LIBRARY' if the file is not found. If, after +searching in both directories, the file still is not found, 'gawk' +appends the suffix '.awk' to the file name and retries the file search. +If 'AWK_LIBRARY' has no definition, a default value of 'SYS$LIBRARY:' is +used for it. + + +File: gawk.info, Node: VMS Running, Next: VMS GNV, Prev: VMS Installation Details, Up: VMS Installation + +B.3.2.4 Running 'gawk' on VMS +............................. + +Command-line parsing and quoting conventions are significantly different +on VMS, so examples in this Info file or from other sources often need +minor changes. They _are_ minor though, and all 'awk' programs should +run correctly. + + Here are a couple of trivial tests: + + $ gawk -- "BEGIN {print ""Hello, World!""}" + $ gawk -"W" version + ! could also be -"W version" or "-W version" + +Note that uppercase and mixed-case text must be quoted. + + The VMS port of 'gawk' includes a 'DCL'-style interface in addition +to the original shell-style interface (see the help entry for details). +One side effect of dual command-line parsing is that if there is only a +single parameter (as in the quoted string program), the command becomes +ambiguous. To work around this, the normally optional '--' flag is +required to force Unix-style parsing rather than 'DCL' parsing. If any +other dash-type options (or multiple parameters such as data files to +process) are present, there is no ambiguity and '--' can be omitted. + + The 'exit' value is a Unix-style value and is encoded into a VMS exit +status value when the program exits. + + The VMS severity bits will be set based on the 'exit' value. A +failure is indicated by 1, and VMS sets the 'ERROR' status. A fatal +error is indicated by 2, and VMS sets the 'FATAL' status. All other +values will have the 'SUCCESS' status. The exit value is encoded to +comply with VMS coding standards and will have the 'C_FACILITY_NO' of +'0x350000' with the constant '0xA000' added to the number shifted over +by 3 bits to make room for the severity codes. + + To extract the actual 'gawk' exit code from the VMS status, use: + + unix_status = (vms_status .and. %x7f8) / 8 + +A C program that uses 'exec()' to call 'gawk' will get the original +Unix-style exit value. + + Older versions of 'gawk' for VMS treated a Unix exit code 0 as 1, a +failure as 2, a fatal error as 4, and passed all the other numbers +through. This violated the VMS exit status coding requirements. + + VAX/VMS floating point uses unbiased rounding. *Note Round +Function::. + + VMS reports time values in GMT unless one of the 'SYS$TIMEZONE_RULE' +or 'TZ' logical names is set. Older versions of VMS, such as VAX/VMS +7.3, do not set these logical names. + + The default search path, when looking for 'awk' program files +specified by the '-f' option, is '"SYS$DISK:[],AWK_LIBRARY:"'. The +logical name 'AWKPATH' can be used to override this default. The format +of 'AWKPATH' is a comma-separated list of directory specifications. +When defining it, the value should be quoted so that it retains a single +translation and not a multitranslation 'RMS' searchlist. + + This restriction also applies to running 'gawk' under GNV, as +redirection is always to a DCL command. + + If you are redirecting data to a VMS command or utility, the current +implementation requires that setting up a VMS foreign command that runs +a command file before invoking 'gawk'. (This restriction may be removed +in a future release of 'gawk' on VMS.) + + Without this command file, the input data will also appear prepended +to the output data. + + This also allows simulating POSIX commands that are not found on VMS +or the use of GNV utilities. + + The example below is for 'gawk' redirecting data to the VMS 'sort' +command. + + $ sort = "@device:[dir]vms_gawk_sort.com" + + The command file needs to be of the format in the example below. + + The first line inhibits the passed input data from also showing up in +the output. It must be in the format in the example. + + The next line creates a foreign command that overrides the outer +foreign command which prevents an infinite recursion of command files. + + The next to the last command redirects 'sys$input' to be +'sys$command', in order to pick up the data that is being redirected to +the command. + + The last line runs the actual command. It must be the last command +as the data redirected from 'gawk' will be read when the command file +ends. + + $!'f$verify(0,0)' + $ sort := sort + $ define/user sys$input sys$command: + $ sort sys$input: sys$output: + + +File: gawk.info, Node: VMS GNV, Next: VMS Old Gawk, Prev: VMS Running, Up: VMS Installation + +B.3.2.5 The VMS GNV Project +........................... + +The VMS GNV package provides a build environment similar to POSIX with +ports of a collection of open source tools. The 'gawk' found in the GNV +base kit is an older port. Currently, the GNV project is being +reorganized to supply individual PCSI packages for each component. See +<https://sourceforge.net/p/gnv/wiki/InstallingGNVPackages/>. + + The normal build procedure for 'gawk' produces a program that is +suitable for use with GNV. + + The file 'vms/gawk_build_steps.txt' in the distribution documents the +procedure for building a VMS PCSI kit that is compatible with GNV. + + +File: gawk.info, Node: VMS Old Gawk, Prev: VMS GNV, Up: VMS Installation + +B.3.2.6 Some VMS Systems Have An Old Version of 'gawk' +...................................................... + +Some versions of VMS have an old version of 'gawk'. To access it, +define a symbol, as follows: + + $ gawk :== $sys$common:[syshlp.examples.tcpip.snmp]gawk.exe + + This is apparently version 2.15.6, which is extremely old. We +recommend compiling and using the current version. + + +File: gawk.info, Node: Bugs, Next: Other Versions, Prev: Non-Unix Installation, Up: Installation + +B.4 Reporting Problems and Bugs +=============================== + + There is nothing more dangerous than a bored archaeologist. + -- _Douglas Adams, 'The Hitchhiker's Guide to the Galaxy'_ + + If you have problems with 'gawk' or think that you have found a bug, +report it to the developers; we cannot promise to do anything, but we +might well want to fix it. + +* Menu: + +* Bug address:: Where to send reports to. +* Usenet:: Where not to send reports to. +* Maintainers:: Maintainers of non-*nix ports. + + +File: gawk.info, Node: Bug address, Next: Usenet, Up: Bugs + +B.4.1 Submitting Bug Reports +---------------------------- + +Before reporting a bug, make sure you have really found a genuine bug. +Carefully reread the documentation and see if it says you can do what +you're trying to do. If it's not clear whether you should be able to do +something or not, report that too; it's a bug in the documentation! + + Before reporting a bug or trying to fix it yourself, try to isolate +it to the smallest possible 'awk' program and input data file that +reproduce the problem. Then send us the program and data file, some +idea of what kind of Unix system you're using, the compiler you used to +compile 'gawk', and the exact results 'gawk' gave you. Also say what +you expected to occur; this helps us decide whether the problem is +really in the documentation. + + Make sure to include the version number of 'gawk' you are using. You +can get this information with the command 'gawk --version'. + + Once you have a precise problem description, send email to +<bug-gawk@gnu.org>. + + The 'gawk' maintainers subscribe to this address, and thus they will +receive your bug report. Although you can send mail to the maintainers +directly, the bug reporting address is preferred because the email list +is archived at the GNU Project. _All email must be in English. This is +the only language understood in common by all the maintainers._ In +addition, please be sure to send all mail in _plain text_, not (or not +exclusively) in HTML. + + NOTE: Many distributions of GNU/Linux and the various BSD-based + operating systems have their own bug reporting systems. If you + report a bug using your distribution's bug reporting system, you + should also send a copy to <bug-gawk@gnu.org>. + + This is for two reasons. First, although some distributions + forward bug reports "upstream" to the GNU mailing list, many don't, + so there is a good chance that the 'gawk' maintainers won't even + see the bug report! Second, mail to the GNU list is archived, and + having everything at the GNU Project keeps things self-contained + and not dependent on other organizations. + + Non-bug suggestions are always welcome as well. If you have +questions about things that are unclear in the documentation or are just +obscure features, ask on the bug list; we will try to help you out if we +can. + + +File: gawk.info, Node: Usenet, Next: Maintainers, Prev: Bug address, Up: Bugs + +B.4.2 Please Don't Post Bug Reports to USENET +--------------------------------------------- + + I gave up on Usenet a couple of years ago and haven't really looked + back. It's like sports talk radio--you feel smarter for not having + read it. + -- _Chet Ramey_ + + Please do _not_ try to report bugs in 'gawk' by posting to the +Usenet/Internet newsgroup 'comp.lang.awk'. Although some of the 'gawk' +developers occasionally read this newgroup, the primary 'gawk' +maintainer no longer does. Thus it's virtually guaranteed that he will +_not_ see your posting. The steps described here are the only +officially recognized way for reporting bugs. Really. + + +File: gawk.info, Node: Maintainers, Prev: Usenet, Up: Bugs + +B.4.3 Reporting Problems with Non-Unix Ports +-------------------------------------------- + +If you find bugs in one of the non-Unix ports of 'gawk', send an email +to the bug list, with a copy to the person who maintains that port. The +maintainers are named in the following list, as well as in the 'README' +file in the 'gawk' distribution. Information in the 'README' file +should be considered authoritative if it conflicts with this Info file. + + The people maintaining the various 'gawk' ports are: + +Unix and POSIX Arnold Robbins, <arnold@skeeve.com> +systems +MS-Windows with MinGW Eli Zaretskii, <eliz@gnu.org> + +OS/2 Andreas Buening, <andreas.buening@nexgo.de> + +VMS John Malmberg, <wb8tyw@qsl.net> + +z/OS (OS/390) Daniel Richard G. <skunk@iSKUNK.ORG> + Dave Pitts (Maintainer Emeritus), <dpitts@cozx.com> + + If your bug is also reproducible under Unix, send a copy of your +report to the <bug-gawk@gnu.org> email list as well. + + The DJGPP port is no longer supported; it will remain in the code +base for a while in case a volunteer wishes to take it over. If this +does not happen, then eventually code for this port will be removed. + + +File: gawk.info, Node: Other Versions, Next: Installation summary, Prev: Bugs, Up: Installation + +B.5 Other Freely Available 'awk' Implementations +================================================ + + It's kind of fun to put comments like this in your awk code: + '// Do C++ comments work? answer: yes! of course' + -- _Michael Brennan_ + + There are a number of other freely available 'awk' implementations. +This minor node briefly describes where to get them: + +Unix 'awk' + Brian Kernighan, one of the original designers of Unix 'awk', has + made his implementation of 'awk' freely available. You can + retrieve this version via his home page + (http://www.cs.princeton.edu/~bwk). It is available in several + archive formats: + + Shell archive + <http://www.cs.princeton.edu/~bwk/btl.mirror/awk.shar> + + Compressed 'tar' file + <http://www.cs.princeton.edu/~bwk/btl.mirror/awk.tar.gz> + + Zip file + <http://www.cs.princeton.edu/~bwk/btl.mirror/awk.zip> + + You can also retrieve it from GitHub: + + git clone git://github.com/onetrueawk/awk bwkawk + + This command creates a copy of the Git (http://git-scm.com) + repository in a directory named 'bwkawk'. If you leave that + argument off the 'git' command line, the repository copy is created + in a directory named 'awk'. + + This version requires an ISO C (1990 standard) compiler; the C + compiler from GCC (the GNU Compiler Collection) works quite nicely. + + *Note Common Extensions:: for a list of extensions in this 'awk' + that are not in POSIX 'awk'. + + As a side note, Dan Bornstein has created a Git repository tracking + all the versions of BWK 'awk' that he could find. It's available + at <git://github.com/danfuzz/one-true-awk>. + +'mawk' + Michael Brennan wrote an independent implementation of 'awk', + called 'mawk'. It is available under the GPL (*note Copying::), + just as 'gawk' is. + + The original distribution site for the 'mawk' source code no longer + has it. A copy is available at + <http://www.skeeve.com/gawk/mawk1.3.3.tar.gz>. + + In 2009, Thomas Dickey took on 'mawk' maintenance. Basic + information is available on the project's web page + (http://www.invisible-island.net/mawk). The download URL is + <http://invisible-island.net/datafiles/release/mawk.tar.gz>. + + Once you have it, 'gunzip' may be used to decompress this file. + Installation is similar to 'gawk''s (*note Unix Installation::). + + *Note Common Extensions:: for a list of extensions in 'mawk' that + are not in POSIX 'awk'. + +'awka' + Written by Andrew Sumner, 'awka' translates 'awk' programs into C, + compiles them, and links them with a library of functions that + provide the core 'awk' functionality. It also has a number of + extensions. + + The 'awk' translator is released under the GPL, and the library is + under the LGPL. + + To get 'awka', go to <http://sourceforge.net/projects/awka>. + + The project seems to be frozen; no new code changes have been made + since approximately 2001. + +'pawk' + Nelson H.F. Beebe at the University of Utah has modified BWK 'awk' + to provide timing and profiling information. It is different from + 'gawk' with the '--profile' option (*note Profiling::) in that it + uses CPU-based profiling, not line-count profiling. You may find + it at either + <ftp://ftp.math.utah.edu/pub/pawk/pawk-20030606.tar.gz> or + <http://www.math.utah.edu/pub/pawk/pawk-20030606.tar.gz>. + +BusyBox 'awk' + BusyBox is a GPL-licensed program providing small versions of many + applications within a single executable. It is aimed at embedded + systems. It includes a full implementation of POSIX 'awk'. When + building it, be careful not to do 'make install' as it will + overwrite copies of other applications in your '/usr/local/bin'. + For more information, see the project's home page + (http://busybox.net). + +The OpenSolaris POSIX 'awk' + The versions of 'awk' in '/usr/xpg4/bin' and '/usr/xpg6/bin' on + Solaris are more or less POSIX-compliant. They are based on the + 'awk' from Mortice Kern Systems for PCs. We were able to make this + code compile and work under GNU/Linux with 1-2 hours of work. + Making it more generally portable (using GNU Autoconf and/or + Automake) would take more work, and this has not been done, at + least to our knowledge. + + The source code used to be available from the OpenSolaris website. + However, that project was ended and the website shut down. + Fortunately, the Illumos project + (http://wiki.illumos.org/display/illumos/illumos+Home) makes this + implementation available. You can view the files one at a time + from + <https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/awk_xpg4>. + +'jawk' + This is an interpreter for 'awk' written in Java. It claims to be + a full interpreter, although because it uses Java facilities for + I/O and for regexp matching, the language it supports is different + from POSIX 'awk'. More information is available on the project's + home page (http://jawk.sourceforge.net). + +Libmawk + This is an embeddable 'awk' interpreter derived from 'mawk'. For + more information, see <http://repo.hu/projects/libmawk/>. + +'pawk' + This is a Python module that claims to bring 'awk'-like features to + Python. See <https://github.com/alecthomas/pawk> for more + information. (This is not related to Nelson Beebe's modified + version of BWK 'awk', described earlier.) + +QSE 'awk' + This is an embeddable 'awk' interpreter. For more information, see + <http://code.google.com/p/qse/> and <http://awk.info/?tools/qse>. + +'QTawk' + This is an independent implementation of 'awk' distributed under + the GPL. It has a large number of extensions over standard 'awk' + and may not be 100% syntactically compatible with it. See + <http://www.quiktrim.org/QTawk.html> for more information, + including the manual. The download link there is out of date; see + <http://www.quiktrim.org/#AdditionalResources> for the latest + download link. + + The project may also be frozen; no new code changes have been made + since approximately 2014. + +Other versions + See also the "Versions and implementations" section of the + Wikipedia article + (http://en.wikipedia.org/wiki/Awk_language#Versions_and_implementations) + on 'awk' for information on additional versions. + + +File: gawk.info, Node: Installation summary, Prev: Other Versions, Up: Installation + +B.6 Summary +=========== + + * The 'gawk' distribution is available from the GNU Project's main + distribution site, 'ftp.gnu.org'. The canonical build recipe is: + + wget http://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.gz + tar -xvpzf gawk-4.1.4.tar.gz + cd gawk-4.1.4 + ./configure && make && make check + + * 'gawk' may be built on non-POSIX systems as well. The currently + supported systems are MS-Windows using MSYS, MinGW, and Cygwin, and + both Vax/VMS and OpenVMS. Instructions for each system are included + in this major node. + + * Bug reports should be sent via email to <bug-gawk@gnu.org>. Bug + reports should be in English and should include the version of + 'gawk', how it was compiled, and a short program and data file that + demonstrate the problem. + + * There are a number of other freely available 'awk' implementations. + Many are POSIX-compliant; others are less so. + + +File: gawk.info, Node: Notes, Next: Basic Concepts, Prev: Installation, Up: Top + +Appendix C Implementation Notes +******************************* + +This appendix contains information mainly of interest to implementers +and maintainers of 'gawk'. Everything in it applies specifically to +'gawk' and not to other implementations. + +* Menu: + +* Compatibility Mode:: How to disable certain 'gawk' + extensions. +* Additions:: Making Additions To 'gawk'. +* Future Extensions:: New features that may be implemented one day. +* Implementation Limitations:: Some limitations of the implementation. +* Extension Design:: Design notes about the extension API. +* Old Extension Mechanism:: Some compatibility for old extensions. +* Notes summary:: Summary of implementation notes. + + +File: gawk.info, Node: Compatibility Mode, Next: Additions, Up: Notes + +C.1 Downward Compatibility and Debugging +======================================== + +*Note POSIX/GNU::, for a summary of the GNU extensions to the 'awk' +language and program. All of these features can be turned off by +invoking 'gawk' with the '--traditional' option or with the '--posix' +option. + + If 'gawk' is compiled for debugging with '-DDEBUG', then there is one +more option available on the command line: + +'-Y' +'--parsedebug' + Print out the parse stack information as the program is being + parsed. + + This option is intended only for serious 'gawk' developers and not +for the casual user. It probably has not even been compiled into your +version of 'gawk', since it slows down execution. + + +File: gawk.info, Node: Additions, Next: Future Extensions, Prev: Compatibility Mode, Up: Notes + +C.2 Making Additions to 'gawk' +============================== + +If you find that you want to enhance 'gawk' in a significant fashion, +you are perfectly free to do so. That is the point of having free +software; the source code is available and you are free to change it as +you want (*note Copying::). + + This minor node discusses the ways you might want to change 'gawk' as +well as any considerations you should bear in mind. + +* Menu: + +* Accessing The Source:: Accessing the Git repository. +* Adding Code:: Adding code to the main body of + 'gawk'. +* New Ports:: Porting 'gawk' to a new operating + system. +* Derived Files:: Why derived files are kept in the Git + repository. + + +File: gawk.info, Node: Accessing The Source, Next: Adding Code, Up: Additions + +C.2.1 Accessing The 'gawk' Git Repository +----------------------------------------- + +As 'gawk' is Free Software, the source code is always available. *note +Gawk Distribution:: describes how to get and build the formal, released +versions of 'gawk'. + + However, if you want to modify 'gawk' and contribute back your +changes, you will probably wish to work with the development version. +To do so, you will need to access the 'gawk' source code repository. +The code is maintained using the Git distributed version control system +(http://git-scm.com). You will need to install it if your system +doesn't have it. Once you have done so, use the command: + + git clone git://git.savannah.gnu.org/gawk.git + +This clones the 'gawk' repository. If you are behind a firewall that +does not allow you to use the Git native protocol, you can still access +the repository using: + + git clone http://git.savannah.gnu.org/r/gawk.git + + Once you have made changes, you can use 'git diff' to produce a +patch, and send that to the 'gawk' maintainer; see *note Bugs::, for how +to do that. + + Once upon a time there was Git-CVS gateway for use by people who +could not install Git. However, this gateway no longer works, so you +may have better luck using a more modern version control system like +Bazaar, that has a Git plug-in for working with Git repositories. + + +File: gawk.info, Node: Adding Code, Next: New Ports, Prev: Accessing The Source, Up: Additions + +C.2.2 Adding New Features +------------------------- + +You are free to add any new features you like to 'gawk'. However, if +you want your changes to be incorporated into the 'gawk' distribution, +there are several steps that you need to take in order to make it +possible to include them: + + 1. Before building the new feature into 'gawk' itself, consider + writing it as an extension (*note Dynamic Extensions::). If that's + not possible, continue with the rest of the steps in this list. + + 2. Be prepared to sign the appropriate paperwork. In order for the + FSF to distribute your changes, you must either place those changes + in the public domain and submit a signed statement to that effect, + or assign the copyright in your changes to the FSF. Both of these + actions are easy to do and _many_ people have done so already. If + you have questions, please contact me (*note Bugs::), or + <assign@gnu.org>. + + 3. Get the latest version. It is much easier for me to integrate + changes if they are relative to the most recent distributed version + of 'gawk', or better yet, relative to the latest code in the Git + repository. If your version of 'gawk' is very old, I may not be + able to integrate your changes at all. (*Note Getting::, for + information on getting the latest version of 'gawk'.) + + 4. See *note (Version, standards, GNU Coding Standards)Top::. This + document describes how GNU software should be written. If you + haven't read it, please do so, preferably _before_ starting to + modify 'gawk'. (The 'GNU Coding Standards' are available from the + GNU Project's website (http://www.gnu.org/prep/standards/). + Texinfo, Info, and DVI versions are also available.) + + 5. Use the 'gawk' coding style. The C code for 'gawk' follows the + instructions in the 'GNU Coding Standards', with minor exceptions. + The code is formatted using the traditional "K&R" style, + particularly as regards to the placement of braces and the use of + TABs. In brief, the coding rules for 'gawk' are as follows: + + * Use ANSI/ISO style (prototype) function headers when defining + functions. + + * Put the name of the function at the beginning of its own line. + + * Use '#elif' instead of nesting '#if' inside '#else'. + + * Put the return type of the function, even if it is 'int', on + the line above the line with the name and arguments of the + function. + + * Put spaces around parentheses used in control structures + ('if', 'while', 'for', 'do', 'switch', and 'return'). + + * Do not put spaces in front of parentheses used in function + calls. + + * Put spaces around all C operators and after commas in function + calls. + + * Do not use the comma operator to produce multiple side + effects, except in 'for' loop initialization and increment + parts, and in macro bodies. + + * Use real TABs for indenting, not spaces. + + * Use the "K&R" brace layout style. + + * Use comparisons against 'NULL' and ''\0'' in the conditions of + 'if', 'while', and 'for' statements, as well as in the 'case's + of 'switch' statements, instead of just the plain pointer or + character value. + + * Use 'true' and 'false' for 'bool' values, the 'NULL' symbolic + constant for pointer values, and the character constant ''\0'' + where appropriate, instead of '1' and '0'. + + * Provide one-line descriptive comments for each function. + + * Do not use the 'alloca()' function for allocating memory off + the stack. Its use causes more portability trouble than is + worth the minor benefit of not having to free the storage. + Instead, use 'malloc()' and 'free()'. + + * Do not use comparisons of the form '! strcmp(a, b)' or + similar. As Henry Spencer once said, "'strcmp()' is not a + boolean!" Instead, use 'strcmp(a, b) == 0'. + + * If adding new bit flag values, use explicit hexadecimal + constants ('0x001', '0x002', '0x004', and son on) instead of + shifting one left by successive amounts ('(1<<0)', '(1<<1)', + and so on). + + NOTE: If I have to reformat your code to follow the coding + style used in 'gawk', I may not bother to integrate your + changes at all. + + 6. Update the documentation. Along with your new code, please supply + new sections and/or chapters for this Info file. If at all + possible, please use real Texinfo, instead of just supplying + unformatted ASCII text (although even that is better than no + documentation at all). Conventions to be followed in 'GAWK: + Effective AWK Programming' are provided after the '@bye' at the end + of the Texinfo source file. If possible, please update the 'man' + page as well. + + You will also have to sign paperwork for your documentation + changes. + + 7. Submit changes as unified diffs. Use 'diff -u -r -N' to compare + the original 'gawk' source tree with your version. I recommend + using the GNU version of 'diff', or best of all, 'git diff' or 'git + format-patch'. Send the output produced by 'diff' to me when you + submit your changes. (*Note Bugs::, for the electronic mail + information.) + + Using this format makes it easy for me to apply your changes to the + master version of the 'gawk' source code (using 'patch'). If I + have to apply the changes manually, using a text editor, I may not + do so, particularly if there are lots of changes. + + 8. Include an entry for the 'ChangeLog' file with your submission. + This helps further minimize the amount of work I have to do, making + it easier for me to accept patches. It is simplest if you just + make this part of your diff. + + Although this sounds like a lot of work, please remember that while +you may write the new code, I have to maintain it and support it. If it +isn't possible for me to do that with a minimum of extra work, then I +probably will not. + + +File: gawk.info, Node: New Ports, Next: Derived Files, Prev: Adding Code, Up: Additions + +C.2.3 Porting 'gawk' to a New Operating System +---------------------------------------------- + +If you want to port 'gawk' to a new operating system, there are several +steps: + + 1. Follow the guidelines in *note Adding Code::, concerning coding + style, submission of diffs, and so on. + + 2. Be prepared to sign the appropriate paperwork. In order for the + FSF to distribute your code, you must either place your code in the + public domain and submit a signed statement to that effect, or + assign the copyright in your code to the FSF. Both of these actions + are easy to do and _many_ people have done so already. If you have + questions, please contact me, or <gnu@gnu.org>. + + 3. When doing a port, bear in mind that your code must coexist + peacefully with the rest of 'gawk' and the other ports. Avoid + gratuitous changes to the system-independent parts of the code. If + at all possible, avoid sprinkling '#ifdef's just for your port + throughout the code. + + If the changes needed for a particular system affect too much of + the code, I probably will not accept them. In such a case, you + can, of course, distribute your changes on your own, as long as you + comply with the GPL (*note Copying::). + + 4. A number of the files that come with 'gawk' are maintained by other + people. Thus, you should not change them unless it is for a very + good reason; i.e., changes are not out of the question, but changes + to these files are scrutinized extra carefully. The files are + 'dfa.c', 'dfa.h', 'getopt.c', 'getopt.h', 'getopt1.c', + 'getopt_int.h', 'gettext.h', 'regcomp.c', 'regex.c', 'regex.h', + 'regex_internal.c', 'regex_internal.h', and 'regexec.c'. + + 5. A number of other files are provided by the GNU Autotools + (Autoconf, Automake, and GNU 'gettext'). You should not change + them either, unless it is for a very good reason. The files are + 'ABOUT-NLS', 'config.guess', 'config.rpath', 'config.sub', + 'depcomp', 'INSTALL', 'install-sh', 'missing', 'mkinstalldirs', + 'xalloc.h', and 'ylwrap'. + + 6. Be willing to continue to maintain the port. Non-Unix operating + systems are supported by volunteers who maintain the code needed to + compile and run 'gawk' on their systems. If no-one volunteers to + maintain a port, it becomes unsupported and it may be necessary to + remove it from the distribution. + + 7. Supply an appropriate 'gawkmisc.???' file. Each port has its own + 'gawkmisc.???' that implements certain operating system specific + functions. This is cleaner than a plethora of '#ifdef's scattered + throughout the code. The 'gawkmisc.c' in the main source directory + includes the appropriate 'gawkmisc.???' file from each + subdirectory. Be sure to update it as well. + + Each port's 'gawkmisc.???' file has a suffix reminiscent of the + machine or operating system for the port--for example, + 'pc/gawkmisc.pc' and 'vms/gawkmisc.vms'. The use of separate + suffixes, instead of plain 'gawkmisc.c', makes it possible to move + files from a port's subdirectory into the main subdirectory, + without accidentally destroying the real 'gawkmisc.c' file. + (Currently, this is only an issue for the PC operating system + ports.) + + 8. Supply a 'Makefile' as well as any other C source and header files + that are necessary for your operating system. All your code should + be in a separate subdirectory, with a name that is the same as, or + reminiscent of, either your operating system or the computer + system. If possible, try to structure things so that it is not + necessary to move files out of the subdirectory into the main + source directory. If that is not possible, then be sure to avoid + using names for your files that duplicate the names of files in the + main source directory. + + 9. Update the documentation. Please write a section (or sections) for + this Info file describing the installation and compilation steps + needed to compile and/or install 'gawk' for your system. + + Following these steps makes it much easier to integrate your changes +into 'gawk' and have them coexist happily with other operating systems' +code that is already there. + + In the code that you supply and maintain, feel free to use a coding +style and brace layout that suits your taste. + + +File: gawk.info, Node: Derived Files, Prev: New Ports, Up: Additions + +C.2.4 Why Generated Files Are Kept In Git +----------------------------------------- + +If you look at the 'gawk' source in the Git repository, you will notice +that it includes files that are automatically generated by GNU +infrastructure tools, such as 'Makefile.in' from Automake and even +'configure' from Autoconf. + + This is different from many Free Software projects that do not store +the derived files, because that keeps the repository less cluttered, and +it is easier to see the substantive changes when comparing versions and +trying to understand what changed between commits. + + However, there are several reasons why the 'gawk' maintainer likes to +have everything in the repository. + + First, because it is then easy to reproduce any given version +completely, without relying upon the availability of (older, likely +obsolete, and maybe even impossible to find) other tools. + + As an extreme example, if you ever even think about trying to +compile, oh, say, the V7 'awk', you will discover that not only do you +have to bootstrap the V7 'yacc' to do so, but you also need the V7 +'lex'. And the latter is pretty much impossible to bring up on a modern +GNU/Linux system.(1) + + (Or, let's say 'gawk' 1.2 required 'bison' whatever-it-was in 1989 +and that there was no 'awkgram.c' file in the repository. Is there a +guarantee that we could find that 'bison' version? Or that _it_ would +build?) + + If the repository has all the generated files, then it's easy to just +check them out and build. (Or _easier_, depending upon how far back we +go.) + + And that brings us to the second (and stronger) reason why all the +files really need to be in Git. It boils down to who do you cater +to--the 'gawk' developer(s), or the user who just wants to check out a +version and try it out? + + The 'gawk' maintainer wants it to be possible for any interested +'awk' user in the world to just clone the repository, check out the +branch of interest and build it. Without their having to have the +correct version(s) of the autotools.(2) That is the point of the +'bootstrap.sh' file. It touches the various other files in the right +order such that + + # The canonical incantation for building GNU software: + ./bootstrap.sh && ./configure && make + +will _just work_. + + This is extremely important for the 'master' and 'gawk-X.Y-stable' +branches. + + Further, the 'gawk' maintainer would argue that it's also important +for the 'gawk' developers. When he tried to check out the 'xgawk' +branch(3) to build it, he couldn't. (No 'ltmain.sh' file, and he had no +idea how to create it, and that was not the only problem.) + + He felt _extremely_ frustrated. With respect to that branch, the +maintainer is no different than Jane User who wants to try to build +'gawk-4.1-stable' or 'master' from the repository. + + Thus, the maintainer thinks that it's not just important, but +critical, that for any given branch, the above incantation _just works_. + + A third reason to have all the files is that without them, using 'git +bisect' to try to find the commit that introduced a bug is exceedingly +difficult. The maintainer tried to do that on another project that +requires running bootstrapping scripts just to create 'configure' and so +on; it was really painful. When the repository is self-contained, using +'git bisect' in it is very easy. + + What are some of the consequences and/or actions to take? + + 1. We don't mind that there are differing files in the different + branches as a result of different versions of the autotools. + + A. It's the maintainer's job to merge them and he will deal with + it. + + B. He is really good at 'git diff x y > /tmp/diff1 ; gvim + /tmp/diff1' to remove the diffs that aren't of interest in + order to review code. + + 2. It would certainly help if everyone used the same versions of the + GNU tools as he does, which in general are the latest released + versions of Automake, Autoconf, 'bison', and GNU 'gettext'. + + Installing from source is quite easy. It's how the maintainer + worked for years (and still works). He had '/usr/local/bin' at the + front of his 'PATH' and just did: + + wget http://ftp.gnu.org/gnu/PACKAGE/PACKAGE-X.Y.Z.tar.gz + tar -xpzvf PACKAGE-X.Y.Z.tar.gz + cd PACKAGE-X.Y.Z + ./configure && make && make check + make install # as root + + Most of the above was originally written by the maintainer to other +'gawk' developers. It raised the objection from one of the developers +"... that anybody pulling down the source from Git is not an end user." + + However, this is not true. There are "power 'awk' users" who can +build 'gawk' (using the magic incantation shown previously) but who +can't program in C. Thus, the major branches should be kept buildable +all the time. + + It was then suggested that there be a 'cron' job to create nightly +tarballs of "the source." Here, the problem is that there are source +trees, corresponding to the various branches! So, nightly tarballs +aren't the answer, especially as the repository can go for weeks without +significant change being introduced. + + Fortunately, the Git server can meet this need. For any given branch +named BRANCHNAME, use: + + wget http://git.savannah.gnu.org/cgit/gawk.git/snapshot/gawk-BRANCHNAME.tar.gz + +to retrieve a snapshot of the given branch. + + ---------- Footnotes ---------- + + (1) We tried. It was painful. + + (2) There is one GNU program that is (in our opinion) severely +difficult to bootstrap from the Git repository. For example, on the +author's old (but still working) PowerPC Macintosh with Mac OS X 10.5, +it was necessary to bootstrap a ton of software, starting with Git +itself, in order to try to work with the latest code. It's not +pleasant, and especially on older systems, it's a big waste of time. + + Starting with the latest tarball was no picnic either. The +maintainers had dropped '.gz' and '.bz2' files and only distribute +'.tar.xz' files. It was necessary to bootstrap 'xz' first! + + (3) A branch (since removed) created by one of the other developers +that did not include the generated files. + + +File: gawk.info, Node: Future Extensions, Next: Implementation Limitations, Prev: Additions, Up: Notes + +C.3 Probable Future Extensions +============================== + + AWK is a language similar to PERL, only considerably more elegant. + -- _Arnold Robbins_ + + Hey! + -- _Larry Wall_ + + The 'TODO' file in the 'master' branch of the 'gawk' Git repository +lists possible future enhancements. Some of these relate to the source +code, and others to possible new features. Please see that file for the +list. *Note Additions::, if you are interested in tackling any of the +projects listed there. + + +File: gawk.info, Node: Implementation Limitations, Next: Extension Design, Prev: Future Extensions, Up: Notes + +C.4 Some Limitations of the Implementation +========================================== + +This following table describes limits of 'gawk' on a Unix-like system +(although it is variable even then). Other systems may have different +limits. + +Item Limit +-------------------------------------------------------------------------- +Characters in a character 2^(number of bits per byte) +class +Length of input record 'MAX_INT' +Length of output record Unlimited +Length of source line Unlimited +Number of fields in a 'MAX_LONG' +record +Number of file redirections Unlimited +Number of input records in 'MAX_LONG' +one file +Number of input records 'MAX_LONG' +total +Number of pipe redirections min(number of processes per user, number + of open files) +Numeric values Double-precision floating point (if not + using MPFR) +Size of a field 'MAX_INT' +Size of a literal string 'MAX_INT' +Size of a printf string 'MAX_INT' + + +File: gawk.info, Node: Extension Design, Next: Old Extension Mechanism, Prev: Implementation Limitations, Up: Notes + +C.5 Extension API Design +======================== + +This minor node documents the design of the extension API, including a +discussion of some of the history and problems that needed to be solved. + + The first version of extensions for 'gawk' was developed in the +mid-1990s and released with 'gawk' 3.1 in the late 1990s. The basic +mechanisms and design remained unchanged for close to 15 years, until +2012. + + The old extension mechanism used data types and functions from 'gawk' +itself, with a "clever hack" to install extension functions. + + 'gawk' included some sample extensions, of which a few were really +useful. However, it was clear from the outset that the extension +mechanism was bolted onto the side and was not really well thought out. + +* Menu: + +* Old Extension Problems:: Problems with the old mechanism. +* Extension New Mechanism Goals:: Goals for the new mechanism. +* Extension Other Design Decisions:: Some other design decisions. +* Extension Future Growth:: Some room for future growth. + + +File: gawk.info, Node: Old Extension Problems, Next: Extension New Mechanism Goals, Up: Extension Design + +C.5.1 Problems With The Old Mechanism +------------------------------------- + +The old extension mechanism had several problems: + + * It depended heavily upon 'gawk' internals. Any time the 'NODE' + structure(1) changed, an extension would have to be recompiled. + Furthermore, to really write extensions required understanding + something about 'gawk''s internal functions. There was some + documentation in this Info file, but it was quite minimal. + + * Being able to call into 'gawk' from an extension required linker + facilities that are common on Unix-derived systems but that did not + work on MS-Windows systems; users wanting extensions on MS-Windows + had to statically link them into 'gawk', even though MS-Windows + supports dynamic loading of shared objects. + + * The API would change occasionally as 'gawk' changed; no + compatibility between versions was ever offered or planned for. + + Despite the drawbacks, the 'xgawk' project developers forked 'gawk' +and developed several significant extensions. They also enhanced +'gawk''s facilities relating to file inclusion and shared object access. + + A new API was desired for a long time, but only in 2012 did the +'gawk' maintainer and the 'xgawk' developers finally start working on it +together. More information about the 'xgawk' project is provided in +*note gawkextlib::. + + ---------- Footnotes ---------- + + (1) A critical central data structure inside 'gawk'. + + +File: gawk.info, Node: Extension New Mechanism Goals, Next: Extension Other Design Decisions, Prev: Old Extension Problems, Up: Extension Design + +C.5.2 Goals For A New Mechanism +------------------------------- + +Some goals for the new API were: + + * The API should be independent of 'gawk' internals. Changes in + 'gawk' internals should not be visible to the writer of an + extension function. + + * The API should provide _binary_ compatibility across 'gawk' + releases as long as the API itself does not change. + + * The API should enable extensions written in C or C++ to have + roughly the same "appearance" to 'awk'-level code as 'awk' + functions do. This means that extensions should have: + + - The ability to access function parameters. + + - The ability to turn an undefined parameter into an array (call + by reference). + + - The ability to create, access and update global variables. + + - Easy access to all the elements of an array at once ("array + flattening") in order to loop over all the element in an easy + fashion for C code. + + - The ability to create arrays (including 'gawk''s true arrays + of arrays). + + Some additional important goals were: + + * The API should use only features in ISO C 90, so that extensions + can be written using the widest range of C and C++ compilers. The + header should include the appropriate '#ifdef __cplusplus' and + 'extern "C"' magic so that a C++ compiler could be used. (If using + C++, the runtime system has to be smart enough to call any + constructors and destructors, as 'gawk' is a C program. As of this + writing, this has not been tested.) + + * The API mechanism should not require access to 'gawk''s symbols(1) + by the compile-time or dynamic linker, in order to enable creation + of extensions that also work on MS-Windows. + + During development, it became clear that there were other features +that should be available to extensions, which were also subsequently +provided: + + * Extensions should have the ability to hook into 'gawk''s I/O + redirection mechanism. In particular, the 'xgawk' developers + provided a so-called "open hook" to take over reading records. + During development, this was generalized to allow extensions to + hook into input processing, output processing, and two-way I/O. + + * An extension should be able to provide a "call back" function to + perform cleanup actions when 'gawk' exits. + + * An extension should be able to provide a version string so that + 'gawk''s '--version' option can provide information about + extensions as well. + + The requirement to avoid access to 'gawk''s symbols is, at first +glance, a difficult one to meet. + + One design, apparently used by Perl and Ruby and maybe others, would +be to make the mainline 'gawk' code into a library, with the 'gawk' +utility a small C 'main()' function linked against the library. + + This seemed like the tail wagging the dog, complicating build and +installation and making a simple copy of the 'gawk' executable from one +system to another (or one place to another on the same system!) into a +chancy operation. + + Pat Rankin suggested the solution that was adopted. *Note Extension +Mechanism Outline::, for the details. + + ---------- Footnotes ---------- + + (1) The "symbols" are the variables and functions defined inside +'gawk'. Access to these symbols by code external to 'gawk' loaded +dynamically at runtime is problematic on MS-Windows. + + +File: gawk.info, Node: Extension Other Design Decisions, Next: Extension Future Growth, Prev: Extension New Mechanism Goals, Up: Extension Design + +C.5.3 Other Design Decisions +---------------------------- + +As an arbitrary design decision, extensions can read the values of +predefined variables and arrays (such as 'ARGV' and 'FS'), but cannot +change them, with the exception of 'PROCINFO'. + + The reason for this is to prevent an extension function from +affecting the flow of an 'awk' program outside its control. While a +real 'awk' function can do what it likes, that is at the discretion of +the programmer. An extension function should provide a service or make +a C API available for use within 'awk', and not mess with 'FS' or 'ARGC' +and 'ARGV'. + + In addition, it becomes easy to start down a slippery slope. How +much access to 'gawk' facilities do extensions need? Do they need +'getline'? What about calling 'gsub()' or compiling regular +expressions? What about calling into 'awk' functions? (_That_ would be +messy.) + + In order to avoid these issues, the 'gawk' developers chose to start +with the simplest, most basic features that are still truly useful. + + Another decision is that although 'gawk' provides nice things like +MPFR, and arrays indexed internally by integers, these features are not +being brought out to the API in order to keep things simple and close to +traditional 'awk' semantics. (In fact, arrays indexed internally by +integers are so transparent that they aren't even documented!) + + Additionally, all functions in the API check that their pointer input +parameters are not 'NULL'. If they are, they return an error. (It is a +good idea for extension code to verify that pointers received from +'gawk' are not 'NULL'. Such a thing should not happen, but the 'gawk' +developers are only human, and they have been known to occasionally make +mistakes.) + + With time, the API will undoubtedly evolve; the 'gawk' developers +expect this to be driven by user needs. For now, the current API seems +to provide a minimal yet powerful set of features for creating +extensions. + + +File: gawk.info, Node: Extension Future Growth, Prev: Extension Other Design Decisions, Up: Extension Design + +C.5.4 Room For Future Growth +---------------------------- + +The API can later be expanded, in two ways: + + * 'gawk' passes an "extension id" into the extension when it first + loads the extension. The extension then passes this id back to + 'gawk' with each function call. This mechanism allows 'gawk' to + identify the extension calling into it, should it need to know. + + * Similarly, the extension passes a "name space" into 'gawk' when it + registers each extension function. This accommodates a possible + future mechanism for grouping extension functions and possibly + avoiding name conflicts. + + Of course, as of this writing, no decisions have been made with +respect to any of the above. + + +File: gawk.info, Node: Old Extension Mechanism, Next: Notes summary, Prev: Extension Design, Up: Notes + +C.6 Compatibility For Old Extensions +==================================== + +*note Dynamic Extensions::, describes the supported API and mechanisms +for writing extensions for 'gawk'. This API was introduced in version +4.1. However, for many years 'gawk' provided an extension mechanism +that required knowledge of 'gawk' internals and that was not as well +designed. + + In order to provide a transition period, 'gawk' version 4.1 continues +to support the original extension mechanism. This will be true for the +life of exactly one major release. This support will be withdrawn, and +removed from the source code, at the next major release. + + Briefly, original-style extensions should be compiled by including +the 'awk.h' header file in the extension source code. Additionally, you +must define the identifier 'GAWK' when building (use '-DGAWK' with +Unix-style compilers). Otherwise, the definitions in 'gawkapi.h' will +cause conflicts with those in 'awk.h' and your extension will not +compile. + + Just as in previous versions, you load an old-style extension with +the 'extension()' built-in function (which is not otherwise documented). +This function in turn finds and loads the shared object file containing +the extension and calls its 'dl_load()' C routine. + + Because original-style and new-style extensions use different +initialization routines ('dl_load()' versus 'dlload()'), they may safely +be installed in the same directory (to be found by 'AWKLIBPATH') without +conflict. + + The 'gawk' development team strongly recommends that you convert any +old extensions that you may have to use the new API described in *note +Dynamic Extensions::. + + +File: gawk.info, Node: Notes summary, Prev: Old Extension Mechanism, Up: Notes + +C.7 Summary +=========== + + * 'gawk''s extensions can be disabled with either the '--traditional' + option or with the '--posix' option. The '--parsedebug' option is + available if 'gawk' is compiled with '-DDEBUG'. + + * The source code for 'gawk' is maintained in a publicly accessible + Git repository. Anyone may check it out and view the source. + + * Contributions to 'gawk' are welcome. Following the steps outlined + in this major node will make it easier to integrate your + contributions into the code base. This applies both to new feature + contributions and to ports to additional operating systems. + + * 'gawk' has some limits--generally those that are imposed by the + machine architecture. + + * The extension API design was intended to solve a number of problems + with the previous extension mechanism, enable features needed by + the 'xgawk' project, and provide binary compatibility going + forward. + + * The previous extension mechanism is still supported in version 4.1 + of 'gawk', but it _will_ be removed in the next major release. + + +File: gawk.info, Node: Basic Concepts, Next: Glossary, Prev: Notes, Up: Top + +Appendix D Basic Programming Concepts +************************************* + +This major node attempts to define some of the basic concepts and terms +that are used throughout the rest of this Info file. As this Info file +is specifically about 'awk', and not about computer programming in +general, the coverage here is by necessity fairly cursory and +simplistic. (If you need more background, there are many other +introductory texts that you should refer to instead.) + +* Menu: + +* Basic High Level:: The high level view. +* Basic Data Typing:: A very quick intro to data types. + + +File: gawk.info, Node: Basic High Level, Next: Basic Data Typing, Up: Basic Concepts + +D.1 What a Program Does +======================= + +At the most basic level, the job of a program is to process some input +data and produce results. See *note Figure D.1: figure-general-flow. + + ++------+ / \\ +---------+ +| Data | -----> < Program > -----> | Results | ++------+ \\_______/ +---------+" + +Figure D.1: General Program Flow + + The "program" in the figure can be either a compiled program(1) (such +as 'ls'), or it may be "interpreted". In the latter case, a +machine-executable program such as 'awk' reads your program, and then +uses the instructions in your program to process the data. + + When you write a program, it usually consists of the following, very +basic set of steps, as shown in *note Figure D.2: figure-process-flow.: + + ++----------------+ / More \\ No +----------+ +| Initialization | -------> < Data > -------> | Clean Up | ++----------------+ ^ \\ ? / +----------+ + | +--+-+ + | | Yes + | | + | V + | +---------+ + +-----+ Process | + +---------+" + +Figure D.2: Basic Program Steps + +Initialization + These are the things you do before actually starting to process + data, such as checking arguments, initializing any data you need to + work with, and so on. This step corresponds to 'awk''s 'BEGIN' + rule (*note BEGIN/END::). + + If you were baking a cake, this might consist of laying out all the + mixing bowls and the baking pan, and making sure you have all the + ingredients that you need. + +Processing + This is where the actual work is done. Your program reads data, + one logical chunk at a time, and processes it as appropriate. + + In most programming languages, you have to manually manage the + reading of data, checking to see if there is more each time you + read a chunk. 'awk''s pattern-action paradigm (*note Getting + Started::) handles the mechanics of this for you. + + In baking a cake, the processing corresponds to the actual labor: + breaking eggs, mixing the flour, water, and other ingredients, and + then putting the cake into the oven. + +Clean Up + Once you've processed all the data, you may have things you need to + do before exiting. This step corresponds to 'awk''s 'END' rule + (*note BEGIN/END::). + + After the cake comes out of the oven, you still have to wrap it in + plastic wrap to keep anyone from tasting it, as well as wash the + mixing bowls and utensils. + + An "algorithm" is a detailed set of instructions necessary to +accomplish a task, or process data. It is much the same as a recipe for +baking a cake. Programs implement algorithms. Often, it is up to you +to design the algorithm and implement it, simultaneously. + + The "logical chunks" we talked about previously are called "records", +similar to the records a company keeps on employees, a school keeps for +students, or a doctor keeps for patients. Each record has many +component parts, such as first and last names, date of birth, address, +and so on. The component parts are referred to as the "fields" of the +record. + + The act of reading data is termed "input", and that of generating +results, not too surprisingly, is termed "output". They are often +referred to together as "input/output," and even more often, as "I/O" +for short. (You will also see "input" and "output" used as verbs.) + + 'awk' manages the reading of data for you, as well as the breaking it +up into records and fields. Your program's job is to tell 'awk' what to +do with the data. You do this by describing "patterns" in the data to +look for, and "actions" to execute when those patterns are seen. This +"data-driven" nature of 'awk' programs usually makes them both easier to +write and easier to read. + + ---------- Footnotes ---------- + + (1) Compiled programs are typically written in lower-level languages +such as C, C++, or Ada, and then translated, or "compiled", into a form +that the computer can execute directly. + + +File: gawk.info, Node: Basic Data Typing, Prev: Basic High Level, Up: Basic Concepts + +D.2 Data Values in a Computer +============================= + +In a program, you keep track of information and values in things called +"variables". A variable is just a name for a given value, such as +'first_name', 'last_name', 'address', and so on. 'awk' has several +predefined variables, and it has special names to refer to the current +input record and the fields of the record. You may also group multiple +associated values under one name, as an array. + + Data, particularly in 'awk', consists of either numeric values, such +as 42 or 3.1415927, or string values. String values are essentially +anything that's not a number, such as a name. Strings are sometimes +referred to as "character data", since they store the individual +characters that comprise them. Individual variables, as well as numeric +and string variables, are referred to as "scalar" values. Groups of +values, such as arrays, are not scalars. + + *note Computer Arithmetic::, provided a basic introduction to numeric +types (integer and floating-point) and how they are used in a computer. +Please review that information, including a number of caveats that were +presented. + + While you are probably used to the idea of a number without a value +(i.e., zero), it takes a bit more getting used to the idea of +zero-length character data. Nevertheless, such a thing exists. It is +called the "null string". The null string is character data that has no +value. In other words, it is empty. It is written in 'awk' programs +like this: '""'. + + Humans are used to working in decimal; i.e., base 10. In base 10, +numbers go from 0 to 9, and then "roll over" into the next column. +(Remember grade school? 42 = 4 x 10 + 2.) + + There are other number bases though. Computers commonly use base 2 +or "binary", base 8 or "octal", and base 16 or "hexadecimal". In +binary, each column represents two times the value in the column to its +right. Each column may contain either a 0 or a 1. Thus, binary 1010 +represents (1 x 8) + (0 x 4) + (1 x 2) + (0 x 1), or decimal 10. Octal +and hexadecimal are discussed more in *note Nondecimal-numbers::. + + At the very lowest level, computers store values as groups of binary +digits, or "bits". Modern computers group bits into groups of eight, +called "bytes". Advanced applications sometimes have to manipulate bits +directly, and 'gawk' provides functions for doing so. + + Programs are written in programming languages. Hundreds, if not +thousands, of programming languages exist. One of the most popular is +the C programming language. The C language had a very strong influence +on the design of the 'awk' language. + + There have been several versions of C. The first is often referred to +as "K&R" C, after the initials of Brian Kernighan and Dennis Ritchie, +the authors of the first book on C. (Dennis Ritchie created the +language, and Brian Kernighan was one of the creators of 'awk'.) + + In the mid-1980s, an effort began to produce an international +standard for C. This work culminated in 1989, with the production of the +ANSI standard for C. This standard became an ISO standard in 1990. In +1999, a revised ISO C standard was approved and released. Where it +makes sense, POSIX 'awk' is compatible with 1999 ISO C. + + +File: gawk.info, Node: Glossary, Next: Copying, Prev: Basic Concepts, Up: Top + +Glossary +******** + +Action + A series of 'awk' statements attached to a rule. If the rule's + pattern matches an input record, 'awk' executes the rule's action. + Actions are always enclosed in braces. (*Note Action Overview::.) + +Ada + A programming language originally defined by the U.S. Department of + Defense for embedded programming. It was designed to enforce good + Software Engineering practices. + +Amazing 'awk' Assembler + Henry Spencer at the University of Toronto wrote a retargetable + assembler completely as 'sed' and 'awk' scripts. It is thousands + of lines long, including machine descriptions for several eight-bit + microcomputers. It is a good example of a program that would have + been better written in another language. You can get it from + <http://awk.info/?awk100/aaa>. + +Amazingly Workable Formatter ('awf') + Henry Spencer at the University of Toronto wrote a formatter that + accepts a large subset of the 'nroff -ms' and 'nroff -man' + formatting commands, using 'awk' and 'sh'. It is available from + <http://awk.info/?tools/awf>. + +Anchor + The regexp metacharacters '^' and '$', which force the match to the + beginning or end of the string, respectively. + +ANSI + The American National Standards Institute. This organization + produces many standards, among them the standards for the C and C++ + programming languages. These standards often become international + standards as well. See also "ISO." + +Argument + An argument can be two different things. It can be an option or a + file name passed to a command while invoking it from the command + line, or it can be something passed to a "function" inside a + program, e.g. inside 'awk'. + + In the latter case, an argument can be passed to a function in two + ways. Either it is given to the called function by value, i.e., a + copy of the value of the variable is made available to the called + function, but the original variable cannot be modified by the + function itself; or it is given by reference, i.e., a pointer to + the interested variable is passed to the function, which can then + directly modify it. In 'awk' scalars are passed by value, and + arrays are passed by reference. See "Pass By Value/Reference." + +Array + A grouping of multiple values under the same name. Most languages + just provide sequential arrays. 'awk' provides associative arrays. + +Assertion + A statement in a program that a condition is true at this point in + the program. Useful for reasoning about how a program is supposed + to behave. + +Assignment + An 'awk' expression that changes the value of some 'awk' variable + or data object. An object that you can assign to is called an + "lvalue". The assigned values are called "rvalues". *Note + Assignment Ops::. + +Associative Array + Arrays in which the indices may be numbers or strings, not just + sequential integers in a fixed range. + +'awk' Language + The language in which 'awk' programs are written. + +'awk' Program + An 'awk' program consists of a series of "patterns" and "actions", + collectively known as "rules". For each input record given to the + program, the program's rules are all processed in turn. 'awk' + programs may also contain function definitions. + +'awk' Script + Another name for an 'awk' program. + +Bash + The GNU version of the standard shell (the Bourne-Again SHell). + See also "Bourne Shell." + +Binary + Base-two notation, where the digits are '0'-'1'. Since electronic + circuitry works "naturally" in base 2 (just think of Off/On), + everything inside a computer is calculated using base 2. Each + digit represents the presence (or absence) of a power of 2 and is + called a "bit". So, for example, the base-two number '10101' is + the same as decimal 21, ((1 x 16) + (1 x 4) + (1 x 1)). + + Since base-two numbers quickly become very long to read and write, + they are usually grouped by 3 (i.e., they are read as octal + numbers), or by 4 (i.e., they are read as hexadecimal numbers). + There is no direct way to insert base 2 numbers in a C program. If + need arises, such numbers are usually inserted as octal or + hexadecimal numbers. The number of base-two digits that fit into + registers used for representing integer numbers in computers is a + rough indication of the computing power of the computer itself. + Most computers nowadays use 64 bits for representing integer + numbers in their registers, but 32-bit, 16-bit and 8-bit registers + have been widely used in the past. *Note Nondecimal-numbers::. +Bit + Short for "Binary Digit." All values in computer memory ultimately + reduce to binary digits: values that are either zero or one. + Groups of bits may be interpreted differently--as integers, + floating-point numbers, character data, addresses of other memory + objects, or other data. 'awk' lets you work with floating-point + numbers and strings. 'gawk' lets you manipulate bit values with + the built-in functions described in *note Bitwise Functions::. + + Computers are often defined by how many bits they use to represent + integer values. Typical systems are 32-bit systems, but 64-bit + systems are becoming increasingly popular, and 16-bit systems have + essentially disappeared. + +Boolean Expression + Named after the English mathematician Boole. See also "Logical + Expression." + +Bourne Shell + The standard shell ('/bin/sh') on Unix and Unix-like systems, + originally written by Steven R. Bourne at Bell Laboratories. Many + shells (Bash, 'ksh', 'pdksh', 'zsh') are generally upwardly + compatible with the Bourne shell. + +Braces + The characters '{' and '}'. Braces are used in 'awk' for + delimiting actions, compound statements, and function bodies. + +Bracket Expression + Inside a "regular expression", an expression included in square + brackets, meant to designate a single character as belonging to a + specified character class. A bracket expression can contain a list + of one or more characters, like '[abc]', a range of characters, + like '[A-Z]', or a name, delimited by ':', that designates a known + set of characters, like '[:digit:]'. The form of bracket + expression enclosed between ':' is independent of the underlying + representation of the character themselves, which could utilize the + ASCII, ECBDIC, or Unicode codesets, depending on the architecture + of the computer system, and on localization. See also "Regular + Expression." + +Built-in Function + The 'awk' language provides built-in functions that perform various + numerical, I/O-related, and string computations. Examples are + 'sqrt()' (for the square root of a number) and 'substr()' (for a + substring of a string). 'gawk' provides functions for timestamp + management, bit manipulation, array sorting, type checking, and + runtime string translation. (*Note Built-in::.) + +Built-in Variable + 'ARGC', 'ARGV', 'CONVFMT', 'ENVIRON', 'FILENAME', 'FNR', 'FS', + 'NF', 'NR', 'OFMT', 'OFS', 'ORS', 'RLENGTH', 'RSTART', 'RS', and + 'SUBSEP' are the variables that have special meaning to 'awk'. In + addition, 'ARGIND', 'BINMODE', 'ERRNO', 'FIELDWIDTHS', 'FPAT', + 'IGNORECASE', 'LINT', 'PROCINFO', 'RT', and 'TEXTDOMAIN' are the + variables that have special meaning to 'gawk'. Changing some of + them affects 'awk''s running environment. (*Note Built-in + Variables::.) + +C + The system programming language that most GNU software is written + in. The 'awk' programming language has C-like syntax, and this + Info file points out similarities between 'awk' and C when + appropriate. + + In general, 'gawk' attempts to be as similar to the 1990 version of + ISO C as makes sense. + +C Shell + The C Shell ('csh' or its improved version, 'tcsh') is a Unix shell + that was created by Bill Joy in the late 1970s. The C shell was + differentiated from other shells by its interactive features and + overall style, which looks more like C. The C Shell is not backward + compatible with the Bourne Shell, so special attention is required + when converting scripts written for other Unix shells to the C + shell, especially with regard to the management of shell variables. + See also "Bourne Shell." + +C++ + A popular object-oriented programming language derived from C. + +Character Class + See "Bracket Expression." + +Character List + See "Bracket Expression." + +Character Set + The set of numeric codes used by a computer system to represent the + characters (letters, numbers, punctuation, etc.) of a particular + country or place. The most common character set in use today is + ASCII (American Standard Code for Information Interchange). Many + European countries use an extension of ASCII known as ISO-8859-1 + (ISO Latin-1). The Unicode character set (http://www.unicode.org) + is increasingly popular and standard, and is particularly widely + used on GNU/Linux systems. + +CHEM + A preprocessor for 'pic' that reads descriptions of molecules and + produces 'pic' input for drawing them. It was written in 'awk' by + Brian Kernighan and Jon Bentley, and is available from + <http://netlib.org/typesetting/chem>. + +Comparison Expression + A relation that is either true or false, such as 'a < b'. + Comparison expressions are used in 'if', 'while', 'do', and 'for' + statements, and in patterns to select which input records to + process. (*Note Typing and Comparison::.) + +Compiler + A program that translates human-readable source code into + machine-executable object code. The object code is then executed + directly by the computer. See also "Interpreter." + +Complemented Bracket Expression + The negation of a "bracket expression". All that is _not_ + described by a given bracket expression. The symbol '^' precedes + the negated bracket expression. E.g.: '[[^:digit:]' designates + whatever character is not a digit. '[^bad]' designates whatever + character is not one of the letters 'b', 'a', or 'd'. See "Bracket + Expression." + +Compound Statement + A series of 'awk' statements, enclosed in curly braces. Compound + statements may be nested. (*Note Statements::.) + +Computed Regexps + See "Dynamic Regular Expressions." + +Concatenation + Concatenating two strings means sticking them together, one after + another, producing a new string. For example, the string 'foo' + concatenated with the string 'bar' gives the string 'foobar'. + (*Note Concatenation::.) + +Conditional Expression + An expression using the '?:' ternary operator, such as 'EXPR1 ? + EXPR2 : EXPR3'. The expression EXPR1 is evaluated; if the result + is true, the value of the whole expression is the value of EXPR2; + otherwise the value is EXPR3. In either case, only one of EXPR2 + and EXPR3 is evaluated. (*Note Conditional Exp::.) + +Control Statement + A control statement is an instruction to perform a given operation + or a set of operations inside an 'awk' program, if a given + condition is true. Control statements are: 'if', 'for', 'while', + and 'do' (*note Statements::). + +Cookie + A peculiar goodie, token, saying or remembrance produced by or + presented to a program. (With thanks to Professor Doug McIlroy.) + +Coprocess + A subordinate program with which two-way communications is + possible. + +Curly Braces + See "Braces." + +Dark Corner + An area in the language where specifications often were (or still + are) not clear, leading to unexpected or undesirable behavior. + Such areas are marked in this Info file with "(d.c.)" in the text + and are indexed under the heading "dark corner." + +Data Driven + A description of 'awk' programs, where you specify the data you are + interested in processing, and what to do when that data is seen. + +Data Objects + These are numbers and strings of characters. Numbers are converted + into strings and vice versa, as needed. (*Note Conversion::.) + +Deadlock + The situation in which two communicating processes are each waiting + for the other to perform an action. + +Debugger + A program used to help developers remove "bugs" from (de-bug) their + programs. + +Double Precision + An internal representation of numbers that can have fractional + parts. Double precision numbers keep track of more digits than do + single precision numbers, but operations on them are sometimes more + expensive. This is the way 'awk' stores numeric values. It is the + C type 'double'. + +Dynamic Regular Expression + A dynamic regular expression is a regular expression written as an + ordinary expression. It could be a string constant, such as + '"foo"', but it may also be an expression whose value can vary. + (*Note Computed Regexps::.) + +Empty String + See "Null String." + +Environment + A collection of strings, of the form 'NAME=VAL', that each program + has available to it. Users generally place values into the + environment in order to provide information to various programs. + Typical examples are the environment variables 'HOME' and 'PATH'. + +Epoch + The date used as the "beginning of time" for timestamps. Time + values in most systems are represented as seconds since the epoch, + with library functions available for converting these values into + standard date and time formats. + + The epoch on Unix and POSIX systems is 1970-01-01 00:00:00 UTC. See + also "GMT" and "UTC." + +Escape Sequences + A special sequence of characters used for describing nonprinting + characters, such as '\n' for newline or '\033' for the ASCII ESC + (Escape) character. (*Note Escape Sequences::.) + +Extension + An additional feature or change to a programming language or + utility not defined by that language's or utility's standard. + 'gawk' has (too) many extensions over POSIX 'awk'. + +FDL + See "Free Documentation License." + +Field + When 'awk' reads an input record, it splits the record into pieces + separated by whitespace (or by a separator regexp that you can + change by setting the predefined variable 'FS'). Such pieces are + called fields. If the pieces are of fixed length, you can use the + built-in variable 'FIELDWIDTHS' to describe their lengths. If you + wish to specify the contents of fields instead of the field + separator, you can use the predefined variable 'FPAT' to do so. + (*Note Field Separators::, *note Constant Size::, and *note + Splitting By Content::.) + +Flag + A variable whose truth value indicates the existence or + nonexistence of some condition. + +Floating-Point Number + Often referred to in mathematical terms as a "rational" or real + number, this is just a number that can have a fractional part. See + also "Double Precision" and "Single Precision." + +Format + Format strings control the appearance of output in the 'strftime()' + and 'sprintf()' functions, and in the 'printf' statement as well. + Also, data conversions from numbers to strings are controlled by + the format strings contained in the predefined variables 'CONVFMT' + and 'OFMT'. (*Note Control Letters::.) + +Fortran + Shorthand for FORmula TRANslator, one of the first programming + languages available for scientific calculations. It was created by + John Backus, and has been available since 1957. It is still in use + today. + +Free Documentation License + This document describes the terms under which this Info file is + published and may be copied. (*Note GNU Free Documentation + License::.) + +Free Software Foundation + A nonprofit organization dedicated to the production and + distribution of freely distributable software. It was founded by + Richard M. Stallman, the author of the original Emacs editor. GNU + Emacs is the most widely used version of Emacs today. + +FSF + See "Free Software Foundation." + +Function + A part of an 'awk' program that can be invoked from every point of + the program, to perform a task. 'awk' has several built-in + functions. Users can define their own functions in every part of + the program. Function can be recursive, i.e., they may invoke + themselves. *Note Functions::. In 'gawk' it is also possible to + have functions shared among different programs, and included where + required using the '@include' directive (*note Include Files::). + In 'gawk' the name of the function that should be invoked can be + generated at run time, i.e., dynamically. The 'gawk' extension API + provides constructor functions (*note Constructor Functions::). + +'gawk' + The GNU implementation of 'awk'. + +General Public License + This document describes the terms under which 'gawk' and its source + code may be distributed. (*Note Copying::.) + +GMT + "Greenwich Mean Time." This is the old term for UTC. It is the + time of day used internally for Unix and POSIX systems. See also + "Epoch" and "UTC." + +GNU + "GNU's not Unix". An on-going project of the Free Software + Foundation to create a complete, freely distributable, + POSIX-compliant computing environment. + +GNU/Linux + A variant of the GNU system using the Linux kernel, instead of the + Free Software Foundation's Hurd kernel. The Linux kernel is a + stable, efficient, full-featured clone of Unix that has been ported + to a variety of architectures. It is most popular on PC-class + systems, but runs well on a variety of other systems too. The + Linux kernel source code is available under the terms of the GNU + General Public License, which is perhaps its most important aspect. + +GPL + See "General Public License." + +Hexadecimal + Base 16 notation, where the digits are '0'-'9' and 'A'-'F', with + 'A' representing 10, 'B' representing 11, and so on, up to 'F' for + 15. Hexadecimal numbers are written in C using a leading '0x', to + indicate their base. Thus, '0x12' is 18 ((1 x 16) + 2). *Note + Nondecimal-numbers::. + +I/O + Abbreviation for "Input/Output," the act of moving data into and/or + out of a running program. + +Input Record + A single chunk of data that is read in by 'awk'. Usually, an 'awk' + input record consists of one line of text. (*Note Records::.) + +Integer + A whole number, i.e., a number that does not have a fractional + part. + +Internationalization + The process of writing or modifying a program so that it can use + multiple languages without requiring further source code changes. + +Interpreter + A program that reads human-readable source code directly, and uses + the instructions in it to process data and produce results. 'awk' + is typically (but not always) implemented as an interpreter. See + also "Compiler." + +Interval Expression + A component of a regular expression that lets you specify repeated + matches of some part of the regexp. Interval expressions were not + originally available in 'awk' programs. + +ISO + The International Organization for Standardization. This + organization produces international standards for many things, + including programming languages, such as C and C++. In the + computer arena, important standards like those for C, C++, and + POSIX become both American national and ISO international standards + simultaneously. This Info file refers to Standard C as "ISO C" + throughout. See the ISO website + (http://www.iso.org/iso/home/about.htm) for more information about + the name of the organization and its language-independent + three-letter acronym. + +Java + A modern programming language originally developed by Sun + Microsystems (now Oracle) supporting Object-Oriented programming. + Although usually implemented by compiling to the instructions for a + standard virtual machine (the JVM), the language can be compiled to + native code. + +Keyword + In the 'awk' language, a keyword is a word that has special + meaning. Keywords are reserved and may not be used as variable + names. + + 'gawk''s keywords are: 'BEGIN', 'BEGINFILE', 'END', 'ENDFILE', + 'break', 'case', 'continue', 'default' 'delete', 'do...while', + 'else', 'exit', 'for...in', 'for', 'function', 'func', 'if', + 'next', 'nextfile', 'switch', and 'while'. + +Korn Shell + The Korn Shell ('ksh') is a Unix shell which was developed by David + Korn at Bell Laboratories in the early 1980s. The Korn Shell is + backward-compatible with the Bourne shell and includes many + features of the C shell. See also "Bourne Shell." + +Lesser General Public License + This document describes the terms under which binary library + archives or shared objects, and their source code may be + distributed. + +LGPL + See "Lesser General Public License." + +Linux + See "GNU/Linux." + +Localization + The process of providing the data necessary for an + internationalized program to work in a particular language. + +Logical Expression + An expression using the operators for logic, AND, OR, and NOT, + written '&&', '||', and '!' in 'awk'. Often called Boolean + expressions, after the mathematician who pioneered this kind of + mathematical logic. + +Lvalue + An expression that can appear on the left side of an assignment + operator. In most languages, lvalues can be variables or array + elements. In 'awk', a field designator can also be used as an + lvalue. + +Matching + The act of testing a string against a regular expression. If the + regexp describes the contents of the string, it is said to "match" + it. + +Metacharacters + Characters used within a regexp that do not stand for themselves. + Instead, they denote regular expression operations, such as + repetition, grouping, or alternation. + +Nesting + Nesting is where information is organized in layers, or where + objects contain other similar objects. In 'gawk' the '@include' + directive can be nested. The "natural" nesting of arithmetic and + logical operations can be changed using parentheses (*note + Precedence::). + +No-op + An operation that does nothing. + +Null String + A string with no characters in it. It is represented explicitly in + 'awk' programs by placing two double quote characters next to each + other ('""'). It can appear in input data by having two successive + occurrences of the field separator appear next to each other. + +Number + A numeric-valued data object. Modern 'awk' implementations use + double precision floating-point to represent numbers. Ancient + 'awk' implementations used single precision floating-point. + +Octal + Base-eight notation, where the digits are '0'-'7'. Octal numbers + are written in C using a leading '0', to indicate their base. + Thus, '013' is 11 ((1 x 8) + 3). *Note Nondecimal-numbers::. + +Output Record + A single chunk of data that is written out by 'awk'. Usually, an + 'awk' output record consists of one or more lines of text. *Note + Records::. + +Pattern + Patterns tell 'awk' which input records are interesting to which + rules. + + A pattern is an arbitrary conditional expression against which + input is tested. If the condition is satisfied, the pattern is + said to "match" the input record. A typical pattern might compare + the input record against a regular expression. (*Note Pattern + Overview::.) + +PEBKAC + An acronym describing what is possibly the most frequent source of + computer usage problems. (Problem Exists Between Keyboard And + Chair.) + +Plug-in + See "Extensions." + +POSIX + The name for a series of standards that specify a Portable + Operating System interface. The "IX" denotes the Unix heritage of + these standards. The main standard of interest for 'awk' users is + 'IEEE Standard for Information Technology, Standard 1003.1-2008'. + The 2008 POSIX standard can be found online at + <http://www.opengroup.org/onlinepubs/9699919799/>. + +Precedence + The order in which operations are performed when operators are used + without explicit parentheses. + +Private + Variables and/or functions that are meant for use exclusively by + library functions and not for the main 'awk' program. Special care + must be taken when naming such variables and functions. (*Note + Library Names::.) + +Range (of input lines) + A sequence of consecutive lines from the input file(s). A pattern + can specify ranges of input lines for 'awk' to process or it can + specify single lines. (*Note Pattern Overview::.) + +Record + See "Input record" and "Output record." + +Recursion + When a function calls itself, either directly or indirectly. If + this is clear, stop, and proceed to the next entry. Otherwise, + refer to the entry for "recursion." + +Redirection + Redirection means performing input from something other than the + standard input stream, or performing output to something other than + the standard output stream. + + You can redirect input to the 'getline' statement using the '<', + '|', and '|&' operators. You can redirect the output of the + 'print' and 'printf' statements to a file or a system command, + using the '>', '>>', '|', and '|&' operators. (*Note Getline::, + and *note Redirection::.) + +Reference Counts + An internal mechanism in 'gawk' to minimize the amount of memory + needed to store the value of string variables. If the value + assumed by a variable is used in more than one place, only one copy + of the value itself is kept, and the associated reference count is + increased when the same value is used by an additional variable, + and decreased when the related variable is no longer in use. When + the reference count goes to zero, the memory space used to store + the value of the variable is freed. + +Regexp + See "Regular Expression." + +Regular Expression + A regular expression ("regexp" for short) is a pattern that denotes + a set of strings, possibly an infinite set. For example, the + regular expression 'R.*xp' matches any string starting with the + letter 'R' and ending with the letters 'xp'. In 'awk', regular + expressions are used in patterns and in conditional expressions. + Regular expressions may contain escape sequences. (*Note + Regexp::.) + +Regular Expression Constant + A regular expression constant is a regular expression written + within slashes, such as '/foo/'. This regular expression is chosen + when you write the 'awk' program and cannot be changed during its + execution. (*Note Regexp Usage::.) + +Regular Expression Operators + See "Metacharacters." + +Rounding + Rounding the result of an arithmetic operation can be tricky. More + than one way of rounding exists, and in 'gawk' it is possible to + choose which method should be used in a program. *Note Setting the + rounding mode::. + +Rule + A segment of an 'awk' program that specifies how to process single + input records. A rule consists of a "pattern" and an "action". + 'awk' reads an input record; then, for each rule, if the input + record satisfies the rule's pattern, 'awk' executes the rule's + action. Otherwise, the rule does nothing for that input record. + +Rvalue + A value that can appear on the right side of an assignment + operator. In 'awk', essentially every expression has a value. + These values are rvalues. + +Scalar + A single value, be it a number or a string. Regular variables are + scalars; arrays and functions are not. + +Search Path + In 'gawk', a list of directories to search for 'awk' program source + files. In the shell, a list of directories to search for + executable programs. + +'sed' + See "Stream Editor." + +Seed + The initial value, or starting point, for a sequence of random + numbers. + +Shell + The command interpreter for Unix and POSIX-compliant systems. The + shell works both interactively, and as a programming language for + batch files, or shell scripts. + +Short-Circuit + The nature of the 'awk' logical operators '&&' and '||'. If the + value of the entire expression is determinable from evaluating just + the lefthand side of these operators, the righthand side is not + evaluated. (*Note Boolean Ops::.) + +Side Effect + A side effect occurs when an expression has an effect aside from + merely producing a value. Assignment expressions, increment and + decrement expressions, and function calls have side effects. + (*Note Assignment Ops::.) + +Single Precision + An internal representation of numbers that can have fractional + parts. Single precision numbers keep track of fewer digits than do + double precision numbers, but operations on them are sometimes less + expensive in terms of CPU time. This is the type used by some + ancient versions of 'awk' to store numeric values. It is the C + type 'float'. + +Space + The character generated by hitting the space bar on the keyboard. + +Special File + A file name interpreted internally by 'gawk', instead of being + handed directly to the underlying operating system--for example, + '/dev/stderr'. (*Note Special Files::.) + +Statement + An expression inside an 'awk' program in the action part of a + pattern-action rule, or inside an 'awk' function. A statement can + be a variable assignment, an array operation, a loop, etc. + +Stream Editor + A program that reads records from an input stream and processes + them one or more at a time. This is in contrast with batch + programs, which may expect to read their input files in entirety + before starting to do anything, as well as with interactive + programs which require input from the user. + +String + A datum consisting of a sequence of characters, such as 'I am a + string'. Constant strings are written with double quotes in the + 'awk' language and may contain escape sequences. (*Note Escape + Sequences::.) + +Tab + The character generated by hitting the 'TAB' key on the keyboard. + It usually expands to up to eight spaces upon output. + +Text Domain + A unique name that identifies an application. Used for grouping + messages that are translated at runtime into the local language. + +Timestamp + A value in the "seconds since the epoch" format used by Unix and + POSIX systems. Used for the 'gawk' functions 'mktime()', + 'strftime()', and 'systime()'. See also "Epoch," "GMT," and "UTC." + +Unix + A computer operating system originally developed in the early + 1970's at AT&T Bell Laboratories. It initially became popular in + universities around the world and later moved into commercial + environments as a software development system and network server + system. There are many commercial versions of Unix, as well as + several work-alike systems whose source code is freely available + (such as GNU/Linux, NetBSD (http://www.netbsd.org), FreeBSD + (http://www.freebsd.org), and OpenBSD (http://www.openbsd.org)). + +UTC + The accepted abbreviation for "Universal Coordinated Time." This + is standard time in Greenwich, England, which is used as a + reference time for day and date calculations. See also "Epoch" and + "GMT." + +Variable + A name for a value. In 'awk', variables may be either scalars or + arrays. + +Whitespace + A sequence of space, TAB, or newline characters occurring inside an + input record or a string. + + +File: gawk.info, Node: Copying, Next: GNU Free Documentation License, Prev: Glossary, Up: Top + +GNU General Public License +************************** + + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/> + + Everyone is permitted to copy and distribute verbatim copies of this + license document, but changing it is not allowed. + +Preamble +======== + +The GNU General Public License is a free, copyleft license for software +and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + +TERMS AND CONDITIONS +==================== + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public + License. + + "Copyright" also means copyright-like laws that apply to other + kinds of works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this + License. Each licensee is addressed as "you". "Licensees" and + "recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the + work in a fashion requiring copyright permission, other than the + making of an exact copy. The resulting work is called a "modified + version" of the earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work + based on the Program. + + To "propagate" a work means to do anything with it that, without + permission, would make you directly or secondarily liable for + infringement under applicable copyright law, except executing it on + a computer or modifying a private copy. Propagation includes + copying, distribution (with or without modification), making + available to the public, and in some countries other activities as + well. + + To "convey" a work means any kind of propagation that enables other + parties to make or receive copies. Mere interaction with a user + through a computer network, with no transfer of a copy, is not + conveying. + + An interactive user interface displays "Appropriate Legal Notices" + to the extent that it includes a convenient and prominently visible + feature that (1) displays an appropriate copyright notice, and (2) + tells the user that there is no warranty for the work (except to + the extent that warranties are provided), that licensees may convey + the work under this License, and how to view a copy of this + License. If the interface presents a list of user commands or + options, such as a menu, a prominent item in the list meets this + criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work + for making modifications to it. "Object code" means any non-source + form of a work. + + A "Standard Interface" means an interface that either is an + official standard defined by a recognized standards body, or, in + the case of interfaces specified for a particular programming + language, one that is widely used among developers working in that + language. + + The "System Libraries" of an executable work include anything, + other than the work as a whole, that (a) is included in the normal + form of packaging a Major Component, but which is not part of that + Major Component, and (b) serves only to enable use of the work with + that Major Component, or to implement a Standard Interface for + which an implementation is available to the public in source code + form. A "Major Component", in this context, means a major + essential component (kernel, window system, and so on) of the + specific operating system (if any) on which the executable work + runs, or a compiler used to produce the work, or an object code + interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all + the source code needed to generate, install, and (for an executable + work) run the object code and to modify the work, including scripts + to control those activities. However, it does not include the + work's System Libraries, or general-purpose tools or generally + available free programs which are used unmodified in performing + those activities but which are not part of the work. For example, + Corresponding Source includes interface definition files associated + with source files for the work, and the source code for shared + libraries and dynamically linked subprograms that the work is + specifically designed to require, such as by intimate data + communication or control flow between those subprograms and other + parts of the work. + + The Corresponding Source need not include anything that users can + regenerate automatically from other parts of the Corresponding + Source. + + The Corresponding Source for a work in source code form is that + same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of + copyright on the Program, and are irrevocable provided the stated + conditions are met. This License explicitly affirms your unlimited + permission to run the unmodified Program. The output from running + a covered work is covered by this License only if the output, given + its content, constitutes a covered work. This License acknowledges + your rights of fair use or other equivalent, as provided by + copyright law. + + You may make, run and propagate covered works that you do not + convey, without conditions so long as your license otherwise + remains in force. You may convey covered works to others for the + sole purpose of having them make modifications exclusively for you, + or provide you with facilities for running those works, provided + that you comply with the terms of this License in conveying all + material for which you do not control copyright. Those thus making + or running the covered works for you must do so exclusively on your + behalf, under your direction and control, on terms that prohibit + them from making any copies of your copyrighted material outside + their relationship with you. + + Conveying under any other circumstances is permitted solely under + the conditions stated below. Sublicensing is not allowed; section + 10 makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological + measure under any applicable law fulfilling obligations under + article 11 of the WIPO copyright treaty adopted on 20 December + 1996, or similar laws prohibiting or restricting circumvention of + such measures. + + When you convey a covered work, you waive any legal power to forbid + circumvention of technological measures to the extent such + circumvention is effected by exercising rights under this License + with respect to the covered work, and you disclaim any intention to + limit operation or modification of the work as a means of + enforcing, against the work's users, your or third parties' legal + rights to forbid circumvention of technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you + receive it, in any medium, provided that you conspicuously and + appropriately publish on each copy an appropriate copyright notice; + keep intact all notices stating that this License and any + non-permissive terms added in accord with section 7 apply to the + code; keep intact all notices of the absence of any warranty; and + give all recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, + and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to + produce it from the Program, in the form of source code under the + terms of section 4, provided that you also meet all of these + conditions: + + a. The work must carry prominent notices stating that you + modified it, and giving a relevant date. + + b. The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in + section 4 to "keep intact all notices". + + c. You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable + section 7 additional terms, to the whole of the work, and all + its parts, regardless of how they are packaged. This License + gives no permission to license the work in any other way, but + it does not invalidate such permission if you have separately + received it. + + d. If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has + interactive interfaces that do not display Appropriate Legal + Notices, your work need not make them do so. + + A compilation of a covered work with other separate and independent + works, which are not by their nature extensions of the covered + work, and which are not combined with it such as to form a larger + program, in or on a volume of a storage or distribution medium, is + called an "aggregate" if the compilation and its resulting + copyright are not used to limit the access or legal rights of the + compilation's users beyond what the individual works permit. + Inclusion of a covered work in an aggregate does not cause this + License to apply to the other parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms + of sections 4 and 5, provided that you also convey the + machine-readable Corresponding Source under the terms of this + License, in one of these ways: + + a. Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b. Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that + product model, to give anyone who possesses the object code + either (1) a copy of the Corresponding Source for all the + software in the product that is covered by this License, on a + durable physical medium customarily used for software + interchange, for a price no more than your reasonable cost of + physically performing this conveying of source, or (2) access + to copy the Corresponding Source from a network server at no + charge. + + c. Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, + and only if you received the object code with such an offer, + in accord with subsection 6b. + + d. Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to + the Corresponding Source in the same way through the same + place at no further charge. You need not require recipients + to copy the Corresponding Source along with the object code. + If the place to copy the object code is a network server, the + Corresponding Source may be on a different server (operated by + you or a third party) that supports equivalent copying + facilities, provided you maintain clear directions next to the + object code saying where to find the Corresponding Source. + Regardless of what server hosts the Corresponding Source, you + remain obligated to ensure that it is available for as long as + needed to satisfy these requirements. + + e. Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the + general public at no charge under subsection 6d. + + A separable portion of the object code, whose source code is + excluded from the Corresponding Source as a System Library, need + not be included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means + any tangible personal property which is normally used for personal, + family, or household purposes, or (2) anything designed or sold for + incorporation into a dwelling. In determining whether a product is + a consumer product, doubtful cases shall be resolved in favor of + coverage. For a particular product received by a particular user, + "normally used" refers to a typical or common use of that class of + product, regardless of the status of the particular user or of the + way in which the particular user actually uses, or expects or is + expected to use, the product. A product is a consumer product + regardless of whether the product has substantial commercial, + industrial or non-consumer uses, unless such uses represent the + only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, + procedures, authorization keys, or other information required to + install and execute modified versions of a covered work in that + User Product from a modified version of its Corresponding Source. + The information must suffice to ensure that the continued + functioning of the modified object code is in no case prevented or + interfered with solely because modification has been made. + + If you convey an object code work under this section in, or with, + or specifically for use in, a User Product, and the conveying + occurs as part of a transaction in which the right of possession + and use of the User Product is transferred to the recipient in + perpetuity or for a fixed term (regardless of how the transaction + is characterized), the Corresponding Source conveyed under this + section must be accompanied by the Installation Information. But + this requirement does not apply if neither you nor any third party + retains the ability to install modified object code on the User + Product (for example, the work has been installed in ROM). + + The requirement to provide Installation Information does not + include a requirement to continue to provide support service, + warranty, or updates for a work that has been modified or installed + by the recipient, or for the User Product in which it has been + modified or installed. Access to a network may be denied when the + modification itself materially and adversely affects the operation + of the network or violates the rules and protocols for + communication across the network. + + Corresponding Source conveyed, and Installation Information + provided, in accord with this section must be in a format that is + publicly documented (and with an implementation available to the + public in source code form), and must require no special password + or key for unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of + this License by making exceptions from one or more of its + conditions. Additional permissions that are applicable to the + entire Program shall be treated as though they were included in + this License, to the extent that they are valid under applicable + law. If additional permissions apply only to part of the Program, + that part may be used separately under those permissions, but the + entire Program remains governed by this License without regard to + the additional permissions. + + When you convey a copy of a covered work, you may at your option + remove any additional permissions from that copy, or from any part + of it. (Additional permissions may be written to require their own + removal in certain cases when you modify the work.) You may place + additional permissions on material, added by you to a covered work, + for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material + you add to a covered work, you may (if authorized by the copyright + holders of that material) supplement the terms of this License with + terms: + + a. Disclaiming warranty or limiting liability differently from + the terms of sections 15 and 16 of this License; or + + b. Requiring preservation of specified reasonable legal notices + or author attributions in that material or in the Appropriate + Legal Notices displayed by works containing it; or + + c. Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked + in reasonable ways as different from the original version; or + + d. Limiting the use for publicity purposes of names of licensors + or authors of the material; or + + e. Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f. Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified + versions of it) with contractual assumptions of liability to + the recipient, for any liability that these contractual + assumptions directly impose on those licensors and authors. + + All other non-permissive additional terms are considered "further + restrictions" within the meaning of section 10. If the Program as + you received it, or any part of it, contains a notice stating that + it is governed by this License along with a term that is a further + restriction, you may remove that term. If a license document + contains a further restriction but permits relicensing or conveying + under this License, you may add to a covered work material governed + by the terms of that license document, provided that the further + restriction does not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you + must place, in the relevant source files, a statement of the + additional terms that apply to those files, or a notice indicating + where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in + the form of a separately written license, or stated as exceptions; + the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly + provided under this License. Any attempt otherwise to propagate or + modify it is void, and will automatically terminate your rights + under this License (including any patent licenses granted under the + third paragraph of section 11). + + However, if you cease all violation of this License, then your + license from a particular copyright holder is reinstated (a) + provisionally, unless and until the copyright holder explicitly and + finally terminates your license, and (b) permanently, if the + copyright holder fails to notify you of the violation by some + reasonable means prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is + reinstated permanently if the copyright holder notifies you of the + violation by some reasonable means, this is the first time you have + received notice of violation of this License (for any work) from + that copyright holder, and you cure the violation prior to 30 days + after your receipt of the notice. + + Termination of your rights under this section does not terminate + the licenses of parties who have received copies or rights from you + under this License. If your rights have been terminated and not + permanently reinstated, you do not qualify to receive new licenses + for the same material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or + run a copy of the Program. Ancillary propagation of a covered work + occurring solely as a consequence of using peer-to-peer + transmission to receive a copy likewise does not require + acceptance. However, nothing other than this License grants you + permission to propagate or modify any covered work. These actions + infringe copyright if you do not accept this License. Therefore, + by modifying or propagating a covered work, you indicate your + acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically + receives a license from the original licensors, to run, modify and + propagate that work, subject to this License. You are not + responsible for enforcing compliance by third parties with this + License. + + An "entity transaction" is a transaction transferring control of an + organization, or substantially all assets of one, or subdividing an + organization, or merging organizations. If propagation of a + covered work results from an entity transaction, each party to that + transaction who receives a copy of the work also receives whatever + licenses to the work the party's predecessor in interest had or + could give under the previous paragraph, plus a right to possession + of the Corresponding Source of the work from the predecessor in + interest, if the predecessor has it or can get it with reasonable + efforts. + + You may not impose any further restrictions on the exercise of the + rights granted or affirmed under this License. For example, you + may not impose a license fee, royalty, or other charge for exercise + of rights granted under this License, and you may not initiate + litigation (including a cross-claim or counterclaim in a lawsuit) + alleging that any patent claim is infringed by making, using, + selling, offering for sale, or importing the Program or any portion + of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this + License of the Program or a work on which the Program is based. + The work thus licensed is called the contributor's "contributor + version". + + A contributor's "essential patent claims" are all patent claims + owned or controlled by the contributor, whether already acquired or + hereafter acquired, that would be infringed by some manner, + permitted by this License, of making, using, or selling its + contributor version, but do not include claims that would be + infringed only as a consequence of further modification of the + contributor version. For purposes of this definition, "control" + includes the right to grant patent sublicenses in a manner + consistent with the requirements of this License. + + Each contributor grants you a non-exclusive, worldwide, + royalty-free patent license under the contributor's essential + patent claims, to make, use, sell, offer for sale, import and + otherwise run, modify and propagate the contents of its contributor + version. + + In the following three paragraphs, a "patent license" is any + express agreement or commitment, however denominated, not to + enforce a patent (such as an express permission to practice a + patent or covenant not to sue for patent infringement). To "grant" + such a patent license to a party means to make such an agreement or + commitment not to enforce a patent against the party. + + If you convey a covered work, knowingly relying on a patent + license, and the Corresponding Source of the work is not available + for anyone to copy, free of charge and under the terms of this + License, through a publicly available network server or other + readily accessible means, then you must either (1) cause the + Corresponding Source to be so available, or (2) arrange to deprive + yourself of the benefit of the patent license for this particular + work, or (3) arrange, in a manner consistent with the requirements + of this License, to extend the patent license to downstream + recipients. "Knowingly relying" means you have actual knowledge + that, but for the patent license, your conveying the covered work + in a country, or your recipient's use of the covered work in a + country, would infringe one or more identifiable patents in that + country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or + arrangement, you convey, or propagate by procuring conveyance of, a + covered work, and grant a patent license to some of the parties + receiving the covered work authorizing them to use, propagate, + modify or convey a specific copy of the covered work, then the + patent license you grant is automatically extended to all + recipients of the covered work and works based on it. + + A patent license is "discriminatory" if it does not include within + the scope of its coverage, prohibits the exercise of, or is + conditioned on the non-exercise of one or more of the rights that + are specifically granted under this License. You may not convey a + covered work if you are a party to an arrangement with a third + party that is in the business of distributing software, under which + you make payment to the third party based on the extent of your + activity of conveying the work, and under which the third party + grants, to any of the parties who would receive the covered work + from you, a discriminatory patent license (a) in connection with + copies of the covered work conveyed by you (or copies made from + those copies), or (b) primarily for and in connection with specific + products or compilations that contain the covered work, unless you + entered into that arrangement, or that patent license was granted, + prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting + any implied license or other defenses to infringement that may + otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement + or otherwise) that contradict the conditions of this License, they + do not excuse you from the conditions of this License. If you + cannot convey a covered work so as to satisfy simultaneously your + obligations under this License and any other pertinent obligations, + then as a consequence you may not convey it at all. For example, + if you agree to terms that obligate you to collect a royalty for + further conveying from those to whom you convey the Program, the + only way you could satisfy both those terms and this License would + be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have + permission to link or combine any covered work with a work licensed + under version 3 of the GNU Affero General Public License into a + single combined work, and to convey the resulting work. The terms + of this License will continue to apply to the part which is the + covered work, but the special requirements of the GNU Affero + General Public License, section 13, concerning interaction through + a network will apply to the combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new + versions of the GNU General Public License from time to time. Such + new versions will be similar in spirit to the present version, but + may differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the + Program specifies that a certain numbered version of the GNU + General Public License "or any later version" applies to it, you + have the option of following the terms and conditions either of + that numbered version or of any later version published by the Free + Software Foundation. If the Program does not specify a version + number of the GNU General Public License, you may choose any + version ever published by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future + versions of the GNU General Public License can be used, that + proxy's public statement of acceptance of a version permanently + authorizes you to choose that version for the Program. + + Later license versions may give you additional or different + permissions. However, no additional obligations are imposed on any + author or copyright holder as a result of your choosing to follow a + later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY + APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE + COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" + WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, + INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE + RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. + SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL + NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN + WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES + AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR + DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR + CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE + THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA + BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD + PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER + PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF + THE POSSIBILITY OF SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided + above cannot be given local legal effect according to their terms, + reviewing courts shall apply local law that most closely + approximates an absolute waiver of all civil liability in + connection with the Program, unless a warranty or assumption of + liability accompanies a copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS +=========================== + +How to Apply These Terms to Your New Programs +============================================= + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these +terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + ONE LINE TO GIVE THE PROGRAM'S NAME AND A BRIEF IDEA OF WHAT IT DOES. + Copyright (C) YEAR NAME OF AUTHOR + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or (at + your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + + Also add information on how to contact you by electronic and paper +mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + PROGRAM Copyright (C) YEAR NAME OF AUTHOR + This program comes with ABSOLUTELY NO WARRANTY; for details type 'show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type 'show c' for details. + + The hypothetical commands 'show w' and 'show c' should show the +appropriate parts of the General Public License. Of course, your +program's commands might be different; for a GUI interface, you would +use an "about box". + + You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU GPL, see <http://www.gnu.org/licenses/>. + + The GNU General Public License does not permit incorporating your +program into proprietary programs. If your program is a subroutine +library, you may consider it more useful to permit linking proprietary +applications with the library. If this is what you want to do, use the +GNU Lesser General Public License instead of this License. But first, +please read <http://www.gnu.org/philosophy/why-not-lgpl.html>. + + +File: gawk.info, Node: GNU Free Documentation License, Next: Index, Prev: Copying, Up: Top + +GNU Free Documentation License +****************************** + + Version 1.3, 3 November 2008 + + Copyright (C) 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc. + <http://fsf.org/> + + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + 0. PREAMBLE + + The purpose of this License is to make a manual, textbook, or other + functional and useful document "free" in the sense of freedom: to + assure everyone the effective freedom to copy and redistribute it, + with or without modifying it, either commercially or + noncommercially. Secondarily, this License preserves for the + author and publisher a way to get credit for their work, while not + being considered responsible for modifications made by others. + + This License is a kind of "copyleft", which means that derivative + works of the document must themselves be free in the same sense. + It complements the GNU General Public License, which is a copyleft + license designed for free software. + + We have designed this License in order to use it for manuals for + free software, because free software needs free documentation: a + free program should come with manuals providing the same freedoms + that the software does. But this License is not limited to + software manuals; it can be used for any textual work, regardless + of subject matter or whether it is published as a printed book. We + recommend this License principally for works whose purpose is + instruction or reference. + + 1. APPLICABILITY AND DEFINITIONS + + This License applies to any manual or other work, in any medium, + that contains a notice placed by the copyright holder saying it can + be distributed under the terms of this License. Such a notice + grants a world-wide, royalty-free license, unlimited in duration, + to use that work under the conditions stated herein. The + "Document", below, refers to any such manual or work. Any member + of the public is a licensee, and is addressed as "you". You accept + the license if you copy, modify or distribute the work in a way + requiring permission under copyright law. + + A "Modified Version" of the Document means any work containing the + Document or a portion of it, either copied verbatim, or with + modifications and/or translated into another language. + + A "Secondary Section" is a named appendix or a front-matter section + of the Document that deals exclusively with the relationship of the + publishers or authors of the Document to the Document's overall + subject (or to related matters) and contains nothing that could + fall directly within that overall subject. (Thus, if the Document + is in part a textbook of mathematics, a Secondary Section may not + explain any mathematics.) The relationship could be a matter of + historical connection with the subject or with related matters, or + of legal, commercial, philosophical, ethical or political position + regarding them. + + The "Invariant Sections" are certain Secondary Sections whose + titles are designated, as being those of Invariant Sections, in the + notice that says that the Document is released under this License. + If a section does not fit the above definition of Secondary then it + is not allowed to be designated as Invariant. The Document may + contain zero Invariant Sections. If the Document does not identify + any Invariant Sections then there are none. + + The "Cover Texts" are certain short passages of text that are + listed, as Front-Cover Texts or Back-Cover Texts, in the notice + that says that the Document is released under this License. A + Front-Cover Text may be at most 5 words, and a Back-Cover Text may + be at most 25 words. + + A "Transparent" copy of the Document means a machine-readable copy, + represented in a format whose specification is available to the + general public, that is suitable for revising the document + straightforwardly with generic text editors or (for images composed + of pixels) generic paint programs or (for drawings) some widely + available drawing editor, and that is suitable for input to text + formatters or for automatic translation to a variety of formats + suitable for input to text formatters. A copy made in an otherwise + Transparent file format whose markup, or absence of markup, has + been arranged to thwart or discourage subsequent modification by + readers is not Transparent. An image format is not Transparent if + used for any substantial amount of text. A copy that is not + "Transparent" is called "Opaque". + + Examples of suitable formats for Transparent copies include plain + ASCII without markup, Texinfo input format, LaTeX input format, + SGML or XML using a publicly available DTD, and standard-conforming + simple HTML, PostScript or PDF designed for human modification. + Examples of transparent image formats include PNG, XCF and JPG. + Opaque formats include proprietary formats that can be read and + edited only by proprietary word processors, SGML or XML for which + the DTD and/or processing tools are not generally available, and + the machine-generated HTML, PostScript or PDF produced by some word + processors for output purposes only. + + The "Title Page" means, for a printed book, the title page itself, + plus such following pages as are needed to hold, legibly, the + material this License requires to appear in the title page. For + works in formats which do not have any title page as such, "Title + Page" means the text near the most prominent appearance of the + work's title, preceding the beginning of the body of the text. + + The "publisher" means any person or entity that distributes copies + of the Document to the public. + + A section "Entitled XYZ" means a named subunit of the Document + whose title either is precisely XYZ or contains XYZ in parentheses + following text that translates XYZ in another language. (Here XYZ + stands for a specific section name mentioned below, such as + "Acknowledgements", "Dedications", "Endorsements", or "History".) + To "Preserve the Title" of such a section when you modify the + Document means that it remains a section "Entitled XYZ" according + to this definition. + + The Document may include Warranty Disclaimers next to the notice + which states that this License applies to the Document. These + Warranty Disclaimers are considered to be included by reference in + this License, but only as regards disclaiming warranties: any other + implication that these Warranty Disclaimers may have is void and + has no effect on the meaning of this License. + + 2. VERBATIM COPYING + + You may copy and distribute the Document in any medium, either + commercially or noncommercially, provided that this License, the + copyright notices, and the license notice saying this License + applies to the Document are reproduced in all copies, and that you + add no other conditions whatsoever to those of this License. You + may not use technical measures to obstruct or control the reading + or further copying of the copies you make or distribute. However, + you may accept compensation in exchange for copies. If you + distribute a large enough number of copies you must also follow the + conditions in section 3. + + You may also lend copies, under the same conditions stated above, + and you may publicly display copies. + + 3. COPYING IN QUANTITY + + If you publish printed copies (or copies in media that commonly + have printed covers) of the Document, numbering more than 100, and + the Document's license notice requires Cover Texts, you must + enclose the copies in covers that carry, clearly and legibly, all + these Cover Texts: Front-Cover Texts on the front cover, and + Back-Cover Texts on the back cover. Both covers must also clearly + and legibly identify you as the publisher of these copies. The + front cover must present the full title with all words of the title + equally prominent and visible. You may add other material on the + covers in addition. Copying with changes limited to the covers, as + long as they preserve the title of the Document and satisfy these + conditions, can be treated as verbatim copying in other respects. + + If the required texts for either cover are too voluminous to fit + legibly, you should put the first ones listed (as many as fit + reasonably) on the actual cover, and continue the rest onto + adjacent pages. + + If you publish or distribute Opaque copies of the Document + numbering more than 100, you must either include a machine-readable + Transparent copy along with each Opaque copy, or state in or with + each Opaque copy a computer-network location from which the general + network-using public has access to download using public-standard + network protocols a complete Transparent copy of the Document, free + of added material. If you use the latter option, you must take + reasonably prudent steps, when you begin distribution of Opaque + copies in quantity, to ensure that this Transparent copy will + remain thus accessible at the stated location until at least one + year after the last time you distribute an Opaque copy (directly or + through your agents or retailers) of that edition to the public. + + It is requested, but not required, that you contact the authors of + the Document well before redistributing any large number of copies, + to give them a chance to provide you with an updated version of the + Document. + + 4. MODIFICATIONS + + You may copy and distribute a Modified Version of the Document + under the conditions of sections 2 and 3 above, provided that you + release the Modified Version under precisely this License, with the + Modified Version filling the role of the Document, thus licensing + distribution and modification of the Modified Version to whoever + possesses a copy of it. In addition, you must do these things in + the Modified Version: + + A. Use in the Title Page (and on the covers, if any) a title + distinct from that of the Document, and from those of previous + versions (which should, if there were any, be listed in the + History section of the Document). You may use the same title + as a previous version if the original publisher of that + version gives permission. + + B. List on the Title Page, as authors, one or more persons or + entities responsible for authorship of the modifications in + the Modified Version, together with at least five of the + principal authors of the Document (all of its principal + authors, if it has fewer than five), unless they release you + from this requirement. + + C. State on the Title page the name of the publisher of the + Modified Version, as the publisher. + + D. Preserve all the copyright notices of the Document. + + E. Add an appropriate copyright notice for your modifications + adjacent to the other copyright notices. + + F. Include, immediately after the copyright notices, a license + notice giving the public permission to use the Modified + Version under the terms of this License, in the form shown in + the Addendum below. + + G. Preserve in that license notice the full lists of Invariant + Sections and required Cover Texts given in the Document's + license notice. + + H. Include an unaltered copy of this License. + + I. Preserve the section Entitled "History", Preserve its Title, + and add to it an item stating at least the title, year, new + authors, and publisher of the Modified Version as given on the + Title Page. If there is no section Entitled "History" in the + Document, create one stating the title, year, authors, and + publisher of the Document as given on its Title Page, then add + an item describing the Modified Version as stated in the + previous sentence. + + J. Preserve the network location, if any, given in the Document + for public access to a Transparent copy of the Document, and + likewise the network locations given in the Document for + previous versions it was based on. These may be placed in the + "History" section. You may omit a network location for a work + that was published at least four years before the Document + itself, or if the original publisher of the version it refers + to gives permission. + + K. For any section Entitled "Acknowledgements" or "Dedications", + Preserve the Title of the section, and preserve in the section + all the substance and tone of each of the contributor + acknowledgements and/or dedications given therein. + + L. Preserve all the Invariant Sections of the Document, unaltered + in their text and in their titles. Section numbers or the + equivalent are not considered part of the section titles. + + M. Delete any section Entitled "Endorsements". Such a section + may not be included in the Modified Version. + + N. Do not retitle any existing section to be Entitled + "Endorsements" or to conflict in title with any Invariant + Section. + + O. Preserve any Warranty Disclaimers. + + If the Modified Version includes new front-matter sections or + appendices that qualify as Secondary Sections and contain no + material copied from the Document, you may at your option designate + some or all of these sections as invariant. To do this, add their + titles to the list of Invariant Sections in the Modified Version's + license notice. These titles must be distinct from any other + section titles. + + You may add a section Entitled "Endorsements", provided it contains + nothing but endorsements of your Modified Version by various + parties--for example, statements of peer review or that the text + has been approved by an organization as the authoritative + definition of a standard. + + You may add a passage of up to five words as a Front-Cover Text, + and a passage of up to 25 words as a Back-Cover Text, to the end of + the list of Cover Texts in the Modified Version. Only one passage + of Front-Cover Text and one of Back-Cover Text may be added by (or + through arrangements made by) any one entity. If the Document + already includes a cover text for the same cover, previously added + by you or by arrangement made by the same entity you are acting on + behalf of, you may not add another; but you may replace the old + one, on explicit permission from the previous publisher that added + the old one. + + The author(s) and publisher(s) of the Document do not by this + License give permission to use their names for publicity for or to + assert or imply endorsement of any Modified Version. + + 5. COMBINING DOCUMENTS + + You may combine the Document with other documents released under + this License, under the terms defined in section 4 above for + modified versions, provided that you include in the combination all + of the Invariant Sections of all of the original documents, + unmodified, and list them all as Invariant Sections of your + combined work in its license notice, and that you preserve all + their Warranty Disclaimers. + + The combined work need only contain one copy of this License, and + multiple identical Invariant Sections may be replaced with a single + copy. If there are multiple Invariant Sections with the same name + but different contents, make the title of each such section unique + by adding at the end of it, in parentheses, the name of the + original author or publisher of that section if known, or else a + unique number. Make the same adjustment to the section titles in + the list of Invariant Sections in the license notice of the + combined work. + + In the combination, you must combine any sections Entitled + "History" in the various original documents, forming one section + Entitled "History"; likewise combine any sections Entitled + "Acknowledgements", and any sections Entitled "Dedications". You + must delete all sections Entitled "Endorsements." + + 6. COLLECTIONS OF DOCUMENTS + + You may make a collection consisting of the Document and other + documents released under this License, and replace the individual + copies of this License in the various documents with a single copy + that is included in the collection, provided that you follow the + rules of this License for verbatim copying of each of the documents + in all other respects. + + You may extract a single document from such a collection, and + distribute it individually under this License, provided you insert + a copy of this License into the extracted document, and follow this + License in all other respects regarding verbatim copying of that + document. + + 7. AGGREGATION WITH INDEPENDENT WORKS + + A compilation of the Document or its derivatives with other + separate and independent documents or works, in or on a volume of a + storage or distribution medium, is called an "aggregate" if the + copyright resulting from the compilation is not used to limit the + legal rights of the compilation's users beyond what the individual + works permit. When the Document is included in an aggregate, this + License does not apply to the other works in the aggregate which + are not themselves derivative works of the Document. + + If the Cover Text requirement of section 3 is applicable to these + copies of the Document, then if the Document is less than one half + of the entire aggregate, the Document's Cover Texts may be placed + on covers that bracket the Document within the aggregate, or the + electronic equivalent of covers if the Document is in electronic + form. Otherwise they must appear on printed covers that bracket + the whole aggregate. + + 8. TRANSLATION + + Translation is considered a kind of modification, so you may + distribute translations of the Document under the terms of section + 4. Replacing Invariant Sections with translations requires special + permission from their copyright holders, but you may include + translations of some or all Invariant Sections in addition to the + original versions of these Invariant Sections. You may include a + translation of this License, and all the license notices in the + Document, and any Warranty Disclaimers, provided that you also + include the original English version of this License and the + original versions of those notices and disclaimers. In case of a + disagreement between the translation and the original version of + this License or a notice or disclaimer, the original version will + prevail. + + If a section in the Document is Entitled "Acknowledgements", + "Dedications", or "History", the requirement (section 4) to + Preserve its Title (section 1) will typically require changing the + actual title. + + 9. TERMINATION + + You may not copy, modify, sublicense, or distribute the Document + except as expressly provided under this License. Any attempt + otherwise to copy, modify, sublicense, or distribute it is void, + and will automatically terminate your rights under this License. + + However, if you cease all violation of this License, then your + license from a particular copyright holder is reinstated (a) + provisionally, unless and until the copyright holder explicitly and + finally terminates your license, and (b) permanently, if the + copyright holder fails to notify you of the violation by some + reasonable means prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is + reinstated permanently if the copyright holder notifies you of the + violation by some reasonable means, this is the first time you have + received notice of violation of this License (for any work) from + that copyright holder, and you cure the violation prior to 30 days + after your receipt of the notice. + + Termination of your rights under this section does not terminate + the licenses of parties who have received copies or rights from you + under this License. If your rights have been terminated and not + permanently reinstated, receipt of a copy of some or all of the + same material does not give you any rights to use it. + + 10. FUTURE REVISIONS OF THIS LICENSE + + The Free Software Foundation may publish new, revised versions of + the GNU Free Documentation License from time to time. Such new + versions will be similar in spirit to the present version, but may + differ in detail to address new problems or concerns. See + <http://www.gnu.org/copyleft/>. + + Each version of the License is given a distinguishing version + number. If the Document specifies that a particular numbered + version of this License "or any later version" applies to it, you + have the option of following the terms and conditions either of + that specified version or of any later version that has been + published (not as a draft) by the Free Software Foundation. If the + Document does not specify a version number of this License, you may + choose any version ever published (not as a draft) by the Free + Software Foundation. If the Document specifies that a proxy can + decide which future versions of this License can be used, that + proxy's public statement of acceptance of a version permanently + authorizes you to choose that version for the Document. + + 11. RELICENSING + + "Massive Multiauthor Collaboration Site" (or "MMC Site") means any + World Wide Web server that publishes copyrightable works and also + provides prominent facilities for anybody to edit those works. A + public wiki that anybody can edit is an example of such a server. + A "Massive Multiauthor Collaboration" (or "MMC") contained in the + site means any set of copyrightable works thus published on the MMC + site. + + "CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0 + license published by Creative Commons Corporation, a not-for-profit + corporation with a principal place of business in San Francisco, + California, as well as future copyleft versions of that license + published by that same organization. + + "Incorporate" means to publish or republish a Document, in whole or + in part, as part of another Document. + + An MMC is "eligible for relicensing" if it is licensed under this + License, and if all works that were first published under this + License somewhere other than this MMC, and subsequently + incorporated in whole or in part into the MMC, (1) had no cover + texts or invariant sections, and (2) were thus incorporated prior + to November 1, 2008. + + The operator of an MMC Site may republish an MMC contained in the + site under CC-BY-SA on the same site at any time before August 1, + 2009, provided the MMC is eligible for relicensing. + +ADDENDUM: How to use this License for your documents +==================================================== + +To use this License in a document you have written, include a copy of +the License in the document and put the following copyright and license +notices just after the title page: + + Copyright (C) YEAR YOUR NAME. + Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.3 + or any later version published by the Free Software Foundation; + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + Texts. A copy of the license is included in the section entitled ``GNU + Free Documentation License''. + + If you have Invariant Sections, Front-Cover Texts and Back-Cover +Texts, replace the "with...Texts." line with this: + + with the Invariant Sections being LIST THEIR TITLES, with + the Front-Cover Texts being LIST, and with the Back-Cover Texts + being LIST. + + If you have Invariant Sections without Cover Texts, or some other +combination of the three, merge those two alternatives to suit the +situation. + + If your document contains nontrivial examples of program code, we +recommend releasing these examples in parallel under your choice of free +software license, such as the GNU General Public License, to permit +their use in free software. + + +File: gawk.info, Node: Index, Prev: GNU Free Documentation License, Up: Top + +Index +***** + + +* Menu: + +* ! (exclamation point), ! operator: Boolean Ops. (line 69) +* ! (exclamation point), ! operator <1>: Precedence. (line 51) +* ! (exclamation point), ! operator <2>: Ranges. (line 47) +* ! (exclamation point), ! operator <3>: Egrep Program. (line 174) +* ! (exclamation point), != operator: Comparison Operators. + (line 11) +* ! (exclamation point), != operator <1>: Precedence. (line 64) +* ! (exclamation point), !~ operator: Regexp Usage. (line 19) +* ! (exclamation point), !~ operator <1>: Computed Regexps. (line 6) +* ! (exclamation point), !~ operator <2>: Case-sensitivity. (line 26) +* ! (exclamation point), !~ operator <3>: Regexp Constants. (line 6) +* ! (exclamation point), !~ operator <4>: Comparison Operators. + (line 11) +* ! (exclamation point), !~ operator <5>: Comparison Operators. + (line 98) +* ! (exclamation point), !~ operator <6>: Precedence. (line 79) +* ! (exclamation point), !~ operator <7>: Expression Patterns. + (line 24) +* " (double quote), in regexp constants: Computed Regexps. (line 30) +* " (double quote), in shell commands: Quoting. (line 54) +* # (number sign), #! (executable scripts): Executable Scripts. + (line 6) +* # (number sign), commenting: Comments. (line 6) +* $ (dollar sign), $ field operator: Fields. (line 19) +* $ (dollar sign), $ field operator <1>: Precedence. (line 42) +* $ (dollar sign), incrementing fields and arrays: Increment Ops. + (line 30) +* $ (dollar sign), regexp operator: Regexp Operators. (line 35) +* % (percent sign), % operator: Precedence. (line 54) +* % (percent sign), %= operator: Assignment Ops. (line 129) +* % (percent sign), %= operator <1>: Precedence. (line 94) +* & (ampersand), && operator: Boolean Ops. (line 59) +* & (ampersand), && operator <1>: Precedence. (line 85) +* & (ampersand), gsub()/gensub()/sub() functions and: Gory Details. + (line 6) +* ' (single quote): One-shot. (line 15) +* ' (single quote) in gawk command lines: Long. (line 35) +* ' (single quote), in shell commands: Quoting. (line 48) +* ' (single quote), vs. apostrophe: Comments. (line 27) +* ' (single quote), with double quotes: Quoting. (line 73) +* () (parentheses), in a profile: Profiling. (line 146) +* () (parentheses), regexp operator: Regexp Operators. (line 81) +* * (asterisk), * operator, as multiplication operator: Precedence. + (line 54) +* * (asterisk), * operator, as regexp operator: Regexp Operators. + (line 89) +* * (asterisk), * operator, null strings, matching: String Functions. + (line 537) +* * (asterisk), ** operator: Arithmetic Ops. (line 81) +* * (asterisk), ** operator <1>: Precedence. (line 48) +* * (asterisk), **= operator: Assignment Ops. (line 129) +* * (asterisk), **= operator <1>: Precedence. (line 94) +* * (asterisk), *= operator: Assignment Ops. (line 129) +* * (asterisk), *= operator <1>: Precedence. (line 94) +* + (plus sign), + operator: Precedence. (line 51) +* + (plus sign), + operator <1>: Precedence. (line 57) +* + (plus sign), ++ operator: Increment Ops. (line 11) +* + (plus sign), ++ operator <1>: Increment Ops. (line 40) +* + (plus sign), ++ operator <2>: Precedence. (line 45) +* + (plus sign), += operator: Assignment Ops. (line 81) +* + (plus sign), += operator <1>: Precedence. (line 94) +* + (plus sign), regexp operator: Regexp Operators. (line 105) +* , (comma), in range patterns: Ranges. (line 6) +* - (hyphen), - operator: Precedence. (line 51) +* - (hyphen), - operator <1>: Precedence. (line 57) +* - (hyphen), -- operator: Increment Ops. (line 48) +* - (hyphen), -- operator <1>: Precedence. (line 45) +* - (hyphen), -= operator: Assignment Ops. (line 129) +* - (hyphen), -= operator <1>: Precedence. (line 94) +* - (hyphen), filenames beginning with: Options. (line 60) +* - (hyphen), in bracket expressions: Bracket Expressions. (line 25) +* --assign option: Options. (line 32) +* --bignum option: Options. (line 203) +* --characters-as-bytes option: Options. (line 69) +* --copyright option: Options. (line 89) +* --debug option: Options. (line 108) +* --disable-extensions configuration option: Additional Configuration Options. + (line 9) +* --disable-lint configuration option: Additional Configuration Options. + (line 15) +* --disable-nls configuration option: Additional Configuration Options. + (line 32) +* --dump-variables option: Options. (line 94) +* --dump-variables option, using for library functions: Library Names. + (line 45) +* --exec option: Options. (line 125) +* --field-separator option: Options. (line 21) +* --file option: Options. (line 25) +* --gen-pot option: Options. (line 147) +* --gen-pot option <1>: String Extraction. (line 6) +* --gen-pot option <2>: String Extraction. (line 6) +* --help option: Options. (line 154) +* --include option: Options. (line 159) +* --lint option: Command Line. (line 20) +* --lint option <1>: Options. (line 184) +* --lint-old option: Options. (line 299) +* --load option: Options. (line 172) +* --no-optimize option: Options. (line 285) +* --non-decimal-data option: Options. (line 209) +* --non-decimal-data option <1>: Nondecimal Data. (line 6) +* --non-decimal-data option, strtonum() function and: Nondecimal Data. + (line 35) +* --optimize option: Options. (line 234) +* --posix option: Options. (line 257) +* --posix option, --traditional option and: Options. (line 272) +* --pretty-print option: Options. (line 223) +* --profile option: Options. (line 245) +* --profile option <1>: Profiling. (line 12) +* --re-interval option: Options. (line 278) +* --sandbox option: Options. (line 290) +* --sandbox option, disabling system() function: I/O Functions. + (line 129) +* --sandbox option, input redirection with getline: Getline. (line 19) +* --sandbox option, output redirection with print, printf: Redirection. + (line 6) +* --source option: Options. (line 117) +* --traditional option: Options. (line 82) +* --traditional option, --posix option and: Options. (line 272) +* --use-lc-numeric option: Options. (line 218) +* --version option: Options. (line 304) +* --with-whiny-user-strftime configuration option: Additional Configuration Options. + (line 37) +* -b option: Options. (line 69) +* -c option: Options. (line 82) +* -C option: Options. (line 89) +* -d option: Options. (line 94) +* -D option: Options. (line 108) +* -e option: Options. (line 117) +* -E option: Options. (line 125) +* -e option <1>: Options. (line 340) +* -f option: Long. (line 12) +* -F option: Options. (line 21) +* -f option <1>: Options. (line 25) +* -F option, -Ft sets FS to TAB: Options. (line 312) +* -F option, command-line: Command Line Field Separator. + (line 6) +* -f option, multiple uses: Options. (line 317) +* -g option: Options. (line 147) +* -h option: Options. (line 154) +* -i option: Options. (line 159) +* -l option: Options. (line 172) +* -l option <1>: Options. (line 184) +* -L option: Options. (line 299) +* -M option: Options. (line 203) +* -n option: Options. (line 209) +* -N option: Options. (line 218) +* -o option: Options. (line 223) +* -O option: Options. (line 234) +* -p option: Options. (line 245) +* -P option: Options. (line 257) +* -r option: Options. (line 278) +* -s option: Options. (line 285) +* -S option: Options. (line 290) +* -v option: Options. (line 32) +* -V option: Options. (line 304) +* -v option <1>: Assignment Options. (line 12) +* -W option: Options. (line 47) +* . (period), regexp operator: Regexp Operators. (line 44) +* .gmo files: Explaining gettext. (line 42) +* .gmo files, specifying directory of: Explaining gettext. (line 54) +* .gmo files, specifying directory of <1>: Programmer i18n. (line 48) +* .mo files, converting from .po: I18N Example. (line 66) +* .po files: Explaining gettext. (line 37) +* .po files <1>: Translator i18n. (line 6) +* .po files, converting to .mo: I18N Example. (line 66) +* .pot files: Explaining gettext. (line 31) +* / (forward slash) to enclose regular expressions: Regexp. (line 10) +* / (forward slash), / operator: Precedence. (line 54) +* / (forward slash), /= operator: Assignment Ops. (line 129) +* / (forward slash), /= operator <1>: Precedence. (line 94) +* / (forward slash), /= operator, vs. /=.../ regexp constant: Assignment Ops. + (line 149) +* / (forward slash), patterns and: Expression Patterns. (line 24) +* /= operator vs. /=.../ regexp constant: Assignment Ops. (line 149) +* /dev/... special files: Special FD. (line 48) +* /dev/fd/N special files (gawk): Special FD. (line 48) +* /inet/... special files (gawk): TCP/IP Networking. (line 6) +* /inet4/... special files (gawk): TCP/IP Networking. (line 6) +* /inet6/... special files (gawk): TCP/IP Networking. (line 6) +* ; (semicolon), AWKPATH variable and: PC Using. (line 9) +* ; (semicolon), separating statements in actions: Statements/Lines. + (line 90) +* ; (semicolon), separating statements in actions <1>: Action Overview. + (line 19) +* ; (semicolon), separating statements in actions <2>: Statements. + (line 10) +* < (left angle bracket), < operator: Comparison Operators. + (line 11) +* < (left angle bracket), < operator <1>: Precedence. (line 64) +* < (left angle bracket), < operator (I/O): Getline/File. (line 6) +* < (left angle bracket), <= operator: Comparison Operators. + (line 11) +* < (left angle bracket), <= operator <1>: Precedence. (line 64) +* = (equals sign), = operator: Assignment Ops. (line 6) +* = (equals sign), == operator: Comparison Operators. + (line 11) +* = (equals sign), == operator <1>: Precedence. (line 64) +* > (right angle bracket), > operator: Comparison Operators. + (line 11) +* > (right angle bracket), > operator <1>: Precedence. (line 64) +* > (right angle bracket), > operator (I/O): Redirection. (line 22) +* > (right angle bracket), >= operator: Comparison Operators. + (line 11) +* > (right angle bracket), >= operator <1>: Precedence. (line 64) +* > (right angle bracket), >> operator (I/O): Redirection. (line 50) +* > (right angle bracket), >> operator (I/O) <1>: Precedence. (line 64) +* ? (question mark), ?: operator: Precedence. (line 91) +* ? (question mark), regexp operator: Regexp Operators. (line 111) +* ? (question mark), regexp operator <1>: GNU Regexp Operators. + (line 62) +* @-notation for indirect function calls: Indirect Calls. (line 47) +* @include directive: Include Files. (line 8) +* @load directive: Loading Shared Libraries. + (line 8) +* [] (square brackets), regexp operator: Regexp Operators. (line 56) +* \ (backslash): Comments. (line 50) +* \ (backslash), as field separator: Command Line Field Separator. + (line 24) +* \ (backslash), continuing lines and: Statements/Lines. (line 19) +* \ (backslash), continuing lines and, comments and: Statements/Lines. + (line 75) +* \ (backslash), continuing lines and, in csh: Statements/Lines. + (line 43) +* \ (backslash), gsub()/gensub()/sub() functions and: Gory Details. + (line 6) +* \ (backslash), in bracket expressions: Bracket Expressions. (line 25) +* \ (backslash), in escape sequences: Escape Sequences. (line 6) +* \ (backslash), in escape sequences <1>: Escape Sequences. (line 103) +* \ (backslash), in escape sequences, POSIX and: Escape Sequences. + (line 108) +* \ (backslash), in regexp constants: Computed Regexps. (line 30) +* \ (backslash), in shell commands: Quoting. (line 48) +* \ (backslash), regexp operator: Regexp Operators. (line 18) +* \ (backslash), \" escape sequence: Escape Sequences. (line 85) +* \ (backslash), \' operator (gawk): GNU Regexp Operators. + (line 59) +* \ (backslash), \/ escape sequence: Escape Sequences. (line 76) +* \ (backslash), \< operator (gawk): GNU Regexp Operators. + (line 33) +* \ (backslash), \> operator (gawk): GNU Regexp Operators. + (line 37) +* \ (backslash), \a escape sequence: Escape Sequences. (line 34) +* \ (backslash), \b escape sequence: Escape Sequences. (line 38) +* \ (backslash), \B operator (gawk): GNU Regexp Operators. + (line 46) +* \ (backslash), \f escape sequence: Escape Sequences. (line 41) +* \ (backslash), \n escape sequence: Escape Sequences. (line 44) +* \ (backslash), \NNN escape sequence: Escape Sequences. (line 56) +* \ (backslash), \r escape sequence: Escape Sequences. (line 47) +* \ (backslash), \s operator (gawk): GNU Regexp Operators. + (line 13) +* \ (backslash), \S operator (gawk): GNU Regexp Operators. + (line 17) +* \ (backslash), \t escape sequence: Escape Sequences. (line 50) +* \ (backslash), \v escape sequence: Escape Sequences. (line 53) +* \ (backslash), \w operator (gawk): GNU Regexp Operators. + (line 22) +* \ (backslash), \W operator (gawk): GNU Regexp Operators. + (line 28) +* \ (backslash), \x escape sequence: Escape Sequences. (line 61) +* \ (backslash), \y operator (gawk): GNU Regexp Operators. + (line 41) +* \ (backslash), \` operator (gawk): GNU Regexp Operators. + (line 57) +* ^ (caret), in bracket expressions: Bracket Expressions. (line 25) +* ^ (caret), in FS: Regexp Field Splitting. + (line 59) +* ^ (caret), regexp operator: Regexp Operators. (line 22) +* ^ (caret), regexp operator <1>: GNU Regexp Operators. + (line 62) +* ^ (caret), ^ operator: Precedence. (line 48) +* ^ (caret), ^= operator: Assignment Ops. (line 129) +* ^ (caret), ^= operator <1>: Precedence. (line 94) +* _ (underscore), C macro: Explaining gettext. (line 71) +* _ (underscore), in names of private variables: Library Names. + (line 29) +* _ (underscore), translatable string: Programmer i18n. (line 69) +* _gr_init() user-defined function: Group Functions. (line 83) +* _ord_init() user-defined function: Ordinal Functions. (line 16) +* _pw_init() user-defined function: Passwd Functions. (line 105) +* {} (braces): Profiling. (line 142) +* {} (braces), actions and: Action Overview. (line 19) +* {} (braces), statements, grouping: Statements. (line 10) +* | (vertical bar): Regexp Operators. (line 70) +* | (vertical bar), | operator (I/O): Getline/Pipe. (line 10) +* | (vertical bar), | operator (I/O) <1>: Redirection. (line 57) +* | (vertical bar), | operator (I/O) <2>: Precedence. (line 64) +* | (vertical bar), |& operator (I/O): Getline/Coprocess. (line 6) +* | (vertical bar), |& operator (I/O) <1>: Redirection. (line 96) +* | (vertical bar), |& operator (I/O) <2>: Precedence. (line 64) +* | (vertical bar), |& operator (I/O) <3>: Two-way I/O. (line 27) +* | (vertical bar), |& operator (I/O), pipes, closing: Close Files And Pipes. + (line 120) +* | (vertical bar), || operator: Boolean Ops. (line 59) +* | (vertical bar), || operator <1>: Precedence. (line 88) +* ~ (tilde), ~ operator: Regexp Usage. (line 19) +* ~ (tilde), ~ operator <1>: Computed Regexps. (line 6) +* ~ (tilde), ~ operator <2>: Case-sensitivity. (line 26) +* ~ (tilde), ~ operator <3>: Regexp Constants. (line 6) +* ~ (tilde), ~ operator <4>: Comparison Operators. + (line 11) +* ~ (tilde), ~ operator <5>: Comparison Operators. + (line 98) +* ~ (tilde), ~ operator <6>: Precedence. (line 79) +* ~ (tilde), ~ operator <7>: Expression Patterns. (line 24) +* accessing fields: Fields. (line 6) +* accessing global variables from extensions: Symbol Table Access. + (line 6) +* account information: Passwd Functions. (line 16) +* account information <1>: Group Functions. (line 6) +* actions: Action Overview. (line 6) +* actions, control statements in: Statements. (line 6) +* actions, default: Very Simple. (line 35) +* actions, empty: Very Simple. (line 40) +* Ada programming language: Glossary. (line 11) +* adding, features to gawk: Adding Code. (line 6) +* adding, fields: Changing Fields. (line 53) +* advanced features, fixed-width data: Constant Size. (line 6) +* advanced features, gawk: Advanced Features. (line 6) +* advanced features, network programming: TCP/IP Networking. (line 6) +* advanced features, nondecimal input data: Nondecimal Data. (line 6) +* advanced features, processes, communicating with: Two-way I/O. + (line 6) +* advanced features, specifying field content: Splitting By Content. + (line 9) +* Aho, Alfred: History. (line 17) +* Aho, Alfred <1>: Contributors. (line 12) +* alarm clock example program: Alarm Program. (line 11) +* alarm.awk program: Alarm Program. (line 31) +* algorithms: Basic High Level. (line 57) +* allocating memory for extensions: Memory Allocation Functions. + (line 6) +* amazing awk assembler (aaa): Glossary. (line 16) +* amazingly workable formatter (awf): Glossary. (line 24) +* ambiguity, syntactic: /= operator vs. /=.../ regexp constant: Assignment Ops. + (line 149) +* ampersand (&), && operator: Boolean Ops. (line 59) +* ampersand (&), && operator <1>: Precedence. (line 85) +* ampersand (&), gsub()/gensub()/sub() functions and: Gory Details. + (line 6) +* anagram.awk program: Anagram Program. (line 21) +* anagrams, finding: Anagram Program. (line 6) +* and: Bitwise Functions. (line 40) +* AND bitwise operation: Bitwise Functions. (line 6) +* and Boolean-logic operator: Boolean Ops. (line 6) +* ANSI: Glossary. (line 34) +* API informational variables: Extension API Informational Variables. + (line 6) +* API version: Extension Versioning. + (line 6) +* arbitrary precision: Arbitrary Precision Arithmetic. + (line 6) +* arbitrary precision integers: Arbitrary Precision Integers. + (line 6) +* archaeologists: Bugs. (line 6) +* arctangent: Numeric Functions. (line 12) +* ARGC/ARGV variables: Auto-set. (line 15) +* ARGC/ARGV variables, command-line arguments: Other Arguments. + (line 15) +* ARGC/ARGV variables, how to use: ARGC and ARGV. (line 6) +* ARGC/ARGV variables, portability and: Executable Scripts. (line 59) +* ARGIND variable: Auto-set. (line 44) +* ARGIND variable, command-line arguments: Other Arguments. (line 15) +* arguments, command-line: Other Arguments. (line 6) +* arguments, command-line <1>: Auto-set. (line 15) +* arguments, command-line <2>: ARGC and ARGV. (line 6) +* arguments, command-line, invoking awk: Command Line. (line 6) +* arguments, in function calls: Function Calls. (line 18) +* arguments, processing: Getopt Function. (line 6) +* ARGV array, indexing into: Other Arguments. (line 15) +* arithmetic operators: Arithmetic Ops. (line 6) +* array manipulation in extensions: Array Manipulation. (line 6) +* array members: Reference to Elements. + (line 6) +* array scanning order, controlling: Controlling Scanning. + (line 14) +* array, number of elements: String Functions. (line 200) +* arrays: Arrays. (line 6) +* arrays of arrays: Arrays of Arrays. (line 6) +* arrays, an example of using: Array Example. (line 6) +* arrays, and IGNORECASE variable: Array Intro. (line 100) +* arrays, as parameters to functions: Pass By Value/Reference. + (line 44) +* arrays, associative: Array Intro. (line 48) +* arrays, associative, library functions and: Library Names. (line 58) +* arrays, deleting entire contents: Delete. (line 39) +* arrays, elements that don't exist: Reference to Elements. + (line 23) +* arrays, elements, assigning values: Assigning Elements. (line 6) +* arrays, elements, deleting: Delete. (line 6) +* arrays, elements, order of access by in operator: Scanning an Array. + (line 48) +* arrays, elements, retrieving number of: String Functions. (line 42) +* arrays, for statement and: Scanning an Array. (line 20) +* arrays, indexing: Array Intro. (line 48) +* arrays, merging into strings: Join Function. (line 6) +* arrays, multidimensional: Multidimensional. (line 10) +* arrays, multidimensional, scanning: Multiscanning. (line 11) +* arrays, numeric subscripts: Numeric Array Subscripts. + (line 6) +* arrays, referencing elements: Reference to Elements. + (line 6) +* arrays, scanning: Scanning an Array. (line 6) +* arrays, sorting: Array Sorting Functions. + (line 6) +* arrays, sorting, and IGNORECASE variable: Array Sorting Functions. + (line 83) +* arrays, sparse: Array Intro. (line 76) +* arrays, subscripts, uninitialized variables as: Uninitialized Subscripts. + (line 6) +* arrays, unassigned elements: Reference to Elements. + (line 18) +* artificial intelligence, gawk and: Distribution contents. + (line 52) +* ASCII: Ordinal Functions. (line 45) +* ASCII <1>: Glossary. (line 196) +* asort: String Functions. (line 42) +* asort <1>: Array Sorting Functions. + (line 6) +* asort() function (gawk), arrays, sorting: Array Sorting Functions. + (line 6) +* asorti: String Functions. (line 42) +* asorti <1>: Array Sorting Functions. + (line 6) +* asorti() function (gawk), arrays, sorting: Array Sorting Functions. + (line 6) +* assert() function (C library): Assert Function. (line 6) +* assert() user-defined function: Assert Function. (line 28) +* assertions: Assert Function. (line 6) +* assign values to variables, in debugger: Viewing And Changing Data. + (line 58) +* assignment operators: Assignment Ops. (line 6) +* assignment operators, evaluation order: Assignment Ops. (line 110) +* assignment operators, lvalues/rvalues: Assignment Ops. (line 31) +* assignments as filenames: Ignoring Assigns. (line 6) +* associative arrays: Array Intro. (line 48) +* asterisk (*), * operator, as multiplication operator: Precedence. + (line 54) +* asterisk (*), * operator, as regexp operator: Regexp Operators. + (line 89) +* asterisk (*), * operator, null strings, matching: String Functions. + (line 537) +* asterisk (*), ** operator: Arithmetic Ops. (line 81) +* asterisk (*), ** operator <1>: Precedence. (line 48) +* asterisk (*), **= operator: Assignment Ops. (line 129) +* asterisk (*), **= operator <1>: Precedence. (line 94) +* asterisk (*), *= operator: Assignment Ops. (line 129) +* asterisk (*), *= operator <1>: Precedence. (line 94) +* atan2: Numeric Functions. (line 12) +* automatic displays, in debugger: Debugger Info. (line 24) +* awf (amazingly workable formatter) program: Glossary. (line 24) +* awk debugging, enabling: Options. (line 108) +* awk language, POSIX version: Assignment Ops. (line 138) +* awk profiling, enabling: Options. (line 245) +* awk programs: Getting Started. (line 12) +* awk programs <1>: Executable Scripts. (line 6) +* awk programs <2>: Two Rules. (line 6) +* awk programs, complex: When. (line 27) +* awk programs, documenting: Comments. (line 6) +* awk programs, documenting <1>: Library Names. (line 6) +* awk programs, examples of: Sample Programs. (line 6) +* awk programs, execution of: Next Statement. (line 16) +* awk programs, internationalizing: I18N Functions. (line 6) +* awk programs, internationalizing <1>: Programmer i18n. (line 6) +* awk programs, lengthy: Long. (line 6) +* awk programs, lengthy, assertions: Assert Function. (line 6) +* awk programs, location of: Options. (line 25) +* awk programs, location of <1>: Options. (line 125) +* awk programs, location of <2>: Options. (line 159) +* awk programs, one-line examples: Very Simple. (line 46) +* awk programs, profiling: Profiling. (line 6) +* awk programs, running: Running gawk. (line 6) +* awk programs, running <1>: Long. (line 6) +* awk programs, running, from shell scripts: One-shot. (line 22) +* awk programs, running, without input files: Read Terminal. (line 16) +* awk programs, shell variables in: Using Shell Variables. + (line 6) +* awk, function of: Getting Started. (line 6) +* awk, gawk and: Preface. (line 21) +* awk, gawk and <1>: This Manual. (line 14) +* awk, history of: History. (line 17) +* awk, implementation issues, pipes: Redirection. (line 129) +* awk, implementations: Other Versions. (line 6) +* awk, implementations, limits: Getline Notes. (line 14) +* awk, invoking: Command Line. (line 6) +* awk, new vs. old: Names. (line 6) +* awk, new vs. old, OFMT variable: Strings And Numbers. (line 56) +* awk, POSIX and: Preface. (line 21) +* awk, POSIX and, See Also POSIX awk: Preface. (line 21) +* awk, regexp constants and: Comparison Operators. + (line 103) +* awk, See Also gawk: Preface. (line 34) +* awk, terms describing: This Manual. (line 6) +* awk, uses for: Preface. (line 21) +* awk, uses for <1>: Getting Started. (line 12) +* awk, uses for <2>: When. (line 6) +* awk, versions of: V7/SVR3.1. (line 6) +* awk, versions of, changes between SVR3.1 and SVR4: SVR4. (line 6) +* awk, versions of, changes between SVR4 and POSIX awk: POSIX. + (line 6) +* awk, versions of, changes between V7 and SVR3.1: V7/SVR3.1. (line 6) +* awk, versions of, See Also Brian Kernighan's awk: BTL. (line 6) +* awk, versions of, See Also Brian Kernighan's awk <1>: Other Versions. + (line 13) +* awka compiler for awk: Other Versions. (line 68) +* AWKLIBPATH environment variable: AWKLIBPATH Variable. (line 6) +* AWKPATH environment variable: AWKPATH Variable. (line 6) +* AWKPATH environment variable <1>: PC Using. (line 9) +* awkprof.out file: Profiling. (line 6) +* awksed.awk program: Simple Sed. (line 25) +* awkvars.out file: Options. (line 94) +* b debugger command (alias for break): Breakpoint Control. (line 11) +* backslash (\): Comments. (line 50) +* backslash (\), as field separator: Command Line Field Separator. + (line 24) +* backslash (\), continuing lines and: Statements/Lines. (line 19) +* backslash (\), continuing lines and, comments and: Statements/Lines. + (line 75) +* backslash (\), continuing lines and, in csh: Statements/Lines. + (line 43) +* backslash (\), gsub()/gensub()/sub() functions and: Gory Details. + (line 6) +* backslash (\), in bracket expressions: Bracket Expressions. (line 25) +* backslash (\), in escape sequences: Escape Sequences. (line 6) +* backslash (\), in escape sequences <1>: Escape Sequences. (line 103) +* backslash (\), in escape sequences, POSIX and: Escape Sequences. + (line 108) +* backslash (\), in regexp constants: Computed Regexps. (line 30) +* backslash (\), in shell commands: Quoting. (line 48) +* backslash (\), regexp operator: Regexp Operators. (line 18) +* backslash (\), \" escape sequence: Escape Sequences. (line 85) +* backslash (\), \' operator (gawk): GNU Regexp Operators. + (line 59) +* backslash (\), \/ escape sequence: Escape Sequences. (line 76) +* backslash (\), \< operator (gawk): GNU Regexp Operators. + (line 33) +* backslash (\), \> operator (gawk): GNU Regexp Operators. + (line 37) +* backslash (\), \a escape sequence: Escape Sequences. (line 34) +* backslash (\), \b escape sequence: Escape Sequences. (line 38) +* backslash (\), \B operator (gawk): GNU Regexp Operators. + (line 46) +* backslash (\), \f escape sequence: Escape Sequences. (line 41) +* backslash (\), \n escape sequence: Escape Sequences. (line 44) +* backslash (\), \NNN escape sequence: Escape Sequences. (line 56) +* backslash (\), \r escape sequence: Escape Sequences. (line 47) +* backslash (\), \s operator (gawk): GNU Regexp Operators. + (line 13) +* backslash (\), \S operator (gawk): GNU Regexp Operators. + (line 17) +* backslash (\), \t escape sequence: Escape Sequences. (line 50) +* backslash (\), \v escape sequence: Escape Sequences. (line 53) +* backslash (\), \w operator (gawk): GNU Regexp Operators. + (line 22) +* backslash (\), \W operator (gawk): GNU Regexp Operators. + (line 28) +* backslash (\), \x escape sequence: Escape Sequences. (line 61) +* backslash (\), \y operator (gawk): GNU Regexp Operators. + (line 41) +* backslash (\), \` operator (gawk): GNU Regexp Operators. + (line 57) +* backtrace debugger command: Execution Stack. (line 13) +* Beebe, Nelson H.F.: Acknowledgments. (line 60) +* Beebe, Nelson H.F. <1>: Other Versions. (line 82) +* BEGIN pattern: Field Separators. (line 44) +* BEGIN pattern <1>: BEGIN/END. (line 6) +* BEGIN pattern <2>: Using BEGIN/END. (line 6) +* BEGIN pattern, and profiling: Profiling. (line 62) +* BEGIN pattern, assert() user-defined function and: Assert Function. + (line 83) +* BEGIN pattern, Boolean patterns and: Expression Patterns. (line 70) +* BEGIN pattern, exit statement and: Exit Statement. (line 12) +* BEGIN pattern, getline and: Getline Notes. (line 19) +* BEGIN pattern, headings, adding: Print Examples. (line 42) +* BEGIN pattern, next/nextfile statements and: I/O And BEGIN/END. + (line 36) +* BEGIN pattern, next/nextfile statements and <1>: Next Statement. + (line 44) +* BEGIN pattern, OFS/ORS variables, assigning values to: Output Separators. + (line 20) +* BEGIN pattern, operators and: Using BEGIN/END. (line 17) +* BEGIN pattern, print statement and: I/O And BEGIN/END. (line 15) +* BEGIN pattern, pwcat program: Passwd Functions. (line 143) +* BEGIN pattern, running awk programs and: Cut Program. (line 63) +* BEGIN pattern, TEXTDOMAIN variable and: Programmer i18n. (line 60) +* BEGINFILE pattern: BEGINFILE/ENDFILE. (line 6) +* BEGINFILE pattern, Boolean patterns and: Expression Patterns. + (line 70) +* beginfile() user-defined function: Filetrans Function. (line 62) +* Bentley, Jon: Glossary. (line 206) +* Benzinger, Michael: Contributors. (line 98) +* Berry, Karl: Acknowledgments. (line 33) +* Berry, Karl <1>: Acknowledgments. (line 75) +* Berry, Karl <2>: Ranges and Locales. (line 74) +* binary input/output: User-modified. (line 15) +* bindtextdomain: I18N Functions. (line 11) +* bindtextdomain <1>: Programmer i18n. (line 48) +* bindtextdomain() function (C library): Explaining gettext. (line 50) +* bindtextdomain() function (gawk), portability and: I18N Portability. + (line 33) +* BINMODE variable: User-modified. (line 15) +* BINMODE variable <1>: PC Using. (line 16) +* bit-manipulation functions: Bitwise Functions. (line 6) +* bits2str() user-defined function: Bitwise Functions. (line 69) +* bitwise AND: Bitwise Functions. (line 40) +* bitwise complement: Bitwise Functions. (line 44) +* bitwise OR: Bitwise Functions. (line 50) +* bitwise XOR: Bitwise Functions. (line 57) +* bitwise, complement: Bitwise Functions. (line 25) +* bitwise, operations: Bitwise Functions. (line 6) +* bitwise, shift: Bitwise Functions. (line 32) +* body, in actions: Statements. (line 10) +* body, in loops: While Statement. (line 14) +* Boolean expressions: Boolean Ops. (line 6) +* Boolean expressions, as patterns: Expression Patterns. (line 39) +* Boolean operators, See Boolean expressions: Boolean Ops. (line 6) +* Bourne shell, quoting rules for: Quoting. (line 18) +* braces ({}): Profiling. (line 142) +* braces ({}), actions and: Action Overview. (line 19) +* braces ({}), statements, grouping: Statements. (line 10) +* bracket expressions: Regexp Operators. (line 56) +* bracket expressions <1>: Bracket Expressions. (line 6) +* bracket expressions, character classes: Bracket Expressions. + (line 40) +* bracket expressions, collating elements: Bracket Expressions. + (line 86) +* bracket expressions, collating symbols: Bracket Expressions. + (line 93) +* bracket expressions, complemented: Regexp Operators. (line 64) +* bracket expressions, equivalence classes: Bracket Expressions. + (line 99) +* bracket expressions, non-ASCII: Bracket Expressions. (line 86) +* bracket expressions, range expressions: Bracket Expressions. + (line 6) +* break debugger command: Breakpoint Control. (line 11) +* break statement: Break Statement. (line 6) +* breakpoint: Debugging Terms. (line 33) +* breakpoint at location, how to delete: Breakpoint Control. (line 36) +* breakpoint commands: Debugger Execution Control. + (line 10) +* breakpoint condition: Breakpoint Control. (line 54) +* breakpoint, delete by number: Breakpoint Control. (line 64) +* breakpoint, how to disable or enable: Breakpoint Control. (line 69) +* breakpoint, setting: Breakpoint Control. (line 11) +* Brennan, Michael: Foreword3. (line 84) +* Brennan, Michael <1>: Foreword4. (line 33) +* Brennan, Michael <2>: Acknowledgments. (line 79) +* Brennan, Michael <3>: Delete. (line 56) +* Brennan, Michael <4>: Simple Sed. (line 25) +* Brennan, Michael <5>: Other Versions. (line 6) +* Brennan, Michael <6>: Other Versions. (line 48) +* Brian Kernighan's awk: When. (line 21) +* Brian Kernighan's awk <1>: Escape Sequences. (line 112) +* Brian Kernighan's awk <2>: GNU Regexp Operators. + (line 85) +* Brian Kernighan's awk <3>: Regexp Field Splitting. + (line 67) +* Brian Kernighan's awk <4>: Getline/Pipe. (line 62) +* Brian Kernighan's awk <5>: Concatenation. (line 36) +* Brian Kernighan's awk <6>: I/O And BEGIN/END. (line 15) +* Brian Kernighan's awk <7>: Break Statement. (line 51) +* Brian Kernighan's awk <8>: Continue Statement. (line 44) +* Brian Kernighan's awk <9>: Nextfile Statement. (line 47) +* Brian Kernighan's awk <10>: Delete. (line 51) +* Brian Kernighan's awk <11>: String Functions. (line 493) +* Brian Kernighan's awk <12>: Gory Details. (line 19) +* Brian Kernighan's awk <13>: I/O Functions. (line 43) +* Brian Kernighan's awk, extensions: BTL. (line 6) +* Brian Kernighan's awk, source code: Other Versions. (line 13) +* Brini, Davide: Signature Program. (line 6) +* Brink, Jeroen: DOS Quoting. (line 10) +* Broder, Alan J.: Contributors. (line 89) +* Brown, Martin: Contributors. (line 83) +* BSD-based operating systems: Glossary. (line 748) +* bt debugger command (alias for backtrace): Execution Stack. (line 13) +* Buening, Andreas: Acknowledgments. (line 60) +* Buening, Andreas <1>: Contributors. (line 93) +* Buening, Andreas <2>: Maintainers. (line 14) +* buffering, input/output: I/O Functions. (line 166) +* buffering, input/output <1>: Two-way I/O. (line 53) +* buffering, interactive vs. noninteractive: I/O Functions. (line 76) +* buffers, flushing: I/O Functions. (line 32) +* buffers, flushing <1>: I/O Functions. (line 166) +* buffers, operators for: GNU Regexp Operators. + (line 51) +* bug reports, email address, bug-gawk@gnu.org: Bug address. (line 22) +* bug-gawk@gnu.org bug reporting address: Bug address. (line 22) +* built-in functions: Functions. (line 6) +* built-in functions, evaluation order: Calling Built-in. (line 30) +* BusyBox Awk: Other Versions. (line 92) +* c.e., See common extensions: Conventions. (line 51) +* call by reference: Pass By Value/Reference. + (line 44) +* call by value: Pass By Value/Reference. + (line 15) +* call stack, display in debugger: Execution Stack. (line 13) +* caret (^), in bracket expressions: Bracket Expressions. (line 25) +* caret (^), regexp operator: Regexp Operators. (line 22) +* caret (^), regexp operator <1>: GNU Regexp Operators. + (line 62) +* caret (^), ^ operator: Precedence. (line 48) +* caret (^), ^= operator: Assignment Ops. (line 129) +* caret (^), ^= operator <1>: Precedence. (line 94) +* case keyword: Switch Statement. (line 6) +* case sensitivity, and regexps: User-modified. (line 76) +* case sensitivity, and string comparisons: User-modified. (line 76) +* case sensitivity, array indices and: Array Intro. (line 100) +* case sensitivity, converting case: String Functions. (line 523) +* case sensitivity, example programs: Library Functions. (line 53) +* case sensitivity, gawk: Case-sensitivity. (line 26) +* case sensitivity, regexps and: Case-sensitivity. (line 6) +* CGI, awk scripts for: Options. (line 125) +* character classes, See bracket expressions: Regexp Operators. + (line 56) +* character lists in regular expression: Bracket Expressions. (line 6) +* character lists, See bracket expressions: Regexp Operators. (line 56) +* character sets (machine character encodings): Ordinal Functions. + (line 45) +* character sets (machine character encodings) <1>: Glossary. (line 196) +* character sets, See Also bracket expressions: Regexp Operators. + (line 56) +* characters, counting: Wc Program. (line 6) +* characters, transliterating: Translate Program. (line 6) +* characters, values of as numbers: Ordinal Functions. (line 6) +* Chassell, Robert J.: Acknowledgments. (line 33) +* chdir() extension function: Extension Sample File Functions. + (line 12) +* chem utility: Glossary. (line 206) +* chr() extension function: Extension Sample Ord. + (line 15) +* chr() user-defined function: Ordinal Functions. (line 16) +* clear debugger command: Breakpoint Control. (line 36) +* Cliff random numbers: Cliff Random Function. + (line 6) +* cliff_rand() user-defined function: Cliff Random Function. + (line 12) +* close: Close Files And Pipes. + (line 18) +* close <1>: I/O Functions. (line 10) +* close file or coprocess: I/O Functions. (line 10) +* close() function, portability: Close Files And Pipes. + (line 81) +* close() function, return value: Close Files And Pipes. + (line 132) +* close() function, two-way pipes and: Two-way I/O. (line 60) +* Close, Diane: Manual History. (line 34) +* Close, Diane <1>: Contributors. (line 21) +* Collado, Manuel: Acknowledgments. (line 60) +* collating elements: Bracket Expressions. (line 86) +* collating symbols: Bracket Expressions. (line 93) +* Colombo, Antonio: Acknowledgments. (line 60) +* Colombo, Antonio <1>: Contributors. (line 141) +* columns, aligning: Print Examples. (line 69) +* columns, cutting: Cut Program. (line 6) +* comma (,), in range patterns: Ranges. (line 6) +* command completion, in debugger: Readline Support. (line 6) +* command line, arguments: Other Arguments. (line 6) +* command line, arguments <1>: Auto-set. (line 15) +* command line, arguments <2>: ARGC and ARGV. (line 6) +* command line, directories on: Command-line directories. + (line 6) +* command line, formats: Running gawk. (line 12) +* command line, FS on, setting: Command Line Field Separator. + (line 6) +* command line, invoking awk from: Command Line. (line 6) +* command line, option -f: Long. (line 12) +* command line, options: Options. (line 6) +* command line, options, end of: Options. (line 55) +* command line, variables, assigning on: Assignment Options. (line 6) +* command-line options, processing: Getopt Function. (line 6) +* command-line options, string extraction: String Extraction. (line 6) +* commands debugger command: Debugger Execution Control. + (line 10) +* commands to execute at breakpoint: Debugger Execution Control. + (line 10) +* commenting: Comments. (line 6) +* commenting, backslash continuation and: Statements/Lines. (line 75) +* common extensions, ** operator: Arithmetic Ops. (line 30) +* common extensions, **= operator: Assignment Ops. (line 138) +* common extensions, /dev/stderr special file: Special FD. (line 48) +* common extensions, /dev/stdin special file: Special FD. (line 48) +* common extensions, /dev/stdout special file: Special FD. (line 48) +* common extensions, BINMODE variable: PC Using. (line 16) +* common extensions, delete to delete entire arrays: Delete. (line 39) +* common extensions, func keyword: Definition Syntax. (line 99) +* common extensions, length() applied to an array: String Functions. + (line 200) +* common extensions, RS as a regexp: gawk split records. (line 6) +* common extensions, single character fields: Single Character Fields. + (line 6) +* common extensions, \x escape sequence: Escape Sequences. (line 61) +* comp.lang.awk newsgroup: Usenet. (line 11) +* comparison expressions: Typing and Comparison. + (line 9) +* comparison expressions, as patterns: Expression Patterns. (line 14) +* comparison expressions, string vs. regexp: Comparison Operators. + (line 79) +* compatibility mode (gawk), extensions: POSIX/GNU. (line 6) +* compatibility mode (gawk), file names: Special Caveats. (line 9) +* compatibility mode (gawk), hexadecimal numbers: Nondecimal-numbers. + (line 59) +* compatibility mode (gawk), octal numbers: Nondecimal-numbers. + (line 59) +* compatibility mode (gawk), specifying: Options. (line 82) +* compiled programs: Basic High Level. (line 13) +* compiled programs <1>: Glossary. (line 218) +* compiling gawk for Cygwin: Cygwin. (line 6) +* compiling gawk for MS-Windows: PC Compiling. (line 11) +* compiling gawk for VMS: VMS Compilation. (line 6) +* compl: Bitwise Functions. (line 44) +* complement, bitwise: Bitwise Functions. (line 25) +* compound statements, control statements and: Statements. (line 10) +* concatenating: Concatenation. (line 9) +* condition debugger command: Breakpoint Control. (line 54) +* conditional expressions: Conditional Exp. (line 6) +* configuration option, --disable-extensions: Additional Configuration Options. + (line 9) +* configuration option, --disable-lint: Additional Configuration Options. + (line 15) +* configuration option, --disable-nls: Additional Configuration Options. + (line 32) +* configuration option, --with-whiny-user-strftime: Additional Configuration Options. + (line 37) +* configuration options, gawk: Additional Configuration Options. + (line 6) +* constant regexps: Regexp Usage. (line 57) +* constants, nondecimal: Nondecimal Data. (line 6) +* constants, numeric: Scalar Constants. (line 6) +* constants, types of: Constants. (line 6) +* continue program, in debugger: Debugger Execution Control. + (line 33) +* continue statement: Continue Statement. (line 6) +* control statements: Statements. (line 6) +* controlling array scanning order: Controlling Scanning. + (line 14) +* convert string to lower case: String Functions. (line 524) +* convert string to number: String Functions. (line 391) +* convert string to upper case: String Functions. (line 530) +* converting integer array subscripts: Numeric Array Subscripts. + (line 31) +* converting, dates to timestamps: Time Functions. (line 76) +* converting, numbers to strings: Strings And Numbers. (line 6) +* converting, numbers to strings <1>: Bitwise Functions. (line 108) +* converting, strings to numbers: Strings And Numbers. (line 6) +* converting, strings to numbers <1>: Bitwise Functions. (line 108) +* CONVFMT variable: Strings And Numbers. (line 29) +* CONVFMT variable <1>: User-modified. (line 30) +* CONVFMT variable, and array subscripts: Numeric Array Subscripts. + (line 6) +* cookie: Glossary. (line 257) +* coprocesses: Redirection. (line 96) +* coprocesses <1>: Two-way I/O. (line 27) +* coprocesses, closing: Close Files And Pipes. + (line 6) +* coprocesses, getline from: Getline/Coprocess. (line 6) +* cos: Numeric Functions. (line 16) +* cosine: Numeric Functions. (line 16) +* counting: Wc Program. (line 6) +* csh utility: Statements/Lines. (line 43) +* csh utility, POSIXLY_CORRECT environment variable: Options. (line 358) +* csh utility, |& operator, comparison with: Two-way I/O. (line 27) +* ctime() user-defined function: Function Example. (line 74) +* currency symbols, localization: Explaining gettext. (line 104) +* current system time: Time Functions. (line 66) +* custom.h file: Configuration Philosophy. + (line 30) +* customized input parser: Input Parsers. (line 6) +* customized output wrapper: Output Wrappers. (line 6) +* customized two-way processor: Two-way processors. (line 6) +* cut utility: Cut Program. (line 6) +* cut utility <1>: Cut Program. (line 6) +* cut.awk program: Cut Program. (line 45) +* d debugger command (alias for delete): Breakpoint Control. (line 64) +* d.c., See dark corner: Conventions. (line 42) +* dark corner: Conventions. (line 42) +* dark corner <1>: Glossary. (line 268) +* dark corner, "0" is actually true: Truth Values. (line 24) +* dark corner, /= operator vs. /=.../ regexp constant: Assignment Ops. + (line 149) +* dark corner, array subscripts: Uninitialized Subscripts. + (line 43) +* dark corner, break statement: Break Statement. (line 51) +* dark corner, close() function: Close Files And Pipes. + (line 132) +* dark corner, command-line arguments: Assignment Options. (line 43) +* dark corner, continue statement: Continue Statement. (line 44) +* dark corner, CONVFMT variable: Strings And Numbers. (line 39) +* dark corner, escape sequences: Other Arguments. (line 38) +* dark corner, escape sequences, for metacharacters: Escape Sequences. + (line 144) +* dark corner, exit statement: Exit Statement. (line 30) +* dark corner, field separators: Full Line Fields. (line 22) +* dark corner, FILENAME variable: Getline Notes. (line 19) +* dark corner, FILENAME variable <1>: Auto-set. (line 108) +* dark corner, FNR/NR variables: Auto-set. (line 357) +* dark corner, format-control characters: Control Letters. (line 18) +* dark corner, format-control characters <1>: Control Letters. + (line 93) +* dark corner, FS as null string: Single Character Fields. + (line 20) +* dark corner, input files: awk split records. (line 110) +* dark corner, invoking awk: Command Line. (line 16) +* dark corner, length() function: String Functions. (line 186) +* dark corner, locale's decimal point character: Locale influences conversions. + (line 17) +* dark corner, multiline records: Multiple Line. (line 35) +* dark corner, NF variable, decrementing: Changing Fields. (line 107) +* dark corner, OFMT variable: OFMT. (line 27) +* dark corner, regexp as second argument to index(): String Functions. + (line 164) +* dark corner, regexp constants: Using Constant Regexps. + (line 6) +* dark corner, regexp constants, /= operator and: Assignment Ops. + (line 149) +* dark corner, regexp constants, as arguments to user-defined functions: Using Constant Regexps. + (line 43) +* dark corner, split() function: String Functions. (line 361) +* dark corner, strings, storing: gawk split records. (line 82) +* dark corner, value of ARGV[0]: Auto-set. (line 39) +* dark corner, ^, in FS: Regexp Field Splitting. + (line 59) +* data, fixed-width: Constant Size. (line 6) +* data-driven languages: Basic High Level. (line 74) +* database, group, reading: Group Functions. (line 6) +* database, users, reading: Passwd Functions. (line 6) +* date utility, GNU: Time Functions. (line 17) +* date utility, POSIX: Time Functions. (line 253) +* dates, converting to timestamps: Time Functions. (line 76) +* dates, information related to, localization: Explaining gettext. + (line 112) +* Davies, Stephen: Acknowledgments. (line 60) +* Davies, Stephen <1>: Contributors. (line 75) +* Day, Robert P.J.: Acknowledgments. (line 79) +* dcgettext: I18N Functions. (line 21) +* dcgettext <1>: Programmer i18n. (line 20) +* dcgettext() function (gawk), portability and: I18N Portability. + (line 33) +* dcngettext: I18N Functions. (line 27) +* dcngettext <1>: Programmer i18n. (line 37) +* dcngettext() function (gawk), portability and: I18N Portability. + (line 33) +* deadlocks: Two-way I/O. (line 53) +* debugger commands, b (break): Breakpoint Control. (line 11) +* debugger commands, backtrace: Execution Stack. (line 13) +* debugger commands, break: Breakpoint Control. (line 11) +* debugger commands, bt (backtrace): Execution Stack. (line 13) +* debugger commands, c (continue): Debugger Execution Control. + (line 33) +* debugger commands, clear: Breakpoint Control. (line 36) +* debugger commands, commands: Debugger Execution Control. + (line 10) +* debugger commands, condition: Breakpoint Control. (line 54) +* debugger commands, continue: Debugger Execution Control. + (line 33) +* debugger commands, d (delete): Breakpoint Control. (line 64) +* debugger commands, delete: Breakpoint Control. (line 64) +* debugger commands, disable: Breakpoint Control. (line 69) +* debugger commands, display: Viewing And Changing Data. + (line 8) +* debugger commands, down: Execution Stack. (line 23) +* debugger commands, dump: Miscellaneous Debugger Commands. + (line 9) +* debugger commands, e (enable): Breakpoint Control. (line 73) +* debugger commands, enable: Breakpoint Control. (line 73) +* debugger commands, end: Debugger Execution Control. + (line 10) +* debugger commands, eval: Viewing And Changing Data. + (line 23) +* debugger commands, f (frame): Execution Stack. (line 27) +* debugger commands, finish: Debugger Execution Control. + (line 39) +* debugger commands, frame: Execution Stack. (line 27) +* debugger commands, h (help): Miscellaneous Debugger Commands. + (line 69) +* debugger commands, help: Miscellaneous Debugger Commands. + (line 69) +* debugger commands, i (info): Debugger Info. (line 13) +* debugger commands, ignore: Breakpoint Control. (line 87) +* debugger commands, info: Debugger Info. (line 13) +* debugger commands, l (list): Miscellaneous Debugger Commands. + (line 75) +* debugger commands, list: Miscellaneous Debugger Commands. + (line 75) +* debugger commands, n (next): Debugger Execution Control. + (line 43) +* debugger commands, next: Debugger Execution Control. + (line 43) +* debugger commands, nexti: Debugger Execution Control. + (line 49) +* debugger commands, ni (nexti): Debugger Execution Control. + (line 49) +* debugger commands, o (option): Debugger Info. (line 57) +* debugger commands, option: Debugger Info. (line 57) +* debugger commands, p (print): Viewing And Changing Data. + (line 35) +* debugger commands, print: Viewing And Changing Data. + (line 35) +* debugger commands, printf: Viewing And Changing Data. + (line 53) +* debugger commands, q (quit): Miscellaneous Debugger Commands. + (line 102) +* debugger commands, quit: Miscellaneous Debugger Commands. + (line 102) +* debugger commands, r (run): Debugger Execution Control. + (line 62) +* debugger commands, return: Debugger Execution Control. + (line 54) +* debugger commands, run: Debugger Execution Control. + (line 62) +* debugger commands, s (step): Debugger Execution Control. + (line 68) +* debugger commands, set: Viewing And Changing Data. + (line 58) +* debugger commands, si (stepi): Debugger Execution Control. + (line 75) +* debugger commands, silent: Debugger Execution Control. + (line 10) +* debugger commands, step: Debugger Execution Control. + (line 68) +* debugger commands, stepi: Debugger Execution Control. + (line 75) +* debugger commands, t (tbreak): Breakpoint Control. (line 90) +* debugger commands, tbreak: Breakpoint Control. (line 90) +* debugger commands, trace: Miscellaneous Debugger Commands. + (line 110) +* debugger commands, u (until): Debugger Execution Control. + (line 82) +* debugger commands, undisplay: Viewing And Changing Data. + (line 79) +* debugger commands, until: Debugger Execution Control. + (line 82) +* debugger commands, unwatch: Viewing And Changing Data. + (line 83) +* debugger commands, up: Execution Stack. (line 36) +* debugger commands, w (watch): Viewing And Changing Data. + (line 66) +* debugger commands, watch: Viewing And Changing Data. + (line 66) +* debugger commands, where (backtrace): Execution Stack. (line 13) +* debugger default list amount: Debugger Info. (line 69) +* debugger history file: Debugger Info. (line 81) +* debugger history size: Debugger Info. (line 65) +* debugger options: Debugger Info. (line 57) +* debugger prompt: Debugger Info. (line 78) +* debugger, how to start: Debugger Invocation. (line 6) +* debugger, read commands from a file: Debugger Info. (line 97) +* debugging awk programs: Debugger. (line 6) +* debugging gawk, bug reports: Bugs. (line 9) +* decimal point character, locale specific: Options. (line 269) +* decrement operators: Increment Ops. (line 35) +* default keyword: Switch Statement. (line 6) +* Deifik, Scott: Acknowledgments. (line 60) +* Deifik, Scott <1>: Contributors. (line 54) +* Deifik, Scott <2>: Maintainers. (line 14) +* delete ARRAY: Delete. (line 39) +* delete breakpoint at location: Breakpoint Control. (line 36) +* delete breakpoint by number: Breakpoint Control. (line 64) +* delete debugger command: Breakpoint Control. (line 64) +* delete statement: Delete. (line 6) +* delete watchpoint: Viewing And Changing Data. + (line 83) +* deleting elements in arrays: Delete. (line 6) +* deleting entire arrays: Delete. (line 39) +* Demaille, Akim: Acknowledgments. (line 60) +* describe call stack frame, in debugger: Debugger Info. (line 27) +* differences between gawk and awk: String Functions. (line 200) +* differences in awk and gawk, ARGC/ARGV variables: ARGC and ARGV. + (line 89) +* differences in awk and gawk, ARGIND variable: Auto-set. (line 44) +* differences in awk and gawk, array elements, deleting: Delete. + (line 39) +* differences in awk and gawk, AWKLIBPATH environment variable: AWKLIBPATH Variable. + (line 6) +* differences in awk and gawk, AWKPATH environment variable: AWKPATH Variable. + (line 6) +* differences in awk and gawk, BEGIN/END patterns: I/O And BEGIN/END. + (line 15) +* differences in awk and gawk, BEGINFILE/ENDFILE patterns: BEGINFILE/ENDFILE. + (line 6) +* differences in awk and gawk, BINMODE variable: User-modified. + (line 15) +* differences in awk and gawk, BINMODE variable <1>: PC Using. + (line 16) +* differences in awk and gawk, close() function: Close Files And Pipes. + (line 81) +* differences in awk and gawk, close() function <1>: Close Files And Pipes. + (line 132) +* differences in awk and gawk, command-line directories: Command-line directories. + (line 6) +* differences in awk and gawk, ERRNO variable: Auto-set. (line 87) +* differences in awk and gawk, error messages: Special FD. (line 19) +* differences in awk and gawk, FIELDWIDTHS variable: User-modified. + (line 37) +* differences in awk and gawk, FPAT variable: User-modified. (line 43) +* differences in awk and gawk, FUNCTAB variable: Auto-set. (line 134) +* differences in awk and gawk, function arguments (gawk): Calling Built-in. + (line 16) +* differences in awk and gawk, getline command: Getline. (line 19) +* differences in awk and gawk, IGNORECASE variable: User-modified. + (line 76) +* differences in awk and gawk, implementation limitations: Getline Notes. + (line 14) +* differences in awk and gawk, implementation limitations <1>: Redirection. + (line 129) +* differences in awk and gawk, indirect function calls: Indirect Calls. + (line 6) +* differences in awk and gawk, input/output operators: Getline/Coprocess. + (line 6) +* differences in awk and gawk, input/output operators <1>: Redirection. + (line 96) +* differences in awk and gawk, line continuations: Conditional Exp. + (line 34) +* differences in awk and gawk, LINT variable: User-modified. (line 87) +* differences in awk and gawk, match() function: String Functions. + (line 262) +* differences in awk and gawk, print/printf statements: Format Modifiers. + (line 13) +* differences in awk and gawk, PROCINFO array: Auto-set. (line 148) +* differences in awk and gawk, read timeouts: Read Timeout. (line 6) +* differences in awk and gawk, record separators: awk split records. + (line 124) +* differences in awk and gawk, regexp constants: Using Constant Regexps. + (line 43) +* differences in awk and gawk, regular expressions: Case-sensitivity. + (line 26) +* differences in awk and gawk, retrying input: Retrying Input. + (line 6) +* differences in awk and gawk, RS/RT variables: gawk split records. + (line 58) +* differences in awk and gawk, RT variable: Auto-set. (line 295) +* differences in awk and gawk, single-character fields: Single Character Fields. + (line 6) +* differences in awk and gawk, split() function: String Functions. + (line 348) +* differences in awk and gawk, strings: Scalar Constants. (line 20) +* differences in awk and gawk, strings, storing: gawk split records. + (line 76) +* differences in awk and gawk, SYMTAB variable: Auto-set. (line 299) +* differences in awk and gawk, TEXTDOMAIN variable: User-modified. + (line 152) +* differences in awk and gawk, trunc-mod operation: Arithmetic Ops. + (line 66) +* directories, command-line: Command-line directories. + (line 6) +* directories, searching: Programs Exercises. (line 70) +* directories, searching for loadable extensions: AWKLIBPATH Variable. + (line 6) +* directories, searching for source files: AWKPATH Variable. (line 6) +* disable breakpoint: Breakpoint Control. (line 69) +* disable debugger command: Breakpoint Control. (line 69) +* display debugger command: Viewing And Changing Data. + (line 8) +* display debugger options: Debugger Info. (line 57) +* division: Arithmetic Ops. (line 44) +* do-while statement: Do Statement. (line 6) +* do-while statement, use of regexps in: Regexp Usage. (line 19) +* documentation, of awk programs: Library Names. (line 6) +* documentation, online: Manual History. (line 11) +* documents, searching: Dupword Program. (line 6) +* dollar sign ($), $ field operator: Fields. (line 19) +* dollar sign ($), $ field operator <1>: Precedence. (line 42) +* dollar sign ($), incrementing fields and arrays: Increment Ops. + (line 30) +* dollar sign ($), regexp operator: Regexp Operators. (line 35) +* double quote ("), in regexp constants: Computed Regexps. (line 30) +* double quote ("), in shell commands: Quoting. (line 54) +* down debugger command: Execution Stack. (line 23) +* Drepper, Ulrich: Acknowledgments. (line 52) +* Duman, Patrice: Acknowledgments. (line 75) +* dump all variables of a program: Options. (line 94) +* dump debugger command: Miscellaneous Debugger Commands. + (line 9) +* dupword.awk program: Dupword Program. (line 31) +* dynamic profiling: Profiling. (line 177) +* dynamically loaded extensions: Dynamic Extensions. (line 6) +* e debugger command (alias for enable): Breakpoint Control. (line 73) +* EBCDIC: Ordinal Functions. (line 45) +* effective group ID of gawk user: Auto-set. (line 153) +* effective user ID of gawk user: Auto-set. (line 161) +* egrep utility: Bracket Expressions. (line 34) +* egrep utility <1>: Egrep Program. (line 6) +* egrep.awk program: Egrep Program. (line 53) +* elements in arrays, assigning values: Assigning Elements. (line 6) +* elements in arrays, deleting: Delete. (line 6) +* elements in arrays, order of access by in operator: Scanning an Array. + (line 48) +* elements in arrays, scanning: Scanning an Array. (line 6) +* elements of arrays: Reference to Elements. + (line 6) +* email address for bug reports, bug-gawk@gnu.org: Bug address. + (line 22) +* empty array elements: Reference to Elements. + (line 18) +* empty pattern: Empty. (line 6) +* empty strings: awk split records. (line 114) +* empty strings, See null strings: Regexp Field Splitting. + (line 43) +* EMRED: TCP/IP Networking. (line 6) +* enable breakpoint: Breakpoint Control. (line 73) +* enable debugger command: Breakpoint Control. (line 73) +* end debugger command: Debugger Execution Control. + (line 10) +* END pattern: BEGIN/END. (line 6) +* END pattern <1>: Using BEGIN/END. (line 6) +* END pattern, and profiling: Profiling. (line 62) +* END pattern, assert() user-defined function and: Assert Function. + (line 75) +* END pattern, Boolean patterns and: Expression Patterns. (line 70) +* END pattern, exit statement and: Exit Statement. (line 12) +* END pattern, next/nextfile statements and: I/O And BEGIN/END. + (line 36) +* END pattern, next/nextfile statements and <1>: Next Statement. + (line 44) +* END pattern, operators and: Using BEGIN/END. (line 17) +* END pattern, print statement and: I/O And BEGIN/END. (line 15) +* ENDFILE pattern: BEGINFILE/ENDFILE. (line 6) +* ENDFILE pattern, Boolean patterns and: Expression Patterns. (line 70) +* endfile() user-defined function: Filetrans Function. (line 62) +* endgrent() function (C library): Group Functions. (line 213) +* endgrent() user-defined function: Group Functions. (line 216) +* endpwent() function (C library): Passwd Functions. (line 208) +* endpwent() user-defined function: Passwd Functions. (line 211) +* English, Steve: Advanced Features. (line 6) +* ENVIRON array: Auto-set. (line 59) +* environment variables used by gawk: Environment Variables. + (line 6) +* environment variables, in ENVIRON array: Auto-set. (line 59) +* epoch, definition of: Glossary. (line 312) +* equals sign (=), = operator: Assignment Ops. (line 6) +* equals sign (=), == operator: Comparison Operators. + (line 11) +* equals sign (=), == operator <1>: Precedence. (line 64) +* EREs (Extended Regular Expressions): Bracket Expressions. (line 34) +* ERRNO variable: Auto-set. (line 87) +* ERRNO variable <1>: TCP/IP Networking. (line 54) +* ERRNO variable, with BEGINFILE pattern: BEGINFILE/ENDFILE. (line 26) +* ERRNO variable, with close() function: Close Files And Pipes. + (line 140) +* ERRNO variable, with getline command: Getline. (line 19) +* error handling: Special FD. (line 19) +* error handling, ERRNO variable and: Auto-set. (line 87) +* error output: Special FD. (line 6) +* escape processing, gsub()/gensub()/sub() functions: Gory Details. + (line 6) +* escape sequences, in strings: Escape Sequences. (line 6) +* eval debugger command: Viewing And Changing Data. + (line 23) +* evaluate expressions, in debugger: Viewing And Changing Data. + (line 23) +* evaluation order: Increment Ops. (line 60) +* evaluation order, concatenation: Concatenation. (line 41) +* evaluation order, functions: Calling Built-in. (line 30) +* examining fields: Fields. (line 6) +* exclamation point (!), ! operator: Boolean Ops. (line 69) +* exclamation point (!), ! operator <1>: Precedence. (line 51) +* exclamation point (!), ! operator <2>: Egrep Program. (line 174) +* exclamation point (!), != operator: Comparison Operators. + (line 11) +* exclamation point (!), != operator <1>: Precedence. (line 64) +* exclamation point (!), !~ operator: Regexp Usage. (line 19) +* exclamation point (!), !~ operator <1>: Computed Regexps. (line 6) +* exclamation point (!), !~ operator <2>: Case-sensitivity. (line 26) +* exclamation point (!), !~ operator <3>: Regexp Constants. (line 6) +* exclamation point (!), !~ operator <4>: Comparison Operators. + (line 11) +* exclamation point (!), !~ operator <5>: Comparison Operators. + (line 98) +* exclamation point (!), !~ operator <6>: Precedence. (line 79) +* exclamation point (!), !~ operator <7>: Expression Patterns. + (line 24) +* exit debugger command: Miscellaneous Debugger Commands. + (line 66) +* exit statement: Exit Statement. (line 6) +* exit status, of gawk: Exit Status. (line 6) +* exit status, of VMS: VMS Running. (line 28) +* exit the debugger: Miscellaneous Debugger Commands. + (line 66) +* exit the debugger <1>: Miscellaneous Debugger Commands. + (line 102) +* exp: Numeric Functions. (line 19) +* expand utility: Very Simple. (line 73) +* Expat XML parser library: gawkextlib. (line 37) +* exponent: Numeric Functions. (line 19) +* expressions: Expressions. (line 6) +* expressions, as patterns: Expression Patterns. (line 6) +* expressions, assignment: Assignment Ops. (line 6) +* expressions, Boolean: Boolean Ops. (line 6) +* expressions, comparison: Typing and Comparison. + (line 9) +* expressions, conditional: Conditional Exp. (line 6) +* expressions, matching, See comparison expressions: Typing and Comparison. + (line 9) +* expressions, selecting: Conditional Exp. (line 6) +* Extended Regular Expressions (EREs): Bracket Expressions. (line 34) +* extension API: Extension API Description. + (line 6) +* extension API informational variables: Extension API Informational Variables. + (line 6) +* extension API version: Extension Versioning. + (line 6) +* extension API, version number: Auto-set. (line 246) +* extension example: Extension Example. (line 6) +* extension registration: Registration Functions. + (line 6) +* extension search path: Finding Extensions. (line 6) +* extensions distributed with gawk: Extension Samples. (line 6) +* extensions, allocating memory: Memory Allocation Functions. + (line 6) +* extensions, Brian Kernighan's awk: BTL. (line 6) +* extensions, Brian Kernighan's awk <1>: Common Extensions. (line 6) +* extensions, common, ** operator: Arithmetic Ops. (line 30) +* extensions, common, **= operator: Assignment Ops. (line 138) +* extensions, common, /dev/stderr special file: Special FD. (line 48) +* extensions, common, /dev/stdin special file: Special FD. (line 48) +* extensions, common, /dev/stdout special file: Special FD. (line 48) +* extensions, common, BINMODE variable: PC Using. (line 16) +* extensions, common, delete to delete entire arrays: Delete. (line 39) +* extensions, common, fflush() function: I/O Functions. (line 43) +* extensions, common, func keyword: Definition Syntax. (line 99) +* extensions, common, length() applied to an array: String Functions. + (line 200) +* extensions, common, RS as a regexp: gawk split records. (line 6) +* extensions, common, single character fields: Single Character Fields. + (line 6) +* extensions, common, \x escape sequence: Escape Sequences. (line 61) +* extensions, in gawk, not in POSIX awk: POSIX/GNU. (line 6) +* extensions, loading, @load directive: Loading Shared Libraries. + (line 8) +* extensions, mawk: Common Extensions. (line 6) +* extensions, where to find: gawkextlib. (line 6) +* extract.awk program: Extract Program. (line 79) +* extraction, of marked strings (internationalization): String Extraction. + (line 6) +* f debugger command (alias for frame): Execution Stack. (line 27) +* false, logical: Truth Values. (line 6) +* FDL (Free Documentation License): GNU Free Documentation License. + (line 8) +* features, adding to gawk: Adding Code. (line 6) +* features, deprecated: Obsolete. (line 6) +* features, undocumented: Undocumented. (line 6) +* Fenlason, Jay: History. (line 30) +* Fenlason, Jay <1>: Contributors. (line 19) +* fflush: I/O Functions. (line 28) +* field numbers: Nonconstant Fields. (line 6) +* field operator $: Fields. (line 19) +* field operators, dollar sign as: Fields. (line 19) +* field separator, in multiline records: Multiple Line. (line 41) +* field separator, on command line: Command Line Field Separator. + (line 6) +* field separator, POSIX and: Full Line Fields. (line 16) +* field separators: Field Separators. (line 15) +* field separators <1>: User-modified. (line 50) +* field separators <2>: User-modified. (line 113) +* field separators, choice of: Field Separators. (line 50) +* field separators, FIELDWIDTHS variable and: User-modified. (line 37) +* field separators, FPAT variable and: User-modified. (line 43) +* field separators, regular expressions as: Field Separators. (line 50) +* field separators, regular expressions as <1>: Regexp Field Splitting. + (line 6) +* field separators, See Also OFS: Changing Fields. (line 64) +* field separators, spaces as: Cut Program. (line 103) +* fields: Reading Files. (line 14) +* fields <1>: Fields. (line 6) +* fields <2>: Basic High Level. (line 62) +* fields, adding: Changing Fields. (line 53) +* fields, changing contents of: Changing Fields. (line 6) +* fields, cutting: Cut Program. (line 6) +* fields, examining: Fields. (line 6) +* fields, number of: Fields. (line 33) +* fields, numbers: Nonconstant Fields. (line 6) +* fields, printing: Print Examples. (line 20) +* fields, separating: Field Separators. (line 15) +* fields, separating <1>: Field Separators. (line 15) +* fields, single-character: Single Character Fields. + (line 6) +* FIELDWIDTHS variable: Constant Size. (line 22) +* FIELDWIDTHS variable <1>: User-modified. (line 37) +* file descriptors: Special FD. (line 6) +* file inclusion, @include directive: Include Files. (line 8) +* file names, distinguishing: Auto-set. (line 55) +* file names, in compatibility mode: Special Caveats. (line 9) +* file names, standard streams in gawk: Special FD. (line 48) +* FILENAME variable: Reading Files. (line 6) +* FILENAME variable <1>: Auto-set. (line 108) +* FILENAME variable, getline, setting with: Getline Notes. (line 19) +* filenames, assignments as: Ignoring Assigns. (line 6) +* files, .gmo: Explaining gettext. (line 42) +* files, .gmo, specifying directory of: Explaining gettext. (line 54) +* files, .gmo, specifying directory of <1>: Programmer i18n. (line 48) +* files, .mo, converting from .po: I18N Example. (line 66) +* files, .po: Explaining gettext. (line 37) +* files, .po <1>: Translator i18n. (line 6) +* files, .po, converting to .mo: I18N Example. (line 66) +* files, .pot: Explaining gettext. (line 31) +* files, /dev/... special files: Special FD. (line 48) +* files, /inet/... (gawk): TCP/IP Networking. (line 6) +* files, /inet4/... (gawk): TCP/IP Networking. (line 6) +* files, /inet6/... (gawk): TCP/IP Networking. (line 6) +* files, awk programs in: Long. (line 6) +* files, awkprof.out: Profiling. (line 6) +* files, awkvars.out: Options. (line 94) +* files, closing: I/O Functions. (line 10) +* files, descriptors, See file descriptors: Special FD. (line 6) +* files, group: Group Functions. (line 6) +* files, initialization and cleanup: Filetrans Function. (line 6) +* files, input, See input files: Read Terminal. (line 16) +* files, log, timestamps in: Time Functions. (line 6) +* files, managing: Data File Management. + (line 6) +* files, managing, data file boundaries: Filetrans Function. (line 6) +* files, message object: Explaining gettext. (line 42) +* files, message object, converting from portable object files: I18N Example. + (line 66) +* files, message object, specifying directory of: Explaining gettext. + (line 54) +* files, message object, specifying directory of <1>: Programmer i18n. + (line 48) +* files, multiple passes over: Other Arguments. (line 56) +* files, multiple, duplicating output into: Tee Program. (line 6) +* files, output, See output files: Close Files And Pipes. + (line 6) +* files, password: Passwd Functions. (line 16) +* files, portable object: Explaining gettext. (line 37) +* files, portable object <1>: Translator i18n. (line 6) +* files, portable object template: Explaining gettext. (line 31) +* files, portable object, converting to message object files: I18N Example. + (line 66) +* files, portable object, generating: Options. (line 147) +* files, processing, ARGIND variable and: Auto-set. (line 50) +* files, reading: Rewind Function. (line 6) +* files, reading, multiline records: Multiple Line. (line 6) +* files, searching for regular expressions: Egrep Program. (line 6) +* files, skipping: File Checking. (line 6) +* files, source, search path for: Programs Exercises. (line 70) +* files, splitting: Split Program. (line 6) +* files, Texinfo, extracting programs from: Extract Program. (line 6) +* find substring in string: String Functions. (line 155) +* finding extensions: Finding Extensions. (line 6) +* finish debugger command: Debugger Execution Control. + (line 39) +* Fish, Fred: Contributors. (line 51) +* fixed-width data: Constant Size. (line 6) +* flag variables: Boolean Ops. (line 69) +* flag variables <1>: Tee Program. (line 20) +* floating-point, numbers, arbitrary precision: Arbitrary Precision Arithmetic. + (line 6) +* floating-point, VAX/VMS: VMS Running. (line 50) +* flush buffered output: I/O Functions. (line 28) +* fnmatch() extension function: Extension Sample Fnmatch. + (line 12) +* FNR variable: Records. (line 6) +* FNR variable <1>: Auto-set. (line 118) +* FNR variable, changing: Auto-set. (line 357) +* for statement: For Statement. (line 6) +* for statement, looping over arrays: Scanning an Array. (line 20) +* fork() extension function: Extension Sample Fork. + (line 11) +* format specifiers: Basic Printf. (line 15) +* format specifiers, mixing regular with positional specifiers: Printf Ordering. + (line 57) +* format specifiers, printf statement: Control Letters. (line 6) +* format specifiers, strftime() function (gawk): Time Functions. + (line 89) +* format time string: Time Functions. (line 48) +* formats, numeric output: OFMT. (line 6) +* formatting output: Printf. (line 6) +* formatting strings: String Functions. (line 384) +* forward slash (/) to enclose regular expressions: Regexp. (line 10) +* forward slash (/), / operator: Precedence. (line 54) +* forward slash (/), /= operator: Assignment Ops. (line 129) +* forward slash (/), /= operator <1>: Precedence. (line 94) +* forward slash (/), /= operator, vs. /=.../ regexp constant: Assignment Ops. + (line 149) +* forward slash (/), patterns and: Expression Patterns. (line 24) +* FPAT variable: Splitting By Content. + (line 25) +* FPAT variable <1>: User-modified. (line 43) +* frame debugger command: Execution Stack. (line 27) +* Free Documentation License (FDL): GNU Free Documentation License. + (line 8) +* Free Software Foundation (FSF): Manual History. (line 6) +* Free Software Foundation (FSF) <1>: Getting. (line 10) +* Free Software Foundation (FSF) <2>: Glossary. (line 372) +* Free Software Foundation (FSF) <3>: Glossary. (line 405) +* FreeBSD: Glossary. (line 748) +* FS variable: Field Separators. (line 15) +* FS variable <1>: User-modified. (line 50) +* FS variable, --field-separator option and: Options. (line 21) +* FS variable, as null string: Single Character Fields. + (line 20) +* FS variable, as TAB character: Options. (line 266) +* FS variable, changing value of: Field Separators. (line 34) +* FS variable, running awk programs and: Cut Program. (line 63) +* FS variable, setting from command line: Command Line Field Separator. + (line 6) +* FS, containing ^: Regexp Field Splitting. + (line 59) +* FS, in multiline records: Multiple Line. (line 41) +* FSF (Free Software Foundation): Manual History. (line 6) +* FSF (Free Software Foundation) <1>: Getting. (line 10) +* FSF (Free Software Foundation) <2>: Glossary. (line 372) +* FSF (Free Software Foundation) <3>: Glossary. (line 405) +* fts() extension function: Extension Sample File Functions. + (line 60) +* FUNCTAB array: Auto-set. (line 134) +* function calls: Function Calls. (line 6) +* function calls, indirect: Indirect Calls. (line 6) +* function calls, indirect, @-notation for: Indirect Calls. (line 47) +* function definition example: Function Example. (line 6) +* function pointers: Indirect Calls. (line 6) +* functions, arrays as parameters to: Pass By Value/Reference. + (line 44) +* functions, built-in: Function Calls. (line 10) +* functions, built-in <1>: Functions. (line 6) +* functions, built-in, evaluation order: Calling Built-in. (line 30) +* functions, defining: Definition Syntax. (line 10) +* functions, library: Library Functions. (line 6) +* functions, library, assertions: Assert Function. (line 6) +* functions, library, associative arrays and: Library Names. (line 58) +* functions, library, C library: Getopt Function. (line 6) +* functions, library, character values as numbers: Ordinal Functions. + (line 6) +* functions, library, Cliff random numbers: Cliff Random Function. + (line 6) +* functions, library, command-line options: Getopt Function. (line 6) +* functions, library, example program for using: Igawk Program. + (line 6) +* functions, library, group database, reading: Group Functions. + (line 6) +* functions, library, managing data files: Data File Management. + (line 6) +* functions, library, managing time: Getlocaltime Function. + (line 6) +* functions, library, merging arrays into strings: Join Function. + (line 6) +* functions, library, rounding numbers: Round Function. (line 6) +* functions, library, user database, reading: Passwd Functions. + (line 6) +* functions, names of: Definition Syntax. (line 24) +* functions, recursive: Definition Syntax. (line 89) +* functions, string-translation: I18N Functions. (line 6) +* functions, undefined: Pass By Value/Reference. + (line 68) +* functions, user-defined: User-defined. (line 6) +* functions, user-defined, calling: Function Caveats. (line 6) +* functions, user-defined, counts, in a profile: Profiling. (line 137) +* functions, user-defined, library of: Library Functions. (line 6) +* functions, user-defined, next/nextfile statements and: Next Statement. + (line 44) +* functions, user-defined, next/nextfile statements and <1>: Nextfile Statement. + (line 47) +* G-d: Acknowledgments. (line 94) +* G., Daniel Richard: Acknowledgments. (line 60) +* G., Daniel Richard <1>: Maintainers. (line 14) +* Garfinkle, Scott: Contributors. (line 35) +* gawk program, dynamic profiling: Profiling. (line 177) +* gawk version: Auto-set. (line 221) +* gawk, ARGIND variable in: Other Arguments. (line 15) +* gawk, awk and: Preface. (line 21) +* gawk, awk and <1>: This Manual. (line 14) +* gawk, bitwise operations in: Bitwise Functions. (line 40) +* gawk, break statement in: Break Statement. (line 51) +* gawk, character classes and: Bracket Expressions. (line 108) +* gawk, coding style in: Adding Code. (line 37) +* gawk, command-line options, and regular expressions: GNU Regexp Operators. + (line 73) +* gawk, configuring: Configuration Philosophy. + (line 6) +* gawk, configuring, options: Additional Configuration Options. + (line 6) +* gawk, continue statement in: Continue Statement. (line 44) +* gawk, distribution: Distribution contents. + (line 6) +* gawk, ERRNO variable in: Getline. (line 19) +* gawk, ERRNO variable in <1>: Close Files And Pipes. + (line 140) +* gawk, ERRNO variable in <2>: BEGINFILE/ENDFILE. (line 26) +* gawk, ERRNO variable in <3>: Auto-set. (line 87) +* gawk, ERRNO variable in <4>: TCP/IP Networking. (line 54) +* gawk, escape sequences: Escape Sequences. (line 121) +* gawk, extensions, disabling: Options. (line 257) +* gawk, features, adding: Adding Code. (line 6) +* gawk, features, advanced: Advanced Features. (line 6) +* gawk, field separators and: User-modified. (line 71) +* gawk, FIELDWIDTHS variable in: Constant Size. (line 22) +* gawk, FIELDWIDTHS variable in <1>: User-modified. (line 37) +* gawk, file names in: Special Files. (line 6) +* gawk, format-control characters: Control Letters. (line 18) +* gawk, format-control characters <1>: Control Letters. (line 93) +* gawk, FPAT variable in: Splitting By Content. + (line 25) +* gawk, FPAT variable in <1>: User-modified. (line 43) +* gawk, FUNCTAB array in: Auto-set. (line 134) +* gawk, function arguments and: Calling Built-in. (line 16) +* gawk, hexadecimal numbers and: Nondecimal-numbers. (line 41) +* gawk, IGNORECASE variable in: Case-sensitivity. (line 26) +* gawk, IGNORECASE variable in <1>: User-modified. (line 76) +* gawk, IGNORECASE variable in <2>: Array Intro. (line 100) +* gawk, IGNORECASE variable in <3>: String Functions. (line 58) +* gawk, IGNORECASE variable in <4>: Array Sorting Functions. + (line 83) +* gawk, implementation issues: Notes. (line 6) +* gawk, implementation issues, debugging: Compatibility Mode. (line 6) +* gawk, implementation issues, downward compatibility: Compatibility Mode. + (line 6) +* gawk, implementation issues, limits: Getline Notes. (line 14) +* gawk, implementation issues, pipes: Redirection. (line 129) +* gawk, installing: Installation. (line 6) +* gawk, internationalization and, See internationalization: Internationalization. + (line 13) +* gawk, interpreter, adding code to: Using Internal File Ops. + (line 6) +* gawk, interval expressions and: Regexp Operators. (line 139) +* gawk, line continuation in: Conditional Exp. (line 34) +* gawk, LINT variable in: User-modified. (line 87) +* gawk, list of contributors to: Contributors. (line 6) +* gawk, MS-Windows version of: PC Using. (line 9) +* gawk, newlines in: Statements/Lines. (line 12) +* gawk, octal numbers and: Nondecimal-numbers. (line 41) +* gawk, predefined variables and: Built-in Variables. (line 14) +* gawk, PROCINFO array in: Auto-set. (line 148) +* gawk, PROCINFO array in <1>: Time Functions. (line 47) +* gawk, PROCINFO array in <2>: Two-way I/O. (line 114) +* gawk, regexp constants and: Using Constant Regexps. + (line 28) +* gawk, regular expressions, case sensitivity: Case-sensitivity. + (line 26) +* gawk, regular expressions, operators: GNU Regexp Operators. + (line 6) +* gawk, regular expressions, precedence: Regexp Operators. (line 161) +* gawk, RT variable in: awk split records. (line 124) +* gawk, RT variable in <1>: Multiple Line. (line 130) +* gawk, RT variable in <2>: Auto-set. (line 295) +* gawk, See Also awk: Preface. (line 34) +* gawk, source code, obtaining: Getting. (line 6) +* gawk, splitting fields and: Constant Size. (line 86) +* gawk, string-translation functions: I18N Functions. (line 6) +* gawk, SYMTAB array in: Auto-set. (line 299) +* gawk, TEXTDOMAIN variable in: User-modified. (line 152) +* gawk, timestamps: Time Functions. (line 6) +* gawk, uses for: Preface. (line 34) +* gawk, versions of, information about, printing: Options. (line 304) +* gawk, VMS version of: VMS Installation. (line 6) +* gawk, word-boundary operator: GNU Regexp Operators. + (line 66) +* gawkextlib: gawkextlib. (line 6) +* gawkextlib project: gawkextlib. (line 6) +* gawklibpath_append shell function: Shell Startup Files. (line 29) +* gawklibpath_default shell function: Shell Startup Files. (line 22) +* gawklibpath_prepend shell function: Shell Startup Files. (line 25) +* gawkpath_append shell function: Shell Startup Files. (line 19) +* gawkpath_default shell function: Shell Startup Files. (line 12) +* gawkpath_prepend shell function: Shell Startup Files. (line 15) +* General Public License (GPL): Glossary. (line 396) +* General Public License, See GPL: Manual History. (line 11) +* generate time values: Time Functions. (line 25) +* gensub: Using Constant Regexps. + (line 43) +* gensub <1>: String Functions. (line 89) +* gensub() function (gawk), escape processing: Gory Details. (line 6) +* getaddrinfo() function (C library): TCP/IP Networking. (line 39) +* getgrent() function (C library): Group Functions. (line 6) +* getgrent() function (C library) <1>: Group Functions. (line 202) +* getgrent() user-defined function: Group Functions. (line 6) +* getgrent() user-defined function <1>: Group Functions. (line 205) +* getgrgid() function (C library): Group Functions. (line 184) +* getgrgid() user-defined function: Group Functions. (line 187) +* getgrnam() function (C library): Group Functions. (line 173) +* getgrnam() user-defined function: Group Functions. (line 178) +* getgruser() function (C library): Group Functions. (line 193) +* getgruser() function, user-defined: Group Functions. (line 196) +* getline command: Reading Files. (line 20) +* getline command, coprocesses, using from: Getline/Coprocess. + (line 6) +* getline command, coprocesses, using from <1>: Close Files And Pipes. + (line 6) +* getline command, deadlock and: Two-way I/O. (line 53) +* getline command, explicit input with: Getline. (line 6) +* getline command, FILENAME variable and: Getline Notes. (line 19) +* getline command, return values: Getline. (line 19) +* getline command, variants: Getline Summary. (line 6) +* getline command, _gr_init() user-defined function: Group Functions. + (line 83) +* getline command, _pw_init() function: Passwd Functions. (line 154) +* getline from a file: Getline/File. (line 6) +* getline into a variable: Getline/Variable. (line 6) +* getline statement, BEGINFILE/ENDFILE patterns and: BEGINFILE/ENDFILE. + (line 53) +* getlocaltime() user-defined function: Getlocaltime Function. + (line 16) +* getopt() function (C library): Getopt Function. (line 15) +* getopt() user-defined function: Getopt Function. (line 108) +* getopt() user-defined function <1>: Getopt Function. (line 134) +* getpwent() function (C library): Passwd Functions. (line 16) +* getpwent() function (C library) <1>: Passwd Functions. (line 196) +* getpwent() user-defined function: Passwd Functions. (line 16) +* getpwent() user-defined function <1>: Passwd Functions. (line 200) +* getpwnam() function (C library): Passwd Functions. (line 175) +* getpwnam() user-defined function: Passwd Functions. (line 180) +* getpwuid() function (C library): Passwd Functions. (line 186) +* getpwuid() user-defined function: Passwd Functions. (line 190) +* gettext library: Explaining gettext. (line 6) +* gettext library, locale categories: Explaining gettext. (line 81) +* gettext() function (C library): Explaining gettext. (line 63) +* gettimeofday() extension function: Extension Sample Time. + (line 12) +* git utility: gawkextlib. (line 31) +* git utility <1>: Other Versions. (line 29) +* git utility <2>: Accessing The Source. + (line 10) +* git utility <3>: Adding Code. (line 112) +* Git, use of for gawk source code: Derived Files. (line 6) +* GNITS mailing list: Acknowledgments. (line 52) +* GNU awk, See gawk: Preface. (line 51) +* GNU Free Documentation License: GNU Free Documentation License. + (line 8) +* GNU General Public License: Glossary. (line 396) +* GNU Lesser General Public License: Glossary. (line 491) +* GNU long options: Command Line. (line 13) +* GNU long options <1>: Options. (line 6) +* GNU long options, printing list of: Options. (line 154) +* GNU Project: Manual History. (line 11) +* GNU Project <1>: Glossary. (line 405) +* GNU/Linux: Manual History. (line 28) +* GNU/Linux <1>: I18N Example. (line 57) +* GNU/Linux <2>: Glossary. (line 748) +* Gordon, Assaf: Contributors. (line 106) +* GPL (General Public License): Manual History. (line 11) +* GPL (General Public License) <1>: Glossary. (line 396) +* GPL (General Public License), printing: Options. (line 89) +* grcat program: Group Functions. (line 16) +* Grigera, Juan: Contributors. (line 58) +* group database, reading: Group Functions. (line 6) +* group file: Group Functions. (line 6) +* group ID of gawk user: Auto-set. (line 170) +* groups, information about: Group Functions. (line 6) +* gsub: Using Constant Regexps. + (line 43) +* gsub <1>: String Functions. (line 139) +* gsub() function, arguments of: String Functions. (line 463) +* gsub() function, escape processing: Gory Details. (line 6) +* h debugger command (alias for help): Miscellaneous Debugger Commands. + (line 69) +* Hankerson, Darrel: Acknowledgments. (line 60) +* Hankerson, Darrel <1>: Contributors. (line 61) +* Haque, John: Contributors. (line 109) +* Hartholz, Elaine: Acknowledgments. (line 38) +* Hartholz, Marshall: Acknowledgments. (line 38) +* Hasegawa, Isamu: Contributors. (line 95) +* help debugger command: Miscellaneous Debugger Commands. + (line 69) +* hexadecimal numbers: Nondecimal-numbers. (line 6) +* hexadecimal values, enabling interpretation of: Options. (line 209) +* history expansion, in debugger: Readline Support. (line 6) +* histsort.awk program: History Sorting. (line 25) +* Hughes, Phil: Acknowledgments. (line 43) +* HUP signal, for dynamic profiling: Profiling. (line 209) +* hyphen (-), - operator: Precedence. (line 51) +* hyphen (-), - operator <1>: Precedence. (line 57) +* hyphen (-), -- operator: Increment Ops. (line 48) +* hyphen (-), -- operator <1>: Precedence. (line 45) +* hyphen (-), -= operator: Assignment Ops. (line 129) +* hyphen (-), -= operator <1>: Precedence. (line 94) +* hyphen (-), filenames beginning with: Options. (line 60) +* hyphen (-), in bracket expressions: Bracket Expressions. (line 25) +* i debugger command (alias for info): Debugger Info. (line 13) +* id utility: Id Program. (line 6) +* id.awk program: Id Program. (line 31) +* if statement: If Statement. (line 6) +* if statement, actions, changing: Ranges. (line 25) +* if statement, use of regexps in: Regexp Usage. (line 19) +* igawk.sh program: Igawk Program. (line 124) +* ignore breakpoint: Breakpoint Control. (line 87) +* ignore debugger command: Breakpoint Control. (line 87) +* IGNORECASE variable: User-modified. (line 76) +* IGNORECASE variable, and array indices: Array Intro. (line 100) +* IGNORECASE variable, and array sorting functions: Array Sorting Functions. + (line 83) +* IGNORECASE variable, in example programs: Library Functions. + (line 53) +* IGNORECASE variable, with ~ and !~ operators: Case-sensitivity. + (line 26) +* Illumos: Other Versions. (line 109) +* Illumos, POSIX-compliant awk: Other Versions. (line 109) +* implementation issues, gawk: Notes. (line 6) +* implementation issues, gawk, debugging: Compatibility Mode. (line 6) +* implementation issues, gawk, limits: Getline Notes. (line 14) +* implementation issues, gawk, limits <1>: Redirection. (line 129) +* in operator: Comparison Operators. + (line 11) +* in operator <1>: Precedence. (line 82) +* in operator <2>: For Statement. (line 75) +* in operator, index existence in multidimensional arrays: Multidimensional. + (line 41) +* in operator, order of array access: Scanning an Array. (line 48) +* in operator, testing if array element exists: Reference to Elements. + (line 38) +* in operator, use in loops: Scanning an Array. (line 17) +* including files, @include directive: Include Files. (line 8) +* increment operators: Increment Ops. (line 6) +* index: String Functions. (line 155) +* indexing arrays: Array Intro. (line 48) +* indirect function calls: Indirect Calls. (line 6) +* indirect function calls, @-notation: Indirect Calls. (line 47) +* infinite precision: Arbitrary Precision Arithmetic. + (line 6) +* info debugger command: Debugger Info. (line 13) +* initialization, automatic: More Complex. (line 39) +* inplace extension: Extension Sample Inplace. + (line 6) +* input files: Reading Files. (line 6) +* input files, closing: Close Files And Pipes. + (line 6) +* input files, counting elements in: Wc Program. (line 6) +* input files, examples: Sample Data Files. (line 6) +* input files, reading: Reading Files. (line 6) +* input files, running awk without: Read Terminal. (line 6) +* input files, running awk without <1>: Read Terminal. (line 16) +* input files, variable assignments and: Other Arguments. (line 26) +* input pipeline: Getline/Pipe. (line 10) +* input record, length of: String Functions. (line 177) +* input redirection: Getline/File. (line 6) +* input, data, nondecimal: Nondecimal Data. (line 6) +* input, explicit: Getline. (line 6) +* input, files, See input files: Multiple Line. (line 6) +* input, multiline records: Multiple Line. (line 6) +* input, splitting into records: Records. (line 6) +* input, standard: Read Terminal. (line 6) +* input, standard <1>: Special FD. (line 6) +* input/output functions: I/O Functions. (line 6) +* input/output, binary: User-modified. (line 15) +* input/output, from BEGIN and END: I/O And BEGIN/END. (line 6) +* input/output, two-way: Two-way I/O. (line 27) +* insomnia, cure for: Alarm Program. (line 6) +* installation, VMS: VMS Installation. (line 6) +* installing gawk: Installation. (line 6) +* instruction tracing, in debugger: Debugger Info. (line 90) +* int: Numeric Functions. (line 24) +* INT signal (MS-Windows): Profiling. (line 212) +* intdiv: Numeric Functions. (line 29) +* intdiv <1>: Numeric Functions. (line 29) +* integer array indices: Numeric Array Subscripts. + (line 31) +* integers, arbitrary precision: Arbitrary Precision Integers. + (line 6) +* integers, unsigned: Computer Arithmetic. (line 41) +* interacting with other programs: I/O Functions. (line 107) +* internationalization: I18N Functions. (line 6) +* internationalization <1>: I18N and L10N. (line 6) +* internationalization, localization: User-modified. (line 152) +* internationalization, localization <1>: Internationalization. + (line 13) +* internationalization, localization, character classes: Bracket Expressions. + (line 108) +* internationalization, localization, gawk and: Internationalization. + (line 13) +* internationalization, localization, locale categories: Explaining gettext. + (line 81) +* internationalization, localization, marked strings: Programmer i18n. + (line 13) +* internationalization, localization, portability and: I18N Portability. + (line 6) +* internationalizing a program: Explaining gettext. (line 6) +* interpreted programs: Basic High Level. (line 13) +* interpreted programs <1>: Glossary. (line 445) +* interval expressions, regexp operator: Regexp Operators. (line 116) +* inventory-shipped file: Sample Data Files. (line 32) +* invoke shell command: I/O Functions. (line 107) +* isarray: Type Functions. (line 11) +* ISO: Glossary. (line 456) +* ISO 8859-1: Glossary. (line 196) +* ISO Latin-1: Glossary. (line 196) +* Jacobs, Andrew: Passwd Functions. (line 90) +* Jaegermann, Michal: Acknowledgments. (line 60) +* Jaegermann, Michal <1>: Contributors. (line 46) +* Java implementation of awk: Other Versions. (line 117) +* Java programming language: Glossary. (line 468) +* jawk: Other Versions. (line 117) +* Jedi knights: Undocumented. (line 6) +* Johansen, Chris: Signature Program. (line 25) +* join() user-defined function: Join Function. (line 18) +* Kahrs, Ju"rgen: Acknowledgments. (line 60) +* Kahrs, Ju"rgen <1>: Contributors. (line 71) +* Kasal, Stepan: Acknowledgments. (line 60) +* Kenobi, Obi-Wan: Undocumented. (line 6) +* Kernighan, Brian: History. (line 17) +* Kernighan, Brian <1>: Conventions. (line 38) +* Kernighan, Brian <2>: Acknowledgments. (line 79) +* Kernighan, Brian <3>: Getline/Pipe. (line 6) +* Kernighan, Brian <4>: Concatenation. (line 6) +* Kernighan, Brian <5>: Library Functions. (line 12) +* Kernighan, Brian <6>: BTL. (line 6) +* Kernighan, Brian <7>: Contributors. (line 12) +* Kernighan, Brian <8>: Other Versions. (line 13) +* Kernighan, Brian <9>: Basic Data Typing. (line 54) +* Kernighan, Brian <10>: Glossary. (line 206) +* kill command, dynamic profiling: Profiling. (line 186) +* Knights, jedi: Undocumented. (line 6) +* Kwok, Conrad: Contributors. (line 35) +* l debugger command (alias for list): Miscellaneous Debugger Commands. + (line 75) +* labels.awk program: Labels Program. (line 51) +* Langston, Peter: Advanced Features. (line 6) +* LANGUAGE environment variable: Explaining gettext. (line 120) +* languages, data-driven: Basic High Level. (line 74) +* LC_ALL locale category: Explaining gettext. (line 117) +* LC_COLLATE locale category: Explaining gettext. (line 94) +* LC_CTYPE locale category: Explaining gettext. (line 98) +* LC_MESSAGES locale category: Explaining gettext. (line 88) +* LC_MESSAGES locale category, bindtextdomain() function (gawk): Programmer i18n. + (line 101) +* LC_MONETARY locale category: Explaining gettext. (line 104) +* LC_NUMERIC locale category: Explaining gettext. (line 108) +* LC_TIME locale category: Explaining gettext. (line 112) +* left angle bracket (<), < operator: Comparison Operators. + (line 11) +* left angle bracket (<), < operator <1>: Precedence. (line 64) +* left angle bracket (<), < operator (I/O): Getline/File. (line 6) +* left angle bracket (<), <= operator: Comparison Operators. + (line 11) +* left angle bracket (<), <= operator <1>: Precedence. (line 64) +* left shift: Bitwise Functions. (line 47) +* left shift, bitwise: Bitwise Functions. (line 32) +* leftmost longest match: Multiple Line. (line 26) +* length: String Functions. (line 170) +* length of input record: String Functions. (line 177) +* length of string: String Functions. (line 170) +* Lesser General Public License (LGPL): Glossary. (line 491) +* LGPL (Lesser General Public License): Glossary. (line 491) +* libmawk: Other Versions. (line 125) +* libraries of awk functions: Library Functions. (line 6) +* libraries of awk functions, assertions: Assert Function. (line 6) +* libraries of awk functions, associative arrays and: Library Names. + (line 58) +* libraries of awk functions, character values as numbers: Ordinal Functions. + (line 6) +* libraries of awk functions, command-line options: Getopt Function. + (line 6) +* libraries of awk functions, example program for using: Igawk Program. + (line 6) +* libraries of awk functions, group database, reading: Group Functions. + (line 6) +* libraries of awk functions, managing, data files: Data File Management. + (line 6) +* libraries of awk functions, managing, time: Getlocaltime Function. + (line 6) +* libraries of awk functions, merging arrays into strings: Join Function. + (line 6) +* libraries of awk functions, rounding numbers: Round Function. + (line 6) +* libraries of awk functions, user database, reading: Passwd Functions. + (line 6) +* line breaks: Statements/Lines. (line 6) +* line continuations: Boolean Ops. (line 64) +* line continuations, gawk: Conditional Exp. (line 34) +* line continuations, in print statement: Print Examples. (line 75) +* line continuations, with C shell: More Complex. (line 31) +* lines, blank, printing: Print. (line 22) +* lines, counting: Wc Program. (line 6) +* lines, duplicate, removing: History Sorting. (line 6) +* lines, matching ranges of: Ranges. (line 6) +* lines, skipping between markers: Ranges. (line 43) +* lint checking: User-modified. (line 87) +* lint checking, array elements: Delete. (line 34) +* lint checking, array subscripts: Uninitialized Subscripts. + (line 43) +* lint checking, empty programs: Command Line. (line 16) +* lint checking, issuing warnings: Options. (line 184) +* lint checking, POSIXLY_CORRECT environment variable: Options. + (line 343) +* lint checking, undefined functions: Pass By Value/Reference. + (line 85) +* LINT variable: User-modified. (line 87) +* Linux: Manual History. (line 28) +* Linux <1>: I18N Example. (line 57) +* Linux <2>: Glossary. (line 748) +* list all global variables, in debugger: Debugger Info. (line 48) +* list debugger command: Miscellaneous Debugger Commands. + (line 75) +* list function definitions, in debugger: Debugger Info. (line 30) +* loading extensions, @load directive: Loading Shared Libraries. + (line 8) +* loading, extensions: Options. (line 172) +* local variables, in a function: Variable Scope. (line 6) +* locale categories: Explaining gettext. (line 81) +* locale decimal point character: Options. (line 269) +* locale, definition of: Locales. (line 6) +* localization: I18N and L10N. (line 6) +* localization, See internationalization, localization: I18N and L10N. + (line 6) +* log: Numeric Functions. (line 44) +* log files, timestamps in: Time Functions. (line 6) +* logarithm: Numeric Functions. (line 44) +* logical false/true: Truth Values. (line 6) +* logical operators, See Boolean expressions: Boolean Ops. (line 6) +* login information: Passwd Functions. (line 16) +* long options: Command Line. (line 13) +* loops: While Statement. (line 6) +* loops, break statement and: Break Statement. (line 6) +* loops, continue statements and: For Statement. (line 64) +* loops, count for header, in a profile: Profiling. (line 131) +* loops, do-while: Do Statement. (line 6) +* loops, exiting: Break Statement. (line 6) +* loops, for, array scanning: Scanning an Array. (line 6) +* loops, for, iterative: For Statement. (line 6) +* loops, See Also while statement: While Statement. (line 6) +* loops, while: While Statement. (line 6) +* ls utility: More Complex. (line 15) +* lshift: Bitwise Functions. (line 47) +* lvalues/rvalues: Assignment Ops. (line 31) +* mail-list file: Sample Data Files. (line 6) +* mailing labels, printing: Labels Program. (line 6) +* mailing list, GNITS: Acknowledgments. (line 52) +* Malmberg, John: Acknowledgments. (line 60) +* Malmberg, John <1>: Maintainers. (line 14) +* Malmberg, John E.: Contributors. (line 138) +* mark parity: Ordinal Functions. (line 45) +* marked string extraction (internationalization): String Extraction. + (line 6) +* marked strings, extracting: String Extraction. (line 6) +* Marx, Groucho: Increment Ops. (line 60) +* match: String Functions. (line 210) +* match regexp in string: String Functions. (line 210) +* match() function, RSTART/RLENGTH variables: String Functions. + (line 227) +* matching, expressions, See comparison expressions: Typing and Comparison. + (line 9) +* matching, leftmost longest: Multiple Line. (line 26) +* matching, null strings: String Functions. (line 537) +* mawk utility: Escape Sequences. (line 121) +* mawk utility <1>: Getline/Pipe. (line 62) +* mawk utility <2>: Concatenation. (line 36) +* mawk utility <3>: Nextfile Statement. (line 47) +* mawk utility <4>: Other Versions. (line 48) +* maximum precision supported by MPFR library: Auto-set. (line 235) +* McIlroy, Doug: Glossary. (line 257) +* McPhee, Patrick: Contributors. (line 101) +* message object files: Explaining gettext. (line 42) +* message object files, converting from portable object files: I18N Example. + (line 66) +* message object files, specifying directory of: Explaining gettext. + (line 54) +* message object files, specifying directory of <1>: Programmer i18n. + (line 48) +* messages from extensions: Printing Messages. (line 6) +* metacharacters in regular expressions: Regexp Operators. (line 6) +* metacharacters, escape sequences for: Escape Sequences. (line 140) +* minimum precision required by MPFR library: Auto-set. (line 238) +* mktime: Time Functions. (line 25) +* modifiers, in format specifiers: Format Modifiers. (line 6) +* monetary information, localization: Explaining gettext. (line 104) +* Moore, Duncan: Getline Notes. (line 40) +* msgfmt utility: I18N Example. (line 66) +* multiple precision: Arbitrary Precision Arithmetic. + (line 6) +* multiple-line records: Multiple Line. (line 6) +* n debugger command (alias for next): Debugger Execution Control. + (line 43) +* names, arrays/variables: Library Names. (line 6) +* names, functions: Definition Syntax. (line 24) +* names, functions <1>: Library Names. (line 6) +* namespace issues: Library Names. (line 6) +* namespace issues, functions: Definition Syntax. (line 24) +* NetBSD: Glossary. (line 748) +* networks, programming: TCP/IP Networking. (line 6) +* networks, support for: Special Network. (line 6) +* newlines: Statements/Lines. (line 6) +* newlines <1>: Options. (line 263) +* newlines <2>: Boolean Ops. (line 69) +* newlines, as record separators: awk split records. (line 12) +* newlines, in dynamic regexps: Computed Regexps. (line 60) +* newlines, in regexp constants: Computed Regexps. (line 70) +* newlines, printing: Print Examples. (line 11) +* newlines, separating statements in actions: Action Overview. + (line 19) +* newlines, separating statements in actions <1>: Statements. (line 10) +* next debugger command: Debugger Execution Control. + (line 43) +* next file statement: Feature History. (line 168) +* next statement: Boolean Ops. (line 95) +* next statement <1>: Next Statement. (line 6) +* next statement, BEGIN/END patterns and: I/O And BEGIN/END. (line 36) +* next statement, BEGINFILE/ENDFILE patterns and: BEGINFILE/ENDFILE. + (line 49) +* next statement, user-defined functions and: Next Statement. (line 44) +* nextfile statement: Nextfile Statement. (line 6) +* nextfile statement, BEGIN/END patterns and: I/O And BEGIN/END. + (line 36) +* nextfile statement, BEGINFILE/ENDFILE patterns and: BEGINFILE/ENDFILE. + (line 26) +* nextfile statement, user-defined functions and: Nextfile Statement. + (line 47) +* nexti debugger command: Debugger Execution Control. + (line 49) +* NF variable: Fields. (line 33) +* NF variable <1>: Auto-set. (line 123) +* NF variable, decrementing: Changing Fields. (line 107) +* ni debugger command (alias for nexti): Debugger Execution Control. + (line 49) +* noassign.awk program: Ignoring Assigns. (line 15) +* non-existent array elements: Reference to Elements. + (line 23) +* not Boolean-logic operator: Boolean Ops. (line 6) +* NR variable: Records. (line 6) +* NR variable <1>: Auto-set. (line 143) +* NR variable, changing: Auto-set. (line 357) +* null strings: awk split records. (line 114) +* null strings <1>: Regexp Field Splitting. + (line 43) +* null strings <2>: Truth Values. (line 6) +* null strings <3>: Basic Data Typing. (line 26) +* null strings in gawk arguments, quoting and: Quoting. (line 82) +* null strings, and deleting array elements: Delete. (line 27) +* null strings, as array subscripts: Uninitialized Subscripts. + (line 43) +* null strings, converting numbers to strings: Strings And Numbers. + (line 21) +* null strings, matching: String Functions. (line 537) +* number as string of bits: Bitwise Functions. (line 108) +* number of array elements: String Functions. (line 200) +* number sign (#), #! (executable scripts): Executable Scripts. + (line 6) +* number sign (#), commenting: Comments. (line 6) +* numbers, as array subscripts: Numeric Array Subscripts. + (line 6) +* numbers, as values of characters: Ordinal Functions. (line 6) +* numbers, Cliff random: Cliff Random Function. + (line 6) +* numbers, converting: Strings And Numbers. (line 6) +* numbers, converting <1>: Bitwise Functions. (line 108) +* numbers, converting, to strings: User-modified. (line 30) +* numbers, converting, to strings <1>: User-modified. (line 104) +* numbers, hexadecimal: Nondecimal-numbers. (line 6) +* numbers, octal: Nondecimal-numbers. (line 6) +* numbers, rounding: Round Function. (line 6) +* numeric constants: Scalar Constants. (line 6) +* numeric functions: Numeric Functions. (line 6) +* numeric, output format: OFMT. (line 6) +* numeric, strings: Variable Typing. (line 6) +* o debugger command (alias for option): Debugger Info. (line 57) +* obsolete features: Obsolete. (line 6) +* octal numbers: Nondecimal-numbers. (line 6) +* octal values, enabling interpretation of: Options. (line 209) +* OFMT variable: OFMT. (line 15) +* OFMT variable <1>: Strings And Numbers. (line 56) +* OFMT variable <2>: User-modified. (line 104) +* OFMT variable, POSIX awk and: OFMT. (line 27) +* OFS variable: Changing Fields. (line 64) +* OFS variable <1>: Output Separators. (line 6) +* OFS variable <2>: User-modified. (line 113) +* OpenBSD: Glossary. (line 748) +* OpenSolaris: Other Versions. (line 100) +* operating systems, BSD-based: Manual History. (line 28) +* operating systems, PC, gawk on: PC Using. (line 6) +* operating systems, PC, gawk on, installing: PC Installation. + (line 6) +* operating systems, porting gawk to: New Ports. (line 6) +* operating systems, See Also GNU/Linux, PC operating systems, Unix: Installation. + (line 6) +* operations, bitwise: Bitwise Functions. (line 6) +* operators, arithmetic: Arithmetic Ops. (line 6) +* operators, assignment: Assignment Ops. (line 6) +* operators, assignment <1>: Assignment Ops. (line 31) +* operators, assignment, evaluation order: Assignment Ops. (line 110) +* operators, Boolean, See Boolean expressions: Boolean Ops. (line 6) +* operators, decrement/increment: Increment Ops. (line 6) +* operators, GNU-specific: GNU Regexp Operators. + (line 6) +* operators, input/output: Getline/File. (line 6) +* operators, input/output <1>: Getline/Pipe. (line 10) +* operators, input/output <2>: Getline/Coprocess. (line 6) +* operators, input/output <3>: Redirection. (line 22) +* operators, input/output <4>: Redirection. (line 96) +* operators, input/output <5>: Precedence. (line 64) +* operators, input/output <6>: Precedence. (line 64) +* operators, input/output <7>: Precedence. (line 64) +* operators, logical, See Boolean expressions: Boolean Ops. (line 6) +* operators, precedence: Increment Ops. (line 60) +* operators, precedence <1>: Precedence. (line 6) +* operators, relational, See operators, comparison: Typing and Comparison. + (line 9) +* operators, short-circuit: Boolean Ops. (line 59) +* operators, string: Concatenation. (line 9) +* operators, string-matching: Regexp Usage. (line 19) +* operators, string-matching, for buffers: GNU Regexp Operators. + (line 51) +* operators, word-boundary (gawk): GNU Regexp Operators. + (line 66) +* option debugger command: Debugger Info. (line 57) +* options, command-line: Options. (line 6) +* options, command-line, end of: Options. (line 55) +* options, command-line, invoking awk: Command Line. (line 6) +* options, command-line, processing: Getopt Function. (line 6) +* options, deprecated: Obsolete. (line 6) +* options, long: Command Line. (line 13) +* options, long <1>: Options. (line 6) +* options, printing list of: Options. (line 154) +* or: Bitwise Functions. (line 50) +* OR bitwise operation: Bitwise Functions. (line 6) +* or Boolean-logic operator: Boolean Ops. (line 6) +* ord() extension function: Extension Sample Ord. + (line 12) +* ord() user-defined function: Ordinal Functions. (line 16) +* order of evaluation, concatenation: Concatenation. (line 41) +* ORS variable: Output Separators. (line 20) +* ORS variable <1>: User-modified. (line 119) +* output field separator, See OFS variable: Changing Fields. (line 64) +* output record separator, See ORS variable: Output Separators. + (line 20) +* output redirection: Redirection. (line 6) +* output wrapper: Output Wrappers. (line 6) +* output, buffering: I/O Functions. (line 32) +* output, buffering <1>: I/O Functions. (line 166) +* output, duplicating into files: Tee Program. (line 6) +* output, files, closing: Close Files And Pipes. + (line 6) +* output, format specifier, OFMT: OFMT. (line 15) +* output, formatted: Printf. (line 6) +* output, pipes: Redirection. (line 57) +* output, printing, See printing: Printing. (line 6) +* output, records: Output Separators. (line 20) +* output, standard: Special FD. (line 6) +* p debugger command (alias for print): Viewing And Changing Data. + (line 35) +* Papadopoulos, Panos: Contributors. (line 129) +* parent process ID of gawk process: Auto-set. (line 210) +* parentheses (), in a profile: Profiling. (line 146) +* parentheses (), regexp operator: Regexp Operators. (line 81) +* password file: Passwd Functions. (line 16) +* patsplit: String Functions. (line 296) +* patterns: Patterns and Actions. + (line 6) +* patterns, comparison expressions as: Expression Patterns. (line 14) +* patterns, counts, in a profile: Profiling. (line 118) +* patterns, default: Very Simple. (line 35) +* patterns, empty: Empty. (line 6) +* patterns, expressions as: Regexp Patterns. (line 6) +* patterns, ranges in: Ranges. (line 6) +* patterns, regexp constants as: Expression Patterns. (line 34) +* patterns, types of: Pattern Overview. (line 15) +* pawk (profiling version of Brian Kernighan's awk): Other Versions. + (line 82) +* pawk, awk-like facilities for Python: Other Versions. (line 129) +* PC operating systems, gawk on: PC Using. (line 6) +* PC operating systems, gawk on, installing: PC Installation. (line 6) +* percent sign (%), % operator: Precedence. (line 54) +* percent sign (%), %= operator: Assignment Ops. (line 129) +* percent sign (%), %= operator <1>: Precedence. (line 94) +* period (.), regexp operator: Regexp Operators. (line 44) +* Perl: Future Extensions. (line 6) +* Peters, Arno: Contributors. (line 86) +* Peterson, Hal: Contributors. (line 40) +* pipe, closing: Close Files And Pipes. + (line 6) +* pipe, input: Getline/Pipe. (line 10) +* pipe, output: Redirection. (line 57) +* Pitts, Dave: Acknowledgments. (line 60) +* Pitts, Dave <1>: Maintainers. (line 14) +* Plauger, P.J.: Library Functions. (line 12) +* plug-in: Extension Intro. (line 6) +* plus sign (+), + operator: Precedence. (line 51) +* plus sign (+), + operator <1>: Precedence. (line 57) +* plus sign (+), ++ operator: Increment Ops. (line 11) +* plus sign (+), ++ operator <1>: Increment Ops. (line 40) +* plus sign (+), ++ operator <2>: Precedence. (line 45) +* plus sign (+), += operator: Assignment Ops. (line 81) +* plus sign (+), += operator <1>: Precedence. (line 94) +* plus sign (+), regexp operator: Regexp Operators. (line 105) +* pointers to functions: Indirect Calls. (line 6) +* portability: Escape Sequences. (line 103) +* portability, #! (executable scripts): Executable Scripts. (line 33) +* portability, ** operator and: Arithmetic Ops. (line 81) +* portability, **= operator and: Assignment Ops. (line 144) +* portability, ARGV variable: Executable Scripts. (line 59) +* portability, backslash continuation and: Statements/Lines. (line 30) +* portability, backslash in escape sequences: Escape Sequences. + (line 108) +* portability, close() function and: Close Files And Pipes. + (line 81) +* portability, data files as single record: gawk split records. + (line 65) +* portability, deleting array elements: Delete. (line 56) +* portability, example programs: Library Functions. (line 42) +* portability, functions, defining: Definition Syntax. (line 114) +* portability, gawk: New Ports. (line 6) +* portability, gettext library and: Explaining gettext. (line 11) +* portability, internationalization and: I18N Portability. (line 6) +* portability, length() function: String Functions. (line 179) +* portability, new awk vs. old awk: Strings And Numbers. (line 56) +* portability, next statement in user-defined functions: Pass By Value/Reference. + (line 88) +* portability, NF variable, decrementing: Changing Fields. (line 115) +* portability, operators: Increment Ops. (line 60) +* portability, operators, not in POSIX awk: Precedence. (line 97) +* portability, POSIXLY_CORRECT environment variable: Options. (line 363) +* portability, substr() function: String Functions. (line 513) +* portable object files: Explaining gettext. (line 37) +* portable object files <1>: Translator i18n. (line 6) +* portable object files, converting to message object files: I18N Example. + (line 66) +* portable object files, generating: Options. (line 147) +* portable object template files: Explaining gettext. (line 31) +* porting gawk: New Ports. (line 6) +* positional specifiers, printf statement: Format Modifiers. (line 13) +* positional specifiers, printf statement <1>: Printf Ordering. + (line 6) +* positional specifiers, printf statement, mixing with regular formats: Printf Ordering. + (line 57) +* POSIX awk: This Manual. (line 14) +* POSIX awk <1>: Assignment Ops. (line 138) +* POSIX awk, ** operator and: Precedence. (line 97) +* POSIX awk, **= operator and: Assignment Ops. (line 144) +* POSIX awk, < operator and: Getline/File. (line 26) +* POSIX awk, arithmetic operators and: Arithmetic Ops. (line 30) +* POSIX awk, backslashes in string constants: Escape Sequences. + (line 108) +* POSIX awk, BEGIN/END patterns: I/O And BEGIN/END. (line 15) +* POSIX awk, bracket expressions and: Bracket Expressions. (line 34) +* POSIX awk, bracket expressions and, character classes: Bracket Expressions. + (line 40) +* POSIX awk, bracket expressions and, character classes <1>: Bracket Expressions. + (line 108) +* POSIX awk, break statement and: Break Statement. (line 51) +* POSIX awk, changes in awk versions: POSIX. (line 6) +* POSIX awk, continue statement and: Continue Statement. (line 44) +* POSIX awk, CONVFMT variable and: User-modified. (line 30) +* POSIX awk, date utility and: Time Functions. (line 253) +* POSIX awk, field separators and: Full Line Fields. (line 16) +* POSIX awk, function keyword in: Definition Syntax. (line 99) +* POSIX awk, functions and, gsub()/sub(): Gory Details. (line 90) +* POSIX awk, functions and, length(): String Functions. (line 179) +* POSIX awk, GNU long options and: Options. (line 15) +* POSIX awk, interval expressions in: Regexp Operators. (line 135) +* POSIX awk, next/nextfile statements and: Next Statement. (line 44) +* POSIX awk, numeric strings and: Variable Typing. (line 6) +* POSIX awk, OFMT variable and: OFMT. (line 27) +* POSIX awk, OFMT variable and <1>: Strings And Numbers. (line 56) +* POSIX awk, period (.), using: Regexp Operators. (line 51) +* POSIX awk, printf format strings and: Format Modifiers. (line 157) +* POSIX awk, regular expressions and: Regexp Operators. (line 161) +* POSIX awk, timestamps and: Time Functions. (line 6) +* POSIX awk, | I/O operator and: Getline/Pipe. (line 56) +* POSIX mode: Options. (line 257) +* POSIX mode <1>: Options. (line 343) +* POSIX, awk and: Preface. (line 21) +* POSIX, gawk extensions not included in: POSIX/GNU. (line 6) +* POSIX, programs, implementing in awk: Clones. (line 6) +* POSIXLY_CORRECT environment variable: Options. (line 343) +* PREC variable: User-modified. (line 124) +* precedence: Increment Ops. (line 60) +* precedence <1>: Precedence. (line 6) +* precedence, regexp operators: Regexp Operators. (line 156) +* predefined variables: Built-in Variables. (line 6) +* predefined variables, -v option, setting with: Options. (line 41) +* predefined variables, conveying information: Auto-set. (line 6) +* predefined variables, user-modifiable: User-modified. (line 6) +* print debugger command: Viewing And Changing Data. + (line 35) +* print statement: Printing. (line 16) +* print statement, BEGIN/END patterns and: I/O And BEGIN/END. (line 15) +* print statement, commas, omitting: Print Examples. (line 30) +* print statement, I/O operators in: Precedence. (line 70) +* print statement, line continuations and: Print Examples. (line 75) +* print statement, OFMT variable and: User-modified. (line 113) +* print statement, See Also redirection, of output: Redirection. + (line 17) +* print statement, sprintf() function and: Round Function. (line 6) +* print variables, in debugger: Viewing And Changing Data. + (line 35) +* printf debugger command: Viewing And Changing Data. + (line 53) +* printf statement: Printing. (line 16) +* printf statement <1>: Printf. (line 6) +* printf statement, columns, aligning: Print Examples. (line 69) +* printf statement, format-control characters: Control Letters. + (line 6) +* printf statement, I/O operators in: Precedence. (line 70) +* printf statement, modifiers: Format Modifiers. (line 6) +* printf statement, positional specifiers: Format Modifiers. (line 13) +* printf statement, positional specifiers <1>: Printf Ordering. + (line 6) +* printf statement, positional specifiers, mixing with regular formats: Printf Ordering. + (line 57) +* printf statement, See Also redirection, of output: Redirection. + (line 17) +* printf statement, sprintf() function and: Round Function. (line 6) +* printf statement, syntax of: Basic Printf. (line 6) +* printing: Printing. (line 6) +* printing messages from extensions: Printing Messages. (line 6) +* printing, list of options: Options. (line 154) +* printing, mailing labels: Labels Program. (line 6) +* printing, unduplicated lines of text: Uniq Program. (line 6) +* printing, user information: Id Program. (line 6) +* private variables: Library Names. (line 11) +* process group ID of gawk process: Auto-set. (line 204) +* process ID of gawk process: Auto-set. (line 207) +* processes, two-way communications with: Two-way I/O. (line 6) +* processing data: Basic High Level. (line 6) +* PROCINFO array: Auto-set. (line 148) +* PROCINFO array <1>: Time Functions. (line 47) +* PROCINFO array <2>: Passwd Functions. (line 6) +* PROCINFO array, and communications via ptys: Two-way I/O. (line 114) +* PROCINFO array, and group membership: Group Functions. (line 6) +* PROCINFO array, and user and group ID numbers: Id Program. (line 15) +* PROCINFO array, testing the field splitting: Passwd Functions. + (line 154) +* PROCINFO, values of sorted_in: Controlling Scanning. + (line 26) +* profiling awk programs: Profiling. (line 6) +* profiling awk programs, dynamically: Profiling. (line 177) +* program identifiers: Auto-set. (line 173) +* program, definition of: Getting Started. (line 21) +* programming conventions, --non-decimal-data option: Nondecimal Data. + (line 35) +* programming conventions, ARGC/ARGV variables: Auto-set. (line 35) +* programming conventions, exit statement: Exit Statement. (line 38) +* programming conventions, function parameters: Return Statement. + (line 44) +* programming conventions, functions, calling: Calling Built-in. + (line 10) +* programming conventions, functions, writing: Definition Syntax. + (line 71) +* programming conventions, gawk extensions: Internal File Ops. + (line 45) +* programming conventions, private variable names: Library Names. + (line 23) +* programming language, recipe for: History. (line 6) +* programming languages, Ada: Glossary. (line 11) +* programming languages, data-driven vs. procedural: Getting Started. + (line 12) +* programming languages, Java: Glossary. (line 468) +* programming, basic steps: Basic High Level. (line 18) +* programming, concepts: Basic Concepts. (line 6) +* programming, concepts <1>: Basic Concepts. (line 6) +* pwcat program: Passwd Functions. (line 23) +* q debugger command (alias for quit): Miscellaneous Debugger Commands. + (line 102) +* QSE awk: Other Versions. (line 135) +* Quanstrom, Erik: Alarm Program. (line 8) +* question mark (?), ?: operator: Precedence. (line 91) +* question mark (?), regexp operator: Regexp Operators. (line 111) +* question mark (?), regexp operator <1>: GNU Regexp Operators. + (line 62) +* QuikTrim Awk: Other Versions. (line 139) +* quit debugger command: Miscellaneous Debugger Commands. + (line 102) +* QUIT signal (MS-Windows): Profiling. (line 212) +* quoting in gawk command lines: Long. (line 26) +* quoting in gawk command lines, tricks for: Quoting. (line 91) +* quoting, for small awk programs: Comments. (line 27) +* r debugger command (alias for run): Debugger Execution Control. + (line 62) +* Rakitzis, Byron: History Sorting. (line 25) +* Ramey, Chet: Acknowledgments. (line 60) +* Ramey, Chet <1>: General Data Types. (line 6) +* rand: Numeric Functions. (line 49) +* random numbers, Cliff: Cliff Random Function. + (line 6) +* random numbers, rand()/srand() functions: Numeric Functions. + (line 49) +* random numbers, seed of: Numeric Functions. (line 79) +* range expressions (regexps): Bracket Expressions. (line 6) +* range patterns: Ranges. (line 6) +* range patterns, line continuation and: Ranges. (line 64) +* Rankin, Pat: Acknowledgments. (line 60) +* Rankin, Pat <1>: Assignment Ops. (line 99) +* Rankin, Pat <2>: Contributors. (line 38) +* reada() extension function: Extension Sample Read write array. + (line 18) +* readable data files, checking: File Checking. (line 6) +* readable.awk program: File Checking. (line 11) +* readdir extension: Extension Sample Readdir. + (line 9) +* readfile() extension function: Extension Sample Readfile. + (line 12) +* readfile() user-defined function: Readfile Function. (line 30) +* reading input files: Reading Files. (line 6) +* recipe for a programming language: History. (line 6) +* record separators: awk split records. (line 6) +* record separators <1>: User-modified. (line 133) +* record separators, changing: awk split records. (line 85) +* record separators, regular expressions as: awk split records. + (line 124) +* record separators, with multiline records: Multiple Line. (line 10) +* records: Reading Files. (line 14) +* records <1>: Basic High Level. (line 62) +* records, multiline: Multiple Line. (line 6) +* records, printing: Print. (line 22) +* records, splitting input into: Records. (line 6) +* records, terminating: awk split records. (line 124) +* records, treating files as: gawk split records. (line 92) +* recursive functions: Definition Syntax. (line 89) +* redirect gawk output, in debugger: Debugger Info. (line 73) +* redirection of input: Getline/File. (line 6) +* redirection of output: Redirection. (line 6) +* redirection on VMS: VMS Running. (line 64) +* reference counting, sorting arrays: Array Sorting Functions. + (line 77) +* regexp: Regexp. (line 6) +* regexp constants: Regexp Usage. (line 57) +* regexp constants <1>: Regexp Constants. (line 6) +* regexp constants <2>: Comparison Operators. + (line 103) +* regexp constants, /=.../, /= operator and: Assignment Ops. (line 149) +* regexp constants, as patterns: Expression Patterns. (line 34) +* regexp constants, in gawk: Using Constant Regexps. + (line 28) +* regexp constants, slashes vs. quotes: Computed Regexps. (line 30) +* regexp constants, vs. string constants: Computed Regexps. (line 40) +* register extension: Registration Functions. + (line 6) +* regular expressions: Regexp. (line 6) +* regular expressions as field separators: Field Separators. (line 50) +* regular expressions, anchors in: Regexp Operators. (line 22) +* regular expressions, as field separators: Regexp Field Splitting. + (line 6) +* regular expressions, as patterns: Regexp Usage. (line 6) +* regular expressions, as patterns <1>: Regexp Patterns. (line 6) +* regular expressions, as record separators: awk split records. + (line 124) +* regular expressions, case sensitivity: Case-sensitivity. (line 6) +* regular expressions, case sensitivity <1>: User-modified. (line 76) +* regular expressions, computed: Computed Regexps. (line 6) +* regular expressions, constants, See regexp constants: Regexp Usage. + (line 57) +* regular expressions, dynamic: Computed Regexps. (line 6) +* regular expressions, dynamic, with embedded newlines: Computed Regexps. + (line 60) +* regular expressions, gawk, command-line options: GNU Regexp Operators. + (line 73) +* regular expressions, interval expressions and: Options. (line 278) +* regular expressions, leftmost longest match: Leftmost Longest. + (line 6) +* regular expressions, operators: Regexp Usage. (line 19) +* regular expressions, operators <1>: Regexp Operators. (line 6) +* regular expressions, operators, for buffers: GNU Regexp Operators. + (line 51) +* regular expressions, operators, for words: GNU Regexp Operators. + (line 6) +* regular expressions, operators, gawk: GNU Regexp Operators. + (line 6) +* regular expressions, operators, precedence of: Regexp Operators. + (line 156) +* regular expressions, searching for: Egrep Program. (line 6) +* relational operators, See comparison operators: Typing and Comparison. + (line 9) +* replace in string: String Functions. (line 409) +* retrying input: Retrying Input. (line 6) +* return debugger command: Debugger Execution Control. + (line 54) +* return statement, user-defined functions: Return Statement. (line 6) +* return value, close() function: Close Files And Pipes. + (line 132) +* rev() user-defined function: Function Example. (line 54) +* revoutput extension: Extension Sample Revout. + (line 11) +* revtwoway extension: Extension Sample Rev2way. + (line 12) +* rewind() user-defined function: Rewind Function. (line 15) +* right angle bracket (>), > operator: Comparison Operators. + (line 11) +* right angle bracket (>), > operator <1>: Precedence. (line 64) +* right angle bracket (>), > operator (I/O): Redirection. (line 22) +* right angle bracket (>), >= operator: Comparison Operators. + (line 11) +* right angle bracket (>), >= operator <1>: Precedence. (line 64) +* right angle bracket (>), >> operator (I/O): Redirection. (line 50) +* right angle bracket (>), >> operator (I/O) <1>: Precedence. (line 64) +* right shift: Bitwise Functions. (line 54) +* right shift, bitwise: Bitwise Functions. (line 32) +* Ritchie, Dennis: Basic Data Typing. (line 54) +* RLENGTH variable: Auto-set. (line 282) +* RLENGTH variable, match() function and: String Functions. (line 227) +* Robbins, Arnold: Command Line Field Separator. + (line 71) +* Robbins, Arnold <1>: Getline/Pipe. (line 40) +* Robbins, Arnold <2>: Passwd Functions. (line 90) +* Robbins, Arnold <3>: Alarm Program. (line 6) +* Robbins, Arnold <4>: General Data Types. (line 6) +* Robbins, Arnold <5>: Contributors. (line 145) +* Robbins, Arnold <6>: Maintainers. (line 14) +* Robbins, Arnold <7>: Future Extensions. (line 6) +* Robbins, Bill: Getline/Pipe. (line 40) +* Robbins, Harry: Acknowledgments. (line 94) +* Robbins, Jean: Acknowledgments. (line 94) +* Robbins, Miriam: Acknowledgments. (line 94) +* Robbins, Miriam <1>: Getline/Pipe. (line 40) +* Robbins, Miriam <2>: Passwd Functions. (line 90) +* Rommel, Kai Uwe: Contributors. (line 43) +* round to nearest integer: Numeric Functions. (line 24) +* round() user-defined function: Round Function. (line 16) +* rounding numbers: Round Function. (line 6) +* ROUNDMODE variable: User-modified. (line 128) +* RS variable: awk split records. (line 12) +* RS variable <1>: User-modified. (line 133) +* RS variable, multiline records and: Multiple Line. (line 17) +* rshift: Bitwise Functions. (line 54) +* RSTART variable: Auto-set. (line 288) +* RSTART variable, match() function and: String Functions. (line 227) +* RT variable: awk split records. (line 124) +* RT variable <1>: Multiple Line. (line 130) +* RT variable <2>: Auto-set. (line 295) +* Rubin, Paul: History. (line 30) +* Rubin, Paul <1>: Contributors. (line 16) +* rule, definition of: Getting Started. (line 21) +* run debugger command: Debugger Execution Control. + (line 62) +* rvalues/lvalues: Assignment Ops. (line 31) +* s debugger command (alias for step): Debugger Execution Control. + (line 68) +* sample debugging session: Sample Debugging Session. + (line 6) +* sandbox mode: Options. (line 290) +* save debugger options: Debugger Info. (line 85) +* scalar or array: Type Functions. (line 11) +* scalar values: Basic Data Typing. (line 13) +* scanning arrays: Scanning an Array. (line 6) +* scanning multidimensional arrays: Multiscanning. (line 11) +* Schorr, Andrew: Acknowledgments. (line 60) +* Schorr, Andrew <1>: Auto-set. (line 327) +* Schorr, Andrew <2>: Contributors. (line 134) +* Schreiber, Bert: Acknowledgments. (line 38) +* Schreiber, Rita: Acknowledgments. (line 38) +* search and replace in strings: String Functions. (line 89) +* search in string: String Functions. (line 155) +* search paths: Programs Exercises. (line 70) +* search paths <1>: PC Using. (line 9) +* search paths <2>: VMS Running. (line 57) +* search paths, for loadable extensions: AWKLIBPATH Variable. (line 6) +* search paths, for source files: AWKPATH Variable. (line 6) +* search paths, for source files <1>: Programs Exercises. (line 70) +* search paths, for source files <2>: PC Using. (line 9) +* search paths, for source files <3>: VMS Running. (line 57) +* searching, files for regular expressions: Egrep Program. (line 6) +* searching, for words: Dupword Program. (line 6) +* sed utility: Full Line Fields. (line 22) +* sed utility <1>: Simple Sed. (line 6) +* sed utility <2>: Glossary. (line 16) +* seeding random number generator: Numeric Functions. (line 79) +* semicolon (;), AWKPATH variable and: PC Using. (line 9) +* semicolon (;), separating statements in actions: Statements/Lines. + (line 90) +* semicolon (;), separating statements in actions <1>: Action Overview. + (line 19) +* semicolon (;), separating statements in actions <2>: Statements. + (line 10) +* separators, field: User-modified. (line 50) +* separators, field <1>: User-modified. (line 113) +* separators, field, FIELDWIDTHS variable and: User-modified. (line 37) +* separators, field, FPAT variable and: User-modified. (line 43) +* separators, for records: awk split records. (line 6) +* separators, for records <1>: awk split records. (line 85) +* separators, for records <2>: User-modified. (line 133) +* separators, for records, regular expressions as: awk split records. + (line 124) +* separators, for statements in actions: Action Overview. (line 19) +* separators, subscript: User-modified. (line 146) +* set breakpoint: Breakpoint Control. (line 11) +* set debugger command: Viewing And Changing Data. + (line 58) +* set directory of message catalogs: I18N Functions. (line 11) +* set watchpoint: Viewing And Changing Data. + (line 66) +* shadowing of variable values: Definition Syntax. (line 77) +* shell quoting, rules for: Quoting. (line 6) +* shells, piping commands into: Redirection. (line 136) +* shells, quoting: Using Shell Variables. + (line 12) +* shells, quoting, rules for: Quoting. (line 18) +* shells, scripts: One-shot. (line 22) +* shells, sea: Undocumented. (line 9) +* shells, variables: Using Shell Variables. + (line 6) +* shift, bitwise: Bitwise Functions. (line 32) +* short-circuit operators: Boolean Ops. (line 59) +* show all source files, in debugger: Debugger Info. (line 45) +* show breakpoints: Debugger Info. (line 21) +* show function arguments, in debugger: Debugger Info. (line 18) +* show local variables, in debugger: Debugger Info. (line 34) +* show name of current source file, in debugger: Debugger Info. + (line 37) +* show watchpoints: Debugger Info. (line 51) +* si debugger command (alias for stepi): Debugger Execution Control. + (line 75) +* side effects: Concatenation. (line 41) +* side effects <1>: Increment Ops. (line 11) +* side effects <2>: Increment Ops. (line 75) +* side effects, array indexing: Reference to Elements. + (line 43) +* side effects, asort() function: Array Sorting Functions. + (line 24) +* side effects, assignment expressions: Assignment Ops. (line 22) +* side effects, Boolean operators: Boolean Ops. (line 30) +* side effects, conditional expressions: Conditional Exp. (line 22) +* side effects, decrement/increment operators: Increment Ops. (line 11) +* side effects, FILENAME variable: Getline Notes. (line 19) +* side effects, function calls: Function Calls. (line 57) +* side effects, statements: Action Overview. (line 32) +* sidebar, A Constant's Base Does Not Affect Its Value: Nondecimal-numbers. + (line 63) +* sidebar, Backslash Before Regular Characters: Escape Sequences. + (line 106) +* sidebar, Beware The Smoke and Mirrors!: Bitwise Functions. (line 126) +* sidebar, Changing FS Does Not Affect the Fields: Full Line Fields. + (line 14) +* sidebar, Changing NR and FNR: Auto-set. (line 355) +* sidebar, Controlling Output Buffering with system(): I/O Functions. + (line 164) +* sidebar, Escape Sequences for Metacharacters: Escape Sequences. + (line 138) +* sidebar, FS and IGNORECASE: Field Splitting Summary. + (line 37) +* sidebar, Interactive Versus Noninteractive Buffering: I/O Functions. + (line 74) +* sidebar, Matching the Null String: String Functions. (line 535) +* sidebar, Operator Evaluation Order: Increment Ops. (line 58) +* sidebar, Piping into sh: Redirection. (line 134) +* sidebar, Pre-POSIX awk Used OFMT for String Conversion: Strings And Numbers. + (line 54) +* sidebar, Recipe for a Programming Language: History. (line 6) +* sidebar, RS = "\0" Is Not Portable: gawk split records. (line 63) +* sidebar, So Why Does gawk Have BEGINFILE and ENDFILE?: Filetrans Function. + (line 83) +* sidebar, Syntactic Ambiguities Between /= and Regular Expressions: Assignment Ops. + (line 147) +* sidebar, Understanding #!: Executable Scripts. (line 31) +* sidebar, Understanding $0: Changing Fields. (line 134) +* sidebar, Using close()'s Return Value: Close Files And Pipes. + (line 130) +* sidebar, Using \n in Bracket Expressions of Dynamic Regexps: Computed Regexps. + (line 58) +* SIGHUP signal, for dynamic profiling: Profiling. (line 209) +* SIGINT signal (MS-Windows): Profiling. (line 212) +* signals, HUP/SIGHUP, for profiling: Profiling. (line 209) +* signals, INT/SIGINT (MS-Windows): Profiling. (line 212) +* signals, QUIT/SIGQUIT (MS-Windows): Profiling. (line 212) +* signals, USR1/SIGUSR1, for profiling: Profiling. (line 186) +* signature program: Signature Program. (line 6) +* SIGQUIT signal (MS-Windows): Profiling. (line 212) +* SIGUSR1 signal, for dynamic profiling: Profiling. (line 186) +* silent debugger command: Debugger Execution Control. + (line 10) +* sin: Numeric Functions. (line 90) +* sine: Numeric Functions. (line 90) +* single quote ('): One-shot. (line 15) +* single quote (') in gawk command lines: Long. (line 35) +* single quote ('), in shell commands: Quoting. (line 48) +* single quote ('), vs. apostrophe: Comments. (line 27) +* single quote ('), with double quotes: Quoting. (line 73) +* single-character fields: Single Character Fields. + (line 6) +* single-step execution, in the debugger: Debugger Execution Control. + (line 43) +* Skywalker, Luke: Undocumented. (line 6) +* sleep utility: Alarm Program. (line 109) +* sleep() extension function: Extension Sample Time. + (line 22) +* Solaris, POSIX-compliant awk: Other Versions. (line 100) +* sort array: String Functions. (line 42) +* sort array indices: String Functions. (line 42) +* sort function, arrays, sorting: Array Sorting Functions. + (line 6) +* sort utility: Word Sorting. (line 50) +* sort utility, coprocesses and: Two-way I/O. (line 66) +* sorting characters in different languages: Explaining gettext. + (line 94) +* source code, awka: Other Versions. (line 68) +* source code, Brian Kernighan's awk: Other Versions. (line 13) +* source code, BusyBox Awk: Other Versions. (line 92) +* source code, gawk: Gawk Distribution. (line 6) +* source code, Illumos awk: Other Versions. (line 109) +* source code, jawk: Other Versions. (line 117) +* source code, libmawk: Other Versions. (line 125) +* source code, mawk: Other Versions. (line 48) +* source code, mixing: Options. (line 117) +* source code, pawk: Other Versions. (line 82) +* source code, pawk (Python version): Other Versions. (line 129) +* source code, QSE awk: Other Versions. (line 135) +* source code, QuikTrim Awk: Other Versions. (line 139) +* source code, Solaris awk: Other Versions. (line 100) +* source files, search path for: Programs Exercises. (line 70) +* sparse arrays: Array Intro. (line 76) +* Spencer, Henry: Glossary. (line 16) +* split: String Functions. (line 315) +* split string into array: String Functions. (line 296) +* split utility: Split Program. (line 6) +* split() function, array elements, deleting: Delete. (line 61) +* split.awk program: Split Program. (line 30) +* sprintf: OFMT. (line 15) +* sprintf <1>: String Functions. (line 384) +* sprintf() function, OFMT variable and: User-modified. (line 113) +* sprintf() function, print/printf statements and: Round Function. + (line 6) +* sqrt: Numeric Functions. (line 93) +* square brackets ([]), regexp operator: Regexp Operators. (line 56) +* square root: Numeric Functions. (line 93) +* srand: Numeric Functions. (line 97) +* stack frame: Debugging Terms. (line 10) +* Stallman, Richard: Manual History. (line 6) +* Stallman, Richard <1>: Acknowledgments. (line 18) +* Stallman, Richard <2>: Contributors. (line 24) +* Stallman, Richard <3>: Glossary. (line 372) +* standard error: Special FD. (line 6) +* standard input: Read Terminal. (line 6) +* standard input <1>: Special FD. (line 6) +* standard output: Special FD. (line 6) +* starting the debugger: Debugger Invocation. (line 6) +* stat() extension function: Extension Sample File Functions. + (line 18) +* statements, compound, control statements and: Statements. (line 10) +* statements, control, in actions: Statements. (line 6) +* statements, multiple: Statements/Lines. (line 90) +* step debugger command: Debugger Execution Control. + (line 68) +* stepi debugger command: Debugger Execution Control. + (line 75) +* stop automatic display, in debugger: Viewing And Changing Data. + (line 79) +* stream editors: Full Line Fields. (line 22) +* stream editors <1>: Simple Sed. (line 6) +* strftime: Time Functions. (line 48) +* string constants: Scalar Constants. (line 15) +* string constants, vs. regexp constants: Computed Regexps. (line 40) +* string extraction (internationalization): String Extraction. + (line 6) +* string length: String Functions. (line 170) +* string operators: Concatenation. (line 9) +* string, regular expression match: String Functions. (line 210) +* string-manipulation functions: String Functions. (line 6) +* string-matching operators: Regexp Usage. (line 19) +* string-translation functions: I18N Functions. (line 6) +* strings splitting, example: String Functions. (line 334) +* strings, converting: Strings And Numbers. (line 6) +* strings, converting <1>: Bitwise Functions. (line 108) +* strings, converting letter case: String Functions. (line 523) +* strings, converting, numbers to: User-modified. (line 30) +* strings, converting, numbers to <1>: User-modified. (line 104) +* strings, empty, See null strings: awk split records. (line 114) +* strings, extracting: String Extraction. (line 6) +* strings, for localization: Programmer i18n. (line 13) +* strings, length limitations: Scalar Constants. (line 20) +* strings, merging arrays into: Join Function. (line 6) +* strings, null: Regexp Field Splitting. + (line 43) +* strings, numeric: Variable Typing. (line 6) +* strtonum: String Functions. (line 391) +* strtonum() function (gawk), --non-decimal-data option and: Nondecimal Data. + (line 35) +* sub: Using Constant Regexps. + (line 43) +* sub <1>: String Functions. (line 409) +* sub() function, arguments of: String Functions. (line 463) +* sub() function, escape processing: Gory Details. (line 6) +* subscript separators: User-modified. (line 146) +* subscripts in arrays, multidimensional: Multidimensional. (line 10) +* subscripts in arrays, multidimensional, scanning: Multiscanning. + (line 11) +* subscripts in arrays, numbers as: Numeric Array Subscripts. + (line 6) +* subscripts in arrays, uninitialized variables as: Uninitialized Subscripts. + (line 6) +* SUBSEP variable: User-modified. (line 146) +* SUBSEP variable, and multidimensional arrays: Multidimensional. + (line 16) +* substitute in string: String Functions. (line 89) +* substr: String Functions. (line 482) +* substring: String Functions. (line 482) +* Sumner, Andrew: Other Versions. (line 68) +* supplementary groups of gawk process: Auto-set. (line 251) +* switch statement: Switch Statement. (line 6) +* SYMTAB array: Auto-set. (line 299) +* syntactic ambiguity: /= operator vs. /=.../ regexp constant: Assignment Ops. + (line 149) +* system: I/O Functions. (line 107) +* systime: Time Functions. (line 66) +* t debugger command (alias for tbreak): Breakpoint Control. (line 90) +* tbreak debugger command: Breakpoint Control. (line 90) +* Tcl: Library Names. (line 58) +* TCP/IP: TCP/IP Networking. (line 6) +* TCP/IP, support for: Special Network. (line 6) +* tee utility: Tee Program. (line 6) +* tee.awk program: Tee Program. (line 26) +* temporary breakpoint: Breakpoint Control. (line 90) +* terminating records: awk split records. (line 124) +* testbits.awk program: Bitwise Functions. (line 69) +* testext extension: Extension Sample API Tests. + (line 6) +* Texinfo: Conventions. (line 6) +* Texinfo <1>: Library Functions. (line 33) +* Texinfo <2>: Dupword Program. (line 17) +* Texinfo <3>: Extract Program. (line 12) +* Texinfo <4>: Distribution contents. + (line 77) +* Texinfo <5>: Adding Code. (line 100) +* Texinfo, chapter beginnings in files: Regexp Operators. (line 22) +* Texinfo, extracting programs from source files: Extract Program. + (line 6) +* text, printing: Print. (line 22) +* text, printing, unduplicated lines of: Uniq Program. (line 6) +* TEXTDOMAIN variable: User-modified. (line 152) +* TEXTDOMAIN variable <1>: Programmer i18n. (line 8) +* TEXTDOMAIN variable, BEGIN pattern and: Programmer i18n. (line 60) +* TEXTDOMAIN variable, portability and: I18N Portability. (line 20) +* textdomain() function (C library): Explaining gettext. (line 28) +* tilde (~), ~ operator: Regexp Usage. (line 19) +* tilde (~), ~ operator <1>: Computed Regexps. (line 6) +* tilde (~), ~ operator <2>: Case-sensitivity. (line 26) +* tilde (~), ~ operator <3>: Regexp Constants. (line 6) +* tilde (~), ~ operator <4>: Comparison Operators. + (line 11) +* tilde (~), ~ operator <5>: Comparison Operators. + (line 98) +* tilde (~), ~ operator <6>: Precedence. (line 79) +* tilde (~), ~ operator <7>: Expression Patterns. (line 24) +* time functions: Time Functions. (line 6) +* time, alarm clock example program: Alarm Program. (line 11) +* time, localization and: Explaining gettext. (line 112) +* time, managing: Getlocaltime Function. + (line 6) +* time, retrieving: Time Functions. (line 17) +* timeout, reading input: Read Timeout. (line 6) +* timestamps: Time Functions. (line 6) +* timestamps <1>: Time Functions. (line 66) +* timestamps, converting dates to: Time Functions. (line 76) +* timestamps, formatted: Getlocaltime Function. + (line 6) +* tolower: String Functions. (line 524) +* toupper: String Functions. (line 530) +* tr utility: Translate Program. (line 6) +* trace debugger command: Miscellaneous Debugger Commands. + (line 110) +* traceback, display in debugger: Execution Stack. (line 13) +* translate string: I18N Functions. (line 21) +* translate.awk program: Translate Program. (line 55) +* treating files, as single records: gawk split records. (line 92) +* troubleshooting, --non-decimal-data option: Options. (line 209) +* troubleshooting, == operator: Comparison Operators. + (line 37) +* troubleshooting, awk uses FS not IFS: Field Separators. (line 29) +* troubleshooting, backslash before nonspecial character: Escape Sequences. + (line 108) +* troubleshooting, division: Arithmetic Ops. (line 44) +* troubleshooting, fatal errors, field widths, specifying: Constant Size. + (line 22) +* troubleshooting, fatal errors, printf format strings: Format Modifiers. + (line 157) +* troubleshooting, fflush() function: I/O Functions. (line 63) +* troubleshooting, function call syntax: Function Calls. (line 30) +* troubleshooting, gawk: Compatibility Mode. (line 6) +* troubleshooting, gawk, bug reports: Bugs. (line 9) +* troubleshooting, gawk, fatal errors, function arguments: Calling Built-in. + (line 16) +* troubleshooting, getline function: File Checking. (line 25) +* troubleshooting, gsub()/sub() functions: String Functions. (line 473) +* troubleshooting, match() function: String Functions. (line 291) +* troubleshooting, print statement, omitting commas: Print Examples. + (line 30) +* troubleshooting, printing: Redirection. (line 112) +* troubleshooting, quotes with file names: Special FD. (line 62) +* troubleshooting, readable data files: File Checking. (line 6) +* troubleshooting, regexp constants vs. string constants: Computed Regexps. + (line 40) +* troubleshooting, string concatenation: Concatenation. (line 27) +* troubleshooting, substr() function: String Functions. (line 500) +* troubleshooting, system() function: I/O Functions. (line 129) +* troubleshooting, typographical errors, global variables: Options. + (line 99) +* true, logical: Truth Values. (line 6) +* Trueman, David: History. (line 30) +* Trueman, David <1>: Acknowledgments. (line 47) +* Trueman, David <2>: Contributors. (line 31) +* trunc-mod operation: Arithmetic Ops. (line 66) +* truth values: Truth Values. (line 6) +* type conversion: Strings And Numbers. (line 21) +* type, of variable: Type Functions. (line 14) +* typeof: Type Functions. (line 14) +* u debugger command (alias for until): Debugger Execution Control. + (line 82) +* unassigned array elements: Reference to Elements. + (line 18) +* undefined functions: Pass By Value/Reference. + (line 68) +* underscore (_), C macro: Explaining gettext. (line 71) +* underscore (_), in names of private variables: Library Names. + (line 29) +* underscore (_), translatable string: Programmer i18n. (line 69) +* undisplay debugger command: Viewing And Changing Data. + (line 79) +* undocumented features: Undocumented. (line 6) +* Unicode: Ordinal Functions. (line 45) +* Unicode <1>: Ranges and Locales. (line 61) +* Unicode <2>: Glossary. (line 196) +* uninitialized variables, as array subscripts: Uninitialized Subscripts. + (line 6) +* uniq utility: Uniq Program. (line 6) +* uniq.awk program: Uniq Program. (line 65) +* Unix: Glossary. (line 748) +* Unix awk, backslashes in escape sequences: Escape Sequences. + (line 121) +* Unix awk, close() function and: Close Files And Pipes. + (line 132) +* Unix awk, password files, field separators and: Command Line Field Separator. + (line 62) +* Unix, awk scripts and: Executable Scripts. (line 6) +* unsigned integers: Computer Arithmetic. (line 41) +* until debugger command: Debugger Execution Control. + (line 82) +* unwatch debugger command: Viewing And Changing Data. + (line 83) +* up debugger command: Execution Stack. (line 36) +* user database, reading: Passwd Functions. (line 6) +* user-defined functions: User-defined. (line 6) +* user-defined, functions, counts, in a profile: Profiling. (line 137) +* user-defined, variables: Variables. (line 6) +* user-modifiable variables: User-modified. (line 6) +* users, information about, printing: Id Program. (line 6) +* users, information about, retrieving: Passwd Functions. (line 16) +* USR1 signal, for dynamic profiling: Profiling. (line 186) +* values, numeric: Basic Data Typing. (line 13) +* values, string: Basic Data Typing. (line 13) +* variable assignments and input files: Other Arguments. (line 26) +* variable type: Type Functions. (line 14) +* variable typing: Typing and Comparison. + (line 9) +* variables: Other Features. (line 6) +* variables <1>: Basic Data Typing. (line 6) +* variables, assigning on command line: Assignment Options. (line 6) +* variables, built-in: Using Variables. (line 23) +* variables, flag: Boolean Ops. (line 69) +* variables, getline command into, using: Getline/Variable. (line 6) +* variables, getline command into, using <1>: Getline/Variable/File. + (line 6) +* variables, getline command into, using <2>: Getline/Variable/Pipe. + (line 6) +* variables, getline command into, using <3>: Getline/Variable/Coprocess. + (line 6) +* variables, global, for library functions: Library Names. (line 11) +* variables, global, printing list of: Options. (line 94) +* variables, initializing: Using Variables. (line 23) +* variables, local to a function: Variable Scope. (line 6) +* variables, predefined: Built-in Variables. (line 6) +* variables, predefined -v option, setting with: Options. (line 41) +* variables, predefined conveying information: Auto-set. (line 6) +* variables, private: Library Names. (line 11) +* variables, setting: Options. (line 32) +* variables, shadowing: Definition Syntax. (line 77) +* variables, types of: Assignment Ops. (line 39) +* variables, types of, comparison expressions and: Typing and Comparison. + (line 9) +* variables, uninitialized, as array subscripts: Uninitialized Subscripts. + (line 6) +* variables, user-defined: Variables. (line 6) +* version of gawk: Auto-set. (line 221) +* version of gawk extension API: Auto-set. (line 246) +* version of GNU MP library: Auto-set. (line 229) +* version of GNU MPFR library: Auto-set. (line 231) +* vertical bar (|): Regexp Operators. (line 70) +* vertical bar (|), | operator (I/O): Getline/Pipe. (line 10) +* vertical bar (|), | operator (I/O) <1>: Precedence. (line 64) +* vertical bar (|), |& operator (I/O): Getline/Coprocess. (line 6) +* vertical bar (|), |& operator (I/O) <1>: Precedence. (line 64) +* vertical bar (|), |& operator (I/O) <2>: Two-way I/O. (line 27) +* vertical bar (|), || operator: Boolean Ops. (line 59) +* vertical bar (|), || operator <1>: Precedence. (line 88) +* Vinschen, Corinna: Acknowledgments. (line 60) +* w debugger command (alias for watch): Viewing And Changing Data. + (line 66) +* w utility: Constant Size. (line 22) +* wait() extension function: Extension Sample Fork. + (line 22) +* waitpid() extension function: Extension Sample Fork. + (line 18) +* walk_array() user-defined function: Walking Arrays. (line 14) +* Wall, Larry: Array Intro. (line 6) +* Wall, Larry <1>: Future Extensions. (line 6) +* Wallin, Anders: Contributors. (line 104) +* warnings, issuing: Options. (line 184) +* watch debugger command: Viewing And Changing Data. + (line 66) +* watchpoint: Debugging Terms. (line 42) +* wc utility: Wc Program. (line 6) +* wc.awk program: Wc Program. (line 46) +* Weinberger, Peter: History. (line 17) +* Weinberger, Peter <1>: Contributors. (line 12) +* where debugger command: Execution Stack. (line 13) +* where debugger command (alias for backtrace): Execution Stack. + (line 13) +* while statement: While Statement. (line 6) +* while statement, use of regexps in: Regexp Usage. (line 19) +* whitespace, as field separators: Default Field Splitting. + (line 6) +* whitespace, functions, calling: Calling Built-in. (line 10) +* whitespace, newlines as: Options. (line 263) +* Williams, Kent: Contributors. (line 35) +* Woehlke, Matthew: Contributors. (line 80) +* Woods, John: Contributors. (line 28) +* word boundaries, matching: GNU Regexp Operators. + (line 41) +* word, regexp definition of: GNU Regexp Operators. + (line 6) +* word-boundary operator (gawk): GNU Regexp Operators. + (line 66) +* wordfreq.awk program: Word Sorting. (line 56) +* words, counting: Wc Program. (line 6) +* words, duplicate, searching for: Dupword Program. (line 6) +* words, usage counts, generating: Word Sorting. (line 6) +* writea() extension function: Extension Sample Read write array. + (line 12) +* xgettext utility: String Extraction. (line 13) +* xor: Bitwise Functions. (line 57) +* XOR bitwise operation: Bitwise Functions. (line 6) +* Yawitz, Efraim: Contributors. (line 132) +* Zaretskii, Eli: Acknowledgments. (line 60) +* Zaretskii, Eli <1>: Contributors. (line 56) +* Zaretskii, Eli <2>: Maintainers. (line 14) +* zerofile.awk program: Empty Files. (line 20) +* Zoulas, Christos: Contributors. (line 67) + + + +Tag Table: +Node: Top1200 +Node: Foreword342530 +Node: Foreword446972 +Node: Preface48504 +Ref: Preface-Footnote-151363 +Ref: Preface-Footnote-251470 +Ref: Preface-Footnote-351704 +Node: History51846 +Node: Names54198 +Ref: Names-Footnote-155292 +Node: This Manual55439 +Ref: This Manual-Footnote-161924 +Node: Conventions62024 +Node: Manual History64378 +Ref: Manual History-Footnote-167373 +Ref: Manual History-Footnote-267414 +Node: How To Contribute67488 +Node: Acknowledgments68617 +Node: Getting Started73503 +Node: Running gawk75942 +Node: One-shot77132 +Node: Read Terminal78395 +Node: Long80388 +Node: Executable Scripts81901 +Ref: Executable Scripts-Footnote-184696 +Node: Comments84799 +Node: Quoting87283 +Node: DOS Quoting92800 +Node: Sample Data Files93475 +Node: Very Simple96070 +Node: Two Rules100972 +Node: More Complex102857 +Node: Statements/Lines105723 +Ref: Statements/Lines-Footnote-1110182 +Node: Other Features110447 +Node: When111383 +Ref: When-Footnote-1113137 +Node: Intro Summary113202 +Node: Invoking Gawk114086 +Node: Command Line115600 +Node: Options116398 +Ref: Options-Footnote-1132497 +Ref: Options-Footnote-2132727 +Node: Other Arguments132752 +Node: Naming Standard Input135699 +Node: Environment Variables136792 +Node: AWKPATH Variable137350 +Ref: AWKPATH Variable-Footnote-1140761 +Ref: AWKPATH Variable-Footnote-2140795 +Node: AWKLIBPATH Variable141056 +Node: Other Environment Variables142313 +Node: Exit Status146134 +Node: Include Files146811 +Node: Loading Shared Libraries150406 +Node: Obsolete151834 +Node: Undocumented152526 +Node: Invoking Summary152823 +Node: Regexp154483 +Node: Regexp Usage156002 +Node: Escape Sequences158039 +Node: Regexp Operators164271 +Ref: Regexp Operators-Footnote-1171687 +Ref: Regexp Operators-Footnote-2171834 +Node: Bracket Expressions171932 +Ref: table-char-classes174408 +Node: Leftmost Longest177545 +Node: Computed Regexps178848 +Node: GNU Regexp Operators182275 +Node: Case-sensitivity185954 +Ref: Case-sensitivity-Footnote-1188850 +Ref: Case-sensitivity-Footnote-2189085 +Node: Strong Regexp Constants189193 +Node: Regexp Summary189982 +Node: Reading Files191457 +Node: Records193620 +Node: awk split records194353 +Node: gawk split records199284 +Ref: gawk split records-Footnote-1203824 +Node: Fields203861 +Node: Nonconstant Fields206602 +Ref: Nonconstant Fields-Footnote-1208838 +Node: Changing Fields209042 +Node: Field Separators214970 +Node: Default Field Splitting217668 +Node: Regexp Field Splitting218786 +Node: Single Character Fields222139 +Node: Command Line Field Separator223199 +Node: Full Line Fields226417 +Ref: Full Line Fields-Footnote-1227939 +Ref: Full Line Fields-Footnote-2227985 +Node: Field Splitting Summary228086 +Node: Constant Size230160 +Node: Splitting By Content234738 +Ref: Splitting By Content-Footnote-1238709 +Node: Multiple Line238872 +Ref: Multiple Line-Footnote-1244754 +Node: Getline244933 +Node: Plain Getline247400 +Node: Getline/Variable250039 +Node: Getline/File251188 +Node: Getline/Variable/File252574 +Ref: Getline/Variable/File-Footnote-1254177 +Node: Getline/Pipe254265 +Node: Getline/Variable/Pipe256970 +Node: Getline/Coprocess258103 +Node: Getline/Variable/Coprocess259368 +Node: Getline Notes260108 +Node: Getline Summary262903 +Ref: table-getline-variants263325 +Node: Read Timeout264073 +Ref: Read Timeout-Footnote-1267979 +Node: Retrying Input268037 +Node: Command-line directories269236 +Node: Input Summary270142 +Node: Input Exercises273314 +Node: Printing274042 +Node: Print275876 +Node: Print Examples277333 +Node: Output Separators280113 +Node: OFMT282130 +Node: Printf283486 +Node: Basic Printf284271 +Node: Control Letters285845 +Node: Format Modifiers289833 +Node: Printf Examples295848 +Node: Redirection298334 +Node: Special FD305175 +Ref: Special FD-Footnote-1308343 +Node: Special Files308417 +Node: Other Inherited Files309034 +Node: Special Network310035 +Node: Special Caveats310895 +Node: Close Files And Pipes311844 +Ref: table-close-pipe-return-values318751 +Ref: Close Files And Pipes-Footnote-1319534 +Ref: Close Files And Pipes-Footnote-2319682 +Node: Nonfatal319834 +Node: Output Summary322159 +Node: Output Exercises323381 +Node: Expressions324060 +Node: Values325248 +Node: Constants325926 +Node: Scalar Constants326617 +Ref: Scalar Constants-Footnote-1327481 +Node: Nondecimal-numbers327731 +Node: Regexp Constants330744 +Node: Using Constant Regexps331270 +Node: Variables334433 +Node: Using Variables335090 +Node: Assignment Options337000 +Node: Conversion338873 +Node: Strings And Numbers339397 +Ref: Strings And Numbers-Footnote-1342460 +Node: Locale influences conversions342569 +Ref: table-locale-affects345327 +Node: All Operators345945 +Node: Arithmetic Ops346574 +Node: Concatenation349080 +Ref: Concatenation-Footnote-1351927 +Node: Assignment Ops352034 +Ref: table-assign-ops357025 +Node: Increment Ops358338 +Node: Truth Values and Conditions361798 +Node: Truth Values362872 +Node: Typing and Comparison363920 +Node: Variable Typing364740 +Node: Comparison Operators368364 +Ref: table-relational-ops368783 +Node: POSIX String Comparison372278 +Ref: POSIX String Comparison-Footnote-1373973 +Ref: POSIX String Comparison-Footnote-2374112 +Node: Boolean Ops374196 +Ref: Boolean Ops-Footnote-1378678 +Node: Conditional Exp378770 +Node: Function Calls380506 +Node: Precedence384383 +Node: Locales388042 +Node: Expressions Summary389674 +Node: Patterns and Actions392247 +Node: Pattern Overview393367 +Node: Regexp Patterns395044 +Node: Expression Patterns395586 +Node: Ranges399367 +Node: BEGIN/END402475 +Node: Using BEGIN/END403236 +Ref: Using BEGIN/END-Footnote-1405972 +Node: I/O And BEGIN/END406078 +Node: BEGINFILE/ENDFILE408392 +Node: Empty411299 +Node: Using Shell Variables411616 +Node: Action Overview413890 +Node: Statements416215 +Node: If Statement418063 +Node: While Statement419558 +Node: Do Statement421586 +Node: For Statement422734 +Node: Switch Statement425892 +Node: Break Statement428278 +Node: Continue Statement430370 +Node: Next Statement432197 +Node: Nextfile Statement434580 +Node: Exit Statement437232 +Node: Built-in Variables439635 +Node: User-modified440768 +Node: Auto-set448354 +Ref: Auto-set-Footnote-1463007 +Ref: Auto-set-Footnote-2463213 +Node: ARGC and ARGV463269 +Node: Pattern Action Summary467482 +Node: Arrays469912 +Node: Array Basics471241 +Node: Array Intro472085 +Ref: figure-array-elements474060 +Ref: Array Intro-Footnote-1476764 +Node: Reference to Elements476892 +Node: Assigning Elements479356 +Node: Array Example479847 +Node: Scanning an Array481606 +Node: Controlling Scanning484628 +Ref: Controlling Scanning-Footnote-1490027 +Node: Numeric Array Subscripts490343 +Node: Uninitialized Subscripts492527 +Node: Delete494146 +Ref: Delete-Footnote-1496898 +Node: Multidimensional496955 +Node: Multiscanning500050 +Node: Arrays of Arrays501641 +Node: Arrays Summary506408 +Node: Functions508501 +Node: Built-in509539 +Node: Calling Built-in510620 +Node: Numeric Functions512616 +Ref: Numeric Functions-Footnote-1517449 +Ref: Numeric Functions-Footnote-2517806 +Ref: Numeric Functions-Footnote-3517854 +Node: String Functions518126 +Ref: String Functions-Footnote-1541630 +Ref: String Functions-Footnote-2541758 +Ref: String Functions-Footnote-3542006 +Node: Gory Details542093 +Ref: table-sub-escapes543884 +Ref: table-sub-proposed545403 +Ref: table-posix-sub546766 +Ref: table-gensub-escapes548307 +Ref: Gory Details-Footnote-1549130 +Node: I/O Functions549284 +Ref: table-system-return-values555866 +Ref: I/O Functions-Footnote-1557846 +Ref: I/O Functions-Footnote-2557994 +Node: Time Functions558114 +Ref: Time Functions-Footnote-1568636 +Ref: Time Functions-Footnote-2568704 +Ref: Time Functions-Footnote-3568862 +Ref: Time Functions-Footnote-4568973 +Ref: Time Functions-Footnote-5569085 +Ref: Time Functions-Footnote-6569312 +Node: Bitwise Functions569578 +Ref: table-bitwise-ops570172 +Ref: Bitwise Functions-Footnote-1576217 +Ref: Bitwise Functions-Footnote-2576390 +Node: Type Functions576581 +Node: I18N Functions579113 +Node: User-defined580764 +Node: Definition Syntax581569 +Ref: Definition Syntax-Footnote-1587256 +Node: Function Example587327 +Ref: Function Example-Footnote-1590249 +Node: Function Caveats590271 +Node: Calling A Function590789 +Node: Variable Scope591747 +Node: Pass By Value/Reference594741 +Node: Return Statement598240 +Node: Dynamic Typing601219 +Node: Indirect Calls602149 +Ref: Indirect Calls-Footnote-1612400 +Node: Functions Summary612528 +Node: Library Functions615233 +Ref: Library Functions-Footnote-1618840 +Ref: Library Functions-Footnote-2618983 +Node: Library Names619154 +Ref: Library Names-Footnote-1622614 +Ref: Library Names-Footnote-2622837 +Node: General Functions622923 +Node: Strtonum Function624026 +Node: Assert Function627048 +Node: Round Function630374 +Node: Cliff Random Function631915 +Node: Ordinal Functions632931 +Ref: Ordinal Functions-Footnote-1635994 +Ref: Ordinal Functions-Footnote-2636246 +Node: Join Function636456 +Ref: Join Function-Footnote-1638226 +Node: Getlocaltime Function638426 +Node: Readfile Function642168 +Node: Shell Quoting644140 +Node: Data File Management645541 +Node: Filetrans Function646173 +Node: Rewind Function650269 +Node: File Checking652175 +Ref: File Checking-Footnote-1653509 +Node: Empty Files653710 +Node: Ignoring Assigns655689 +Node: Getopt Function657239 +Ref: Getopt Function-Footnote-1668708 +Node: Passwd Functions668908 +Ref: Passwd Functions-Footnote-1677747 +Node: Group Functions677835 +Ref: Group Functions-Footnote-1685733 +Node: Walking Arrays685940 +Node: Library Functions Summary688948 +Node: Library Exercises690354 +Node: Sample Programs690819 +Node: Running Examples691589 +Node: Clones692317 +Node: Cut Program693541 +Node: Egrep Program703470 +Ref: Egrep Program-Footnote-1710982 +Node: Id Program711092 +Node: Split Program714772 +Ref: Split Program-Footnote-1718231 +Node: Tee Program718360 +Node: Uniq Program721150 +Node: Wc Program728576 +Ref: Wc Program-Footnote-1732831 +Node: Miscellaneous Programs732925 +Node: Dupword Program734138 +Node: Alarm Program736168 +Node: Translate Program741023 +Ref: Translate Program-Footnote-1745588 +Node: Labels Program745858 +Ref: Labels Program-Footnote-1749209 +Node: Word Sorting749293 +Node: History Sorting753365 +Node: Extract Program755200 +Node: Simple Sed762729 +Node: Igawk Program765803 +Ref: Igawk Program-Footnote-1780134 +Ref: Igawk Program-Footnote-2780336 +Ref: Igawk Program-Footnote-3780458 +Node: Anagram Program780573 +Node: Signature Program783635 +Node: Programs Summary784882 +Node: Programs Exercises786096 +Ref: Programs Exercises-Footnote-1790225 +Node: Advanced Features790316 +Node: Nondecimal Data792306 +Node: Array Sorting793897 +Node: Controlling Array Traversal794597 +Ref: Controlling Array Traversal-Footnote-1802964 +Node: Array Sorting Functions803082 +Ref: Array Sorting Functions-Footnote-1808173 +Node: Two-way I/O808369 +Ref: Two-way I/O-Footnote-1814919 +Ref: Two-way I/O-Footnote-2815106 +Node: TCP/IP Networking815188 +Node: Profiling818306 +Ref: Profiling-Footnote-1826799 +Node: Advanced Features Summary827122 +Node: Internationalization828966 +Node: I18N and L10N830446 +Node: Explaining gettext831133 +Ref: Explaining gettext-Footnote-1837025 +Ref: Explaining gettext-Footnote-2837210 +Node: Programmer i18n837375 +Ref: Programmer i18n-Footnote-1842230 +Node: Translator i18n842279 +Node: String Extraction843073 +Ref: String Extraction-Footnote-1844205 +Node: Printf Ordering844291 +Ref: Printf Ordering-Footnote-1847077 +Node: I18N Portability847141 +Ref: I18N Portability-Footnote-1849597 +Node: I18N Example849660 +Ref: I18N Example-Footnote-1852466 +Node: Gawk I18N852539 +Node: I18N Summary853184 +Node: Debugger854525 +Node: Debugging855547 +Node: Debugging Concepts855988 +Node: Debugging Terms857797 +Node: Awk Debugging860372 +Node: Sample Debugging Session861278 +Node: Debugger Invocation861812 +Node: Finding The Bug863198 +Node: List of Debugger Commands869676 +Node: Breakpoint Control871009 +Node: Debugger Execution Control874703 +Node: Viewing And Changing Data878065 +Node: Execution Stack881439 +Node: Debugger Info883076 +Node: Miscellaneous Debugger Commands887147 +Node: Readline Support892235 +Node: Limitations893131 +Ref: Limitations-Footnote-1897362 +Node: Debugging Summary897413 +Node: Arbitrary Precision Arithmetic898692 +Node: Computer Arithmetic900108 +Ref: table-numeric-ranges903699 +Ref: Computer Arithmetic-Footnote-1904421 +Node: Math Definitions904478 +Ref: table-ieee-formats907792 +Ref: Math Definitions-Footnote-1908395 +Node: MPFR features908500 +Node: FP Math Caution910217 +Ref: FP Math Caution-Footnote-1911289 +Node: Inexactness of computations911658 +Node: Inexact representation912618 +Node: Comparing FP Values913978 +Node: Errors accumulate915060 +Node: Getting Accuracy916493 +Node: Try To Round919203 +Node: Setting precision920102 +Ref: table-predefined-precision-strings920799 +Node: Setting the rounding mode922629 +Ref: table-gawk-rounding-modes923003 +Ref: Setting the rounding mode-Footnote-1926411 +Node: Arbitrary Precision Integers926590 +Ref: Arbitrary Precision Integers-Footnote-1931507 +Node: POSIX Floating Point Problems931656 +Ref: POSIX Floating Point Problems-Footnote-1935538 +Node: Floating point summary935576 +Node: Dynamic Extensions937766 +Node: Extension Intro939319 +Node: Plugin License940585 +Node: Extension Mechanism Outline941382 +Ref: figure-load-extension941821 +Ref: figure-register-new-function943386 +Ref: figure-call-new-function944478 +Node: Extension API Description946540 +Node: Extension API Functions Introduction948072 +Node: General Data Types952931 +Ref: General Data Types-Footnote-1958886 +Node: Memory Allocation Functions959185 +Ref: Memory Allocation Functions-Footnote-1962030 +Node: Constructor Functions962129 +Node: Registration Functions963874 +Node: Extension Functions964559 +Node: Exit Callback Functions967182 +Node: Extension Version String968432 +Node: Input Parsers969095 +Node: Output Wrappers978977 +Node: Two-way processors983489 +Node: Printing Messages985754 +Ref: Printing Messages-Footnote-1986925 +Node: Updating ERRNO987078 +Node: Requesting Values987817 +Ref: table-value-types-returned988554 +Node: Accessing Parameters989437 +Node: Symbol Table Access990672 +Node: Symbol table by name991184 +Node: Symbol table by cookie993205 +Ref: Symbol table by cookie-Footnote-1997357 +Node: Cached values997421 +Ref: Cached values-Footnote-11000928 +Node: Array Manipulation1001019 +Ref: Array Manipulation-Footnote-11002110 +Node: Array Data Types1002147 +Ref: Array Data Types-Footnote-11004805 +Node: Array Functions1004897 +Node: Flattening Arrays1008755 +Node: Creating Arrays1015663 +Node: Redirection API1020432 +Node: Extension API Variables1023263 +Node: Extension Versioning1023896 +Ref: gawk-api-version1024333 +Node: Extension API Informational Variables1026089 +Node: Extension API Boilerplate1027153 +Node: Finding Extensions1030967 +Node: Extension Example1031526 +Node: Internal File Description1032324 +Node: Internal File Ops1036404 +Ref: Internal File Ops-Footnote-11048166 +Node: Using Internal File Ops1048306 +Ref: Using Internal File Ops-Footnote-11050689 +Node: Extension Samples1050963 +Node: Extension Sample File Functions1052492 +Node: Extension Sample Fnmatch1060141 +Node: Extension Sample Fork1061628 +Node: Extension Sample Inplace1062846 +Node: Extension Sample Ord1066056 +Node: Extension Sample Readdir1066892 +Ref: table-readdir-file-types1067781 +Node: Extension Sample Revout1068586 +Node: Extension Sample Rev2way1069175 +Node: Extension Sample Read write array1069915 +Node: Extension Sample Readfile1071857 +Node: Extension Sample Time1072952 +Node: Extension Sample API Tests1074300 +Node: gawkextlib1074792 +Node: Extension summary1077239 +Node: Extension Exercises1080941 +Node: Language History1082439 +Node: V7/SVR3.11084095 +Node: SVR41086247 +Node: POSIX1087681 +Node: BTL1089060 +Node: POSIX/GNU1089789 +Node: Feature History1095651 +Node: Common Extensions1110021 +Node: Ranges and Locales1111304 +Ref: Ranges and Locales-Footnote-11115920 +Ref: Ranges and Locales-Footnote-21115947 +Ref: Ranges and Locales-Footnote-31116182 +Node: Contributors1116403 +Node: History summary1121963 +Node: Installation1123343 +Node: Gawk Distribution1124287 +Node: Getting1124771 +Node: Extracting1125732 +Node: Distribution contents1127370 +Node: Unix Installation1133455 +Node: Quick Installation1134137 +Node: Shell Startup Files1136551 +Node: Additional Configuration Options1137629 +Node: Configuration Philosophy1139434 +Node: Non-Unix Installation1141803 +Node: PC Installation1142263 +Node: PC Binary Installation1143101 +Node: PC Compiling1143536 +Node: PC Using1144653 +Node: Cygwin1147698 +Node: MSYS1148468 +Node: VMS Installation1148969 +Node: VMS Compilation1149760 +Ref: VMS Compilation-Footnote-11150989 +Node: VMS Dynamic Extensions1151047 +Node: VMS Installation Details1152732 +Node: VMS Running1154985 +Node: VMS GNV1159264 +Node: VMS Old Gawk1159999 +Node: Bugs1160470 +Node: Bug address1161133 +Node: Usenet1163530 +Node: Maintainers1164305 +Node: Other Versions1165681 +Node: Installation summary1172265 +Node: Notes1173300 +Node: Compatibility Mode1174165 +Node: Additions1174947 +Node: Accessing The Source1175872 +Node: Adding Code1177307 +Node: New Ports1183526 +Node: Derived Files1188014 +Ref: Derived Files-Footnote-11193499 +Ref: Derived Files-Footnote-21193534 +Ref: Derived Files-Footnote-31194132 +Node: Future Extensions1194246 +Node: Implementation Limitations1194904 +Node: Extension Design1196087 +Node: Old Extension Problems1197241 +Ref: Old Extension Problems-Footnote-11198759 +Node: Extension New Mechanism Goals1198816 +Ref: Extension New Mechanism Goals-Footnote-11202180 +Node: Extension Other Design Decisions1202369 +Node: Extension Future Growth1204482 +Node: Old Extension Mechanism1205318 +Node: Notes summary1207081 +Node: Basic Concepts1208263 +Node: Basic High Level1208944 +Ref: figure-general-flow1209226 +Ref: figure-process-flow1209911 +Ref: Basic High Level-Footnote-11213212 +Node: Basic Data Typing1213397 +Node: Glossary1216725 +Node: Copying1248672 +Node: GNU Free Documentation License1286211 +Node: Index1311329 + +End Tag Table diff --git a/doc/gawkinet.info b/doc/gawkinet.info new file mode 100644 index 00000000..d5a7abf8 --- /dev/null +++ b/doc/gawkinet.info @@ -0,0 +1,4406 @@ +This is gawkinet.info, produced by makeinfo version 6.1 from +gawkinet.texi. + +This is Edition 1.4 of 'TCP/IP Internetworking with 'gawk'', for the +4.1.4 (or later) version of the GNU implementation of AWK. + + + Copyright (C) 2000, 2001, 2002, 2004, 2009, 2010, 2016 Free Software +Foundation, Inc. + + + Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with the +Invariant Sections being "GNU General Public License", the Front-Cover +texts being (a) (see below), and with the Back-Cover Texts being (b) +(see below). A copy of the license is included in the section entitled +"GNU Free Documentation License". + + a. "A GNU Manual" + + b. "You have the freedom to copy and modify this GNU manual. Buying + copies from the FSF supports it in developing GNU and promoting + software freedom." +INFO-DIR-SECTION Network applications +START-INFO-DIR-ENTRY +* Gawkinet: (gawkinet). TCP/IP Internetworking With 'gawk'. +END-INFO-DIR-ENTRY + + This file documents the networking features in GNU 'awk'. + + This is Edition 1.4 of 'TCP/IP Internetworking with 'gawk'', for the +4.1.4 (or later) version of the GNU implementation of AWK. + + + Copyright (C) 2000, 2001, 2002, 2004, 2009, 2010, 2016 Free Software +Foundation, Inc. + + + Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with the +Invariant Sections being "GNU General Public License", the Front-Cover +texts being (a) (see below), and with the Back-Cover Texts being (b) +(see below). A copy of the license is included in the section entitled +"GNU Free Documentation License". + + a. "A GNU Manual" + + b. "You have the freedom to copy and modify this GNU manual. Buying + copies from the FSF supports it in developing GNU and promoting + software freedom." + + +File: gawkinet.info, Node: Top, Next: Preface, Prev: (dir), Up: (dir) + +General Introduction +******************** + +This file documents the networking features in GNU Awk ('gawk') version +4.0 and later. + + This is Edition 1.4 of 'TCP/IP Internetworking with 'gawk'', for the +4.1.4 (or later) version of the GNU implementation of AWK. + + + Copyright (C) 2000, 2001, 2002, 2004, 2009, 2010, 2016 Free Software +Foundation, Inc. + + + Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with the +Invariant Sections being "GNU General Public License", the Front-Cover +texts being (a) (see below), and with the Back-Cover Texts being (b) +(see below). A copy of the license is included in the section entitled +"GNU Free Documentation License". + + a. "A GNU Manual" + + b. "You have the freedom to copy and modify this GNU manual. Buying + copies from the FSF supports it in developing GNU and promoting + software freedom." + +* Menu: + +* Preface:: About this document. +* Introduction:: About networking. +* Using Networking:: Some examples. +* Some Applications and Techniques:: More extended examples. +* Links:: Where to find the stuff mentioned in this + document. +* GNU Free Documentation License:: The license for this document. +* Index:: The index. + +* Stream Communications:: Sending data streams. +* Datagram Communications:: Sending self-contained messages. +* The TCP/IP Protocols:: How these models work in the Internet. +* Basic Protocols:: The basic protocols. +* Ports:: The idea behind ports. +* Making Connections:: Making TCP/IP connections. +* Gawk Special Files:: How to do 'gawk' networking. +* Special File Fields:: The fields in the special file name. +* Comparing Protocols:: Differences between the protocols. +* File /inet/tcp:: The TCP special file. +* File /inet/udp:: The UDP special file. +* TCP Connecting:: Making a TCP connection. +* Troubleshooting:: Troubleshooting TCP/IP connections. +* Interacting:: Interacting with a service. +* Setting Up:: Setting up a service. +* Email:: Reading email. +* Web page:: Reading a Web page. +* Primitive Service:: A primitive Web service. +* Interacting Service:: A Web service with interaction. +* CGI Lib:: A simple CGI library. +* Simple Server:: A simple Web server. +* Caveats:: Network programming caveats. +* Challenges:: Where to go from here. +* PANIC:: An Emergency Web Server. +* GETURL:: Retrieving Web Pages. +* REMCONF:: Remote Configuration Of Embedded Systems. +* URLCHK:: Look For Changed Web Pages. +* WEBGRAB:: Extract Links From A Page. +* STATIST:: Graphing A Statistical Distribution. +* MAZE:: Walking Through A Maze In Virtual Reality. +* MOBAGWHO:: A Simple Mobile Agent. +* STOXPRED:: Stock Market Prediction As A Service. +* PROTBASE:: Searching Through A Protein Database. + + +File: gawkinet.info, Node: Preface, Next: Introduction, Prev: Top, Up: Top + +Preface +******* + +In May of 1997, Ju"rgen Kahrs felt the need for network access from +'awk', and, with a little help from me, set about adding features to do +this for 'gawk'. At that time, he wrote the bulk of this Info file. + + The code and documentation were added to the 'gawk' 3.1 development +tree, and languished somewhat until I could finally get down to some +serious work on that version of 'gawk'. This finally happened in the +middle of 2000. + + Meantime, Ju"rgen wrote an article about the Internet special files +and '|&' operator for 'Linux Journal', and made a networking patch for +the production versions of 'gawk' available from his home page. In +August of 2000 (for 'gawk' 3.0.6), this patch also made it to the main +GNU 'ftp' distribution site. + + For release with 'gawk', I edited Ju"rgen's prose for English grammar +and style, as he is not a native English speaker. I also rearranged the +material somewhat for what I felt was a better order of presentation, +and (re)wrote some of the introductory material. + + The majority of this document and the code are his work, and the high +quality and interesting ideas speak for themselves. It is my hope that +these features will be of significant value to the 'awk' community. + + +Arnold Robbins +Nof Ayalon, ISRAEL +March, 2001 + + +File: gawkinet.info, Node: Introduction, Next: Using Networking, Prev: Preface, Up: Top + +1 Networking Concepts +********************* + +This major node provides a (necessarily) brief introduction to computer +networking concepts. For many applications of 'gawk' to TCP/IP +networking, we hope that this is enough. For more advanced tasks, you +will need deeper background, and it may be necessary to switch to +lower-level programming in C or C++. + + There are two real-life models for the way computers send messages to +each other over a network. While the analogies are not perfect, they +are close enough to convey the major concepts. These two models are the +phone system (reliable byte-stream communications), and the postal +system (best-effort datagrams). + +* Menu: + +* Stream Communications:: Sending data streams. +* Datagram Communications:: Sending self-contained messages. +* The TCP/IP Protocols:: How these models work in the Internet. +* Making Connections:: Making TCP/IP connections. + + +File: gawkinet.info, Node: Stream Communications, Next: Datagram Communications, Prev: Introduction, Up: Introduction + +1.1 Reliable Byte-streams (Phone Calls) +======================================= + +When you make a phone call, the following steps occur: + + 1. You dial a number. + + 2. The phone system connects to the called party, telling them there + is an incoming call. (Their phone rings.) + + 3. The other party answers the call, or, in the case of a computer + network, refuses to answer the call. + + 4. Assuming the other party answers, the connection between you is now + a "duplex" (two-way), "reliable" (no data lost), sequenced (data + comes out in the order sent) data stream. + + 5. You and your friend may now talk freely, with the phone system + moving the data (your voices) from one end to the other. From your + point of view, you have a direct end-to-end connection with the + person on the other end. + + The same steps occur in a duplex reliable computer networking +connection. There is considerably more overhead in setting up the +communications, but once it's done, data moves in both directions, +reliably, in sequence. + + +File: gawkinet.info, Node: Datagram Communications, Next: The TCP/IP Protocols, Prev: Stream Communications, Up: Introduction + +1.2 Best-effort Datagrams (Mailed Letters) +========================================== + +Suppose you mail three different documents to your office on the other +side of the country on two different days. Doing so entails the +following. + + 1. Each document travels in its own envelope. + + 2. Each envelope contains both the sender and the recipient address. + + 3. Each envelope may travel a different route to its destination. + + 4. The envelopes may arrive in a different order from the one in which + they were sent. + + 5. One or more may get lost in the mail. (Although, fortunately, this + does not occur very often.) + + 6. In a computer network, one or more "packets" may also arrive + multiple times. (This doesn't happen with the postal system!) + + The important characteristics of datagram communications, like those +of the postal system are thus: + + * Delivery is "best effort;" the data may never get there. + + * Each message is self-contained, including the source and + destination addresses. + + * Delivery is _not_ sequenced; packets may arrive out of order, + and/or multiple times. + + * Unlike the phone system, overhead is considerably lower. It is not + necessary to set up the call first. + + The price the user pays for the lower overhead of datagram +communications is exactly the lower reliability; it is often necessary +for user-level protocols that use datagram communications to add their +own reliability features on top of the basic communications. + + +File: gawkinet.info, Node: The TCP/IP Protocols, Next: Making Connections, Prev: Datagram Communications, Up: Introduction + +1.3 The Internet Protocols +========================== + +The Internet Protocol Suite (usually referred to as just TCP/IP)(1) +consists of a number of different protocols at different levels or +"layers." For our purposes, three protocols provide the fundamental +communications mechanisms. All other defined protocols are referred to +as user-level protocols (e.g., HTTP, used later in this Info file). + +* Menu: + +* Basic Protocols:: The basic protocols. +* Ports:: The idea behind ports. + + ---------- Footnotes ---------- + + (1) It should be noted that although the Internet seems to have +conquered the world, there are other networking protocol suites in +existence and in use. + + +File: gawkinet.info, Node: Basic Protocols, Next: Ports, Prev: The TCP/IP Protocols, Up: The TCP/IP Protocols + +1.3.1 The Basic Internet Protocols +---------------------------------- + +IP + The Internet Protocol. This protocol is almost never used directly + by applications. It provides the basic packet delivery and routing + infrastructure of the Internet. Much like the phone company's + switching centers or the Post Office's trucks, it is not of much + day-to-day interest to the regular user (or programmer). It + happens to be a best effort datagram protocol. In the early + twenty-first century, there are two versions of this protocol in + use: + + IPv4 + The original version of the Internet Protocol, with 32-bit + addresses, on which most of the current Internet is based. + + IPv6 + The "next generation" of the Internet Protocol, with 128-bit + addresses. This protocol is in wide use in certain parts of + the world, but has not yet replaced IPv4.(1) + + Versions of the other protocols that sit "atop" IP exist for both + IPv4 and IPv6. However, as the IPv6 versions are fundamentally the + same as the original IPv4 versions, we will not distinguish further + between them. + +UDP + The User Datagram Protocol. This is a best effort datagram + protocol. It provides a small amount of extra reliability over IP, + and adds the notion of "ports", described in *note TCP and UDP + Ports: Ports. + +TCP + The Transmission Control Protocol. This is a duplex, reliable, + sequenced byte-stream protocol, again layered on top of IP, and + also providing the notion of ports. This is the protocol that you + will most likely use when using 'gawk' for network programming. + + All other user-level protocols use either TCP or UDP to do their +basic communications. Examples are SMTP (Simple Mail Transfer +Protocol), FTP (File Transfer Protocol), and HTTP (HyperText Transfer +Protocol). + + ---------- Footnotes ---------- + + (1) There isn't an IPv5. + + +File: gawkinet.info, Node: Ports, Prev: Basic Protocols, Up: The TCP/IP Protocols + +1.3.2 TCP and UDP Ports +----------------------- + +In the postal system, the address on an envelope indicates a physical +location, such as a residence or office building. But there may be more +than one person at the location; thus you have to further quantify the +recipient by putting a person or company name on the envelope. + + In the phone system, one phone number may represent an entire +company, in which case you need a person's extension number in order to +reach that individual directly. Or, when you call a home, you have to +say, "May I please speak to ..." before talking to the person directly. + + IP networking provides the concept of addressing. An IP address +represents a particular computer, but no more. In order to reach the +mail service on a system, or the FTP or WWW service on a system, you +must have some way to further specify which service you want. In the +Internet Protocol suite, this is done with "port numbers", which +represent the services, much like an extension number used with a phone +number. + + Port numbers are 16-bit integers. Unix and Unix-like systems reserve +ports below 1024 for "well known" services, such as SMTP, FTP, and HTTP. +Numbers 1024 and above may be used by any application, although there is +no promise made that a particular port number is always available. + + +File: gawkinet.info, Node: Making Connections, Prev: The TCP/IP Protocols, Up: Introduction + +1.4 Making TCP/IP Connections (And Some Terminology) +==================================================== + +Two terms come up repeatedly when discussing networking: "client" and +"server". For now, we'll discuss these terms at the "connection level", +when first establishing connections between two processes on different +systems over a network. (Once the connection is established, the higher +level, or "application level" protocols, such as HTTP or FTP, determine +who is the client and who is the server. Often, it turns out that the +client and server are the same in both roles.) + + The "server" is the system providing the service, such as the web +server or email server. It is the "host" (system) which is _connected +to_ in a transaction. For this to work though, the server must be +expecting connections. Much as there has to be someone at the office +building to answer the phone(1), the server process (usually) has to be +started first and be waiting for a connection. + + The "client" is the system requesting the service. It is the system +_initiating the connection_ in a transaction. (Just as when you pick up +the phone to call an office or store.) + + In the TCP/IP framework, each end of a connection is represented by a +pair of (ADDRESS, PORT) pairs. For the duration of the connection, the +ports in use at each end are unique, and cannot be used simultaneously +by other processes on the same system. (Only after closing a connection +can a new one be built up on the same port. This is contrary to the +usual behavior of fully developed web servers which have to avoid +situations in which they are not reachable. We have to pay this price +in order to enjoy the benefits of a simple communication paradigm in +'gawk'.) + + Furthermore, once the connection is established, communications are +"synchronous".(2) I.e., each end waits on the other to finish +transmitting, before replying. This is much like two people in a phone +conversation. While both could talk simultaneously, doing so usually +doesn't work too well. + + In the case of TCP, the synchronicity is enforced by the protocol +when sending data. Data writes "block" until the data have been +received on the other end. For both TCP and UDP, data reads block until +there is incoming data waiting to be read. This is summarized in the +following table, where an "X" indicates that the given action blocks. + +TCP X X +UDP X + + ---------- Footnotes ---------- + + (1) In the days before voice mail systems! + + (2) For the technically savvy, data reads block--if there's no +incoming data, the program is made to wait until there is, instead of +receiving a "there's no data" error return. + + +File: gawkinet.info, Node: Using Networking, Next: Some Applications and Techniques, Prev: Introduction, Up: Top + +2 Networking With 'gawk' +************************ + +The 'awk' programming language was originally developed as a +pattern-matching language for writing short programs to perform data +manipulation tasks. 'awk''s strength is the manipulation of textual +data that is stored in files. It was never meant to be used for +networking purposes. To exploit its features in a networking context, +it's necessary to use an access mode for network connections that +resembles the access of files as closely as possible. + + 'awk' is also meant to be a prototyping language. It is used to +demonstrate feasibility and to play with features and user interfaces. +This can be done with file-like handling of network connections. 'gawk' +trades the lack of many of the advanced features of the TCP/IP family of +protocols for the convenience of simple connection handling. The +advanced features are available when programming in C or Perl. In fact, +the network programming in this major node is very similar to what is +described in books such as 'Internet Programming with Python', 'Advanced +Perl Programming', or 'Web Client Programming with Perl'. + + However, you can do the programming here without first having to +learn object-oriented ideology; underlying languages such as Tcl/Tk, +Perl, Python; or all of the libraries necessary to extend these +languages before they are ready for the Internet. + + This major node demonstrates how to use the TCP protocol. The UDP +protocol is much less important for most users. + +* Menu: + +* Gawk Special Files:: How to do 'gawk' networking. +* TCP Connecting:: Making a TCP connection. +* Troubleshooting:: Troubleshooting TCP/IP connections. +* Interacting:: Interacting with a service. +* Setting Up:: Setting up a service. +* Email:: Reading email. +* Web page:: Reading a Web page. +* Primitive Service:: A primitive Web service. +* Interacting Service:: A Web service with interaction. +* Simple Server:: A simple Web server. +* Caveats:: Network programming caveats. +* Challenges:: Where to go from here. + + +File: gawkinet.info, Node: Gawk Special Files, Next: TCP Connecting, Prev: Using Networking, Up: Using Networking + +2.1 'gawk''s Networking Mechanisms +================================== + +The '|&' operator for use in communicating with a "coprocess" is +described in *note Two-way Communications With Another Process: +(gawk)Two-way I/O. It shows how to do two-way I/O to a separate process, +sending it data with 'print' or 'printf' and reading data with +'getline'. If you haven't read it already, you should detour there to +do so. + + 'gawk' transparently extends the two-way I/O mechanism to simple +networking through the use of special file names. When a "coprocess" +that matches the special files we are about to describe is started, +'gawk' creates the appropriate network connection, and then two-way I/O +proceeds as usual. + + At the C, C++, and Perl level, networking is accomplished via +"sockets", an Application Programming Interface (API) originally +developed at the University of California at Berkeley that is now used +almost universally for TCP/IP networking. Socket level programming, +while fairly straightforward, requires paying attention to a number of +details, as well as using binary data. It is not well-suited for use +from a high-level language like 'awk'. The special files provided in +'gawk' hide the details from the programmer, making things much simpler +and easier to use. + + The special file name for network access is made up of several +fields, all of which are mandatory: + + /NET-TYPE/PROTOCOL/LOCALPORT/HOSTNAME/REMOTEPORT + + The NET-TYPE field lets you specify IPv4 versus IPv6, or lets you +allow the system to choose. + +* Menu: + +* Special File Fields:: The fields in the special file name. +* Comparing Protocols:: Differences between the protocols. + + +File: gawkinet.info, Node: Special File Fields, Next: Comparing Protocols, Prev: Gawk Special Files, Up: Gawk Special Files + +2.1.1 The Fields of the Special File Name +----------------------------------------- + +This node explains the meaning of all the other fields, as well as the +range of values and the defaults. All of the fields are mandatory. To +let the system pick a value, or if the field doesn't apply to the +protocol, specify it as '0': + +NET-TYPE + This is one of 'inet4' for IPv4, 'inet6' for IPv6, or 'inet' to use + the system default (which is likely to be IPv4). For the rest of + this document, we will use the generic '/inet' in our descriptions + of how 'gawk''s networking works. + +PROTOCOL + Determines which member of the TCP/IP family of protocols is + selected to transport the data across the network. There are two + possible values (always written in lowercase): 'tcp' and 'udp'. + The exact meaning of each is explained later in this node. + +LOCALPORT + Determines which port on the local machine is used to communicate + across the network. Application-level clients usually use '0' to + indicate they do not care which local port is used--instead they + specify a remote port to connect to. It is vital for + application-level servers to use a number different from '0' here + because their service has to be available at a specific publicly + known port number. It is possible to use a name from + '/etc/services' here. + +HOSTNAME + Determines which remote host is to be at the other end of the + connection. Application-level servers must fill this field with a + '0' to indicate their being open for all other hosts to connect to + them and enforce connection level server behavior this way. It is + not possible for an application-level server to restrict its + availability to one remote host by entering a host name here. + Application-level clients must enter a name different from '0'. + The name can be either symbolic (e.g., 'jpl-devvax.jpl.nasa.gov') + or numeric (e.g., '128.149.1.143'). + +REMOTEPORT + Determines which port on the remote machine is used to communicate + across the network. For '/inet/tcp' and '/inet/udp', + application-level clients _must_ use a number other than '0' to + indicate to which port on the remote machine they want to connect. + Application-level servers must not fill this field with a '0'. + Instead they specify a local port to which clients connect. It is + possible to use a name from '/etc/services' here. + + Experts in network programming will notice that the usual +client/server asymmetry found at the level of the socket API is not +visible here. This is for the sake of simplicity of the high-level +concept. If this asymmetry is necessary for your application, use +another language. For 'gawk', it is more important to enable users to +write a client program with a minimum of code. What happens when first +accessing a network connection is seen in the following pseudocode: + + if ((name of remote host given) && (other side accepts connection)) { + rendez-vous successful; transmit with getline or print + } else { + if ((other side did not accept) && (localport == 0)) + exit unsuccessful + if (TCP) { + set up a server accepting connections + this means waiting for the client on the other side to connect + } else + ready + } + + The exact behavior of this algorithm depends on the values of the +fields of the special file name. When in doubt, *note Table 2.1: +table-inet-components. gives you the combinations of values and their +meaning. If this table is too complicated, focus on the three lines +printed in *bold*. All the examples in *note Networking With 'gawk': +Using Networking, use only the patterns printed in bold letters. + +PROTOCOL LOCAL HOST NAME REMOTE RESULTING CONNECTION-LEVEL + PORT PORT BEHAVIOR +------------------------------------------------------------------------------ +*tcp* *0* *x* *x* *Dedicated client, fails if + immediately connecting to a + server on the other side + fails* +udp 0 x x Dedicated client +*tcp, *x* *x* *x* *Client, switches to +udp* dedicated server if + necessary* +*tcp, *x* *0* *0* *Dedicated server* +udp* +tcp, udp x x 0 Invalid +tcp, udp 0 0 x Invalid +tcp, udp x 0 x Invalid +tcp, udp 0 0 0 Invalid +tcp, udp 0 x 0 Invalid + +Table 2.1: /inet Special File Components + + In general, TCP is the preferred mechanism to use. It is the +simplest protocol to understand and to use. Use UDP only if +circumstances demand low-overhead. + + +File: gawkinet.info, Node: Comparing Protocols, Prev: Special File Fields, Up: Gawk Special Files + +2.1.2 Comparing Protocols +------------------------- + +This node develops a pair of programs (sender and receiver) that do +nothing but send a timestamp from one machine to another. The sender +and the receiver are implemented with each of the two protocols +available and demonstrate the differences between them. + +* Menu: + +* File /inet/tcp:: The TCP special file. +* File /inet/udp:: The UDP special file. + + +File: gawkinet.info, Node: File /inet/tcp, Next: File /inet/udp, Prev: Comparing Protocols, Up: Comparing Protocols + +2.1.2.1 '/inet/tcp' +................... + +Once again, always use TCP. (Use UDP when low overhead is a necessity, +and use RAW for network experimentation.) The first example is the +sender program: + + # Server + BEGIN { + print strftime() |& "/inet/tcp/8888/0/0" + close("/inet/tcp/8888/0/0") + } + + The receiver is very simple: + + # Client + BEGIN { + "/inet/tcp/0/localhost/8888" |& getline + print $0 + close("/inet/tcp/0/localhost/8888") + } + + TCP guarantees that the bytes arrive at the receiving end in exactly +the same order that they were sent. No byte is lost (except for broken +connections), doubled, or out of order. Some overhead is necessary to +accomplish this, but this is the price to pay for a reliable service. +It does matter which side starts first. The sender/server has to be +started first, and it waits for the receiver to read a line. + + +File: gawkinet.info, Node: File /inet/udp, Prev: File /inet/tcp, Up: Comparing Protocols + +2.1.2.2 '/inet/udp' +................... + +The server and client programs that use UDP are almost identical to +their TCP counterparts; only the PROTOCOL has changed. As before, it +does matter which side starts first. The receiving side blocks and +waits for the sender. In this case, the receiver/client has to be +started first: + + # Server + BEGIN { + print strftime() |& "/inet/udp/8888/0/0" + close("/inet/udp/8888/0/0") + } + + The receiver is almost identical to the TCP receiver: + + # Client + BEGIN { + print "hi!" |& "/inet/udp/0/localhost/8888" + "/inet/udp/0/localhost/8888" |& getline + print $0 + close("/inet/udp/0/localhost/8888") + } + + In the case of UDP, the initial 'print' command is the one that +actually sends data so that there is a connection. UDP and "connection" +sounds strange to anyone who has learned that UDP is a connectionless +protocol. Here, "connection" means that the 'connect()' system call has +completed its work and completed the "association" between a certain +socket and an IP address. Thus there are subtle differences between +'connect()' for TCP and UDP; see the man page for details.(1) + + UDP cannot guarantee that the datagrams at the receiving end will +arrive in exactly the same order they were sent. Some datagrams could +be lost, some doubled, and some out of order. But no overhead is +necessary to accomplish this. This unreliable behavior is good enough +for tasks such as data acquisition, logging, and even stateless services +like the original versions of NFS. + + ---------- Footnotes ---------- + + (1) This subtlety is just one of many details that are hidden in the +socket API, invisible and intractable for the 'gawk' user. The +developers are currently considering how to rework the network +facilities to make them easier to understand and use. + + +File: gawkinet.info, Node: TCP Connecting, Next: Troubleshooting, Prev: Gawk Special Files, Up: Using Networking + +2.2 Establishing a TCP Connection +================================= + +Let's observe a network connection at work. Type in the following +program and watch the output. Within a second, it connects via TCP +('/inet/tcp') to the machine it is running on ('localhost') and asks the +service 'daytime' on the machine what time it is: + + BEGIN { + "/inet/tcp/0/localhost/daytime" |& getline + print $0 + close("/inet/tcp/0/localhost/daytime") + } + + Even experienced 'awk' users will find the second line strange in two +respects: + + * A special file is used as a shell command that pipes its output + into 'getline'. One would rather expect to see the special file + being read like any other file ('getline < + "/inet/tcp/0/localhost/daytime")'. + + * The operator '|&' has not been part of any 'awk' implementation + (until now). It is actually the only extension of the 'awk' + language needed (apart from the special files) to introduce network + access. + + The '|&' operator was introduced in 'gawk' 3.1 in order to overcome +the crucial restriction that access to files and pipes in 'awk' is +always unidirectional. It was formerly impossible to use both access +modes on the same file or pipe. Instead of changing the whole concept +of file access, the '|&' operator behaves exactly like the usual pipe +operator except for two additions: + + * Normal shell commands connected to their 'gawk' program with a '|&' + pipe can be accessed bidirectionally. The '|&' turns out to be a + quite general, useful, and natural extension of 'awk'. + + * Pipes that consist of a special file name for network connections + are not executed as shell commands. Instead, they can be read and + written to, just like a full-duplex network connection. + + In the earlier example, the '|&' operator tells 'getline' to read a +line from the special file '/inet/tcp/0/localhost/daytime'. We could +also have printed a line into the special file. But instead we just +read a line with the time, printed it, and closed the connection. +(While we could just let 'gawk' close the connection by finishing the +program, in this Info file we are pedantic and always explicitly close +the connections.) + + +File: gawkinet.info, Node: Troubleshooting, Next: Interacting, Prev: TCP Connecting, Up: Using Networking + +2.3 Troubleshooting Connection Problems +======================================= + +It may well be that for some reason the program shown in the previous +example does not run on your machine. When looking at possible reasons +for this, you will learn much about typical problems that arise in +network programming. First of all, your implementation of 'gawk' may +not support network access because it is a pre-3.1 version or you do not +have a network interface in your machine. Perhaps your machine uses +some other protocol, such as DECnet or Novell's IPX. For the rest of +this major node, we will assume you work on a Unix machine that supports +TCP/IP. If the previous example program does not run on your machine, it +may help to replace the name 'localhost' with the name of your machine +or its IP address. If it does, you could replace 'localhost' with the +name of another machine in your vicinity--this way, the program connects +to another machine. Now you should see the date and time being printed +by the program, otherwise your machine may not support the 'daytime' +service. Try changing the service to 'chargen' or 'ftp'. This way, the +program connects to other services that should give you some response. +If you are curious, you should have a look at your '/etc/services' file. +It could look like this: + + # /etc/services: + # + # Network services, Internet style + # + # Name Number/Protocol Alternate name # Comments + + echo 7/tcp + echo 7/udp + discard 9/tcp sink null + discard 9/udp sink null + daytime 13/tcp + daytime 13/udp + chargen 19/tcp ttytst source + chargen 19/udp ttytst source + ftp 21/tcp + telnet 23/tcp + smtp 25/tcp mail + finger 79/tcp + www 80/tcp http # WorldWideWeb HTTP + www 80/udp # HyperText Transfer Protocol + pop-2 109/tcp postoffice # POP version 2 + pop-2 109/udp + pop-3 110/tcp # POP version 3 + pop-3 110/udp + nntp 119/tcp readnews untp # USENET News + irc 194/tcp # Internet Relay Chat + irc 194/udp + ... + + Here, you find a list of services that traditional Unix machines +usually support. If your GNU/Linux machine does not do so, it may be +that these services are switched off in some startup script. Systems +running some flavor of Microsoft Windows usually do _not_ support these +services. Nevertheless, it _is_ possible to do networking with 'gawk' +on Microsoft Windows.(1) The first column of the file gives the name of +the service, and the second column gives a unique number and the +protocol that one can use to connect to this service. The rest of the +line is treated as a comment. You see that some services ('echo') +support TCP as well as UDP. + + ---------- Footnotes ---------- + + (1) Microsoft preferred to ignore the TCP/IP family of protocols +until 1995. Then came the rise of the Netscape browser as a landmark +"killer application." Microsoft added TCP/IP support and their own +browser to Microsoft Windows 95 at the last minute. They even +back-ported their TCP/IP implementation to Microsoft Windows for +Workgroups 3.11, but it was a rather rudimentary and half-hearted +implementation. Nevertheless, the equivalent of '/etc/services' resides +under 'C:\WINNT\system32\drivers\etc\services' on Microsoft Windows 2000 +and Microsoft Windows XP. + + +File: gawkinet.info, Node: Interacting, Next: Setting Up, Prev: Troubleshooting, Up: Using Networking + +2.4 Interacting with a Network Service +====================================== + +The next program makes use of the possibility to really interact with a +network service by printing something into the special file. It asks +the so-called 'finger' service if a user of the machine is logged in. +When testing this program, try to change 'localhost' to some other +machine name in your local network: + + BEGIN { + NetService = "/inet/tcp/0/localhost/finger" + print "NAME" |& NetService + while ((NetService |& getline) > 0) + print $0 + close(NetService) + } + + After telling the service on the machine which user to look for, the +program repeatedly reads lines that come as a reply. When no more lines +are coming (because the service has closed the connection), the program +also closes the connection. Try replacing '"NAME"' with your login name +(or the name of someone else logged in). For a list of all users +currently logged in, replace NAME with an empty string ('""'). + + The final 'close()' command could be safely deleted from the above +script, because the operating system closes any open connection by +default when a script reaches the end of execution. In order to avoid +portability problems, it is best to always close connections explicitly. +With the Linux kernel, for example, proper closing results in flushing +of buffers. Letting the close happen by default may result in +discarding buffers. + + When looking at '/etc/services' you may have noticed that the +'daytime' service is also available with 'udp'. In the earlier example, +change 'tcp' to 'udp', and change 'finger' to 'daytime'. After starting +the modified program, you see the expected day and time message. The +program then hangs, because it waits for more lines coming from the +service. However, they never come. This behavior is a consequence of +the differences between TCP and UDP. When using UDP, neither party is +automatically informed about the other closing the connection. +Continuing to experiment this way reveals many other subtle differences +between TCP and UDP. To avoid such trouble, one should always remember +the advice Douglas E. Comer and David Stevens give in Volume III of +their series 'Internetworking With TCP' (page 14): + + When designing client-server applications, beginners are strongly + advised to use TCP because it provides reliable, + connection-oriented communication. Programs only use UDP if the + application protocol handles reliability, the application requires + hardware broadcast or multicast, or the application cannot tolerate + virtual circuit overhead. + + +File: gawkinet.info, Node: Setting Up, Next: Email, Prev: Interacting, Up: Using Networking + +2.5 Setting Up a Service +======================== + +The preceding programs behaved as clients that connect to a server +somewhere on the Internet and request a particular service. Now we set +up such a service to mimic the behavior of the 'daytime' service. Such +a server does not know in advance who is going to connect to it over the +network. Therefore, we cannot insert a name for the host to connect to +in our special file name. + + Start the following program in one window. Notice that the service +does not have the name 'daytime', but the number '8888'. From looking +at '/etc/services', you know that names like 'daytime' are just +mnemonics for predetermined 16-bit integers. Only the system +administrator ('root') could enter our new service into '/etc/services' +with an appropriate name. Also notice that the service name has to be +entered into a different field of the special file name because we are +setting up a server, not a client: + + BEGIN { + print strftime() |& "/inet/tcp/8888/0/0" + close("/inet/tcp/8888/0/0") + } + + Now open another window on the same machine. Copy the client program +given as the first example (*note Establishing a TCP Connection: TCP +Connecting.) to a new file and edit it, changing the name 'daytime' to +'8888'. Then start the modified client. You should get a reply like +this: + + Sat Sep 27 19:08:16 CEST 1997 + +Both programs explicitly close the connection. + + Now we will intentionally make a mistake to see what happens when the +name '8888' (the so-called port) is already used by another service. +Start the server program in both windows. The first one works, but the +second one complains that it could not open the connection. Each port +on a single machine can only be used by one server program at a time. +Now terminate the server program and change the name '8888' to 'echo'. +After restarting it, the server program does not run any more, and you +know why: there is already an 'echo' service running on your machine. +But even if this isn't true, you would not get your own 'echo' server +running on a Unix machine, because the ports with numbers smaller than +1024 ('echo' is at port 7) are reserved for 'root'. On machines running +some flavor of Microsoft Windows, there is no restriction that reserves +ports 1 to 1024 for a privileged user; hence, you can start an 'echo' +server there. + + Turning this short server program into something really useful is +simple. Imagine a server that first reads a file name from the client +through the network connection, then does something with the file and +sends a result back to the client. The server-side processing could be: + + BEGIN { + NetService = "/inet/tcp/8888/0/0" + NetService |& getline + CatPipe = ("cat " $1) # sets $0 and the fields + while ((CatPipe | getline) > 0) + print $0 |& NetService + close(NetService) + } + +and we would have a remote copying facility. Such a server reads the +name of a file from any client that connects to it and transmits the +contents of the named file across the net. The server-side processing +could also be the execution of a command that is transmitted across the +network. From this example, you can see how simple it is to open up a +security hole on your machine. If you allow clients to connect to your +machine and execute arbitrary commands, anyone would be free to do 'rm +-rf *'. + + +File: gawkinet.info, Node: Email, Next: Web page, Prev: Setting Up, Up: Using Networking + +2.6 Reading Email +================= + +The distribution of email is usually done by dedicated email servers +that communicate with your machine using special protocols. To receive +email, we will use the Post Office Protocol (POP). Sending can be done +with the much older Simple Mail Transfer Protocol (SMTP). + + When you type in the following program, replace the EMAILHOST by the +name of your local email server. Ask your administrator if the server +has a POP service, and then use its name or number in the program below. +Now the program is ready to connect to your email server, but it will +not succeed in retrieving your mail because it does not yet know your +login name or password. Replace them in the program and it shows you +the first email the server has in store: + + BEGIN { + POPService = "/inet/tcp/0/EMAILHOST/pop3" + RS = ORS = "\r\n" + print "user NAME" |& POPService + POPService |& getline + print "pass PASSWORD" |& POPService + POPService |& getline + print "retr 1" |& POPService + POPService |& getline + if ($1 != "+OK") exit + print "quit" |& POPService + RS = "\r\n\\.\r\n" + POPService |& getline + print $0 + close(POPService) + } + + The record separators 'RS' and 'ORS' are redefined because the +protocol (POP) requires CR-LF to separate lines. After identifying +yourself to the email service, the command 'retr 1' instructs the +service to send the first of all your email messages in line. If the +service replies with something other than '+OK', the program exits; +maybe there is no email. Otherwise, the program first announces that it +intends to finish reading email, and then redefines 'RS' in order to +read the entire email as multiline input in one record. From the POP +RFC, we know that the body of the email always ends with a single line +containing a single dot. The program looks for this using 'RS = +"\r\n\\.\r\n"'. When it finds this sequence in the mail message, it +quits. You can invoke this program as often as you like; it does not +delete the message it reads, but instead leaves it on the server. + + +File: gawkinet.info, Node: Web page, Next: Primitive Service, Prev: Email, Up: Using Networking + +2.7 Reading a Web Page +====================== + +Retrieving a web page from a web server is as simple as retrieving email +from an email server. We only have to use a similar, but not identical, +protocol and a different port. The name of the protocol is HyperText +Transfer Protocol (HTTP) and the port number is usually 80. As in the +preceding node, ask your administrator about the name of your local web +server or proxy web server and its port number for HTTP requests. + + The following program employs a rather crude approach toward +retrieving a web page. It uses the prehistoric syntax of HTTP 0.9, +which almost all web servers still support. The most noticeable thing +about it is that the program directs the request to the local proxy +server whose name you insert in the special file name (which in turn +calls 'www.yahoo.com'): + + BEGIN { + RS = ORS = "\r\n" + HttpService = "/inet/tcp/0/PROXY/80" + print "GET http://www.yahoo.com" |& HttpService + while ((HttpService |& getline) > 0) + print $0 + close(HttpService) + } + + Again, lines are separated by a redefined 'RS' and 'ORS'. The 'GET' +request that we send to the server is the only kind of HTTP request that +existed when the web was created in the early 1990s. HTTP calls this +'GET' request a "method," which tells the service to transmit a web page +(here the home page of the Yahoo! search engine). Version 1.0 added +the request methods 'HEAD' and 'POST'. The current version of HTTP is +1.1,(1) and knows the additional request methods 'OPTIONS', 'PUT', +'DELETE', and 'TRACE'. You can fill in any valid web address, and the +program prints the HTML code of that page to your screen. + + Notice the similarity between the responses of the POP and HTTP +services. First, you get a header that is terminated by an empty line, +and then you get the body of the page in HTML. The lines of the headers +also have the same form as in POP. There is the name of a parameter, +then a colon, and finally the value of that parameter. + + Images ('.png' or '.gif' files) can also be retrieved this way, but +then you get binary data that should be redirected into a file. Another +application is calling a CGI (Common Gateway Interface) script on some +server. CGI scripts are used when the contents of a web page are not +constant, but generated instantly at the moment you send a request for +the page. For example, to get a detailed report about the current +quotes of Motorola stock shares, call a CGI script at Yahoo! with the +following: + + get = "GET http://quote.yahoo.com/q?s=MOT&d=t" + print get |& HttpService + + You can also request weather reports this way. + + ---------- Footnotes ---------- + + (1) Version 1.0 of HTTP was defined in RFC 1945. HTTP 1.1 was +initially specified in RFC 2068. In June 1999, RFC 2068 was made +obsolete by RFC 2616, an update without any substantial changes. + + +File: gawkinet.info, Node: Primitive Service, Next: Interacting Service, Prev: Web page, Up: Using Networking + +2.8 A Primitive Web Service +=========================== + +Now we know enough about HTTP to set up a primitive web service that +just says '"Hello, world"' when someone connects to it with a browser. +Compared to the situation in the preceding node, our program changes the +role. It tries to behave just like the server we have observed. Since +we are setting up a server here, we have to insert the port number in +the 'localport' field of the special file name. The other two fields +(HOSTNAME and REMOTEPORT) have to contain a '0' because we do not know +in advance which host will connect to our service. + + In the early 1990s, all a server had to do was send an HTML document +and close the connection. Here, we adhere to the modern syntax of HTTP. +The steps are as follows: + + 1. Send a status line telling the web browser that everything is okay. + + 2. Send a line to tell the browser how many bytes follow in the body + of the message. This was not necessary earlier because both + parties knew that the document ended when the connection closed. + Nowadays it is possible to stay connected after the transmission of + one web page. This is to avoid the network traffic necessary for + repeatedly establishing TCP connections for requesting several + images. Thus, there is the need to tell the receiving party how + many bytes will be sent. The header is terminated as usual with an + empty line. + + 3. Send the '"Hello, world"' body in HTML. The useless 'while' loop + swallows the request of the browser. We could actually omit the + loop, and on most machines the program would still work. First, + start the following program: + + BEGIN { + RS = ORS = "\r\n" + HttpService = "/inet/tcp/8080/0/0" + Hello = "<HTML><HEAD>" \ + "<TITLE>A Famous Greeting</TITLE></HEAD>" \ + "<BODY><H1>Hello, world</H1></BODY></HTML>" + Len = length(Hello) + length(ORS) + print "HTTP/1.0 200 OK" |& HttpService + print "Content-Length: " Len ORS |& HttpService + print Hello |& HttpService + while ((HttpService |& getline) > 0) + continue; + close(HttpService) + } + + Now, on the same machine, start your favorite browser and let it +point to <http://localhost:8080> (the browser needs to know on which +port our server is listening for requests). If this does not work, the +browser probably tries to connect to a proxy server that does not know +your machine. If so, change the browser's configuration so that the +browser does not try to use a proxy to connect to your machine. + + +File: gawkinet.info, Node: Interacting Service, Next: Simple Server, Prev: Primitive Service, Up: Using Networking + +2.9 A Web Service with Interaction +================================== + +This node shows how to set up a simple web server. The subnode is a +library file that we will use with all the examples in *note Some +Applications and Techniques::. + +* Menu: + +* CGI Lib:: A simple CGI library. + + Setting up a web service that allows user interaction is more +difficult and shows us the limits of network access in 'gawk'. In this +node, we develop a main program (a 'BEGIN' pattern and its action) that +will become the core of event-driven execution controlled by a graphical +user interface (GUI). Each HTTP event that the user triggers by some +action within the browser is received in this central procedure. +Parameters and menu choices are extracted from this request, and an +appropriate measure is taken according to the user's choice. For +example: + + BEGIN { + if (MyHost == "") { + "uname -n" | getline MyHost + close("uname -n") + } + if (MyPort == 0) MyPort = 8080 + HttpService = "/inet/tcp/" MyPort "/0/0" + MyPrefix = "http://" MyHost ":" MyPort + SetUpServer() + while ("awk" != "complex") { + # header lines are terminated this way + RS = ORS = "\r\n" + Status = 200 # this means OK + Reason = "OK" + Header = TopHeader + Document = TopDoc + Footer = TopFooter + if (GETARG["Method"] == "GET") { + HandleGET() + } else if (GETARG["Method"] == "HEAD") { + # not yet implemented + } else if (GETARG["Method"] != "") { + print "bad method", GETARG["Method"] + } + Prompt = Header Document Footer + print "HTTP/1.0", Status, Reason |& HttpService + print "Connection: Close" |& HttpService + print "Pragma: no-cache" |& HttpService + len = length(Prompt) + length(ORS) + print "Content-length:", len |& HttpService + print ORS Prompt |& HttpService + # ignore all the header lines + while ((HttpService |& getline) > 0) + ; + # stop talking to this client + close(HttpService) + # wait for new client request + HttpService |& getline + # do some logging + print systime(), strftime(), $0 + # read request parameters + CGI_setup($1, $2, $3) + } + } + + This web server presents menu choices in the form of HTML links. +Therefore, it has to tell the browser the name of the host it is +residing on. When starting the server, the user may supply the name of +the host from the command line with 'gawk -v MyHost="Rumpelstilzchen"'. +If the user does not do this, the server looks up the name of the host +it is running on for later use as a web address in HTML documents. The +same applies to the port number. These values are inserted later into +the HTML content of the web pages to refer to the home system. + + Each server that is built around this core has to initialize some +application-dependent variables (such as the default home page) in a +procedure 'SetUpServer()', which is called immediately before entering +the infinite loop of the server. For now, we will write an instance +that initiates a trivial interaction. With this home page, the client +user can click on two possible choices, and receive the current date +either in human-readable format or in seconds since 1970: + + function SetUpServer() { + TopHeader = "<HTML><HEAD>" + TopHeader = TopHeader \ + "<title>My name is GAWK, GNU AWK</title></HEAD>" + TopDoc = "<BODY><h2>\ + Do you prefer your date <A HREF=" MyPrefix \ + "/human>human</A> or \ + <A HREF=" MyPrefix "/POSIX>POSIXed</A>?</h2>" ORS ORS + TopFooter = "</BODY></HTML>" + } + + On the first run through the main loop, the default line terminators +are set and the default home page is copied to the actual home page. +Since this is the first run, 'GETARG["Method"]' is not initialized yet, +hence the case selection over the method does nothing. Now that the +home page is initialized, the server can start communicating to a client +browser. + + It does so by printing the HTTP header into the network connection +('print ... |& HttpService'). This command blocks execution of the +server script until a client connects. If this server script is +compared with the primitive one we wrote before, you will notice two +additional lines in the header. The first instructs the browser to +close the connection after each request. The second tells the browser +that it should never try to _remember_ earlier requests that had +identical web addresses (no caching). Otherwise, it could happen that +the browser retrieves the time of day in the previous example just once, +and later it takes the web page from the cache, always displaying the +same time of day although time advances each second. + + Having supplied the initial home page to the browser with a valid +document stored in the parameter 'Prompt', it closes the connection and +waits for the next request. When the request comes, a log line is +printed that allows us to see which request the server receives. The +final step in the loop is to call the function 'CGI_setup()', which +reads all the lines of the request (coming from the browser), processes +them, and stores the transmitted parameters in the array 'PARAM'. The +complete text of these application-independent functions can be found in +*note A Simple CGI Library: CGI Lib. For now, we use a simplified +version of 'CGI_setup()': + + function CGI_setup( method, uri, version, i) { + delete GETARG; delete MENU; delete PARAM + GETARG["Method"] = $1 + GETARG["URI"] = $2 + GETARG["Version"] = $3 + i = index($2, "?") + # is there a "?" indicating a CGI request? + if (i > 0) { + split(substr($2, 1, i-1), MENU, "[/:]") + split(substr($2, i+1), PARAM, "&") + for (i in PARAM) { + j = index(PARAM[i], "=") + GETARG[substr(PARAM[i], 1, j-1)] = \ + substr(PARAM[i], j+1) + } + } else { # there is no "?", no need for splitting PARAMs + split($2, MENU, "[/:]") + } + } + + At first, the function clears all variables used for global storage +of request parameters. The rest of the function serves the purpose of +filling the global parameters with the extracted new values. To +accomplish this, the name of the requested resource is split into parts +and stored for later evaluation. If the request contains a '?', then +the request has CGI variables seamlessly appended to the web address. +Everything in front of the '?' is split up into menu items, and +everything behind the '?' is a list of 'VARIABLE=VALUE' pairs (separated +by '&') that also need splitting. This way, CGI variables are isolated +and stored. This procedure lacks recognition of special characters that +are transmitted in coded form(1). Here, any optional request header and +body parts are ignored. We do not need header parameters and the +request body. However, when refining our approach or working with the +'POST' and 'PUT' methods, reading the header and body becomes +inevitable. Header parameters should then be stored in a global array +as well as the body. + + On each subsequent run through the main loop, one request from a +browser is received, evaluated, and answered according to the user's +choice. This can be done by letting the value of the HTTP method guide +the main loop into execution of the procedure 'HandleGET()', which +evaluates the user's choice. In this case, we have only one +hierarchical level of menus, but in the general case, menus are nested. +The menu choices at each level are separated by '/', just as in file +names. Notice how simple it is to construct menus of arbitrary depth: + + function HandleGET() { + if ( MENU[2] == "human") { + Footer = strftime() TopFooter + } else if (MENU[2] == "POSIX") { + Footer = systime() TopFooter + } + } + + The disadvantage of this approach is that our server is slow and can +handle only one request at a time. Its main advantage, however, is that +the server consists of just one 'gawk' program. No need for installing +an 'httpd', and no need for static separate HTML files, CGI scripts, or +'root' privileges. This is rapid prototyping. This program can be +started on the same host that runs your browser. Then let your browser +point to <http://localhost:8080>. + + It is also possible to include images into the HTML pages. Most +browsers support the not very well-known '.xbm' format, which may +contain only monochrome pictures but is an ASCII format. Binary images +are possible but not so easy to handle. Another way of including images +is to generate them with a tool such as GNUPlot, by calling the tool +with the 'system()' function or through a pipe. + + ---------- Footnotes ---------- + + (1) As defined in RFC 2068. + + +File: gawkinet.info, Node: CGI Lib, Prev: Interacting Service, Up: Interacting Service + +2.9.1 A Simple CGI Library +-------------------------- + + HTTP is like being married: you have to be able to handle whatever + you're given, while being very careful what you send back. + Phil Smith III, + <http://www.netfunny.com/rhf/jokes/99/Mar/http.html> + + In *note A Web Service with Interaction: Interacting Service, we saw +the function 'CGI_setup()' as part of the web server "core logic" +framework. The code presented there handles almost everything necessary +for CGI requests. One thing it doesn't do is handle encoded characters +in the requests. For example, an '&' is encoded as a percent sign +followed by the hexadecimal value: '%26'. These encoded values should +be decoded. Following is a simple library to perform these tasks. This +code is used for all web server examples used throughout the rest of +this Info file. If you want to use it for your own web server, store +the source code into a file named 'inetlib.awk'. Then you can include +these functions into your code by placing the following statement into +your program (on the first line of your script): + + @include inetlib.awk + +But beware, this mechanism is only possible if you invoke your web +server script with 'igawk' instead of the usual 'awk' or 'gawk'. Here +is the code: + + # CGI Library and core of a web server + # Global arrays + # GETARG --- arguments to CGI GET command + # MENU --- menu items (path names) + # PARAM --- parameters of form x=y + + # Optional variable MyHost contains host address + # Optional variable MyPort contains port number + # Needs TopHeader, TopDoc, TopFooter + # Sets MyPrefix, HttpService, Status, Reason + + BEGIN { + if (MyHost == "") { + "uname -n" | getline MyHost + close("uname -n") + } + if (MyPort == 0) MyPort = 8080 + HttpService = "/inet/tcp/" MyPort "/0/0" + MyPrefix = "http://" MyHost ":" MyPort + SetUpServer() + while ("awk" != "complex") { + # header lines are terminated this way + RS = ORS = "\r\n" + Status = 200 # this means OK + Reason = "OK" + Header = TopHeader + Document = TopDoc + Footer = TopFooter + if (GETARG["Method"] == "GET") { + HandleGET() + } else if (GETARG["Method"] == "HEAD") { + # not yet implemented + } else if (GETARG["Method"] != "") { + print "bad method", GETARG["Method"] + } + Prompt = Header Document Footer + print "HTTP/1.0", Status, Reason |& HttpService + print "Connection: Close" |& HttpService + print "Pragma: no-cache" |& HttpService + len = length(Prompt) + length(ORS) + print "Content-length:", len |& HttpService + print ORS Prompt |& HttpService + # ignore all the header lines + while ((HttpService |& getline) > 0) + continue + # stop talking to this client + close(HttpService) + # wait for new client request + HttpService |& getline + # do some logging + print systime(), strftime(), $0 + CGI_setup($1, $2, $3) + } + } + + function CGI_setup( method, uri, version, i) + { + delete GETARG + delete MENU + delete PARAM + GETARG["Method"] = method + GETARG["URI"] = uri + GETARG["Version"] = version + + i = index(uri, "?") + if (i > 0) { # is there a "?" indicating a CGI request? + split(substr(uri, 1, i-1), MENU, "[/:]") + split(substr(uri, i+1), PARAM, "&") + for (i in PARAM) { + PARAM[i] = _CGI_decode(PARAM[i]) + j = index(PARAM[i], "=") + GETARG[substr(PARAM[i], 1, j-1)] = \ + substr(PARAM[i], j+1) + } + } else { # there is no "?", no need for splitting PARAMs + split(uri, MENU, "[/:]") + } + for (i in MENU) # decode characters in path + if (i > 4) # but not those in host name + MENU[i] = _CGI_decode(MENU[i]) + } + + This isolates details in a single function, 'CGI_setup()'. Decoding +of encoded characters is pushed off to a helper function, +'_CGI_decode()'. The use of the leading underscore ('_') in the +function name is intended to indicate that it is an "internal" function, +although there is nothing to enforce this: + + function _CGI_decode(str, hexdigs, i, pre, code1, code2, + val, result) + { + hexdigs = "123456789abcdef" + + i = index(str, "%") + if (i == 0) # no work to do + return str + + do { + pre = substr(str, 1, i-1) # part before %xx + code1 = substr(str, i+1, 1) # first hex digit + code2 = substr(str, i+2, 1) # second hex digit + str = substr(str, i+3) # rest of string + + code1 = tolower(code1) + code2 = tolower(code2) + val = index(hexdigs, code1) * 16 \ + + index(hexdigs, code2) + + result = result pre sprintf("%c", val) + i = index(str, "%") + } while (i != 0) + if (length(str) > 0) + result = result str + return result + } + + This works by splitting the string apart around an encoded character. +The two digits are converted to lowercase characters and looked up in a +string of hex digits. Note that '0' is not in the string on purpose; +'index()' returns zero when it's not found, automatically giving the +correct value! Once the hexadecimal value is converted from characters +in a string into a numerical value, 'sprintf()' converts the value back +into a real character. The following is a simple test harness for the +above functions: + + BEGIN { + CGI_setup("GET", + "http://www.gnu.org/cgi-bin/foo?p1=stuff&p2=stuff%26junk" \ + "&percent=a %25 sign", + "1.0") + for (i in MENU) + printf "MENU[\"%s\"] = %s\n", i, MENU[i] + for (i in PARAM) + printf "PARAM[\"%s\"] = %s\n", i, PARAM[i] + for (i in GETARG) + printf "GETARG[\"%s\"] = %s\n", i, GETARG[i] + } + + And this is the result when we run it: + + $ gawk -f testserv.awk + -| MENU["4"] = www.gnu.org + -| MENU["5"] = cgi-bin + -| MENU["6"] = foo + -| MENU["1"] = http + -| MENU["2"] = + -| MENU["3"] = + -| PARAM["1"] = p1=stuff + -| PARAM["2"] = p2=stuff&junk + -| PARAM["3"] = percent=a % sign + -| GETARG["p1"] = stuff + -| GETARG["percent"] = a % sign + -| GETARG["p2"] = stuff&junk + -| GETARG["Method"] = GET + -| GETARG["Version"] = 1.0 + -| GETARG["URI"] = http://www.gnu.org/cgi-bin/foo?p1=stuff& + p2=stuff%26junk&percent=a %25 sign + + +File: gawkinet.info, Node: Simple Server, Next: Caveats, Prev: Interacting Service, Up: Using Networking + +2.10 A Simple Web Server +======================== + +In the preceding node, we built the core logic for event-driven GUIs. +In this node, we finally extend the core to a real application. No one +would actually write a commercial web server in 'gawk', but it is +instructive to see that it is feasible in principle. + + The application is ELIZA, the famous program by Joseph Weizenbaum +that mimics the behavior of a professional psychotherapist when talking +to you. Weizenbaum would certainly object to this description, but this +is part of the legend around ELIZA. Take the site-independent core logic +and append the following code: + + function SetUpServer() { + SetUpEliza() + TopHeader = \ + "<HTML><title>An HTTP-based System with GAWK</title>\ + <HEAD><META HTTP-EQUIV=\"Content-Type\"\ + CONTENT=\"text/html; charset=iso-8859-1\"></HEAD>\ + <BODY BGCOLOR=\"#ffffff\" TEXT=\"#000000\"\ + LINK=\"#0000ff\" VLINK=\"#0000ff\"\ + ALINK=\"#0000ff\"> <A NAME=\"top\">" + TopDoc = "\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI>\ + <A HREF=" MyPrefix "/AboutServer>About this server</A>\ + </LI><LI>\ + <A HREF=" MyPrefix "/AboutELIZA>About Eliza</A></LI>\ + <LI>\ + <A HREF=" MyPrefix \ + "/StartELIZA>Start talking to Eliza</A></LI></UL>" + TopFooter = "</BODY></HTML>" + } + + 'SetUpServer()' is similar to the previous example, except for +calling another function, 'SetUpEliza()'. This approach can be used to +implement other kinds of servers. The only changes needed to do so are +hidden in the functions 'SetUpServer()' and 'HandleGET()'. Perhaps it +might be necessary to implement other HTTP methods. The 'igawk' program +that comes with 'gawk' may be useful for this process. + + When extending this example to a complete application, the first +thing to do is to implement the function 'SetUpServer()' to initialize +the HTML pages and some variables. These initializations determine the +way your HTML pages look (colors, titles, menu items, etc.). + + The function 'HandleGET()' is a nested case selection that decides +which page the user wants to see next. Each nesting level refers to a +menu level of the GUI. Each case implements a certain action of the +menu. On the deepest level of case selection, the handler essentially +knows what the user wants and stores the answer into the variable that +holds the HTML page contents: + + function HandleGET() { + # A real HTTP server would treat some parts of the URI as a file name. + # We take parts of the URI as menu choices and go on accordingly. + if(MENU[2] == "AboutServer") { + Document = "This is not a CGI script.\ + This is an httpd, an HTML file, and a CGI script all \ + in one GAWK script. It needs no separate www-server, \ + no installation, and no root privileges.\ + <p>To run it, do this:</p><ul>\ + <li> start this script with \"gawk -f httpserver.awk\",</li>\ + <li> and on the same host let your www browser open location\ + \"http://localhost:8080\"</li>\ + </ul>\<p>\ Details of HTTP come from:</p><ul>\ + <li>Hethmon: Illustrated Guide to HTTP</p>\ + <li>RFC 2068</li></ul><p>JK 14.9.1997</p>" + } else if (MENU[2] == "AboutELIZA") { + Document = "This is an implementation of the famous ELIZA\ + program by Joseph Weizenbaum. It is written in GAWK and\ + uses an HTML GUI." + } else if (MENU[2] == "StartELIZA") { + gsub(/\+/, " ", GETARG["YouSay"]) + # Here we also have to substitute coded special characters + Document = "<form method=GET>" \ + "<h3>" ElizaSays(GETARG["YouSay"]) "</h3>\ + <p><input type=text name=YouSay value=\"\" size=60>\ + <br><input type=submit value=\"Tell her about it\"></p></form>" + } + } + + Now we are down to the heart of ELIZA, so you can see how it works. +Initially the user does not say anything; then ELIZA resets its money +counter and asks the user to tell what comes to mind open heartedly. +The subsequent answers are converted to uppercase characters and stored +for later comparison. ELIZA presents the bill when being confronted +with a sentence that contains the phrase "shut up." Otherwise, it looks +for keywords in the sentence, conjugates the rest of the sentence, +remembers the keyword for later use, and finally selects an answer from +the set of possible answers: + + function ElizaSays(YouSay) { + if (YouSay == "") { + cost = 0 + answer = "HI, IM ELIZA, TELL ME YOUR PROBLEM" + } else { + q = toupper(YouSay) + gsub("'", "", q) + if(q == qold) { + answer = "PLEASE DONT REPEAT YOURSELF !" + } else { + if (index(q, "SHUT UP") > 0) { + answer = "WELL, PLEASE PAY YOUR BILL. ITS EXACTLY ... $"\ + int(100*rand()+30+cost/100) + } else { + qold = q + w = "-" # no keyword recognized yet + for (i in k) { # search for keywords + if (index(q, i) > 0) { + w = i + break + } + } + if (w == "-") { # no keyword, take old subject + w = wold + subj = subjold + } else { # find subject + subj = substr(q, index(q, w) + length(w)+1) + wold = w + subjold = subj # remember keyword and subject + } + for (i in conj) + gsub(i, conj[i], q) # conjugation + # from all answers to this keyword, select one randomly + answer = r[indices[int(split(k[w], indices) * rand()) + 1]] + # insert subject into answer + gsub("_", subj, answer) + } + } + } + cost += length(answer) # for later payment : 1 cent per character + return answer + } + + In the long but simple function 'SetUpEliza()', you can see tables +for conjugation, keywords, and answers.(1) The associative array 'k' +contains indices into the array of answers 'r'. To choose an answer, +ELIZA just picks an index randomly: + + function SetUpEliza() { + srand() + wold = "-" + subjold = " " + + # table for conjugation + conj[" ARE " ] = " AM " + conj["WERE " ] = "WAS " + conj[" YOU " ] = " I " + conj["YOUR " ] = "MY " + conj[" IVE " ] =\ + conj[" I HAVE " ] = " YOU HAVE " + conj[" YOUVE " ] =\ + conj[" YOU HAVE "] = " I HAVE " + conj[" IM " ] =\ + conj[" I AM " ] = " YOU ARE " + conj[" YOURE " ] =\ + conj[" YOU ARE " ] = " I AM " + + # table of all answers + r[1] = "DONT YOU BELIEVE THAT I CAN _" + r[2] = "PERHAPS YOU WOULD LIKE TO BE ABLE TO _ ?" + ... + + # table for looking up answers that + # fit to a certain keyword + k["CAN YOU"] = "1 2 3" + k["CAN I"] = "4 5" + k["YOU ARE"] =\ + k["YOURE"] = "6 7 8 9" + ... + } + + Some interesting remarks and details (including the original source +code of ELIZA) are found on Mark Humphrys' home page. Yahoo! also has +a page with a collection of ELIZA-like programs. Many of them are +written in Java, some of them disclosing the Java source code, and a few +even explain how to modify the Java source code. + + ---------- Footnotes ---------- + + (1) The version shown here is abbreviated. The full version comes +with the 'gawk' distribution. + + +File: gawkinet.info, Node: Caveats, Next: Challenges, Prev: Simple Server, Up: Using Networking + +2.11 Network Programming Caveats +================================ + +By now it should be clear that debugging a networked application is more +complicated than debugging a single-process single-hosted application. +The behavior of a networked application sometimes looks noncausal +because it is not reproducible in a strong sense. Whether a network +application works or not sometimes depends on the following: + + * How crowded the underlying network is + + * If the party at the other end is running or not + + * The state of the party at the other end + + The most difficult problems for a beginner arise from the hidden +states of the underlying network. After closing a TCP connection, it's +often necessary to wait a short while before reopening the connection. +Even more difficult is the establishment of a connection that previously +ended with a "broken pipe." Those connections have to "time out" for a +minute or so before they can reopen. Check this with the command +'netstat -a', which provides a list of still "active" connections. + + +File: gawkinet.info, Node: Challenges, Prev: Caveats, Up: Using Networking + +2.12 Where To Go From Here +========================== + +Now, you have learned enough to build your own application. You could, +for example, take part in the Loebner Contest to win $100,000. The +Loebner Prize is the first formal instantiation of a Turing Test. Hugh +Loebner agreed with The Cambridge Center for Behavioral Studies to +underwrite a contest designed to implement the Turing Test. Dr. Loebner +pledged a Grand Prize of $100,000 for the first computer whose responses +were indistinguishable from a human's. Each year an annual prize of +$2000 and a bronze medal is awarded to the _most_ human computer. The +winner of the annual contest is the best entry relative to other entries +that year, irrespective of how good it is in an absolute sense. Here is +an example of a conversation with the winning program of 1997: + + (c)1997 Cambridge Center For Behavioral Studies all rights reserved [CONVERSE] + [Intelligent Research Ltd.] Start at: [1997/04/29 08:59:40] + PROGRAM[10:39:20]Hi, my name is Catherine, what's yours? + JUDGE02[10:39:36]My name is Nathan. + PROGRAM[10:39:37]Well, it's nice to meet you, Nathan. + PROGRAM[10:39:45]Did you see that story on CNN last night about the lesbian + PROGRAM[10:40:03]couple who came out at a White House party on Sunday? + JUDGE02[10:40:45]No. I'm sorry I missed that. How did the Prez take it? + PROGRAM[10:40:47]Ellen Degeneres was one of them - she was kissing her lover + PROGRAM[10:41:08]in front of Bill Clinton. Do you think he was very + PROGRAM[10:41:21]embarrassed? + JUDGE02[10:41:37]You tell me. I didn't see it. + PROGRAM[10:41:38]Are you still a Friend of Bill Clinton? + JUDGE02[10:42:22]Yes. I would like to talk about something else. Do you know anything about sculpture. + PROGRAM[10:42:24]Well, I suppose he's doing his best like most of us, but he + PROGRAM[10:42:35]does seem to attract an UNUSUAL amount of bad publicity, + PROGRAM[10:42:51]wouldnt you say--can it all be bad luck? + JUDGE02[10:43:20]I asked you if you knew anything about sculpture. + PROGRAM[10:43:21]OK, but what happens now--does it all drift on, with + PROGRAM[10:43:36]everyone waiting to see how much his friends have said about + PROGRAM[10:43:47]him under threat of lots and lots of jail-or will something + PROGRAM[10:44:01]dramatic actually happen? + JUDGE02[10:44:19]Do you know what sculpture is? + PROGRAM[10:44:20]Well, youre more a fan of his than I am I guess---though I + PROGRAM[10:44:34]cant SWEAR what Id have done in that locked hotel room, I + PROGRAM[10:44:51]must say. + JUDGE02[10:45:26]Have you ever been to the Salmagundi Club? + PROGRAM[10:45:29]Never. to go back to the original discussion Time to get + PROGRAM[10:45:43]off this one I think and onto something more down to earth! + JUDGE02[10:46:20]What is more down to earth than art? + PROGRAM[10:46:21]I don't know what -- to be honest. - + PROGRAM[10:46:30]I suppose it depends on where you're coming from, but as the + PROGRAM[10:46:46]song goes in My Fair Lady, "Why can't a woman be more like a + PROGRAM[10:47:03]man? " + JUDGE02[10:47:52]I liked that song but I prefer "Why can't children be more like we are" + JUDGE02[10:48:10]from another play the name of which I cannot remember. + PROGRAM[10:48:12]What, if anything, do you enjoy about being a man? + JUDGE02[10:49:01]I would enjoy all aspects of life no matter what my gender. + PROGRAM[10:49:02]Do you think of yourself as being attractive? + + This program insists on always speaking about the same story around +Bill Clinton. You see, even a program with a rather narrow mind can +behave so much like a human being that it can win this prize. It is +quite common to let these programs talk to each other via network +connections. But during the competition itself, the program and its +computer have to be present at the place the competition is held. We +all would love to see a 'gawk' program win in such an event. Maybe it +is up to you to accomplish this? + + Some other ideas for useful networked applications: + * Read the file 'doc/awkforai.txt' in the 'gawk' distribution. It + was written by Ronald P. Loui (at the time, Associate Professor of + Computer Science, at Washington University in St. Louis, + <loui@ai.wustl.edu>) and summarizes why he taught 'gawk' to + students of Artificial Intelligence. Here are some passages from + the text: + + The GAWK manual can be consumed in a single lab session and + the language can be mastered by the next morning by the + average student. GAWK's automatic initialization, implicit + coercion, I/O support and lack of pointers forgive many of the + mistakes that young programmers are likely to make. Those who + have seen C but not mastered it are happy to see that GAWK + retains some of the same sensibilities while adding what must + be regarded as spoonsful of syntactic sugar. + ... + There are further simple answers. Probably the best is the + fact that increasingly, undergraduate AI programming is + involving the Web. Oren Etzioni (University of Washington, + Seattle) has for a while been arguing that the "softbot" is + replacing the mechanical engineers' robot as the most + glamorous AI testbed. If the artifact whose behavior needs to + be controlled in an intelligent way is the software agent, + then a language that is well-suited to controlling the + software environment is the appropriate language. That would + imply a scripting language. If the robot is KAREL, then the + right language is "turn left; turn right." If the robot is + Netscape, then the right language is something that can + generate 'netscape -remote + 'openURL(http://cs.wustl.edu/~loui)'' with elan. + ... + AI programming requires high-level thinking. There have + always been a few gifted programmers who can write high-level + programs in assembly language. Most however need the ambient + abstraction to have a higher floor. + ... + Second, inference is merely the expansion of notation. No + matter whether the logic that underlies an AI program is + fuzzy, probabilistic, deontic, defeasible, or deductive, the + logic merely defines how strings can be transformed into other + strings. A language that provides the best support for string + processing in the end provides the best support for logic, for + the exploration of various logics, and for most forms of + symbolic processing that AI might choose to call "reasoning" + instead of "logic." The implication is that PROLOG, which + saves the AI programmer from having to write a unifier, saves + perhaps two dozen lines of GAWK code at the expense of + strongly biasing the logic and representational expressiveness + of any approach. + + Now that 'gawk' itself can connect to the Internet, it should be + obvious that it is suitable for writing intelligent web agents. + + * 'awk' is strong at pattern recognition and string processing. So, + it is well suited to the classic problem of language translation. + A first try could be a program that knows the 100 most frequent + English words and their counterparts in German or French. The + service could be implemented by regularly reading email with the + program above, replacing each word by its translation and sending + the translation back via SMTP. Users would send English email to + their translation service and get back a translated email message + in return. As soon as this works, more effort can be spent on a + real translation program. + + * Another dialogue-oriented application (on the verge of ridicule) is + the email "support service." Troubled customers write an email to + an automatic 'gawk' service that reads the email. It looks for + keywords in the mail and assembles a reply email accordingly. By + carefully investigating the email header, and repeating these + keywords through the reply email, it is rather simple to give the + customer a feeling that someone cares. Ideally, such a service + would search a database of previous cases for solutions. If none + exists, the database could, for example, consist of all the + newsgroups, mailing lists and FAQs on the Internet. + + +File: gawkinet.info, Node: Some Applications and Techniques, Next: Links, Prev: Using Networking, Up: Top + +3 Some Applications and Techniques +********************************** + +In this major node, we look at a number of self-contained scripts, with +an emphasis on concise networking. Along the way, we work towards +creating building blocks that encapsulate often needed functions of the +networking world, show new techniques that broaden the scope of problems +that can be solved with 'gawk', and explore leading edge technology that +may shape the future of networking. + + We often refer to the site-independent core of the server that we +built in *note A Simple Web Server: Simple Server. When building new +and nontrivial servers, we always copy this building block and append +new instances of the two functions 'SetUpServer()' and 'HandleGET()'. + + This makes a lot of sense, since this scheme of event-driven +execution provides 'gawk' with an interface to the most widely accepted +standard for GUIs: the web browser. Now, 'gawk' can rival even Tcl/Tk. + + Tcl and 'gawk' have much in common. Both are simple scripting +languages that allow us to quickly solve problems with short programs. +But Tcl has Tk on top of it, and 'gawk' had nothing comparable up to +now. While Tcl needs a large and ever-changing library (Tk, which was +bound to the X Window System until recently), 'gawk' needs just the +networking interface and some kind of browser on the client's side. +Besides better portability, the most important advantage of this +approach (embracing well-established standards such HTTP and HTML) is +that _we do not need to change the language_. We let others do the work +of fighting over protocols and standards. We can use HTML, JavaScript, +VRML, or whatever else comes along to do our work. + +* Menu: + +* PANIC:: An Emergency Web Server. +* GETURL:: Retrieving Web Pages. +* REMCONF:: Remote Configuration Of Embedded Systems. +* URLCHK:: Look For Changed Web Pages. +* WEBGRAB:: Extract Links From A Page. +* STATIST:: Graphing A Statistical Distribution. +* MAZE:: Walking Through A Maze In Virtual Reality. +* MOBAGWHO:: A Simple Mobile Agent. +* STOXPRED:: Stock Market Prediction As A Service. +* PROTBASE:: Searching Through A Protein Database. + + +File: gawkinet.info, Node: PANIC, Next: GETURL, Prev: Some Applications and Techniques, Up: Some Applications and Techniques + +3.1 PANIC: An Emergency Web Server +================================== + +At first glance, the '"Hello, world"' example in *note A Primitive Web +Service: Primitive Service, seems useless. By adding just a few lines, +we can turn it into something useful. + + The PANIC program tells everyone who connects that the local site is +not working. When a web server breaks down, it makes a difference if +customers get a strange "network unreachable" message, or a short +message telling them that the server has a problem. In such an +emergency, the hard disk and everything on it (including the regular web +service) may be unavailable. Rebooting the web server off a diskette +makes sense in this setting. + + To use the PANIC program as an emergency web server, all you need are +the 'gawk' executable and the program below on a diskette. By default, +it connects to port 8080. A different value may be supplied on the +command line: + + BEGIN { + RS = ORS = "\r\n" + if (MyPort == 0) MyPort = 8080 + HttpService = "/inet/tcp/" MyPort "/0/0" + Hello = "<HTML><HEAD><TITLE>Out Of Service</TITLE>" \ + "</HEAD><BODY><H1>" \ + "This site is temporarily out of service." \ + "</H1></BODY></HTML>" + Len = length(Hello) + length(ORS) + while ("awk" != "complex") { + print "HTTP/1.0 200 OK" |& HttpService + print "Content-Length: " Len ORS |& HttpService + print Hello |& HttpService + while ((HttpService |& getline) > 0) + continue; + close(HttpService) + } + } + + +File: gawkinet.info, Node: GETURL, Next: REMCONF, Prev: PANIC, Up: Some Applications and Techniques + +3.2 GETURL: Retrieving Web Pages +================================ + +GETURL is a versatile building block for shell scripts that need to +retrieve files from the Internet. It takes a web address as a +command-line parameter and tries to retrieve the contents of this +address. The contents are printed to standard output, while the header +is printed to '/dev/stderr'. A surrounding shell script could analyze +the contents and extract the text or the links. An ASCII browser could +be written around GETURL. But more interestingly, web robots are +straightforward to write on top of GETURL. On the Internet, you can find +several programs of the same name that do the same job. They are +usually much more complex internally and at least 10 times longer. + + At first, GETURL checks if it was called with exactly one web +address. Then, it checks if the user chose to use a special proxy +server whose name is handed over in a variable. By default, it is +assumed that the local machine serves as proxy. GETURL uses the 'GET' +method by default to access the web page. By handing over the name of a +different method (such as 'HEAD'), it is possible to choose a different +behavior. With the 'HEAD' method, the user does not receive the body of +the page content, but does receive the header: + + BEGIN { + if (ARGC != 2) { + print "GETURL - retrieve Web page via HTTP 1.0" + print "IN:\n the URL as a command-line parameter" + print "PARAM(S):\n -v Proxy=MyProxy" + print "OUT:\n the page content on stdout" + print " the page header on stderr" + print "JK 16.05.1997" + print "ADR 13.08.2000" + exit + } + URL = ARGV[1]; ARGV[1] = "" + if (Proxy == "") Proxy = "127.0.0.1" + if (ProxyPort == 0) ProxyPort = 80 + if (Method == "") Method = "GET" + HttpService = "/inet/tcp/0/" Proxy "/" ProxyPort + ORS = RS = "\r\n\r\n" + print Method " " URL " HTTP/1.0" |& HttpService + HttpService |& getline Header + print Header > "/dev/stderr" + while ((HttpService |& getline) > 0) + printf "%s", $0 + close(HttpService) + } + + This program can be changed as needed, but be careful with the last +lines. Make sure transmission of binary data is not corrupted by +additional line breaks. Even as it is now, the byte sequence +'"\r\n\r\n"' would disappear if it were contained in binary data. Don't +get caught in a trap when trying a quick fix on this one. + + +File: gawkinet.info, Node: REMCONF, Next: URLCHK, Prev: GETURL, Up: Some Applications and Techniques + +3.3 REMCONF: Remote Configuration of Embedded Systems +===================================================== + +Today, you often find powerful processors in embedded systems. +Dedicated network routers and controllers for all kinds of machinery are +examples of embedded systems. Processors like the Intel 80x86 or the +AMD Elan are able to run multitasking operating systems, such as XINU or +GNU/Linux in embedded PCs. These systems are small and usually do not +have a keyboard or a display. Therefore it is difficult to set up their +configuration. There are several widespread ways to set them up: + + * DIP switches + + * Read Only Memories such as EPROMs + + * Serial lines or some kind of keyboard + + * Network connections via 'telnet' or SNMP + + * HTTP connections with HTML GUIs + + In this node, we look at a solution that uses HTTP connections to +control variables of an embedded system that are stored in a file. +Since embedded systems have tight limits on resources like memory, it is +difficult to employ advanced techniques such as SNMP and HTTP servers. +'gawk' fits in quite nicely with its single executable which needs just +a short script to start working. The following program stores the +variables in a file, and a concurrent process in the embedded system may +read the file. The program uses the site-independent part of the simple +web server that we developed in *note A Web Service with Interaction: +Interacting Service. As mentioned there, all we have to do is to write +two new procedures 'SetUpServer()' and 'HandleGET()': + + function SetUpServer() { + TopHeader = "<HTML><title>Remote Configuration</title>" + TopDoc = "<BODY>\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\ + <LI><A HREF=" MyPrefix "/ReadConfig>Read Configuration</A></LI>\ + <LI><A HREF=" MyPrefix "/CheckConfig>Check Configuration</A></LI>\ + <LI><A HREF=" MyPrefix "/ChangeConfig>Change Configuration</A></LI>\ + <LI><A HREF=" MyPrefix "/SaveConfig>Save Configuration</A></LI>\ + </UL>" + TopFooter = "</BODY></HTML>" + if (ConfigFile == "") ConfigFile = "config.asc" + } + + The function 'SetUpServer()' initializes the top level HTML texts as +usual. It also initializes the name of the file that contains the +configuration parameters and their values. In case the user supplies a +name from the command line, that name is used. The file is expected to +contain one parameter per line, with the name of the parameter in column +one and the value in column two. + + The function 'HandleGET()' reflects the structure of the menu tree as +usual. The first menu choice tells the user what this is all about. +The second choice reads the configuration file line by line and stores +the parameters and their values. Notice that the record separator for +this file is '"\n"', in contrast to the record separator for HTTP. The +third menu choice builds an HTML table to show the contents of the +configuration file just read. The fourth choice does the real work of +changing parameters, and the last one just saves the configuration into +a file: + + function HandleGET() { + if(MENU[2] == "AboutServer") { + Document = "This is a GUI for remote configuration of an\ + embedded system. It is is implemented as one GAWK script." + } else if (MENU[2] == "ReadConfig") { + RS = "\n" + while ((getline < ConfigFile) > 0) + config[$1] = $2; + close(ConfigFile) + RS = "\r\n" + Document = "Configuration has been read." + } else if (MENU[2] == "CheckConfig") { + Document = "<TABLE BORDER=1 CELLPADDING=5>" + for (i in config) + Document = Document "<TR><TD>" i "</TD>" \ + "<TD>" config[i] "</TD></TR>" + Document = Document "</TABLE>" + } else if (MENU[2] == "ChangeConfig") { + if ("Param" in GETARG) { # any parameter to set? + if (GETARG["Param"] in config) { # is parameter valid? + config[GETARG["Param"]] = GETARG["Value"] + Document = (GETARG["Param"] " = " GETARG["Value"] ".") + } else { + Document = "Parameter <b>" GETARG["Param"] "</b> is invalid." + } + } else { + Document = "<FORM method=GET><h4>Change one parameter</h4>\ + <TABLE BORDER CELLPADDING=5>\ + <TR><TD>Parameter</TD><TD>Value</TD></TR>\ + <TR><TD><input type=text name=Param value=\"\" size=20></TD>\ + <TD><input type=text name=Value value=\"\" size=40></TD>\ + </TR></TABLE><input type=submit value=\"Set\"></FORM>" + } + } else if (MENU[2] == "SaveConfig") { + for (i in config) + printf("%s %s\n", i, config[i]) > ConfigFile + close(ConfigFile) + Document = "Configuration has been saved." + } + } + + We could also view the configuration file as a database. From this +point of view, the previous program acts like a primitive database +server. Real SQL database systems also make a service available by +providing a TCP port that clients can connect to. But the application +level protocols they use are usually proprietary and also change from +time to time. This is also true for the protocol that MiniSQL uses. + + +File: gawkinet.info, Node: URLCHK, Next: WEBGRAB, Prev: REMCONF, Up: Some Applications and Techniques + +3.4 URLCHK: Look for Changed Web Pages +====================================== + +Most people who make heavy use of Internet resources have a large +bookmark file with pointers to interesting web sites. It is impossible +to regularly check by hand if any of these sites have changed. A +program is needed to automatically look at the headers of web pages and +tell which ones have changed. URLCHK does the comparison after using +GETURL with the 'HEAD' method to retrieve the header. + + Like GETURL, this program first checks that it is called with exactly +one command-line parameter. URLCHK also takes the same command-line +variables 'Proxy' and 'ProxyPort' as GETURL, because these variables are +handed over to GETURL for each URL that gets checked. The one and only +parameter is the name of a file that contains one line for each URL. In +the first column, we find the URL, and the second and third columns hold +the length of the URL's body when checked for the two last times. Now, +we follow this plan: + + 1. Read the URLs from the file and remember their most recent lengths + + 2. Delete the contents of the file + + 3. For each URL, check its new length and write it into the file + + 4. If the most recent and the new length differ, tell the user + + It may seem a bit peculiar to read the URLs from a file together with +their two most recent lengths, but this approach has several advantages. +You can call the program again and again with the same file. After +running the program, you can regenerate the changed URLs by extracting +those lines that differ in their second and third columns: + + BEGIN { + if (ARGC != 2) { + print "URLCHK - check if URLs have changed" + print "IN:\n the file with URLs as a command-line parameter" + print " file contains URL, old length, new length" + print "PARAMS:\n -v Proxy=MyProxy -v ProxyPort=8080" + print "OUT:\n same as file with URLs" + print "JK 02.03.1998" + exit + } + URLfile = ARGV[1]; ARGV[1] = "" + if (Proxy != "") Proxy = " -v Proxy=" Proxy + if (ProxyPort != "") ProxyPort = " -v ProxyPort=" ProxyPort + while ((getline < URLfile) > 0) + Length[$1] = $3 + 0 + close(URLfile) # now, URLfile is read in and can be updated + GetHeader = "gawk " Proxy ProxyPort " -v Method=\"HEAD\" -f geturl.awk " + for (i in Length) { + GetThisHeader = GetHeader i " 2>&1" + while ((GetThisHeader | getline) > 0) + if (toupper($0) ~ /CONTENT-LENGTH/) NewLength = $2 + 0 + close(GetThisHeader) + print i, Length[i], NewLength > URLfile + if (Length[i] != NewLength) # report only changed URLs + print i, Length[i], NewLength + } + close(URLfile) + } + + Another thing that may look strange is the way GETURL is called. +Before calling GETURL, we have to check if the proxy variables need to +be passed on. If so, we prepare strings that will become part of the +command line later. In 'GetHeader()', we store these strings together +with the longest part of the command line. Later, in the loop over the +URLs, 'GetHeader()' is appended with the URL and a redirection operator +to form the command that reads the URL's header over the Internet. +GETURL always produces the headers over '/dev/stderr'. That is the +reason why we need the redirection operator to have the header piped in. + + This program is not perfect because it assumes that changing URLs +results in changed lengths, which is not necessarily true. A more +advanced approach is to look at some other header line that holds time +information. But, as always when things get a bit more complicated, +this is left as an exercise to the reader. + + +File: gawkinet.info, Node: WEBGRAB, Next: STATIST, Prev: URLCHK, Up: Some Applications and Techniques + +3.5 WEBGRAB: Extract Links from a Page +====================================== + +Sometimes it is necessary to extract links from web pages. Browsers do +it, web robots do it, and sometimes even humans do it. Since we have a +tool like GETURL at hand, we can solve this problem with some help from +the Bourne shell: + + BEGIN { RS = "http://[#%&\\+\\-\\./0-9\\:;\\?A-Z_a-z\\~]*" } + RT != "" { + command = ("gawk -v Proxy=MyProxy -f geturl.awk " RT \ + " > doc" NR ".html") + print command + } + + Notice that the regular expression for URLs is rather crude. A +precise regular expression is much more complex. But this one works +rather well. One problem is that it is unable to find internal links of +an HTML document. Another problem is that 'ftp', 'telnet', 'news', +'mailto', and other kinds of links are missing in the regular +expression. However, it is straightforward to add them, if doing so is +necessary for other tasks. + + This program reads an HTML file and prints all the HTTP links that it +finds. It relies on 'gawk''s ability to use regular expressions as +record separators. With 'RS' set to a regular expression that matches +links, the second action is executed each time a non-empty link is +found. We can find the matching link itself in 'RT'. + + The action could use the 'system()' function to let another GETURL +retrieve the page, but here we use a different approach. This simple +program prints shell commands that can be piped into 'sh' for execution. +This way it is possible to first extract the links, wrap shell commands +around them, and pipe all the shell commands into a file. After editing +the file, execution of the file retrieves exactly those files that we +really need. In case we do not want to edit, we can retrieve all the +pages like this: + + gawk -f geturl.awk http://www.suse.de | gawk -f webgrab.awk | sh + + After this, you will find the contents of all referenced documents in +files named 'doc*.html' even if they do not contain HTML code. The most +annoying thing is that we always have to pass the proxy to GETURL. If +you do not like to see the headers of the web pages appear on the +screen, you can redirect them to '/dev/null'. Watching the headers +appear can be quite interesting, because it reveals interesting details +such as which web server the companies use. Now, it is clear how the +clever marketing people use web robots to determine the market shares of +Microsoft and Netscape in the web server market. + + Port 80 of any web server is like a small hole in a repellent +firewall. After attaching a browser to port 80, we usually catch a +glimpse of the bright side of the server (its home page). With a tool +like GETURL at hand, we are able to discover some of the more concealed +or even "indecent" services (i.e., lacking conformity to standards of +quality). It can be exciting to see the fancy CGI scripts that lie +there, revealing the inner workings of the server, ready to be called: + + * With a command such as: + + gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/ + + some servers give you a directory listing of the CGI files. + Knowing the names, you can try to call some of them and watch for + useful results. Sometimes there are executables in such + directories (such as Perl interpreters) that you may call remotely. + If there are subdirectories with configuration data of the web + server, this can also be quite interesting to read. + + * The well-known Apache web server usually has its CGI files in the + directory '/cgi-bin'. There you can often find the scripts + 'test-cgi' and 'printenv'. Both tell you some things about the + current connection and the installation of the web server. Just + call: + + gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/test-cgi + gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/printenv + + * Sometimes it is even possible to retrieve system files like the web + server's log file--possibly containing customer data--or even the + file '/etc/passwd'. (We don't recommend this!) + + *Caution:* Although this may sound funny or simply irrelevant, we are +talking about severe security holes. Try to explore your own system +this way and make sure that none of the above reveals too much +information about your system. + + +File: gawkinet.info, Node: STATIST, Next: MAZE, Prev: WEBGRAB, Up: Some Applications and Techniques + +3.6 STATIST: Graphing a Statistical Distribution +================================================ + +In the HTTP server examples we've shown thus far, we never present an +image to the browser and its user. Presenting images is one task. +Generating images that reflect some user input and presenting these +dynamically generated images is another. In this node, we use GNUPlot +for generating '.png', '.ps', or '.gif' files.(1) + + The program we develop takes the statistical parameters of two +samples and computes the t-test statistics. As a result, we get the +probabilities that the means and the variances of both samples are the +same. In order to let the user check plausibility, the program presents +an image of the distributions. The statistical computation follows +'Numerical Recipes in C: The Art of Scientific Computing' by William H. +Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. +Since 'gawk' does not have a built-in function for the computation of +the beta function, we use the 'ibeta()' function of GNUPlot. As a side +effect, we learn how to use GNUPlot as a sophisticated calculator. The +comparison of means is done as in 'tutest', paragraph 14.2, page 613, +and the comparison of variances is done as in 'ftest', page 611 in +'Numerical Recipes'. + + As usual, we take the site-independent code for servers and append +our own functions 'SetUpServer()' and 'HandleGET()': + + function SetUpServer() { + TopHeader = "<HTML><title>Statistics with GAWK</title>" + TopDoc = "<BODY>\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\ + <LI><A HREF=" MyPrefix "/EnterParameters>Enter Parameters</A></LI>\ + </UL>" + TopFooter = "</BODY></HTML>" + GnuPlot = "gnuplot 2>&1" + m1=m2=0; v1=v2=1; n1=n2=10 + } + + Here, you see the menu structure that the user sees. Later, we will +see how the program structure of the 'HandleGET()' function reflects the +menu structure. What is missing here is the link for the image we +generate. In an event-driven environment, request, generation, and +delivery of images are separated. + + Notice the way we initialize the 'GnuPlot' command string for the +pipe. By default, GNUPlot outputs the generated image via standard +output, as well as the results of 'print'(ed) calculations via standard +error. The redirection causes standard error to be mixed into standard +output, enabling us to read results of calculations with 'getline'. By +initializing the statistical parameters with some meaningful defaults, +we make sure the user gets an image the first time he uses the program. + + Following is the rather long function 'HandleGET()', which implements +the contents of this service by reacting to the different kinds of +requests from the browser. Before you start playing with this script, +make sure that your browser supports JavaScript and that it also has +this option switched on. The script uses a short snippet of JavaScript +code for delayed opening of a window with an image. A more detailed +explanation follows: + + function HandleGET() { + if(MENU[2] == "AboutServer") { + Document = "This is a GUI for a statistical computation.\ + It compares means and variances of two distributions.\ + It is implemented as one GAWK script and uses GNUPLOT." + } else if (MENU[2] == "EnterParameters") { + Document = "" + if ("m1" in GETARG) { # are there parameters to compare? + Document = Document "<SCRIPT LANGUAGE=\"JavaScript\">\ + setTimeout(\"window.open(\\\"" MyPrefix "/Image" systime()\ + "\\\",\\\"dist\\\", \\\"status=no\\\");\", 1000); </SCRIPT>" + m1 = GETARG["m1"]; v1 = GETARG["v1"]; n1 = GETARG["n1"] + m2 = GETARG["m2"]; v2 = GETARG["v2"]; n2 = GETARG["n2"] + t = (m1-m2)/sqrt(v1/n1+v2/n2) + df = (v1/n1+v2/n2)*(v1/n1+v2/n2)/((v1/n1)*(v1/n1)/(n1-1) \ + + (v2/n2)*(v2/n2) /(n2-1)) + if (v1>v2) { + f = v1/v2 + df1 = n1 - 1 + df2 = n2 - 1 + } else { + f = v2/v1 + df1 = n2 - 1 + df2 = n1 - 1 + } + print "pt=ibeta(" df/2 ",0.5," df/(df+t*t) ")" |& GnuPlot + print "pF=2.0*ibeta(" df2/2 "," df1/2 "," \ + df2/(df2+df1*f) ")" |& GnuPlot + print "print pt, pF" |& GnuPlot + RS="\n"; GnuPlot |& getline; RS="\r\n" # $1 is pt, $2 is pF + print "invsqrt2pi=1.0/sqrt(2.0*pi)" |& GnuPlot + print "nd(x)=invsqrt2pi/sd*exp(-0.5*((x-mu)/sd)**2)" |& GnuPlot + print "set term png small color" |& GnuPlot + #print "set term postscript color" |& GnuPlot + #print "set term gif medium size 320,240" |& GnuPlot + print "set yrange[-0.3:]" |& GnuPlot + print "set label 'p(m1=m2) =" $1 "' at 0,-0.1 left" |& GnuPlot + print "set label 'p(v1=v2) =" $2 "' at 0,-0.2 left" |& GnuPlot + print "plot mu=" m1 ",sd=" sqrt(v1) ", nd(x) title 'sample 1',\ + mu=" m2 ",sd=" sqrt(v2) ", nd(x) title 'sample 2'" |& GnuPlot + print "quit" |& GnuPlot + GnuPlot |& getline Image + while ((GnuPlot |& getline) > 0) + Image = Image RS $0 + close(GnuPlot) + } + Document = Document "\ + <h3>Do these samples have the same Gaussian distribution?</h3>\ + <FORM METHOD=GET> <TABLE BORDER CELLPADDING=5>\ + <TR>\ + <TD>1. Mean </TD> + <TD><input type=text name=m1 value=" m1 " size=8></TD>\ + <TD>1. Variance</TD> + <TD><input type=text name=v1 value=" v1 " size=8></TD>\ + <TD>1. Count </TD> + <TD><input type=text name=n1 value=" n1 " size=8></TD>\ + </TR><TR>\ + <TD>2. Mean </TD> + <TD><input type=text name=m2 value=" m2 " size=8></TD>\ + <TD>2. Variance</TD> + <TD><input type=text name=v2 value=" v2 " size=8></TD>\ + <TD>2. Count </TD> + <TD><input type=text name=n2 value=" n2 " size=8></TD>\ + </TR> <input type=submit value=\"Compute\">\ + </TABLE></FORM><BR>" + } else if (MENU[2] ~ "Image") { + Reason = "OK" ORS "Content-type: image/png" + #Reason = "OK" ORS "Content-type: application/x-postscript" + #Reason = "OK" ORS "Content-type: image/gif" + Header = Footer = "" + Document = Image + } + } + + As usual, we give a short description of the service in the first +menu choice. The third menu choice shows us that generation and +presentation of an image are two separate actions. While the latter +takes place quite instantly in the third menu choice, the former takes +place in the much longer second choice. Image data passes from the +generating action to the presenting action via the variable 'Image' that +contains a complete '.png' image, which is otherwise stored in a file. +If you prefer '.ps' or '.gif' images over the default '.png' images, you +may select these options by uncommenting the appropriate lines. But +remember to do so in two places: when telling GNUPlot which kind of +images to generate, and when transmitting the image at the end of the +program. + + Looking at the end of the program, the way we pass the 'Content-type' +to the browser is a bit unusual. It is appended to the 'OK' of the +first header line to make sure the type information becomes part of the +header. The other variables that get transmitted across the network are +made empty, because in this case we do not have an HTML document to +transmit, but rather raw image data to contain in the body. + + Most of the work is done in the second menu choice. It starts with a +strange JavaScript code snippet. When first implementing this server, +we used a short '"<IMG SRC=" MyPrefix "/Image>"' here. But then +browsers got smarter and tried to improve on speed by requesting the +image and the HTML code at the same time. When doing this, the browser +tries to build up a connection for the image request while the request +for the HTML text is not yet completed. The browser tries to connect to +the 'gawk' server on port 8080 while port 8080 is still in use for +transmission of the HTML text. The connection for the image cannot be +built up, so the image appears as "broken" in the browser window. We +solved this problem by telling the browser to open a separate window for +the image, but only after a delay of 1000 milliseconds. By this time, +the server should be ready for serving the next request. + + But there is one more subtlety in the JavaScript code. Each time the +JavaScript code opens a window for the image, the name of the image is +appended with a timestamp ('systime()'). Why this constant change of +name for the image? Initially, we always named the image 'Image', but +then the Netscape browser noticed the name had _not_ changed since the +previous request and displayed the previous image (caching behavior). +The server core is implemented so that browsers are told _not_ to cache +anything. Obviously HTTP requests do not always work as expected. One +way to circumvent the cache of such overly smart browsers is to change +the name of the image with each request. These three lines of +JavaScript caused us a lot of trouble. + + The rest can be broken down into two phases. At first, we check if +there are statistical parameters. When the program is first started, +there usually are no parameters because it enters the page coming from +the top menu. Then, we only have to present the user a form that he can +use to change statistical parameters and submit them. Subsequently, the +submission of the form causes the execution of the first phase because +_now_ there _are_ parameters to handle. + + Now that we have parameters, we know there will be an image +available. Therefore we insert the JavaScript code here to initiate the +opening of the image in a separate window. Then, we prepare some +variables that will be passed to GNUPlot for calculation of the +probabilities. Prior to reading the results, we must temporarily change +'RS' because GNUPlot separates lines with newlines. After instructing +GNUPlot to generate a '.png' (or '.ps' or '.gif') image, we initiate the +insertion of some text, explaining the resulting probabilities. The +final 'plot' command actually generates the image data. This raw binary +has to be read in carefully without adding, changing, or deleting a +single byte. Hence the unusual initialization of 'Image' and completion +with a 'while' loop. + + When using this server, it soon becomes clear that it is far from +being perfect. It mixes source code of six scripting languages or +protocols: + + * GNU 'awk' implements a server for the protocol: + * HTTP which transmits: + * HTML text which contains a short piece of: + * JavaScript code opening a separate window. + * A Bourne shell script is used for piping commands into: + * GNUPlot to generate the image to be opened. + + After all this work, the GNUPlot image opens in the JavaScript window +where it can be viewed by the user. + + It is probably better not to mix up so many different languages. The +result is not very readable. Furthermore, the statistical part of the +server does not take care of invalid input. Among others, using +negative variances will cause invalid results. + + ---------- Footnotes ---------- + + (1) Due to licensing problems, the default installation of GNUPlot +disables the generation of '.gif' files. If your installed version does +not accept 'set term gif', just download and install the most recent +version of GNUPlot and the GD library (http://www.boutell.com/gd/) by +Thomas Boutell. Otherwise you still have the chance to generate some +ASCII-art style images with GNUPlot by using 'set term dumb'. (We tried +it and it worked.) + + +File: gawkinet.info, Node: MAZE, Next: MOBAGWHO, Prev: STATIST, Up: Some Applications and Techniques + +3.7 MAZE: Walking Through a Maze In Virtual Reality +=================================================== + + In the long run, every program becomes rococo, and then rubble. + Alan Perlis + + By now, we know how to present arbitrary 'Content-type's to a +browser. In this node, our server will present a 3D world to our +browser. The 3D world is described in a scene description language +(VRML, Virtual Reality Modeling Language) that allows us to travel +through a perspective view of a 2D maze with our browser. Browsers with +a VRML plugin enable exploration of this technology. We could do one of +those boring 'Hello world' examples here, that are usually presented +when introducing novices to VRML. If you have never written any VRML +code, have a look at the VRML FAQ. Presenting a static VRML scene is a +bit trivial; in order to expose 'gawk''s new capabilities, we will +present a dynamically generated VRML scene. The function +'SetUpServer()' is very simple because it only sets the default HTML +page and initializes the random number generator. As usual, the +surrounding server lets you browse the maze. + + function SetUpServer() { + TopHeader = "<HTML><title>Walk through a maze</title>" + TopDoc = "\ + <h2>Please choose one of the following actions:</h2>\ + <UL>\ + <LI><A HREF=" MyPrefix "/AboutServer>About this server</A>\ + <LI><A HREF=" MyPrefix "/VRMLtest>Watch a simple VRML scene</A>\ + </UL>" + TopFooter = "</HTML>" + srand() + } + + The function 'HandleGET()' is a bit longer because it first computes +the maze and afterwards generates the VRML code that is sent across the +network. As shown in the STATIST example (*note STATIST::), we set the +type of the content to VRML and then store the VRML representation of +the maze as the page content. We assume that the maze is stored in a 2D +array. Initially, the maze consists of walls only. Then, we add an +entry and an exit to the maze and let the rest of the work be done by +the function 'MakeMaze()'. Now, only the wall fields are left in the +maze. By iterating over the these fields, we generate one line of VRML +code for each wall field. + + function HandleGET() { + if (MENU[2] == "AboutServer") { + Document = "If your browser has a VRML 2 plugin,\ + this server shows you a simple VRML scene." + } else if (MENU[2] == "VRMLtest") { + XSIZE = YSIZE = 11 # initially, everything is wall + for (y = 0; y < YSIZE; y++) + for (x = 0; x < XSIZE; x++) + Maze[x, y] = "#" + delete Maze[0, 1] # entry is not wall + delete Maze[XSIZE-1, YSIZE-2] # exit is not wall + MakeMaze(1, 1) + Document = "\ + #VRML V2.0 utf8\n\ + Group {\n\ + children [\n\ + PointLight {\n\ + ambientIntensity 0.2\n\ + color 0.7 0.7 0.7\n\ + location 0.0 8.0 10.0\n\ + }\n\ + DEF B1 Background {\n\ + skyColor [0 0 0, 1.0 1.0 1.0 ]\n\ + skyAngle 1.6\n\ + groundColor [1 1 1, 0.8 0.8 0.8, 0.2 0.2 0.2 ]\n\ + groundAngle [ 1.2 1.57 ]\n\ + }\n\ + DEF Wall Shape {\n\ + geometry Box {size 1 1 1}\n\ + appearance Appearance { material Material { diffuseColor 0 0 1 } }\n\ + }\n\ + DEF Entry Viewpoint {\n\ + position 0.5 1.0 5.0\n\ + orientation 0.0 0.0 -1.0 0.52\n\ + }\n" + for (i in Maze) { + split(i, t, SUBSEP) + Document = Document " Transform { translation " + Document = Document t[1] " 0 -" t[2] " children USE Wall }\n" + } + Document = Document " ] # end of group for world\n}" + Reason = "OK" ORS "Content-type: model/vrml" + Header = Footer = "" + } + } + + Finally, we have a look at 'MakeMaze()', the function that generates +the 'Maze' array. When entered, this function assumes that the array +has been initialized so that each element represents a wall element and +the maze is initially full of wall elements. Only the entrance and the +exit of the maze should have been left free. The parameters of the +function tell us which element must be marked as not being a wall. +After this, we take a look at the four neighboring elements and remember +which we have already treated. Of all the neighboring elements, we take +one at random and walk in that direction. Therefore, the wall element +in that direction has to be removed and then, we call the function +recursively for that element. The maze is only completed if we iterate +the above procedure for _all_ neighboring elements (in random order) and +for our present element by recursively calling the function for the +present element. This last iteration could have been done in a loop, +but it is done much simpler recursively. + + Notice that elements with coordinates that are both odd are assumed +to be on our way through the maze and the generating process cannot +terminate as long as there is such an element not being 'delete'd. All +other elements are potentially part of the wall. + + function MakeMaze(x, y) { + delete Maze[x, y] # here we are, we have no wall here + p = 0 # count unvisited fields in all directions + if (x-2 SUBSEP y in Maze) d[p++] = "-x" + if (x SUBSEP y-2 in Maze) d[p++] = "-y" + if (x+2 SUBSEP y in Maze) d[p++] = "+x" + if (x SUBSEP y+2 in Maze) d[p++] = "+y" + if (p>0) { # if there are unvisited fields, go there + p = int(p*rand()) # choose one unvisited field at random + if (d[p] == "-x") { delete Maze[x - 1, y]; MakeMaze(x - 2, y) + } else if (d[p] == "-y") { delete Maze[x, y - 1]; MakeMaze(x, y - 2) + } else if (d[p] == "+x") { delete Maze[x + 1, y]; MakeMaze(x + 2, y) + } else if (d[p] == "+y") { delete Maze[x, y + 1]; MakeMaze(x, y + 2) + } # we are back from recursion + MakeMaze(x, y); # try again while there are unvisited fields + } + } + + +File: gawkinet.info, Node: MOBAGWHO, Next: STOXPRED, Prev: MAZE, Up: Some Applications and Techniques + +3.8 MOBAGWHO: a Simple Mobile Agent +=================================== + + There are two ways of constructing a software design: One way is to + make it so simple that there are obviously no deficiencies, and the + other way is to make it so complicated that there are no obvious + deficiencies. + C. A. R. Hoare + + A "mobile agent" is a program that can be dispatched from a computer +and transported to a remote server for execution. This is called +"migration", which means that a process on another system is started +that is independent from its originator. Ideally, it wanders through a +network while working for its creator or owner. In places like the UMBC +Agent Web, people are quite confident that (mobile) agents are a +software engineering paradigm that enables us to significantly increase +the efficiency of our work. Mobile agents could become the mediators +between users and the networking world. For an unbiased view at this +technology, see the remarkable paper 'Mobile Agents: Are they a good +idea?'.(1) + + When trying to migrate a process from one system to another, a server +process is needed on the receiving side. Depending on the kind of +server process, several ways of implementation come to mind. How the +process is implemented depends upon the kind of server process: + + * HTTP can be used as the protocol for delivery of the migrating + process. In this case, we use a common web server as the receiving + server process. A universal CGI script mediates between migrating + process and web server. Each server willing to accept migrating + agents makes this universal service available. HTTP supplies the + 'POST' method to transfer some data to a file on the web server. + When a CGI script is called remotely with the 'POST' method instead + of the usual 'GET' method, data is transmitted from the client + process to the standard input of the server's CGI script. So, to + implement a mobile agent, we must not only write the agent program + to start on the client side, but also the CGI script to receive the + agent on the server side. + + * The 'PUT' method can also be used for migration. HTTP does not + require a CGI script for migration via 'PUT'. However, with common + web servers there is no advantage to this solution, because web + servers such as Apache require explicit activation of a special + 'PUT' script. + + * 'Agent Tcl' pursues a different course; it relies on a dedicated + server process with a dedicated protocol specialized for receiving + mobile agents. + + Our agent example abuses a common web server as a migration tool. +So, it needs a universal CGI script on the receiving side (the web +server). The receiving script is activated with a 'POST' request when +placed into a location like '/httpd/cgi-bin/PostAgent.sh'. Make sure +that the server system uses a version of 'gawk' that supports network +access (Version 3.1 or later; verify with 'gawk --version'). + + #!/bin/sh + MobAg=/tmp/MobileAgent.$$ + # direct script to mobile agent file + cat > $MobAg + # execute agent concurrently + gawk -f $MobAg $MobAg > /dev/null & + # HTTP header, terminator and body + gawk 'BEGIN { print "\r\nAgent started" }' + rm $MobAg # delete script file of agent + + By making its process id ('$$') part of the unique file name, the +script avoids conflicts between concurrent instances of the script. +First, all lines from standard input (the mobile agent's source code) +are copied into this unique file. Then, the agent is started as a +concurrent process and a short message reporting this fact is sent to +the submitting client. Finally, the script file of the mobile agent is +removed because it is no longer needed. Although it is a short script, +there are several noteworthy points: + +Security + _There is none_. In fact, the CGI script should never be made + available on a server that is part of the Internet because everyone + would be allowed to execute arbitrary commands with it. This + behavior is acceptable only when performing rapid prototyping. + +Self-Reference + Each migrating instance of an agent is started in a way that + enables it to read its own source code from standard input and use + the code for subsequent migrations. This is necessary because it + needs to treat the agent's code as data to transmit. 'gawk' is not + the ideal language for such a job. Lisp and Tcl are more suitable + because they do not make a distinction between program code and + data. + +Independence + After migration, the agent is not linked to its former home in any + way. By reporting 'Agent started', it waves "Goodbye" to its + origin. The originator may choose to terminate or not. + + The originating agent itself is started just like any other +command-line script, and reports the results on standard output. By +letting the name of the original host migrate with the agent, the agent +that migrates to a host far away from its origin can report the result +back home. Having arrived at the end of the journey, the agent +establishes a connection and reports the results. This is the reason +for determining the name of the host with 'uname -n' and storing it in +'MyOrigin' for later use. We may also set variables with the '-v' +option from the command line. This interactivity is only of importance +in the context of starting a mobile agent; therefore this 'BEGIN' +pattern and its action do not take part in migration: + + BEGIN { + if (ARGC != 2) { + print "MOBAG - a simple mobile agent" + print "CALL:\n gawk -f mobag.awk mobag.awk" + print "IN:\n the name of this script as a command-line parameter" + print "PARAM:\n -v MyOrigin=myhost.com" + print "OUT:\n the result on stdout" + print "JK 29.03.1998 01.04.1998" + exit + } + if (MyOrigin == "") { + "uname -n" | getline MyOrigin + close("uname -n") + } + } + + Since 'gawk' cannot manipulate and transmit parts of the program +directly, the source code is read and stored in strings. Therefore, the +program scans itself for the beginning and the ending of functions. +Each line in between is appended to the code string until the end of the +function has been reached. A special case is this part of the program +itself. It is not a function. Placing a similar framework around it +causes it to be treated like a function. Notice that this mechanism +works for all the functions of the source code, but it cannot guarantee +that the order of the functions is preserved during migration: + + #ReadMySelf + /^function / { FUNC = $2 } + /^END/ || /^#ReadMySelf/ { FUNC = $1 } + FUNC != "" { MOBFUN[FUNC] = MOBFUN[FUNC] RS $0 } + (FUNC != "") && (/^}/ || /^#EndOfMySelf/) \ + { FUNC = "" } + #EndOfMySelf + + The web server code in *note A Web Service with Interaction: +Interacting Service, was first developed as a site-independent core. +Likewise, the 'gawk'-based mobile agent starts with an agent-independent +core, to which can be appended application-dependent functions. What +follows is the only application-independent function needed for the +mobile agent: + + function migrate(Destination, MobCode, Label) { + MOBVAR["Label"] = Label + MOBVAR["Destination"] = Destination + RS = ORS = "\r\n" + HttpService = "/inet/tcp/0/" Destination + for (i in MOBFUN) + MobCode = (MobCode "\n" MOBFUN[i]) + MobCode = MobCode "\n\nBEGIN {" + for (i in MOBVAR) + MobCode = (MobCode "\n MOBVAR[\"" i "\"] = \"" MOBVAR[i] "\"") + MobCode = MobCode "\n}\n" + print "POST /cgi-bin/PostAgent.sh HTTP/1.0" |& HttpService + print "Content-length:", length(MobCode) ORS |& HttpService + printf "%s", MobCode |& HttpService + while ((HttpService |& getline) > 0) + print $0 + close(HttpService) + } + + The 'migrate()' function prepares the aforementioned strings +containing the program code and transmits them to a server. A +consequence of this modular approach is that the 'migrate()' function +takes some parameters that aren't needed in this application, but that +will be in future ones. Its mandatory parameter 'Destination' holds the +name (or IP address) of the server that the agent wants as a host for +its code. The optional parameter 'MobCode' may contain some 'gawk' code +that is inserted during migration in front of all other code. The +optional parameter 'Label' may contain a string that tells the agent +what to do in program execution after arrival at its new home site. One +of the serious obstacles in implementing a framework for mobile agents +is that it does not suffice to migrate the code. It is also necessary +to migrate the state of execution of the agent. In contrast to 'Agent +Tcl', this program does not try to migrate the complete set of +variables. The following conventions are used: + + * Each variable in an agent program is local to the current host and + does _not_ migrate. + + * The array 'MOBFUN' shown above is an exception. It is handled by + the function 'migrate()' and does migrate with the application. + + * The other exception is the array 'MOBVAR'. Each variable that + takes part in migration has to be an element of this array. + 'migrate()' also takes care of this. + + Now it's clear what happens to the 'Label' parameter of the function +'migrate()'. It is copied into 'MOBVAR["Label"]' and travels alongside +the other data. Since travelling takes place via HTTP, records must be +separated with '"\r\n"' in 'RS' and 'ORS' as usual. The code assembly +for migration takes place in three steps: + + * Iterate over 'MOBFUN' to collect all functions verbatim. + + * Prepare a 'BEGIN' pattern and put assignments to mobile variables + into the action part. + + * Transmission itself resembles GETURL: the header with the request + and the 'Content-length' is followed by the body. In case there is + any reply over the network, it is read completely and echoed to + standard output to avoid irritating the server. + + The application-independent framework is now almost complete. What +follows is the 'END' pattern that is executed when the mobile agent has +finished reading its own code. First, it checks whether it is already +running on a remote host or not. In case initialization has not yet +taken place, it starts 'MyInit()'. Otherwise (later, on a remote host), +it starts 'MyJob()': + + END { + if (ARGC != 2) exit # stop when called with wrong parameters + if (MyOrigin != "") # is this the originating host? + MyInit() # if so, initialize the application + else # we are on a host with migrated data + MyJob() # so we do our job + } + + All that's left to extend the framework into a complete application +is to write two application-specific functions: 'MyInit()' and +'MyJob()'. Keep in mind that the former is executed once on the +originating host, while the latter is executed after each migration: + + function MyInit() { + MOBVAR["MyOrigin"] = MyOrigin + MOBVAR["Machines"] = "localhost/80 max/80 moritz/80 castor/80" + split(MOBVAR["Machines"], Machines) # which host is the first? + migrate(Machines[1], "", "") # go to the first host + while (("/inet/tcp/8080/0/0" |& getline) > 0) # wait for result + print $0 # print result + close("/inet/tcp/8080/0/0") + } + + As mentioned earlier, this agent takes the name of its origin +('MyOrigin') with it. Then, it takes the name of its first destination +and goes there for further work. Notice that this name has the port +number of the web server appended to the name of the server, because the +function 'migrate()' needs it this way to create the 'HttpService' +variable. Finally, it waits for the result to arrive. The 'MyJob()' +function runs on the remote host: + + function MyJob() { + # forget this host + sub(MOBVAR["Destination"], "", MOBVAR["Machines"]) + MOBVAR["Result"]=MOBVAR["Result"] SUBSEP SUBSEP MOBVAR["Destination"] ":" + while (("who" | getline) > 0) # who is logged in? + MOBVAR["Result"] = MOBVAR["Result"] SUBSEP $0 + close("who") + if (index(MOBVAR["Machines"], "/") > 0) { # any more machines to visit? + split(MOBVAR["Machines"], Machines) # which host is next? + migrate(Machines[1], "", "") # go there + } else { # no more machines + gsub(SUBSEP, "\n", MOBVAR["Result"]) # send result to origin + print MOBVAR["Result"] |& "/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080" + close("/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080") + } + } + + After migrating, the first thing to do in 'MyJob()' is to delete the +name of the current host from the list of hosts to visit. Now, it is +time to start the real work by appending the host's name to the result +string, and reading line by line who is logged in on this host. A very +annoying circumstance is the fact that the elements of 'MOBVAR' cannot +hold the newline character ('"\n"'). If they did, migration of this +string did not work because the string didn't obey the syntax rule for a +string in 'gawk'. 'SUBSEP' is used as a temporary replacement. If the +list of hosts to visit holds at least one more entry, the agent migrates +to that place to go on working there. Otherwise, we replace the +'SUBSEP's with a newline character in the resulting string, and report +it to the originating host, whose name is stored in +'MOBVAR["MyOrigin"]'. + + ---------- Footnotes ---------- + + (1) <http://www.research.ibm.com/massive/mobag.ps> + + +File: gawkinet.info, Node: STOXPRED, Next: PROTBASE, Prev: MOBAGWHO, Up: Some Applications and Techniques + +3.9 STOXPRED: Stock Market Prediction As A Service +================================================== + + Far out in the uncharted backwaters of the unfashionable end of the + Western Spiral arm of the Galaxy lies a small unregarded yellow + sun. + + Orbiting this at a distance of roughly ninety-two million miles is + an utterly insignificant little blue-green planet whose + ape-descendent life forms are so amazingly primitive that they + still think digital watches are a pretty neat idea. + + This planet has -- or rather had -- a problem, which was this: most + of the people living on it were unhappy for pretty much of the + time. Many solutions were suggested for this problem, but most of + these were largely concerned with the movements of small green + pieces of paper, which is odd because it wasn't the small green + pieces of paper that were unhappy. + Douglas Adams, 'The Hitch Hiker's Guide to the Galaxy' + + Valuable services on the Internet are usually _not_ implemented as +mobile agents. There are much simpler ways of implementing services. +All Unix systems provide, for example, the 'cron' service. Unix system +users can write a list of tasks to be done each day, each week, twice a +day, or just once. The list is entered into a file named 'crontab'. +For example, to distribute a newsletter on a daily basis this way, use +'cron' for calling a script each day early in the morning. + + # run at 8 am on weekdays, distribute the newsletter + 0 8 * * 1-5 $HOME/bin/daily.job >> $HOME/log/newsletter 2>&1 + + The script first looks for interesting information on the Internet, +assembles it in a nice form and sends the results via email to the +customers. + + The following is an example of a primitive newsletter on stock market +prediction. It is a report which first tries to predict the change of +each share in the Dow Jones Industrial Index for the particular day. +Then it mentions some especially promising shares as well as some shares +which look remarkably bad on that day. The report ends with the usual +disclaimer which tells every child _not_ to try this at home and hurt +anybody. + + Good morning Uncle Scrooge, + + This is your daily stock market report for Monday, October 16, 2000. + Here are the predictions for today: + + AA neutral + GE up + JNJ down + MSFT neutral + ... + UTX up + DD down + IBM up + MO down + WMT up + DIS up + INTC up + MRK down + XOM down + EK down + IP down + + The most promising shares for today are these: + + INTC http://biz.yahoo.com/n/i/intc.html + + The stock shares to avoid today are these: + + EK http://biz.yahoo.com/n/e/ek.html + IP http://biz.yahoo.com/n/i/ip.html + DD http://biz.yahoo.com/n/d/dd.html + ... + + The script as a whole is rather long. In order to ease the pain of +studying other people's source code, we have broken the script up into +meaningful parts which are invoked one after the other. The basic +structure of the script is as follows: + + BEGIN { + Init() + ReadQuotes() + CleanUp() + Prediction() + Report() + SendMail() + } + + The earlier parts store data into variables and arrays which are +subsequently used by later parts of the script. The 'Init()' function +first checks if the script is invoked correctly (without any +parameters). If not, it informs the user of the correct usage. What +follows are preparations for the retrieval of the historical quote data. +The names of the 30 stock shares are stored in an array 'name' along +with the current date in 'day', 'month', and 'year'. + + All users who are separated from the Internet by a firewall and have +to direct their Internet accesses to a proxy must supply the name of the +proxy to this script with the '-v Proxy=NAME' option. For most users, +the default proxy and port number should suffice. + + function Init() { + if (ARGC != 1) { + print "STOXPRED - daily stock share prediction" + print "IN:\n no parameters, nothing on stdin" + print "PARAM:\n -v Proxy=MyProxy -v ProxyPort=80" + print "OUT:\n commented predictions as email" + print "JK 09.10.2000" + exit + } + # Remember ticker symbols from Dow Jones Industrial Index + StockCount = split("AA GE JNJ MSFT AXP GM JPM PG BA HD KO \ + SBC C HON MCD T CAT HWP MMM UTX DD IBM MO WMT DIS INTC \ + MRK XOM EK IP", name); + # Remember the current date as the end of the time series + day = strftime("%d") + month = strftime("%m") + year = strftime("%Y") + if (Proxy == "") Proxy = "chart.yahoo.com" + if (ProxyPort == 0) ProxyPort = 80 + YahooData = "/inet/tcp/0/" Proxy "/" ProxyPort + } + + There are two really interesting parts in the script. One is the +function which reads the historical stock quotes from an Internet +server. The other is the one that does the actual prediction. In the +following function we see how the quotes are read from the Yahoo server. +The data which comes from the server is in CSV format (comma-separated +values): + + Date,Open,High,Low,Close,Volume + 9-Oct-00,22.75,22.75,21.375,22.375,7888500 + 6-Oct-00,23.8125,24.9375,21.5625,22,10701100 + 5-Oct-00,24.4375,24.625,23.125,23.50,5810300 + + Lines contain values of the same time instant, whereas columns are +separated by commas and contain the kind of data that is described in +the header (first) line. At first, 'gawk' is instructed to separate +columns by commas ('FS = ","'). In the loop that follows, a connection +to the Yahoo server is first opened, then a download takes place, and +finally the connection is closed. All this happens once for each ticker +symbol. In the body of this loop, an Internet address is built up as a +string according to the rules of the Yahoo server. The starting and +ending date are chosen to be exactly the same, but one year apart in the +past. All the action is initiated within the 'printf' command which +transmits the request for data to the Yahoo server. + + In the inner loop, the server's data is first read and then scanned +line by line. Only lines which have six columns and the name of a month +in the first column contain relevant data. This data is stored in the +two-dimensional array 'quote'; one dimension being time, the other being +the ticker symbol. During retrieval of the first stock's data, the +calendar names of the time instances are stored in the array 'day' +because we need them later. + + function ReadQuotes() { + # Retrieve historical data for each ticker symbol + FS = "," + for (stock = 1; stock <= StockCount; stock++) { + URL = "http://chart.yahoo.com/table.csv?s=" name[stock] \ + "&a=" month "&b=" day "&c=" year-1 \ + "&d=" month "&e=" day "&f=" year \ + "g=d&q=q&y=0&z=" name[stock] "&x=.csv" + printf("GET " URL " HTTP/1.0\r\n\r\n") |& YahooData + while ((YahooData |& getline) > 0) { + if (NF == 6 && $1 ~ /Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec/) { + if (stock == 1) + days[++daycount] = $1; + quote[$1, stock] = $5 + } + } + close(YahooData) + } + FS = " " + } + + Now that we _have_ the data, it can be checked once again to make +sure that no individual stock is missing or invalid, and that all the +stock quotes are aligned correctly. Furthermore, we renumber the time +instances. The most recent day gets day number 1 and all other days get +consecutive numbers. All quotes are rounded toward the nearest whole +number in US Dollars. + + function CleanUp() { + # clean up time series; eliminate incomplete data sets + for (d = 1; d <= daycount; d++) { + for (stock = 1; stock <= StockCount; stock++) + if (! ((days[d], stock) in quote)) + stock = StockCount + 10 + if (stock > StockCount + 1) + continue + datacount++ + for (stock = 1; stock <= StockCount; stock++) + data[datacount, stock] = int(0.5 + quote[days[d], stock]) + } + delete quote + delete days + } + + Now we have arrived at the second really interesting part of the +whole affair. What we present here is a very primitive prediction +algorithm: _If a stock fell yesterday, assume it will also fall today; +if it rose yesterday, assume it will rise today_. (Feel free to replace +this algorithm with a smarter one.) If a stock changed in the same +direction on two consecutive days, this is an indication which should be +highlighted. Two-day advances are stored in 'hot' and two-day declines +in 'avoid'. + + The rest of the function is a sanity check. It counts the number of +correct predictions in relation to the total number of predictions one +could have made in the year before. + + function Prediction() { + # Predict each ticker symbol by prolonging yesterday's trend + for (stock = 1; stock <= StockCount; stock++) { + if (data[1, stock] > data[2, stock]) { + predict[stock] = "up" + } else if (data[1, stock] < data[2, stock]) { + predict[stock] = "down" + } else { + predict[stock] = "neutral" + } + if ((data[1, stock] > data[2, stock]) && (data[2, stock] > data[3, stock])) + hot[stock] = 1 + if ((data[1, stock] < data[2, stock]) && (data[2, stock] < data[3, stock])) + avoid[stock] = 1 + } + # Do a plausibility check: how many predictions proved correct? + for (s = 1; s <= StockCount; s++) { + for (d = 1; d <= datacount-2; d++) { + if (data[d+1, s] > data[d+2, s]) { + UpCount++ + } else if (data[d+1, s] < data[d+2, s]) { + DownCount++ + } else { + NeutralCount++ + } + if (((data[d, s] > data[d+1, s]) && (data[d+1, s] > data[d+2, s])) || + ((data[d, s] < data[d+1, s]) && (data[d+1, s] < data[d+2, s])) || + ((data[d, s] == data[d+1, s]) && (data[d+1, s] == data[d+2, s]))) + CorrectCount++ + } + } + } + + At this point the hard work has been done: the array 'predict' +contains the predictions for all the ticker symbols. It is up to the +function 'Report()' to find some nice words to introduce the desired +information. + + function Report() { + # Generate report + report = "\nThis is your daily " + report = report "stock market report for "strftime("%A, %B %d, %Y")".\n" + report = report "Here are the predictions for today:\n\n" + for (stock = 1; stock <= StockCount; stock++) + report = report "\t" name[stock] "\t" predict[stock] "\n" + for (stock in hot) { + if (HotCount++ == 0) + report = report "\nThe most promising shares for today are these:\n\n" + report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \ + tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n" + } + for (stock in avoid) { + if (AvoidCount++ == 0) + report = report "\nThe stock shares to avoid today are these:\n\n" + report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \ + tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n" + } + report = report "\nThis sums up to " HotCount+0 " winners and " AvoidCount+0 + report = report " losers. When using this kind\nof prediction scheme for" + report = report " the 12 months which lie behind us,\nwe get " UpCount + report = report " 'ups' and " DownCount " 'downs' and " NeutralCount + report = report " 'neutrals'. Of all\nthese " UpCount+DownCount+NeutralCount + report = report " predictions " CorrectCount " proved correct next day.\n" + report = report "A success rate of "\ + int(100*CorrectCount/(UpCount+DownCount+NeutralCount)) "%.\n" + report = report "Random choice would have produced a 33% success rate.\n" + report = report "Disclaimer: Like every other prediction of the stock\n" + report = report "market, this report is, of course, complete nonsense.\n" + report = report "If you are stupid enough to believe these predictions\n" + report = report "you should visit a doctor who can treat your ailment." + } + + The function 'SendMail()' goes through the list of customers and +opens a pipe to the 'mail' command for each of them. Each one receives +an email message with a proper subject heading and is addressed with his +full name. + + function SendMail() { + # send report to customers + customer["uncle.scrooge@ducktown.gov"] = "Uncle Scrooge" + customer["more@utopia.org" ] = "Sir Thomas More" + customer["spinoza@denhaag.nl" ] = "Baruch de Spinoza" + customer["marx@highgate.uk" ] = "Karl Marx" + customer["keynes@the.long.run" ] = "John Maynard Keynes" + customer["bierce@devil.hell.org" ] = "Ambrose Bierce" + customer["laplace@paris.fr" ] = "Pierre Simon de Laplace" + for (c in customer) { + MailPipe = "mail -s 'Daily Stock Prediction Newsletter'" c + print "Good morning " customer[c] "," | MailPipe + print report "\n.\n" | MailPipe + close(MailPipe) + } + } + + Be patient when running the script by hand. Retrieving the data for +all the ticker symbols and sending the emails may take several minutes +to complete, depending upon network traffic and the speed of the +available Internet link. The quality of the prediction algorithm is +likely to be disappointing. Try to find a better one. Should you find +one with a success rate of more than 50%, please tell us about it! It +is only for the sake of curiosity, of course. ':-)' + + +File: gawkinet.info, Node: PROTBASE, Prev: STOXPRED, Up: Some Applications and Techniques + +3.10 PROTBASE: Searching Through A Protein Database +=================================================== + + Hoare's Law of Large Problems: Inside every large problem is a + small problem struggling to get out. + + Yahoo's database of stock market data is just one among the many +large databases on the Internet. Another one is located at NCBI +(National Center for Biotechnology Information). Established in 1988 as +a national resource for molecular biology information, NCBI creates +public databases, conducts research in computational biology, develops +software tools for analyzing genome data, and disseminates biomedical +information. In this section, we look at one of NCBI's public services, +which is called BLAST (Basic Local Alignment Search Tool). + + You probably know that the information necessary for reproducing +living cells is encoded in the genetic material of the cells. The +genetic material is a very long chain of four base nucleotides. It is +the order of appearance (the sequence) of nucleotides which contains the +information about the substance to be produced. Scientists in +biotechnology often find a specific fragment, determine the nucleotide +sequence, and need to know where the sequence at hand comes from. This +is where the large databases enter the game. At NCBI, databases store +the knowledge about which sequences have ever been found and where they +have been found. When the scientist sends his sequence to the BLAST +service, the server looks for regions of genetic material in its +database which look the most similar to the delivered nucleotide +sequence. After a search time of some seconds or minutes the server +sends an answer to the scientist. In order to make access simple, NCBI +chose to offer their database service through popular Internet +protocols. There are four basic ways to use the so-called BLAST +services: + + * The easiest way to use BLAST is through the web. Users may simply + point their browsers at the NCBI home page and link to the BLAST + pages. NCBI provides a stable URL that may be used to perform + BLAST searches without interactive use of a web browser. This is + what we will do later in this section. A demonstration client and + a 'README' file demonstrate how to access this URL. + + * Currently, 'blastcl3' is the standard network BLAST client. You + can download 'blastcl3' from the anonymous FTP location. + + * BLAST 2.0 can be run locally as a full executable and can be used + to run BLAST searches against private local databases, or + downloaded copies of the NCBI databases. BLAST 2.0 executables may + be found on the NCBI anonymous FTP server. + + * The NCBI BLAST Email server is the best option for people without + convenient access to the web. A similarity search can be performed + by sending a properly formatted mail message containing the + nucleotide or protein query sequence to <blast@ncbi.nlm.nih.gov>. + The query sequence is compared against the specified database using + the BLAST algorithm and the results are returned in an email + message. For more information on formulating email BLAST searches, + you can send a message consisting of the word "HELP" to the same + address, <blast@ncbi.nlm.nih.gov>. + + Our starting point is the demonstration client mentioned in the first +option. The 'README' file that comes along with the client explains the +whole process in a nutshell. In the rest of this section, we first show +what such requests look like. Then we show how to use 'gawk' to +implement a client in about 10 lines of code. Finally, we show how to +interpret the result returned from the service. + + Sequences are expected to be represented in the standard IUB/IUPAC +amino acid and nucleic acid codes, with these exceptions: lower-case +letters are accepted and are mapped into upper-case; a single hyphen or +dash can be used to represent a gap of indeterminate length; and in +amino acid sequences, 'U' and '*' are acceptable letters (see below). +Before submitting a request, any numerical digits in the query sequence +should either be removed or replaced by appropriate letter codes (e.g., +'N' for unknown nucleic acid residue or 'X' for unknown amino acid +residue). The nucleic acid codes supported are: + + A --> adenosine M --> A C (amino) + C --> cytidine S --> G C (strong) + G --> guanine W --> A T (weak) + T --> thymidine B --> G T C + U --> uridine D --> G A T + R --> G A (purine) H --> A C T + Y --> T C (pyrimidine) V --> G C A + K --> G T (keto) N --> A G C T (any) + - gap of indeterminate length + + Now you know the alphabet of nucleotide sequences. The last two +lines of the following example query show you such a sequence, which is +obviously made up only of elements of the alphabet just described. +Store this example query into a file named 'protbase.request'. You are +now ready to send it to the server with the demonstration client. + + PROGRAM blastn + DATALIB month + EXPECT 0.75 + BEGIN + >GAWK310 the gawking gene GNU AWK + tgcttggctgaggagccataggacgagagcttcctggtgaagtgtgtttcttgaaatcat + caccaccatggacagcaaa + + The actual search request begins with the mandatory parameter +'PROGRAM' in the first column followed by the value 'blastn' (the name +of the program) for searching nucleic acids. The next line contains the +mandatory search parameter 'DATALIB' with the value 'month' for the +newest nucleic acid sequences. The third line contains an optional +'EXPECT' parameter and the value desired for it. The fourth line +contains the mandatory 'BEGIN' directive, followed by the query sequence +in FASTA/Pearson format. Each line of information must be less than 80 +characters in length. + + The "month" database contains all new or revised sequences released +in the last 30 days and is useful for searching against new sequences. +There are five different blast programs, 'blastn' being the one that +compares a nucleotide query sequence against a nucleotide sequence +database. + + The last server directive that must appear in every request is the +'BEGIN' directive. The query sequence should immediately follow the +'BEGIN' directive and must appear in FASTA/Pearson format. A sequence +in FASTA/Pearson format begins with a single-line description. The +description line, which is required, is distinguished from the lines of +sequence data that follow it by having a greater-than ('>') symbol in +the first column. For the purposes of the BLAST server, the text of the +description is arbitrary. + + If you prefer to use a client written in 'gawk', just store the +following 10 lines of code into a file named 'protbase.awk' and use this +client instead. Invoke it with 'gawk -f protbase.awk protbase.request'. +Then wait a minute and watch the result coming in. In order to +replicate the demonstration client's behavior as closely as possible, +this client does not use a proxy server. We could also have extended +the client program in *note Retrieving Web Pages: GETURL, to implement +the client request from 'protbase.awk' as a special case. + + { request = request "\n" $0 } + + END { + BLASTService = "/inet/tcp/0/www.ncbi.nlm.nih.gov/80" + printf "POST /cgi-bin/BLAST/nph-blast_report HTTP/1.0\n" |& BLASTService + printf "Content-Length: " length(request) "\n\n" |& BLASTService + printf request |& BLASTService + while ((BLASTService |& getline) > 0) + print $0 + close(BLASTService) + } + + The demonstration client from NCBI is 214 lines long (written in C) +and it is not immediately obvious what it does. Our client is so short +that it _is_ obvious what it does. First it loops over all lines of the +query and stores the whole query into a variable. Then the script +establishes an Internet connection to the NCBI server and transmits the +query by framing it with a proper HTTP request. Finally it receives and +prints the complete result coming from the server. + + Now, let us look at the result. It begins with an HTTP header, which +you can ignore. Then there are some comments about the query having +been filtered to avoid spuriously high scores. After this, there is a +reference to the paper that describes the software being used for +searching the data base. After a repetition of the original query's +description we find the list of significant alignments: + + Sequences producing significant alignments: (bits) Value + + gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733... 38 0.20 + gb|AC021056.12|AC021056 Homo sapiens chromosome 3 clone RP11-115... 38 0.20 + emb|AL160278.10|AL160278 Homo sapiens chromosome 9 clone RP11-57... 38 0.20 + emb|AL391139.11|AL391139 Homo sapiens chromosome X clone RP11-35... 38 0.20 + emb|AL365192.6|AL365192 Homo sapiens chromosome 6 clone RP3-421H... 38 0.20 + emb|AL138812.9|AL138812 Homo sapiens chromosome 11 clone RP1-276... 38 0.20 + gb|AC073881.3|AC073881 Homo sapiens chromosome 15 clone CTD-2169... 38 0.20 + + This means that the query sequence was found in seven human +chromosomes. But the value 0.20 (20%) means that the probability of an +accidental match is rather high (20%) in all cases and should be taken +into account. You may wonder what the first column means. It is a key +to the specific database in which this occurrence was found. The unique +sequence identifiers reported in the search results can be used as +sequence retrieval keys via the NCBI server. The syntax of sequence +header lines used by the NCBI BLAST server depends on the database from +which each sequence was obtained. The table below lists the identifiers +for the databases from which the sequences were derived. + + Database Name Identifier Syntax + ============================ ======================== + GenBank gb|accession|locus + EMBL Data Library emb|accession|locus + DDBJ, DNA Database of Japan dbj|accession|locus + NBRF PIR pir||entry + Protein Research Foundation prf||name + SWISS-PROT sp|accession|entry name + Brookhaven Protein Data Bank pdb|entry|chain + Kabat's Sequences of Immuno... gnl|kabat|identifier + Patents pat|country|number + GenInfo Backbone Id bbs|number + + For example, an identifier might be 'gb|AC021182.14|AC021182', where +the 'gb' tag indicates that the identifier refers to a GenBank sequence, +'AC021182.14' is its GenBank ACCESSION, and 'AC021182' is the GenBank +LOCUS. The identifier contains no spaces, so that a space indicates the +end of the identifier. + + Let us continue in the result listing. Each of the seven alignments +mentioned above is subsequently described in detail. We will have a +closer look at the first of them. + + >gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733N23, WORKING DRAFT SEQUENCE, 4 + unordered pieces + Length = 176383 + + Score = 38.2 bits (19), Expect = 0.20 + Identities = 19/19 (100%) + Strand = Plus / Plus + + Query: 35 tggtgaagtgtgtttcttg 53 + ||||||||||||||||||| + Sbjct: 69786 tggtgaagtgtgtttcttg 69804 + + This alignment was located on the human chromosome 7. The fragment +on which part of the query was found had a total length of 176383. Only +19 of the nucleotides matched and the matching sequence ran from +character 35 to 53 in the query sequence and from 69786 to 69804 in the +fragment on chromosome 7. If you are still reading at this point, you +are probably interested in finding out more about Computational Biology +and you might appreciate the following hints. + + 1. There is a book called 'Introduction to Computational Biology' by + Michael S. Waterman, which is worth reading if you are seriously + interested. You can find a good book review on the Internet. + + 2. While Waterman's book can explain to you the algorithms employed + internally in the database search engines, most practitioners + prefer to approach the subject differently. The applied side of + Computational Biology is called Bioinformatics, and emphasizes the + tools available for day-to-day work as well as how to actually + _use_ them. One of the very few affordable books on Bioinformatics + is 'Developing Bioinformatics Computer Skills'. + + 3. The sequences _gawk_ and _gnuawk_ are in widespread use in the + genetic material of virtually every earthly living being. Let us + take this as a clear indication that the divine creator has + intended 'gawk' to prevail over other scripting languages such as + 'perl', 'tcl', or 'python' which are not even proper sequences. + (:-) + + +File: gawkinet.info, Node: Links, Next: GNU Free Documentation License, Prev: Some Applications and Techniques, Up: Top + +4 Related Links +*************** + +This section lists the URLs for various items discussed in this major +node. They are presented in the order in which they appear. + +'Internet Programming with Python' + <http://www.fsbassociates.com/books/python.htm> + +'Advanced Perl Programming' + <http://www.oreilly.com/catalog/advperl> + +'Web Client Programming with Perl' + <http://www.oreilly.com/catalog/webclient> + +Richard Stevens's home page and book + <http://www.kohala.com/~rstevens> + +The SPAK home page + <http://www.userfriendly.net/linux/RPM/contrib/libc6/i386/spak-0.6b-1.i386.html> + +Volume III of 'Internetworking with TCP/IP', by Comer and Stevens + <http://www.cs.purdue.edu/homes/dec/tcpip3s.cont.html> + +XBM Graphics File Format + <http://www.wotsit.org/download.asp?f=xbm> + +GNUPlot + <http://www.cs.dartmouth.edu/gnuplot_info.html> + +Mark Humphrys' Eliza page + <http://www.compapp.dcu.ie/~humphrys/eliza.html> + +Yahoo! Eliza Information + <http://dir.yahoo.com/Recreation/Games/Computer_Games/Internet_Games/Web_Games/Artificial_Intelligence> + +Java versions of Eliza + <http://www.tjhsst.edu/Psych/ch1/eliza.html> + +Java versions of Eliza with source code + <http://home.adelphia.net/~lifeisgood/eliza/eliza.htm> + +Eliza Programs with Explanations + <http://chayden.net/chayden/eliza/Eliza.shtml> + +Loebner Contest + <http://acm.org/~loebner/loebner-prize.htmlx> + +Tck/Tk Information + <http://www.scriptics.com/> + +Intel 80x86 Processors + <http://developer.intel.com/design/platform/embedpc/what_is.htm> + +AMD Elan Processors + <http://www.amd.com/products/epd/processors/4.32bitcont/32bitcont/index.html> + +XINU + <http://willow.canberra.edu.au/~chrisc/xinu.html> + +GNU/Linux + <http://uclinux.lineo.com/> + +Embedded PCs + <http://dir.yahoo.com/Business_and_Economy/Business_to_Business/Computers/Hardware/Embedded_Control/> + +MiniSQL + <http://www.hughes.com.au/library/> + +Market Share Surveys + <http://www.netcraft.com/survey> + +'Numerical Recipes in C: The Art of Scientific Computing' + <http://www.nr.com> + +VRML + <http://www.vrml.org> + +The VRML FAQ + <http://www.vrml.org/technicalinfo/specifications/specifications.htm#FAQ> + +The UMBC Agent Web + <http://www.cs.umbc.edu/agents> + +Apache Web Server + <http://www.apache.org> + +National Center for Biotechnology Information (NCBI) + <http://www.ncbi.nlm.nih.gov> + +Basic Local Alignment Search Tool (BLAST) + <http://www.ncbi.nlm.nih.gov/BLAST/blast_overview.html> + +NCBI Home Page + <http://www.ncbi.nlm.nih.gov> + +BLAST Pages + <http://www.ncbi.nlm.nih.gov/BLAST> + +BLAST Demonstration Client + <ftp://ncbi.nlm.nih.gov/blast/blasturl/> + +BLAST anonymous FTP location + <ftp://ncbi.nlm.nih.gov/blast/network/netblast/> + +BLAST 2.0 Executables + <ftp://ncbi.nlm.nih.gov/blast/executables/> + +IUB/IUPAC Amino Acid and Nucleic Acid Codes + <http://www.uthscsa.edu/geninfo/blastmail.html#item6> + +FASTA/Pearson Format + <http://www.ncbi.nlm.nih.gov/BLAST/fasta.html> + +Fasta/Pearson Sequence in Java + <http://www.kazusa.or.jp/java/codon_table_java/> + +Book Review of 'Introduction to Computational Biology' + <http://www.acm.org/crossroads/xrds5-1/introcb.html> + +'Developing Bioinformatics Computer Skills' + <http://www.oreilly.com/catalog/bioskills/> + + +File: gawkinet.info, Node: GNU Free Documentation License, Next: Index, Prev: Links, Up: Top + +GNU Free Documentation License +****************************** + + Version 1.3, 3 November 2008 + + Copyright (C) 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc. + <http://fsf.org/> + + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + 0. PREAMBLE + + The purpose of this License is to make a manual, textbook, or other + functional and useful document "free" in the sense of freedom: to + assure everyone the effective freedom to copy and redistribute it, + with or without modifying it, either commercially or + noncommercially. Secondarily, this License preserves for the + author and publisher a way to get credit for their work, while not + being considered responsible for modifications made by others. + + This License is a kind of "copyleft", which means that derivative + works of the document must themselves be free in the same sense. + It complements the GNU General Public License, which is a copyleft + license designed for free software. + + We have designed this License in order to use it for manuals for + free software, because free software needs free documentation: a + free program should come with manuals providing the same freedoms + that the software does. But this License is not limited to + software manuals; it can be used for any textual work, regardless + of subject matter or whether it is published as a printed book. We + recommend this License principally for works whose purpose is + instruction or reference. + + 1. APPLICABILITY AND DEFINITIONS + + This License applies to any manual or other work, in any medium, + that contains a notice placed by the copyright holder saying it can + be distributed under the terms of this License. Such a notice + grants a world-wide, royalty-free license, unlimited in duration, + to use that work under the conditions stated herein. The + "Document", below, refers to any such manual or work. Any member + of the public is a licensee, and is addressed as "you". You accept + the license if you copy, modify or distribute the work in a way + requiring permission under copyright law. + + A "Modified Version" of the Document means any work containing the + Document or a portion of it, either copied verbatim, or with + modifications and/or translated into another language. + + A "Secondary Section" is a named appendix or a front-matter section + of the Document that deals exclusively with the relationship of the + publishers or authors of the Document to the Document's overall + subject (or to related matters) and contains nothing that could + fall directly within that overall subject. (Thus, if the Document + is in part a textbook of mathematics, a Secondary Section may not + explain any mathematics.) The relationship could be a matter of + historical connection with the subject or with related matters, or + of legal, commercial, philosophical, ethical or political position + regarding them. + + The "Invariant Sections" are certain Secondary Sections whose + titles are designated, as being those of Invariant Sections, in the + notice that says that the Document is released under this License. + If a section does not fit the above definition of Secondary then it + is not allowed to be designated as Invariant. The Document may + contain zero Invariant Sections. If the Document does not identify + any Invariant Sections then there are none. + + The "Cover Texts" are certain short passages of text that are + listed, as Front-Cover Texts or Back-Cover Texts, in the notice + that says that the Document is released under this License. A + Front-Cover Text may be at most 5 words, and a Back-Cover Text may + be at most 25 words. + + A "Transparent" copy of the Document means a machine-readable copy, + represented in a format whose specification is available to the + general public, that is suitable for revising the document + straightforwardly with generic text editors or (for images composed + of pixels) generic paint programs or (for drawings) some widely + available drawing editor, and that is suitable for input to text + formatters or for automatic translation to a variety of formats + suitable for input to text formatters. A copy made in an otherwise + Transparent file format whose markup, or absence of markup, has + been arranged to thwart or discourage subsequent modification by + readers is not Transparent. An image format is not Transparent if + used for any substantial amount of text. A copy that is not + "Transparent" is called "Opaque". + + Examples of suitable formats for Transparent copies include plain + ASCII without markup, Texinfo input format, LaTeX input format, + SGML or XML using a publicly available DTD, and standard-conforming + simple HTML, PostScript or PDF designed for human modification. + Examples of transparent image formats include PNG, XCF and JPG. + Opaque formats include proprietary formats that can be read and + edited only by proprietary word processors, SGML or XML for which + the DTD and/or processing tools are not generally available, and + the machine-generated HTML, PostScript or PDF produced by some word + processors for output purposes only. + + The "Title Page" means, for a printed book, the title page itself, + plus such following pages as are needed to hold, legibly, the + material this License requires to appear in the title page. For + works in formats which do not have any title page as such, "Title + Page" means the text near the most prominent appearance of the + work's title, preceding the beginning of the body of the text. + + The "publisher" means any person or entity that distributes copies + of the Document to the public. + + A section "Entitled XYZ" means a named subunit of the Document + whose title either is precisely XYZ or contains XYZ in parentheses + following text that translates XYZ in another language. (Here XYZ + stands for a specific section name mentioned below, such as + "Acknowledgements", "Dedications", "Endorsements", or "History".) + To "Preserve the Title" of such a section when you modify the + Document means that it remains a section "Entitled XYZ" according + to this definition. + + The Document may include Warranty Disclaimers next to the notice + which states that this License applies to the Document. These + Warranty Disclaimers are considered to be included by reference in + this License, but only as regards disclaiming warranties: any other + implication that these Warranty Disclaimers may have is void and + has no effect on the meaning of this License. + + 2. VERBATIM COPYING + + You may copy and distribute the Document in any medium, either + commercially or noncommercially, provided that this License, the + copyright notices, and the license notice saying this License + applies to the Document are reproduced in all copies, and that you + add no other conditions whatsoever to those of this License. You + may not use technical measures to obstruct or control the reading + or further copying of the copies you make or distribute. However, + you may accept compensation in exchange for copies. If you + distribute a large enough number of copies you must also follow the + conditions in section 3. + + You may also lend copies, under the same conditions stated above, + and you may publicly display copies. + + 3. COPYING IN QUANTITY + + If you publish printed copies (or copies in media that commonly + have printed covers) of the Document, numbering more than 100, and + the Document's license notice requires Cover Texts, you must + enclose the copies in covers that carry, clearly and legibly, all + these Cover Texts: Front-Cover Texts on the front cover, and + Back-Cover Texts on the back cover. Both covers must also clearly + and legibly identify you as the publisher of these copies. The + front cover must present the full title with all words of the title + equally prominent and visible. You may add other material on the + covers in addition. Copying with changes limited to the covers, as + long as they preserve the title of the Document and satisfy these + conditions, can be treated as verbatim copying in other respects. + + If the required texts for either cover are too voluminous to fit + legibly, you should put the first ones listed (as many as fit + reasonably) on the actual cover, and continue the rest onto + adjacent pages. + + If you publish or distribute Opaque copies of the Document + numbering more than 100, you must either include a machine-readable + Transparent copy along with each Opaque copy, or state in or with + each Opaque copy a computer-network location from which the general + network-using public has access to download using public-standard + network protocols a complete Transparent copy of the Document, free + of added material. If you use the latter option, you must take + reasonably prudent steps, when you begin distribution of Opaque + copies in quantity, to ensure that this Transparent copy will + remain thus accessible at the stated location until at least one + year after the last time you distribute an Opaque copy (directly or + through your agents or retailers) of that edition to the public. + + It is requested, but not required, that you contact the authors of + the Document well before redistributing any large number of copies, + to give them a chance to provide you with an updated version of the + Document. + + 4. MODIFICATIONS + + You may copy and distribute a Modified Version of the Document + under the conditions of sections 2 and 3 above, provided that you + release the Modified Version under precisely this License, with the + Modified Version filling the role of the Document, thus licensing + distribution and modification of the Modified Version to whoever + possesses a copy of it. In addition, you must do these things in + the Modified Version: + + A. Use in the Title Page (and on the covers, if any) a title + distinct from that of the Document, and from those of previous + versions (which should, if there were any, be listed in the + History section of the Document). You may use the same title + as a previous version if the original publisher of that + version gives permission. + + B. List on the Title Page, as authors, one or more persons or + entities responsible for authorship of the modifications in + the Modified Version, together with at least five of the + principal authors of the Document (all of its principal + authors, if it has fewer than five), unless they release you + from this requirement. + + C. State on the Title page the name of the publisher of the + Modified Version, as the publisher. + + D. Preserve all the copyright notices of the Document. + + E. Add an appropriate copyright notice for your modifications + adjacent to the other copyright notices. + + F. Include, immediately after the copyright notices, a license + notice giving the public permission to use the Modified + Version under the terms of this License, in the form shown in + the Addendum below. + + G. Preserve in that license notice the full lists of Invariant + Sections and required Cover Texts given in the Document's + license notice. + + H. Include an unaltered copy of this License. + + I. Preserve the section Entitled "History", Preserve its Title, + and add to it an item stating at least the title, year, new + authors, and publisher of the Modified Version as given on the + Title Page. If there is no section Entitled "History" in the + Document, create one stating the title, year, authors, and + publisher of the Document as given on its Title Page, then add + an item describing the Modified Version as stated in the + previous sentence. + + J. Preserve the network location, if any, given in the Document + for public access to a Transparent copy of the Document, and + likewise the network locations given in the Document for + previous versions it was based on. These may be placed in the + "History" section. You may omit a network location for a work + that was published at least four years before the Document + itself, or if the original publisher of the version it refers + to gives permission. + + K. For any section Entitled "Acknowledgements" or "Dedications", + Preserve the Title of the section, and preserve in the section + all the substance and tone of each of the contributor + acknowledgements and/or dedications given therein. + + L. Preserve all the Invariant Sections of the Document, unaltered + in their text and in their titles. Section numbers or the + equivalent are not considered part of the section titles. + + M. Delete any section Entitled "Endorsements". Such a section + may not be included in the Modified Version. + + N. Do not retitle any existing section to be Entitled + "Endorsements" or to conflict in title with any Invariant + Section. + + O. Preserve any Warranty Disclaimers. + + If the Modified Version includes new front-matter sections or + appendices that qualify as Secondary Sections and contain no + material copied from the Document, you may at your option designate + some or all of these sections as invariant. To do this, add their + titles to the list of Invariant Sections in the Modified Version's + license notice. These titles must be distinct from any other + section titles. + + You may add a section Entitled "Endorsements", provided it contains + nothing but endorsements of your Modified Version by various + parties--for example, statements of peer review or that the text + has been approved by an organization as the authoritative + definition of a standard. + + You may add a passage of up to five words as a Front-Cover Text, + and a passage of up to 25 words as a Back-Cover Text, to the end of + the list of Cover Texts in the Modified Version. Only one passage + of Front-Cover Text and one of Back-Cover Text may be added by (or + through arrangements made by) any one entity. If the Document + already includes a cover text for the same cover, previously added + by you or by arrangement made by the same entity you are acting on + behalf of, you may not add another; but you may replace the old + one, on explicit permission from the previous publisher that added + the old one. + + The author(s) and publisher(s) of the Document do not by this + License give permission to use their names for publicity for or to + assert or imply endorsement of any Modified Version. + + 5. COMBINING DOCUMENTS + + You may combine the Document with other documents released under + this License, under the terms defined in section 4 above for + modified versions, provided that you include in the combination all + of the Invariant Sections of all of the original documents, + unmodified, and list them all as Invariant Sections of your + combined work in its license notice, and that you preserve all + their Warranty Disclaimers. + + The combined work need only contain one copy of this License, and + multiple identical Invariant Sections may be replaced with a single + copy. If there are multiple Invariant Sections with the same name + but different contents, make the title of each such section unique + by adding at the end of it, in parentheses, the name of the + original author or publisher of that section if known, or else a + unique number. Make the same adjustment to the section titles in + the list of Invariant Sections in the license notice of the + combined work. + + In the combination, you must combine any sections Entitled + "History" in the various original documents, forming one section + Entitled "History"; likewise combine any sections Entitled + "Acknowledgements", and any sections Entitled "Dedications". You + must delete all sections Entitled "Endorsements." + + 6. COLLECTIONS OF DOCUMENTS + + You may make a collection consisting of the Document and other + documents released under this License, and replace the individual + copies of this License in the various documents with a single copy + that is included in the collection, provided that you follow the + rules of this License for verbatim copying of each of the documents + in all other respects. + + You may extract a single document from such a collection, and + distribute it individually under this License, provided you insert + a copy of this License into the extracted document, and follow this + License in all other respects regarding verbatim copying of that + document. + + 7. AGGREGATION WITH INDEPENDENT WORKS + + A compilation of the Document or its derivatives with other + separate and independent documents or works, in or on a volume of a + storage or distribution medium, is called an "aggregate" if the + copyright resulting from the compilation is not used to limit the + legal rights of the compilation's users beyond what the individual + works permit. When the Document is included in an aggregate, this + License does not apply to the other works in the aggregate which + are not themselves derivative works of the Document. + + If the Cover Text requirement of section 3 is applicable to these + copies of the Document, then if the Document is less than one half + of the entire aggregate, the Document's Cover Texts may be placed + on covers that bracket the Document within the aggregate, or the + electronic equivalent of covers if the Document is in electronic + form. Otherwise they must appear on printed covers that bracket + the whole aggregate. + + 8. TRANSLATION + + Translation is considered a kind of modification, so you may + distribute translations of the Document under the terms of section + 4. Replacing Invariant Sections with translations requires special + permission from their copyright holders, but you may include + translations of some or all Invariant Sections in addition to the + original versions of these Invariant Sections. You may include a + translation of this License, and all the license notices in the + Document, and any Warranty Disclaimers, provided that you also + include the original English version of this License and the + original versions of those notices and disclaimers. In case of a + disagreement between the translation and the original version of + this License or a notice or disclaimer, the original version will + prevail. + + If a section in the Document is Entitled "Acknowledgements", + "Dedications", or "History", the requirement (section 4) to + Preserve its Title (section 1) will typically require changing the + actual title. + + 9. TERMINATION + + You may not copy, modify, sublicense, or distribute the Document + except as expressly provided under this License. Any attempt + otherwise to copy, modify, sublicense, or distribute it is void, + and will automatically terminate your rights under this License. + + However, if you cease all violation of this License, then your + license from a particular copyright holder is reinstated (a) + provisionally, unless and until the copyright holder explicitly and + finally terminates your license, and (b) permanently, if the + copyright holder fails to notify you of the violation by some + reasonable means prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is + reinstated permanently if the copyright holder notifies you of the + violation by some reasonable means, this is the first time you have + received notice of violation of this License (for any work) from + that copyright holder, and you cure the violation prior to 30 days + after your receipt of the notice. + + Termination of your rights under this section does not terminate + the licenses of parties who have received copies or rights from you + under this License. If your rights have been terminated and not + permanently reinstated, receipt of a copy of some or all of the + same material does not give you any rights to use it. + + 10. FUTURE REVISIONS OF THIS LICENSE + + The Free Software Foundation may publish new, revised versions of + the GNU Free Documentation License from time to time. Such new + versions will be similar in spirit to the present version, but may + differ in detail to address new problems or concerns. See + <http://www.gnu.org/copyleft/>. + + Each version of the License is given a distinguishing version + number. If the Document specifies that a particular numbered + version of this License "or any later version" applies to it, you + have the option of following the terms and conditions either of + that specified version or of any later version that has been + published (not as a draft) by the Free Software Foundation. If the + Document does not specify a version number of this License, you may + choose any version ever published (not as a draft) by the Free + Software Foundation. If the Document specifies that a proxy can + decide which future versions of this License can be used, that + proxy's public statement of acceptance of a version permanently + authorizes you to choose that version for the Document. + + 11. RELICENSING + + "Massive Multiauthor Collaboration Site" (or "MMC Site") means any + World Wide Web server that publishes copyrightable works and also + provides prominent facilities for anybody to edit those works. A + public wiki that anybody can edit is an example of such a server. + A "Massive Multiauthor Collaboration" (or "MMC") contained in the + site means any set of copyrightable works thus published on the MMC + site. + + "CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0 + license published by Creative Commons Corporation, a not-for-profit + corporation with a principal place of business in San Francisco, + California, as well as future copyleft versions of that license + published by that same organization. + + "Incorporate" means to publish or republish a Document, in whole or + in part, as part of another Document. + + An MMC is "eligible for relicensing" if it is licensed under this + License, and if all works that were first published under this + License somewhere other than this MMC, and subsequently + incorporated in whole or in part into the MMC, (1) had no cover + texts or invariant sections, and (2) were thus incorporated prior + to November 1, 2008. + + The operator of an MMC Site may republish an MMC contained in the + site under CC-BY-SA on the same site at any time before August 1, + 2009, provided the MMC is eligible for relicensing. + +ADDENDUM: How to use this License for your documents +==================================================== + +To use this License in a document you have written, include a copy of +the License in the document and put the following copyright and license +notices just after the title page: + + Copyright (C) YEAR YOUR NAME. + Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.3 + or any later version published by the Free Software Foundation; + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + Texts. A copy of the license is included in the section entitled ``GNU + Free Documentation License''. + + If you have Invariant Sections, Front-Cover Texts and Back-Cover +Texts, replace the "with...Texts." line with this: + + with the Invariant Sections being LIST THEIR TITLES, with + the Front-Cover Texts being LIST, and with the Back-Cover Texts + being LIST. + + If you have Invariant Sections without Cover Texts, or some other +combination of the three, merge those two alternatives to suit the +situation. + + If your document contains nontrivial examples of program code, we +recommend releasing these examples in parallel under your choice of free +software license, such as the GNU General Public License, to permit +their use in free software. + + +File: gawkinet.info, Node: Index, Prev: GNU Free Documentation License, Up: Top + +Index +***** + + +* Menu: + +* /inet/ files (gawk): Gawk Special Files. (line 34) +* /inet/tcp special files (gawk): File /inet/tcp. (line 6) +* /inet/udp special files (gawk): File /inet/udp. (line 6) +* | (vertical bar), |& operator (I/O): TCP Connecting. (line 25) +* advanced features, network connections: Troubleshooting. (line 6) +* agent: Challenges. (line 75) +* agent <1>: MOBAGWHO. (line 6) +* AI: Challenges. (line 75) +* apache: WEBGRAB. (line 72) +* apache <1>: MOBAGWHO. (line 42) +* Bioinformatics: PROTBASE. (line 227) +* BLAST, Basic Local Alignment Search Tool: PROTBASE. (line 6) +* blocking: Making Connections. (line 35) +* Boutell, Thomas: STATIST. (line 6) +* CGI (Common Gateway Interface): MOBAGWHO. (line 42) +* CGI (Common Gateway Interface), dynamic web pages and: Web page. + (line 45) +* CGI (Common Gateway Interface), library: CGI Lib. (line 11) +* clients: Making Connections. (line 21) +* Clinton, Bill: Challenges. (line 58) +* Common Gateway Interface, See CGI: Web page. (line 45) +* Computational Biology: PROTBASE. (line 227) +* contest: Challenges. (line 6) +* cron utility: STOXPRED. (line 23) +* CSV format: STOXPRED. (line 128) +* Dow Jones Industrial Index: STOXPRED. (line 44) +* ELIZA program: Simple Server. (line 11) +* ELIZA program <1>: Simple Server. (line 178) +* email: Email. (line 11) +* FASTA/Pearson format: PROTBASE. (line 102) +* FDL (Free Documentation License): GNU Free Documentation License. + (line 6) +* filenames, for network access: Gawk Special Files. (line 29) +* files, /inet/ (gawk): Gawk Special Files. (line 34) +* files, /inet/tcp (gawk): File /inet/tcp. (line 6) +* files, /inet/udp (gawk): File /inet/udp. (line 6) +* finger utility: Setting Up. (line 22) +* Free Documentation License (FDL): GNU Free Documentation License. + (line 6) +* FTP (File Transfer Protocol): Basic Protocols. (line 45) +* gawk, networking: Using Networking. (line 6) +* gawk, networking, connections: Special File Fields. (line 53) +* gawk, networking, connections <1>: TCP Connecting. (line 6) +* gawk, networking, filenames: Gawk Special Files. (line 29) +* gawk, networking, See Also email: Email. (line 6) +* gawk, networking, service, establishing: Setting Up. (line 6) +* gawk, networking, troubleshooting: Caveats. (line 6) +* gawk, web and, See web service: Interacting Service. (line 6) +* getline command: TCP Connecting. (line 11) +* GETURL program: GETURL. (line 6) +* GIF image format: Web page. (line 45) +* GIF image format <1>: STATIST. (line 6) +* GNU Free Documentation License: GNU Free Documentation License. + (line 6) +* GNU/Linux: Troubleshooting. (line 54) +* GNU/Linux <1>: Interacting. (line 27) +* GNU/Linux <2>: REMCONF. (line 6) +* GNUPlot utility: Interacting Service. (line 189) +* GNUPlot utility <1>: STATIST. (line 6) +* Hoare, C.A.R.: MOBAGWHO. (line 6) +* Hoare, C.A.R. <1>: PROTBASE. (line 6) +* hostname field: Special File Fields. (line 34) +* HTML (Hypertext Markup Language): Web page. (line 29) +* HTTP (Hypertext Transfer Protocol): Basic Protocols. (line 45) +* HTTP (Hypertext Transfer Protocol) <1>: Web page. (line 6) +* HTTP (Hypertext Transfer Protocol), record separators and: Web page. + (line 29) +* HTTP server, core logic: Interacting Service. (line 6) +* HTTP server, core logic <1>: Interacting Service. (line 24) +* Humphrys, Mark: Simple Server. (line 178) +* Hypertext Markup Language (HTML): Web page. (line 29) +* Hypertext Transfer Protocol, See HTTP: Web page. (line 6) +* image format: STATIST. (line 6) +* images, in web pages: Interacting Service. (line 189) +* images, retrieving over networks: Web page. (line 45) +* input/output, two-way, See Also gawk, networking: Gawk Special Files. + (line 19) +* Internet, See networks: Interacting. (line 48) +* JavaScript: STATIST. (line 56) +* Linux: Troubleshooting. (line 54) +* Linux <1>: Interacting. (line 27) +* Linux <2>: REMCONF. (line 6) +* Lisp: MOBAGWHO. (line 98) +* localport field: Gawk Special Files. (line 34) +* Loebner, Hugh: Challenges. (line 6) +* Loui, Ronald: Challenges. (line 75) +* MAZE: MAZE. (line 6) +* Microsoft Windows: WEBGRAB. (line 43) +* Microsoft Windows, networking: Troubleshooting. (line 54) +* Microsoft Windows, networking, ports: Setting Up. (line 37) +* MiniSQL: REMCONF. (line 109) +* MOBAGWHO program: MOBAGWHO. (line 6) +* NCBI, National Center for Biotechnology Information: PROTBASE. + (line 6) +* network type field: Special File Fields. (line 11) +* networks, gawk and: Using Networking. (line 6) +* networks, gawk and, connections: Special File Fields. (line 53) +* networks, gawk and, connections <1>: TCP Connecting. (line 6) +* networks, gawk and, filenames: Gawk Special Files. (line 29) +* networks, gawk and, See Also email: Email. (line 6) +* networks, gawk and, service, establishing: Setting Up. (line 6) +* networks, gawk and, troubleshooting: Caveats. (line 6) +* networks, ports, reserved: Setting Up. (line 37) +* networks, ports, specifying: Special File Fields. (line 24) +* networks, See Also web pages: PANIC. (line 6) +* Numerical Recipes: STATIST. (line 24) +* ORS variable, HTTP and: Web page. (line 29) +* ORS variable, POP and: Email. (line 36) +* PANIC program: PANIC. (line 6) +* Perl: Using Networking. (line 14) +* Perl, gawk networking and: Using Networking. (line 24) +* Perlis, Alan: MAZE. (line 6) +* pipes, networking and: TCP Connecting. (line 30) +* PNG image format: Web page. (line 45) +* PNG image format <1>: STATIST. (line 6) +* POP (Post Office Protocol): Email. (line 6) +* POP (Post Office Protocol) <1>: Email. (line 36) +* Post Office Protocol (POP): Email. (line 6) +* PostScript: STATIST. (line 138) +* PROLOG: Challenges. (line 75) +* PROTBASE: PROTBASE. (line 6) +* protocol field: Special File Fields. (line 17) +* PS image format: STATIST. (line 6) +* Python: Using Networking. (line 14) +* Python, gawk networking and: Using Networking. (line 24) +* record separators, HTTP and: Web page. (line 29) +* record separators, POP and: Email. (line 36) +* REMCONF program: REMCONF. (line 6) +* remoteport field: Gawk Special Files. (line 34) +* RFC 1939: Email. (line 6) +* RFC 1939 <1>: Email. (line 36) +* RFC 1945: Web page. (line 29) +* RFC 2068: Web page. (line 6) +* RFC 2068 <1>: Interacting Service. (line 104) +* RFC 2616: Web page. (line 6) +* RFC 821: Email. (line 6) +* robot: Challenges. (line 84) +* robot <1>: WEBGRAB. (line 6) +* RS variable, HTTP and: Web page. (line 29) +* RS variable, POP and: Email. (line 36) +* servers: Making Connections. (line 14) +* servers <1>: Setting Up. (line 22) +* servers, as hosts: Special File Fields. (line 34) +* servers, HTTP: Interacting Service. (line 6) +* servers, web: Simple Server. (line 6) +* Simple Mail Transfer Protocol (SMTP): Email. (line 6) +* SMTP (Simple Mail Transfer Protocol): Basic Protocols. (line 45) +* SMTP (Simple Mail Transfer Protocol) <1>: Email. (line 6) +* STATIST program: STATIST. (line 6) +* STOXPRED program: STOXPRED. (line 6) +* synchronous communications: Making Connections. (line 35) +* Tcl/Tk: Using Networking. (line 14) +* Tcl/Tk, gawk and: Using Networking. (line 24) +* Tcl/Tk, gawk and <1>: Some Applications and Techniques. + (line 22) +* TCP (Transmission Control Protocol): Using Networking. (line 29) +* TCP (Transmission Control Protocol) <1>: File /inet/tcp. (line 6) +* TCP (Transmission Control Protocol), connection, establishing: TCP Connecting. + (line 6) +* TCP (Transmission Control Protocol), UDP and: Interacting. (line 48) +* TCP/IP, network type, selecting: Special File Fields. (line 11) +* TCP/IP, protocols, selecting: Special File Fields. (line 17) +* TCP/IP, sockets and: Gawk Special Files. (line 19) +* Transmission Control Protocol, See TCP: Using Networking. (line 29) +* troubleshooting, gawk, networks: Caveats. (line 6) +* troubleshooting, networks, connections: Troubleshooting. (line 6) +* troubleshooting, networks, timeouts: Caveats. (line 18) +* UDP (User Datagram Protocol): File /inet/udp. (line 6) +* UDP (User Datagram Protocol), TCP and: Interacting. (line 48) +* Unix, network ports and: Setting Up. (line 37) +* URLCHK program: URLCHK. (line 6) +* User Datagram Protocol, See UDP: File /inet/udp. (line 6) +* vertical bar (|), |& operator (I/O): TCP Connecting. (line 25) +* VRML: MAZE. (line 6) +* web browsers, See web service: Interacting Service. (line 6) +* web pages: Web page. (line 6) +* web pages, images in: Interacting Service. (line 189) +* web pages, retrieving: GETURL. (line 6) +* web servers: Simple Server. (line 6) +* web service: Primitive Service. (line 6) +* web service <1>: PANIC. (line 6) +* WEBGRAB program: WEBGRAB. (line 6) +* Weizenbaum, Joseph: Simple Server. (line 11) +* XBM image format: Interacting Service. (line 189) +* Yahoo!: REMCONF. (line 6) +* Yahoo! <1>: STOXPRED. (line 6) + + + +Tag Table: +Node: Top2022 +Node: Preface5665 +Node: Introduction7040 +Node: Stream Communications8066 +Node: Datagram Communications9240 +Node: The TCP/IP Protocols10870 +Ref: The TCP/IP Protocols-Footnote-111554 +Node: Basic Protocols11711 +Ref: Basic Protocols-Footnote-113756 +Node: Ports13785 +Node: Making Connections15192 +Ref: Making Connections-Footnote-117750 +Ref: Making Connections-Footnote-217797 +Node: Using Networking17978 +Node: Gawk Special Files20301 +Node: Special File Fields22110 +Ref: table-inet-components26003 +Node: Comparing Protocols27312 +Node: File /inet/tcp27846 +Node: File /inet/udp28874 +Ref: File /inet/udp-Footnote-130573 +Node: TCP Connecting30827 +Node: Troubleshooting33173 +Ref: Troubleshooting-Footnote-136232 +Node: Interacting36805 +Node: Setting Up39545 +Node: Email43048 +Node: Web page45380 +Ref: Web page-Footnote-148197 +Node: Primitive Service48395 +Node: Interacting Service51136 +Ref: Interacting Service-Footnote-160303 +Node: CGI Lib60335 +Node: Simple Server67310 +Ref: Simple Server-Footnote-175053 +Node: Caveats75154 +Node: Challenges76299 +Node: Some Applications and Techniques84997 +Node: PANIC87462 +Node: GETURL89186 +Node: REMCONF91819 +Node: URLCHK97314 +Node: WEBGRAB101166 +Node: STATIST105628 +Ref: STATIST-Footnote-1117377 +Node: MAZE117822 +Node: MOBAGWHO124029 +Ref: MOBAGWHO-Footnote-1138047 +Node: STOXPRED138102 +Node: PROTBASE152390 +Node: Links165506 +Node: GNU Free Documentation License168939 +Node: Index194059 + +End Tag Table |