path: root/gawk-info-1

Info file gawk-info, produced by Makeinfo, -*- Text -*- from input
file gawk.texinfo.

This file documents `awk', a program that you can use to select
particular records in a file and perform operations upon them.

Copyright (C) 1989 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.



File: gawk-info,  Node: Top,  Next: Preface,  Prev: (dir),  Up: (dir)

This file documents `awk', a program that you can use to select
particular records in a file and perform operations upon them; it
contains the following chapters:

* Menu:

* Preface::	       What you can do with `awk'; brief history
		       and acknowledgements.

* License::	       Your right to copy and distribute `gawk'.

* This Manual::	       Using this manual.

		       Includes sample input files that you can use.

* Getting Started::    A basic introduction to using `awk'.
                       How to run an `awk' program.  Command line syntax.

* Reading Files::      How to read files and manipulate fields.

* Printing::           How to print using `awk'.  Describes the
                       `print' and `printf' statements.  
		       Also describes redirection of output.

* One-liners::         Short, sample `awk' programs.

* Patterns::           The various types of patterns explained in detail.

* Actions::            The various types of actions are introduced here.
                       Describes expressions and the various operators in
                       detail.  Also describes comparison expressions.

* Statements::         The various control statements are described in
                       detail.

* Arrays::             The description and use of arrays.  Also includes
                       array--oriented control statements.

* User-defined::       User--defined functions are described in detail.

* Built-in::           The built--in functions are summarized here.

* Special::            The special variables are summarized here.

* Sample Program::     A sample `awk' program with a complete explanation.

* Notes::              Something about the implementation of `gawk'.

* Glossary::           An explanation of some unfamiliar terms.

* Index::



File: gawk-info,  Node: Preface,  Next: License,  Prev: Top,  Up: Top

Preface
*******

If you are like many computer users, you frequently would like to
make changes in various text files wherever certain patterns appear,
or extract data from parts of certain lines while discarding the
rest.  To write a program to do this in a language such as C or
Pascal is a time--consuming inconvenience that may take many lines of
code.  The job may be easier with `awk'.

The `awk' utility interprets a special--purpose programming language
that makes it possible to handle simple data--reformatting jobs
easily with just a few lines of code.

The GNU implementation of `awk' is called `gawk'; it is fully upward
compatible with the System V Release 3.1 and later version of `awk'. 
All properly written `awk' programs should work with `gawk'.  So we
usually don't distinguish between `gawk' and other `awk'
implementations in this manual.

This manual teaches you what `awk' does and how you can use `awk'
effectively.  You should already be familiar with basic,
general--purpose, operating system commands such as `ls'.  Using
`awk' you can:

   * manage small, personal databases,

   * generate reports,

   * validate data,

   * produce indexes, and perform other document preparation tasks,

   * even experiment with algorithms that can be adapted later to
     other computer languages!

* Menu:

* History::  The history of gawk and awk.  Acknowledgements.

 

File: gawk-info,  Node: History,  Up: Preface

History of `awk' and `gawk'
===========================

The name `awk' comes from the initials of its designers: Alfred V. 
Aho, Peter J. Weinberger, and Brian W. Kernighan.  The original
version of `awk' was written in 1977.  In 1985 a new version made the
programming language more powerful, introducing user--defined
functions, multiple input streams, and computed regular expressions.

The GNU implementation, `gawk', was written in 1986 by Paul Rubin and
Jay Fenlason, with advice from Richard Stallman.  John Woods
contributed parts of the code as well.  In 1988, David Trueman, with
help from Arnold Robbins, reworked `gawk' for compatibility with the
newer `awk'.

Many people need to be thanked for their assistance in producing this
manual.  Jay Fenlason contributed many ideas and sample programs. 
Richard Mlynarik and Robert Chassell gave helpful comments on drafts
of this manual.  The paper ``A Supplemental Document for `awk''' by
John W.  Pierce of the Chemistry Department at UC San Diego,
pinpointed several issues relevant both to `awk' implementation and
to this manual, that would otherwise have escaped us.

Finally, we would like to thank Brian Kernighan of Bell Labs for
invaluable assistance during the testing and debugging of `gawk', and
for help in clarifying several points about the language.



File: gawk-info,  Node: License,  Next: This Manual,  Prev: Preface,  Up: Top

GNU GENERAL PUBLIC LICENSE
**************************

                        Version 1, February 1989

     Copyright (C) 1989 Free Software Foundation, Inc.
     675 Mass Ave, Cambridge, MA 02139, USA
     
     Everyone is permitted to copy and distribute verbatim copies
     of this license document, but changing it is not allowed.

 Preamble
=========

  The license agreements of most software companies try to keep users
at the mercy of those companies.  By contrast, our General Public
License is intended to guarantee your freedom to share and change
free software--to make sure the software is free for all its users. 
The General Public License applies to the Free Software Foundation's
software and to any other program whose authors commit to using it. 
You can use it for your programs, too.

  When we speak of free software, we are referring to freedom, not
price.  Specifically, the General Public License is designed to make
sure that you have the freedom to give away or sell copies of free
software, that you receive source code or can get it if you want it,
that you can change the software or use pieces of it in new free
programs; and that you know you can do these things.

  To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if
you distribute copies of the software, or if you modify it.

  For example, if you distribute copies of a such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have.  You must make sure that they, too, receive or can get the
source code.  And you must tell them their rights.

  We protect your rights with two steps: (1) copyright the software,
and (2) offer you this license which gives you legal permission to
copy, distribute and/or modify the software.

  Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software.  If the software is modified by someone else and passed on,
we want its recipients to know that what they have is not the
original, so that any problems introduced by others will not reflect
on the original authors' reputations.

  The precise terms and conditions for copying, distribution and
modification follow.

                          TERMS AND CONDITIONS

  1. This License Agreement applies to any program or other work
     which contains a notice placed by the copyright holder saying it
     may be distributed under the terms of this General Public
     License.  The ``Program'', below, refers to any such program or
     work, and a ``work based on the Program'' means either the
     Program or any work containing the Program or a portion of it,
     either verbatim or with modifications.  Each licensee is
     addressed as ``you''.

  2. You may copy and distribute verbatim copies of the Program's
     source code as you receive it, in any medium, provided that you
     conspicuously and appropriately publish on each copy an
     appropriate copyright notice and disclaimer of warranty; keep
     intact all the notices that refer to this General Public License
     and to the absence of any warranty; and give any other
     recipients of the Program a copy of this General Public License
     along with the Program.  You may charge a fee for the physical
     act of transferring a copy.

  3. You may modify your copy or copies of the Program or any portion
     of it, and copy and distribute such modifications under the
     terms of Paragraph 1 above, provided that you also do the
     following:

        * cause the modified files to carry prominent notices stating
          that you changed the files and the date of any change; and

        * cause the whole of any work that you distribute or publish,
          that in whole or in part contains the Program or any part
          thereof, either with or without modifications, to be
          licensed at no charge to all third parties under the terms
          of this General Public License (except that you may choose
          to grant warranty protection to some or all third parties,
          at your option).

        * If the modified program normally reads commands
          interactively when run, you must cause it, when started
          running for such interactive use in the simplest and most
          usual way, to print or display an announcement including an
          appropriate copyright notice and a notice that there is no
          warranty (or else, saying that you provide a warranty) and
          that users may redistribute the program under these
          conditions, and telling the user how to view a copy of this
          General Public License.

        * You may charge a fee for the physical act of transferring a
          copy, and you may at your option offer warranty protection
          in exchange for a fee.

     Mere aggregation of another independent work with the Program
     (or its derivative) on a volume of a storage or distribution
     medium does not bring the other work under the scope of these
     terms.

  4. You may copy and distribute the Program (or a portion or
     derivative of it, under Paragraph 2) in object code or
     executable form under the terms of Paragraphs 1 and 2 above
     provided that you also do one of the following:

        * accompany it with the complete corresponding
          machine-readable source code, which must be distributed
          under the terms of Paragraphs 1 and 2 above; or,

        * accompany it with a written offer, valid for at least three
          years, to give any third party free (except for a nominal
          charge for the cost of distribution) a complete
          machine-readable copy of the corresponding source code, to
          be distributed under the terms of Paragraphs 1 and 2 above;
          or,

        * accompany it with the information you received as to where
          the corresponding source code may be obtained.  (This
          alternative is allowed only for noncommercial distribution
          and only if you received the program in object code or
          executable form alone.)

     Source code for a work means the preferred form of the work for
     making modifications to it.  For an executable file, complete
     source code means all the source code for all modules it
     contains; but, as a special exception, it need not include
     source code for modules which are standard libraries that
     accompany the operating system on which the executable file
     runs, or for standard header files or definitions files that
     accompany that operating system.

  5. You may not copy, modify, sublicense, distribute or transfer the
     Program except as expressly provided under this General Public
     License.  Any attempt otherwise to copy, modify, sublicense,
     distribute or transfer the Program is void, and will
     automatically terminate your rights to use the Program under
     this License.  However, parties who have received copies, or
     rights to use copies, from you under this General Public License
     will not have their licenses terminated so long as such parties
     remain in full compliance.

  6. By copying, distributing or modifying the Program (or any work
     based on the Program) you indicate your acceptance of this
     license to do so, and all its terms and conditions.

  7. Each time you redistribute the Program (or any work based on the
     Program), the recipient automatically receives a license from
     the original licensor to copy, distribute or modify the Program
     subject to these terms and conditions.  You may not impose any
     further restrictions on the recipients' exercise of the rights
     granted herein.

  8. The Free Software Foundation may publish revised and/or new
     versions of the General Public License from time to time.  Such
     new versions will be similar in spirit to the present version,
     but may differ in detail to address new problems or concerns.

     Each version is given a distinguishing version number.  If the
     Program specifies a version number of the license which applies
     to it and ``any later version'', you have the option of
     following the terms and conditions either of that version or of
     any later version published by the Free Software Foundation.  If
     the Program does not specify a version number of the license,
     you may choose any version ever published by the Free Software
     Foundation.

  9. If you wish to incorporate parts of the Program into other free
     programs whose distribution conditions are different, write to
     the author to ask for permission.  For software which is
     copyrighted by the Free Software Foundation, write to the Free
     Software Foundation; we sometimes make exceptions for this.  Our
     decision will be guided by the two goals of preserving the free
     status of all derivatives of our free software and of promoting
     the sharing and reuse of software generally.

                                   NO WARRANTY

 10. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO
     WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE
     LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
     HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM ``AS IS''
     WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED,
     INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
     MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE
     ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS
     WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE
     COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

 11. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
     WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY
     MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE
     LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL,
     INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR
     INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS
     OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
     YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH
     ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN
     ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

                      END OF TERMS AND CONDITIONS

Appendix: How to Apply These Terms to Your New Programs
=======================================================

  If you develop a new program, and you want it to be of the greatest
possible use to humanity, the best way to achieve this is to make it
free software which everyone can redistribute and change under these
terms.

  To do so, attach the following notices to the program.  It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the ``copyright'' line and a pointer to where the full notice is found.

     ONE LINE TO GIVE THE PROGRAM'S NAME AND A BRIEF IDEA OF WHAT IT DOES.
     Copyright (C) 19YY  NAME OF AUTHOR
     
     This program is free software; you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation; either version 1, or (at your option)
     any later version.
     
     This program is distributed in the hope that it will be useful,
     but WITHOUT ANY WARRANTY; without even the implied warranty of
     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     GNU General Public License for more details.
     
     You should have received a copy of the GNU General Public License
     along with this program; if not, write to the Free Software
     Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

 Also add information on how to contact you by electronic and paper
mail.

If the program is interactive, make it output a short notice like
this when it starts in an interactive mode:

     Gnomovision version 69, Copyright (C) 19YY NAME OF AUTHOR
     Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
     This is free software, and you are welcome to redistribute it
     under certain conditions; type `show c' for details.

 The hypothetical commands `show w' and `show c' should show the
appropriate parts of the General Public License.  Of course, the
commands you use may be called something other than `show w' and
`show c'; they could even be mouse-clicks or menu items--whatever
suits your program.

You should also get your employer (if you work as a programmer) or
your school, if any, to sign a ``copyright disclaimer'' for the
program, if necessary.  Here a sample; alter the names:

     Yoyodyne, Inc., hereby disclaims all copyright interest in the
     program `Gnomovision' (a program to direct compilers to make passes
     at assemblers) written by James Hacker.
     
     SIGNATURE OF TY COON, 1 April 1989
     Ty Coon, President of Vice

That's all there is to it!



File: gawk-info,  Node: This Manual,  Next: Getting Started,  Prev: License,  Up: Top

Using This Manual
*****************

The term `gawk' refers to a program (a version of `awk') developed by
the Free Software Foundation, and to the language you use to tell it
what to do.  When we need to be careful, we call the program ``the
`awk' utility'' and the language ``the `awk' language''.  The purpose
of this manual is to explain the `awk' language and how to run the
`awk' utility.

The term "`awk' program" refers to a program written by you in the
`awk' programming language.

*Note Getting Started::, for the bare essentials you need to know to
start using `awk'.

Useful ``one--liners'' are included to give you a feel for the `awk'
language (*note One-liners::.).

A sizable sample `awk' program has been provided for you (*note
Sample Program::.).

If you find terms that you aren't familiar with, try looking them up
in the glossary (*note Glossary::.).

Most of the time complete `awk' programs are used as examples, but in
some of the more advanced sections, only the part of the `awk'
program that illustrates the concept being described is shown.

* Menu:

This chapter contains the following sections:

* The Files::          Sample data files for use in the `awk' programs
                       illustrated in this manual.

 

File: gawk-info,  Node: The Files,  Up: This Manual

Input Files for the Examples
============================

This manual contains many sample programs.  The data for many of
those programs comes from two files.  The first file, called
`BBS-list', represents a list of computer bulletin board systems and
information about those systems.

Each line of this file is one "record".  Each record contains the
name of a computer bulletin board, its phone number, the board's baud
rate, and a code for the number of hours it is operational.  An `A'
in the last column means the board operates 24 hours all week.  A `B'
in the last column means the board operates evening and weekend
hours, only.  A `C' means the board operates only on weekends.

     aardvark     555-5553     1200/300          B
     alpo-net     555-3412     2400/1200/300     A
     barfly       555-7685     1200/300          A
     bites        555-1675     2400/1200/300     A
     camelot      555-0542     300               C
     core         555-2912     1200/300          C
     fooey        555-1234     2400/1200/300     B
     foot         555-6699     1200/300          B
     macfoo       555-6480     1200/300          A
     sdace        555-3430     2400/1200/300     A
     sabafoo      555-2127     1200/300          C

The second data file, called `inventory-shipped', represents
information about shipments during the year.  Each line of this file
is also one record.  Each record contains the month of the year, the
number of green crates shipped, the number of red boxes shipped, the
number of orange bags shipped, and the number of blue packages
shipped, respectively.

     Jan  13  25  15 115
     Feb  15  32  24 226
     Mar  15  24  34 228
     Apr  31  52  63 420
     May  16  34  29 208
     Jun  31  42  75 492
     Jul  24  34  67 436
     Aug  15  34  47 316
     Sep  13  55  37 277
     Oct  29  54  68 525
     Nov  20  87  82 577
     Dec  17  35  61 401
     
     Jan  21  36  64 620
     Feb  26  58  80 652
     Mar  24  75  70 495
     Apr  21  70  74 514

If you are reading this in GNU Emacs using Info, you can copy the
regions of text showing these sample files into your own test files. 
This way you can try out the examples shown in the remainder of this
document.  You do this by using the command `M-x write-region' to
copy text from the Info file into a file for use with `awk' (see your
``GNU Emacs Manual'' for more information).  Using this information,
create your own `BBS-list' and `inventory-shipped' files, and
practice what you learn in this manual.



File: gawk-info,  Node: Getting Started,  Next: Reading Files,  Prev: This Manual,  Up: Top

Getting Started With `awk'
**************************

The basic function of `awk' is to search files for lines (or other
units of text) that contain certain patterns.  When a line matching
any of those patterns is found, `awk' performs specified actions on
that line.  Then `awk' keeps processing input lines until the end of
the file is reached.

An `awk' "program" or "script" consists of a series of "rules". 
(They may also contain "function definitions", but that is an
advanced feature, so let's ignore it for now.  *Note User-defined::.)

A rule contains a "pattern", an "action", or both.  Actions are
enclosed in curly braces to distinguish them from patterns. 
Therefore, an `awk' program is a sequence of rules in the form:

     PATTERN { ACTION }
     PATTERN { ACTION }
     ...

 * Menu:

* Very Simple::      A very simple example.
* Two Rules::        A less simple one--line example with two rules.
* More Complex::     A more complex example.
* Running gawk::     How to run gawk programs; includes command line syntax.
* Comments::         Adding documentation to gawk programs.
* Statements/Lines:: Subdividing or combining statements into lines.

* When::             When to use gawk and when to use other things.

 

File: gawk-info,  Node: Very Simple,  Next: Two Rules,  Up: Getting Started

A Very Simple Example
=====================

The following command runs a simple `awk' program that searches the
input file `BBS-list' for the string of characters: `foo'.  (A string
of characters is usually called, quite simply, a "string".)

     awk '/foo/ { print $0 }' BBS-list

When lines containing `foo' are found, they are printed, because
`print $0' means print the current line.  (Just `print' by itself
also means the same thing, so we could have written that instead.)

You will notice that slashes, `/', surround the string `foo' in the
actual `awk' program.  The slashes indicate that `foo' is a pattern
to search for.  This type of pattern is called a "regular
expression", and is covered in more detail later (*note Regexp::.). 
There are single quotes around the `awk' program so that the shell
won't interpret any of it as special shell characters.

Here is what this program prints:

     fooey        555-1234     2400/1200/300     B
     foot         555-6699     1200/300          B
     macfoo       555-6480     1200/300          A
     sabafoo      555-2127     1200/300          C

In an `awk' rule, either the pattern or the action can be omitted,
but not both.

If the pattern is omitted, then the action is performed for *every*
input line.

If the action is omitted, the default action is to print all lines
that match the pattern.  We could leave out the action (the print
statement and the curly braces) in the above example, and the result
would be the same: all lines matching the pattern `foo' would be
printed.  (By comparison, omitting the print statement but retaining
the curly braces makes an empty action that does nothing; then no
lines would be printed.)



File: gawk-info,  Node: Two Rules,  Next: More Complex,  Prev: Very Simple,  Up: Getting Started

An Example with Two Rules
=========================

The `awk' utility reads the input files one line at a time.  For each
line, `awk' tries the patterns of all the rules.  If several patterns
match then several actions are run, in the order in which they appear
in the `awk' program.  If no patterns match, then no actions are run.

After processing all the rules (perhaps none) that match the line,
`awk' reads the next line (however, *note Next::.).  This continues
until the end of the file is reached.

For example, the `awk' program:

     /12/  { print $0 }
     /21/  { print $0 }

contains two rules.  The first rule has the string `12' as the
pattern and `print $0' as the action.  The second rule has the string
`21' as the pattern and also has `print $0' as the action.  Each
rule's action is enclosed in its own pair of braces.

This `awk' program prints every line that contains the string `12'
*or* the string `21'.  If a line contains both strings, it is printed
twice, once by each rule.

If we run this program on our two sample data files, `BBS-list' and
`inventory-shipped', as shown here:

     awk '/12/ { print $0 }
          /21/ { print $0 }' BBS-list inventory-shipped

we get the following output:

     aardvark     555-5553     1200/300          B
     alpo-net     555-3412     2400/1200/300     A
     barfly       555-7685     1200/300          A
     bites        555-1675     2400/1200/300     A
     core         555-2912     1200/300          C
     fooey        555-1234     2400/1200/300     B
     foot         555-6699     1200/300          B
     macfoo       555-6480     1200/300          A
     sdace        555-3430     2400/1200/300     A
     sabafoo      555-2127     1200/300          C
     sabafoo      555-2127     1200/300          C
     Jan  21  36  64 620
     Apr  21  70  74 514

Note how the line in `BBS-list' beginning with `sabafoo' was printed
twice, once for each rule.



File: gawk-info,  Node: More Complex,  Next: Running gawk,  Prev: Two Rules,  Up: Getting Started

A More Complex Example
======================

Here is an example to give you an idea of what typical `awk' programs
do.  This example shows how `awk' can be used to summarize, select,
and rearrange the output of another utility.  It uses features that
haven't been covered yet, so don't worry if you don't understand all
the details.

     ls -l | awk '$5 == "Nov" { sum += $4 }
                  END { print sum }'

This command prints the total number of bytes in all the files in the
current directory that were last modified in November (of any year). 
(In the C shell you would need to type a semicolon and then a
backslash at the end of the first line; in the Bourne shell you can
type the example as shown.)

The `ls -l' part of this example is a command that gives you a full
listing of all the files in a directory, including file size and date.
Its output looks like this:

     -rw-r--r--  1 close        1933 Nov  7 13:05 Makefile
     -rw-r--r--  1 close       10809 Nov  7 13:03 gawk.h
     -rw-r--r--  1 close         983 Apr 13 12:14 gawk.tab.h
     -rw-r--r--  1 close       31869 Jun 15 12:20 gawk.y
     -rw-r--r--  1 close       22414 Nov  7 13:03 gawk1.c
     -rw-r--r--  1 close       37455 Nov  7 13:03 gawk2.c
     -rw-r--r--  1 close       27511 Dec  9 13:07 gawk3.c
     -rw-r--r--  1 close        7989 Nov  7 13:03 gawk4.c

The first field contains read--write permissions, the second field
contains the number of links to the file, and the third field
identifies the owner of the file.  The fourth field contains the size
of the file in bytes.  The fifth, sixth, and seventh fields contain
the month, day, and time, respectively, that the file was last
modified.  Finally, the eighth field contains the name of the file.

The `$5 == "Nov"' in our `awk' program is an expression that tests
whether the fifth field of the output from `ls -l' matches the string
`Nov'.  Each time a line has the string `Nov' in its fifth field, the
action `{ sum += $4 }' is performed.  This adds the fourth field (the
file size) to the variable `sum'.  As a result, when `awk' has
finished reading all the input lines, `sum' will be the sum of the
sizes of files whose lines matched the pattern.

After the last line of output from `ls' has been processed, the `END'
pattern is executed, and the value of `sum' is printed.  In this
example, the value of `sum' would be 80600.

These more advanced `awk' techniques are covered in later sections
(*note Actions::.).  Before you can move on to more advanced `awk'
programming, you have to know how `awk' interprets your input and
displays your output.  By manipulating "fields" and using special
"print" statements, you can produce some very useful and spectacular
looking reports.



File: gawk-info,  Node: Running gawk,  Next: Comments,  Prev: More Complex,  Up: Getting Started

How to Run `awk' Programs
=========================

There are several ways to run an `awk' program.  If the program is
short, it is easiest to include it in the command that runs `awk',
like this:

     awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ...

 where PROGRAM consists of a series of PATTERNS and ACTIONS, as
described earlier.

When the program is long, you would probably prefer to put it in a
file and run it with a command like this:

     awk -f PROGRAM-FILE INPUT-FILE1 INPUT-FILE2 ...

 * Menu:

* One-shot::      Running a short throw--away `awk' program.
* Read Terminal:: Using no input files (input from terminal instead).
* Long::          Putting permanent `awk' programs in files.
* Executable Scripts:: Making self--contained `awk' programs.
* Command Line::  How the `awk' command line is laid out.

 

File: gawk-info,  Node: One-shot,  Next: Read Terminal,  Up: Running gawk

One--shot Throw--away `awk' Programs
------------------------------------

Once you are familiar with `awk', you will often type simple programs
at the moment you want to use them.  Then you can write the program
as the first argument of the `awk' command, like this:

     awk 'PROGRAM' INPUT-FILE1 INPUT-FILE2 ...

 where PROGRAM consists of a series of PATTERNS and ACTIONS, as
described earlier.

This command format tells the shell to start `awk' and use the
PROGRAM to process records in the input file(s).  There are single
quotes around the PROGRAM so that the shell doesn't interpret any
`awk' characters as special shell characters.  They cause the shell
to treat all of PROGRAM as a single argument for `awk'.  They also
allow PROGRAM to be more than one line long.

This format is also useful for running short or medium--sized `awk'
programs from shell scripts, because it avoids the need for a
separate file for the `awk' program.  A self--contained shell script
is more reliable since there are no other files to misplace.



File: gawk-info,  Node: Read Terminal,  Next: Long,  Prev: One-shot,  Up: Running gawk

Running `awk' without Input Files
---------------------------------

You can also use `awk' without any input files.  If you type the
command line:

     awk 'PROGRAM'

then `awk' applies the PROGRAM to the "standard input", which usually
means whatever you type on the terminal.  This continues until you
indicate end--of--file by typing `Control-d'.

For example, if you type:

     awk '/th/'

whatever you type next will be taken as data for that `awk' program. 
If you go on to type the following data,

     Kathy
     Ben
     Tom
     Beth
     Seth
     Karen
     Thomas
     `Control-d'

then `awk' will print

     Kathy
     Beth
     Seth

as matching the pattern `th'.  Notice that it did not recognize
`Thomas' as matching the pattern.  The `awk' language is "case
sensitive", and matches patterns *exactly*.



File: gawk-info,  Node: Long,  Next: Executable Scripts,  Prev: Read Terminal,  Up: Running gawk

Running Long Programs
---------------------

Sometimes your `awk' programs can be very long.  In this case it is
more convenient to put the program into a separate file.  To tell
`awk' to use that file for its program, you type:

     awk -f SOURCE-FILE INPUT-FILE1 INPUT-FILE2 ...

 The `-f' tells the `awk' utility to get the `awk' program from the
file SOURCE-FILE.  Any file name can be used for SOURCE-FILE.  For
example, you could put the program:

     /th/

into the file `th-prog'.  Then the command:

     awk -f th-prog

does the same thing as this one:

     awk '/th/'

which was explained earlier (*note Read Terminal::.).  Note that you
don't usually need single quotes around the file name that you
specify with `-f', because most file names don't contain any of the
shell's special characters.

If you want to identify your `awk' program files clearly as such, you
can add the extension `.awk' to the filename.  This doesn't affect
the execution of the `awk' program, but it does make ``housekeeping''
easier.



File: gawk-info,  Node: Executable Scripts,  Next: Command Line,  Prev: Long,  Up: Running gawk

Executable `awk' Programs
-------------------------

(The following section assumes that you are already somewhat familiar
with `awk'.)

Once you have learned `awk', you may want to write self--contained
`awk' scripts, using the `#!' script mechanism.  You can do this on
BSD Unix systems and GNU.

For example, you could create a text file named `hello', containing
the following (where `BEGIN' is a feature we have not yet discussed):

     #! /bin/awk -f
     
     # a sample awk program
     
     BEGIN    { print "hello, world" }

After making this file executable (with the `chmod' command), you can
simply type:

     hello

at the shell, and the system will arrange to run `awk' as if you had
typed:

     awk -f hello

Self--contained `awk' scripts are particularly useful for putting
`awk' programs into production on your system, without your users
having to know that they are actually using an `awk' program.

If your system does not support the `#!' mechanism, you can get a
similar effect using a regular shell script.  It would look something
like this:

     : a sample awk program
     
     awk 'PROGRAM' "$@"

Using this technique, it is *vital* to enclose the PROGRAM in single
quotes to protect it from interpretation by the shell.  If you omit
the quotes, only a shell wizard can predict the result.

The `"$@"' causes the shell to forward all the command line arguments
to the `awk' program, without interpretation.



File: gawk-info,  Node: Command Line,  Prev: Executable Scripts,  Up: Running gawk

Details of the `awk' Command Line
---------------------------------

(The following section assumes that you are already familiar with
`awk'.)

There are two ways to run `awk'.  Here are templates for both of
them; items enclosed in `[' and `]' in these templates are optional.

     awk [ -FFS ] [ -- ] 'PROGRAM' FILE ...
     awk [ -FFS ] -f SOURCE-FILE [ -f SOURCE-FILE ... ] [ -- ] FILE ...

 Options begin with a minus sign, and consist of a single character. 
The options and their meanings are as follows:

`-FFS'
     This sets the `FS' variable to FS (*note Special::.).  As a
     special case, if FS is `t', then `FS' will be set to the tab
     character (`"\t"').

`-f SOURCE-FILE'
     Indicates that the `awk' program is to be found in SOURCE-FILE
     instead of in the first non--option argument.

`--'
     This signals the end of the command line options.  If you wish
     to specify an input file named `-f', you can precede it with the
     `--' argument to prevent the `-f' from being interpreted as an
     option.  This handling of `--' follows the POSIX argument
     parsing conventions.

Any other options will be flagged as invalid with a warning message,
but are otherwise ignored.

If the `-f' option is *not* used, then the first non--option command
line argument is expected to be the program text.

The `-f' option may be used more than once on the command line. 
`awk' will read its program source from all of the named files, as if
they had been concatenated together into one big file.  This is
useful for creating libraries of `awk' functions.  Useful functions
can be written once, and then retrieved from a standard place,
instead of having to be included into each individual program.  You
can still type in a program at the terminal and use library
functions, by specifying `/dev/tty' as one of the arguments to a
`-f'.  Type your program, and end it with the keyboard end--of--file
character `Control-d'.

Any additional arguments on the command line are made available to
your `awk' program in the `ARGV' array (*note Special::.).  These
arguments are normally treated as input files to be processed in the
order specified.  However, an argument that has the form VAR`='VALUE,
means to assign the value VALUE to the variable VAR--it does not
specify a file at all.

Command line options and the program text (if present) are omitted
from the `ARGV' array.  All other arguments, including variable
assignments, are included (*note Special::.).

The distinction between file name arguments and variable--assignment
arguments is made when `awk' is about to open the next input file. 
At that point in execution, it checks the ``file name'' to see
whether it is really a variable assignment; if so, instead of trying
to read a file it will, *at that point in the execution*, assign the
variable.

Therefore, the variables actually receive the specified values after
all previously specified files have been read.  In particular, the
values of variables assigned in this fashion are *not* available
inside a `BEGIN' rule (*note BEGIN/END::.), since such rules are run
before `awk' begins scanning the argument list.

The variable assignment feature is most useful for assigning to
variables such as `RS', `OFS', and `ORS', which control input and
output formats, before listing the data files.  It is also useful for
controlling state if multiple passes are needed over a data file. 
For example:

     awk 'pass == 1  { PASS 1 STUFF }
          pass == 2  { PASS 2 STUFF }' pass=1 datafile pass=2 datafile



File: gawk-info,  Node: Comments,  Next: Statements/Lines,  Prev: Running gawk,  Up: Getting Started

Comments in `awk' Programs
==========================

When you write a complicated `awk' program, you can put "comments" in
the program file to help you remember what the program does, and how
it works.

A comment starts with the the sharp sign character, `#', and
continues to the end of the line.  The `awk' language ignores the
rest of a line following a sharp sign.  For example, we could have
put the following into `th-prog':

     # This program finds records containing the pattern `th'.  This is how
     # you continue comments on additional lines.
     /th/

You can put comment lines into keyboard--composed throw--away `awk'
programs also, but this usually isn't very useful; the purpose of a
comment is to help yourself or another person understand the program
at another time.



File: gawk-info,  Node: Statements/Lines,  Next: When,  Prev: Comments,  Up: Getting Started

`awk' Statements versus Lines
=============================

Most often, each line in an `awk' program is a separate statement or
separate rule, like this:

     awk '/12/  { print $0 }
          /21/  { print $0 }' BBS-list inventory-shipped

But sometimes statements can be more than one line, and lines can
contain several statements.

You can split a statement into multiple lines by inserting a newline
after any of the following:

     ,    {    ?    :    ||    &&

Lines ending in `do' or `else' automatically have their statements
continued on the following line(s).  A newline at any other point
ends the statement.

If you would like to split a single statement into two lines at a
point where a newline would terminate it, you can "continue" it by
ending the first line with a backslash character, `\'.  This is
allowed absolutely anywhere in the statement, even in the middle of a
string or regular expression.  For example:

     awk '/This program is too long, so continue it\
      on the next line/ { print $1 }'

We have generally not used backslash continuation in the sample
programs in this manual.  Since there is no limit on the length of a
line, it is never strictly necessary; it just makes programs
prettier.  We have preferred to make them even more pretty by keeping
the statements short.  Backslash continuation is most useful when
your `awk' program is in a separate source file, instead of typed in
on the command line.

*Warning: this does not work if you are using the C shell.*
Continuation with backslash works for `awk' programs in files, and
also for one--shot programs *provided* you are using the Bourne
shell, the Korn shell, or the Bourne--again shell.  But the C shell
used on Berkeley Unix behaves differently!  There, you must use two
backslashes in a row, followed by a newline.

When `awk' statements within one rule are short, you might want to
put more than one of them on a line.  You do this by separating the
statements with semicolons, `;'.  This also applies to the rules
themselves.  Thus, the above example program could have been written:

     /12/ { print $0 } ; /21/ { print $0 }

*Note:* It is a new requirement that rules on the same line require
semicolons as a separator in the `awk' language; it was done for
consistency with the statements in the action part of rules.



File: gawk-info,  Node: When,  Prev: Statements/Lines,  Up: Getting Started

When to Use `awk'
=================

What use is all of this to me, you might ask?  Using additional
operating system utilities, more advanced patterns, field separators,
arithmetic statements, and other selection criteria, you can produce
much more complex output.  The `awk' language is very useful for
producing reports from large amounts of raw data, like summarizing
information from the output of standard operating system programs
such as `ls'.  (*Note A More Complex Example: More Complex.)

Programs written with `awk' are usually much smaller than they would
be in other languages.  This makes `awk' programs easy to compose and
use.  Often `awk' programs can be quickly composed at your terminal,
used once, and thrown away.  Since `awk' programs are interpreted,
you can avoid the usually lengthy edit--compile--test--debug cycle of
software development.

Complex programs have been written in `awk', including a complete
retargetable assembler for 8--bit microprocessors (*note Glossary::.
for more information) and a microcode assembler for a special purpose
Prolog computer.  However, `awk''s capabilities are strained by tasks
of such complexity.

If you find yourself writing `awk' scripts of more than, say, a few
hundred lines, you might consider using a different programming
language.  Emacs Lisp is a good choice if you need sophisticated
string or pattern matching capabilities.  The shell is also good at
string and pattern matching; in addition it allows powerful use of
the standard utilities.  More conventional languages like C, C++, or
Lisp offer better facilities for system programming and for managing
the complexity of large programs.  Programs in these languages may
require more lines of source code than the equivalent `awk' programs,
but they will be easier to maintain and usually run more efficiently.



File: gawk-info,  Node: Reading Files,  Next: Printing,  Prev: Getting Started,  Up: Top

Reading Files (Input)
*********************

In the typical `awk' program, all input is read either from the
standard input (usually the keyboard) or from files whose names you
specify on the `awk' command line.  If you specify input files, `awk'
reads data from the first one until it reaches the end; then it reads
the second file until it reaches the end, and so on.  The name of the
current input file can be found in the special variable `FILENAME'
(*note Special::.).

The input is split automatically into "records", and processed by the
rules one record at a time.  (Records are the units of text mentioned
in the introduction; by default, a record is a line of text.) Each
record read is split automatically into "fields", to make it more
convenient for a rule to work on parts of the record under
consideration.

On rare occasions you will need to use the `getline' command, which
can do explicit input from any number of files.

* Menu:

* Records::            Controlling how data is split into records.
* Fields::             An introduction to fields.
* Field Separators::   The field separator and how to change it.
* Multiple::           Reading multi--line records.

* Assignment Options:: Setting variables on the command line and a summary
                       of command line syntax.  This is an advanced method
                       of input.

* Getline::            Reading files under explicit program control
                       using the `getline' function.
* Close Input::        Closing an input file (so you can read from
                       the beginning once more).

 

File: gawk-info,  Node: Records,  Next: Fields,  Up: Reading Files

How Input is Split into Records
===============================

The `awk' language divides its input into records and fields. 
Records are separated from each other by the "record separator".  By
default, the record separator is the "newline" character.  Therefore,
normally, a record is a line of text.

Sometimes you may want to use a different character to separate your
records.  You can use different characters by changing the special
variable `RS'.

The value of `RS' is a string that says how to separate records; the
default value is `"\n"', the string of just a newline character. 
This is why lines of text are the default record.  Although `RS' can
have any string as its value, only the first character of the string
will be used as the record separator.  The other characters are
ignored.  `RS' is exceptional in this regard; `awk' uses the full
value of all its other special variables.

The value of `RS' is changed by "assigning" it a new value (*note
Assignment Ops::.).  One way to do this is at the beginning of your
`awk' program, before any input has been processed, using the special
`BEGIN' pattern (*note BEGIN/END::.).  This way, `RS' is changed to
its new value before any input is read.  The new value of `RS' is
enclosed in quotation marks.  For example:

     awk 'BEGIN { RS = "/" } ; { print $0 }' BBS-list

changes the value of `RS' to `/', the slash character, before reading
any input.  Records are now separated by a slash.  The second rule in
the `awk' program (the action with no pattern) will proceed to print
each record.  Since each `print' statement adds a newline at the end
of its output, the effect of this `awk' program is to copy the input
with each slash changed to a newline.

Another way to change the record separator is on the command line,
using the variable--assignment feature (*note Command Line::.).

     awk '...' RS="/" SOURCE-FILE

`RS' will be set to `/' before processing SOURCE-FILE.

The empty string (a string of no characters) has a special meaning as
the value of `RS': it means that records are separated only by blank
lines.  *Note Multiple::, for more details.

The `awk' utility keeps track of the number of records that have been
read so far from the current input file.  This value is stored in a
special variable called `FNR'.  It is reset to zero when a new file
is started.  Another variable, `NR', is the total number of input
records read so far from all files.  It starts at zero but is never
automatically reset to zero.

If you change the value of `RS' in the middle of an `awk' run, the
new value is used to delimit subsequent records, but the record
currently being processed (and records already finished) are not
affected.