simple awk tutorial

why awk?

awk is small, fast, and simple, unlike, say, perl. awk also has a clean comprehensible C-like input language, unlike, say, perl. And while it can't do everything you can do in perl, it can do most things that are actually text processing, and it's much easier to work with.

what do you do?

In its simplest usage awk is meant for processing column-oriented text data, such as tables, presented to it on standard input. The variables $1, $2, and so forth are the contents of the first, second, etc. column of the current input line. For example, to print the second column of a file, you might use the following simple awk script: 
        awk < file '{ print $2 }'
This means "on every line, print the second field".

To print the second and third columns, you might use 

        awk < file '{ print $2, $3 }'

Input separator

By default awk splits input lines into fields based on whitespace, that is, spaces and tabs. You can change this by using the -F option to awk and supplying another character. For instance, to print the home directories of all users on the system, you might do 
        awk < /etc/passwd -F: '{ print $6 }'
since the password file has fields delimited by colons and the home directory is the 6th field.

Arithmetic

Awk is a weakly typed language; variables can be either strings or numbers, depending on how they're referenced. All numbers are floating-point. So to implement the fahrenheit-to-celsius calculator, you might write 
        awk '{ print ($1-32)*(5/9) }'
which will convert fahrenheit temperatures provided on standard input to celsius until it gets an end-of-file.

The selection of operators is basically the same as in C, although some of C's wilder constructs do not work. String concatenation is accomplished simply by writing two string expressions next to each other. '+' is always addition. Thus 

        echo 5 4 | awk '{ print $1 + $2 }'
prints 9, while 
        echo 5 4 | awk '{ print $1 $2 }'
prints 54. Note that 
        echo 5 4 | awk '{ print $1, $2 }'
prints "5 4".

Variables

awk has some built-in variables that are automatically set; $1 and so on are examples of these. The other builtin variables that are useful for beginners are generally NF, which holds the number of fields in the current input line ($NF gives the last field), and $0, which holds the entire current input line.

You can make your own variables, with whatever names you like (except for reserved words in the awk language) just by using them. You do not have to declare variables. Variables that haven't been explicitly set to anything have the value "" as strings and 0 as numbers.

For example, the following code prints the average of all the numbers on each line: 

        awk '{ tot=0; for (i=1; i<=NF; i++) tot += $i; print	tot/NF; }'
Note the use of $i to retrieve the i'th variable, and the for loop, which works like in C. The reason tot is explicitly initialized at the beginning is that this code is run for every input line, and when starting work on the second line, tot will have the total value from the first line.

Blocks

It might seem silly to do that. Probably, you have only one set of numbers to add up. Why not put each one on its own line? In order to do this you need to be able to print the results when you're done. The way you do this is like this: 
        awk '{ tot += $1; n += 1; }  END { print tot/n; }'
Note the use of two different block statements. The second one has END in front of it; this means to run the block once after all input has been processed. In fact, in general, you can put all kinds of things in front of a block, and the block will only run if they're satisfied. That is, you can say 
        awk ' $1==0 { print $2 }'
which will print the second column for lines of input where the first column is 0. You can also supply regular expressions to match the whole line against: 
        awk ' /^test/ { print $2 }'
If you put no expression, the block is run on every line of input. If multiple blocks have conditions that are true, they are all run. There is no particularly clean way I know of to get it to run exactly one of a bunch of possible blocks of code.

The block conditions BEGIN and END are special and are run before processing any input, and after processing all input, respectively.

Other language constructs

As hinted at above, awk supports loop and conditional statements like in C, that is, for, while, do/while, if, and if/else.

printf

awk includes a printf statement that works essentially like C printf. This can be used when you want to format output neatly or combine things onto one line in more complex ways (print implicitly adds a newline; printf doesn't.)

Here's how you strip the first column off: 

        awk '{ for (i=2; i<=NF; i++) printf "%s ", $i; printf "\n"; }'
Note the use of NF to iterate over all the fields and the use of printf to place newlines explicitly.

Combining awk with other tools

The strength of shell scripting is the ability to combine lots of tools together. Some things are most readily done with successive awk processes pipelined together. awk is also often combined with the sed utility, which does regular expression matching and substitution. You can actually do most common sed operations in awk, but it's usually more convenient to use sed.

In combination with sed, sort, and some other useful shell utilities like paste, awk is quite satisfactory for a lot of numerical data processing, as well as the maintenance of simple databases kept in column-tabular formats.

awk is also extremely useful in a certain style of makefile writing for generating pieces of makefile to include.

Do not script in csh. Use sh, or if you must, ksh.

More stuff

This introduction intentionally covers only the absolute basics of awk. awk can do lots of other useful and powerful things. The manual page for GNU awk (gawk) is a reasonably good reference for looking things up once you know the basics.

The only thing seriously lacking in awk that I've yet run into is that there's no way to tell awk not to do buffering on its input and output, which makes it useless for assorted interactive or network-oriented applications it would otherwise be ideally suited for.