Command Line Control: awk

Built-in variables

awk has a number built-in variables, and while I won't cover all of them, these are the ones that I've found the most useful:

~NR~

Gives you the line number of the current match. Could be used for adding line numbers to an input file: ~awk '{print NR, $0}'~. Or maybe looking for lines that contain 'ice cream' is more your speed: ~awk 'ice cream {print NR}'~.

~NF~

This is the number of fields in a line. Useful if you're looking for the last field, either for finding out how many fields are in a line or for seeing if it contains a pattern: awk '$NF ~ /out/ {print NR}

~FS~

The field separator value. This is what awk will use to split each line into fields. By default this is whitespace. If you have a file full of comma-separated values and want to split each line on commas instead of whitespace: BEGIN {FS=","}

~RS~

This is the line ('record') equivalent of the field separator. The default is \n. Say you want to print your PATH over multiple lines: ~echo $PATH | awk 'BEGIN {RS=":"} {print}'~

BEGIN and END

awk lets you supply commands to be executed at the start and end of a script by using BEGIN and END. While they may seem like it, they're not really special at all. Instead, think of them as being patterns that evaluate to true before anything is evaluated and after everything is evaluated, respectively.

BEGIN could be used to set the field separator or initialize a variable.

END is useful for printing out results accrued through the life of the program, such as a word count. If we bring back our word counting example: ~awk '{word_count += NF} END {print word_count}'~

Functions and conditionals

Like most languages, AWK has a number of built-in functions (such as the print and length functions we saw earlier) and also lets you define your own functions if you so please. This is probably overkill for most trivial operations but could come in handy in certain cases.

And AWK has conditionals too! While you can use the if else construct within actions, I'd like to highlight that you can do conditional statements based on the supplied pattern. I.e. awk '/foo/ {print "FOO"} {print "bar"}'~ will print 'FOO' for lines that match ~/foo/ and 'bar' for lines that don't.

Sorting lines in a file by length

Here's a fun little application: Let's take a file, count the number of characters in each line, and then sort it based on the number of characters:

awk '{print length, $0}' <file> | sort -n

And if you want to exclude empty lines, try this:

awk 'length > 0 {print length, $0}' <file> | sort -n

Friendly PATH

This is the same one that was listed above, and is useful if you're looking for certain entries that can easily get lost when the path is on one line:

# bash, zsh
echo $PATH | awk 'BEGIN {RS=":"} {print}'

# fish
echo $PATH | awk 'BEGIN {RS=" "} {print}'

Counting words

Another example that was used previously. Count the words in a file or any input stream:

awk '{word_count += NF} END {print word_count}'

Parsing git logs

Maybe your team has agreed that all commit messages should start with #<task_no> if it relates to a task and #-- if it doesn't. To find all commits that relate to a specific task---say #42---we could do this:

git log --oneline | awk '/#42/ {print $1}'

Or how about finding the ratio of commits that belong to a task versus those that don't?

git log --oneline --no-decorate | \
awk '$2 ~ /#[0-9]+/ {task += 1} {notask +=1} \
  END {printf "Tasks: %d\nNo-tasks: %d\nTask to no-task ratio: %f", \
  task, notask, task / notask }'

(Yeah, the printf function works pretty much the same as in C---so much so that I haven't looked it up yet!)

Killing processes

This is actually what triggered me to look into awk and what eventually led to this post being brought into existence.

There are a number of ways to kill applications from the command line, and for a while, my default fallback has been the classic ps aux | grep <app> combo.

While this lists out all the relevant processes and their process IDs (PIDs), you then have to manually copy the PID and put it into the kill command to shut it down. This is annoying at best, and gets even worse if the process has spawned children that we want to take down as well.

How do we deal with this? Well:

ps aux | awk '/<app>/ {print $2}' | xargs kill

1. ps lists all processes on the system

1. awk then goes over every line that contains the application name, extracts the second whitespace-delimited word---which in this case is the PID---and prints that.

1. For the last stretch, we use xargs to feed the PIDs into the kill command, thus killing all the processes.

This works just fine, though it comes with the caveat that it'll also try and kill the awk process (because the application name is part of the command, so it gets listed by ps), but that's only a minor annoyance, and I'll leave fixing that as an exercise for the reader.

Now, I'm sure you can make killall do something quite similar to this, but I've found this way to be more effective (by which I mean: closer to what I expect).