#011 Process a File/Stream Line-by-Line

Notes

The bash for..in construct is useful for simple repetitive processing tasks.

When iterating character strings, the behaviour is governed by the internal field separator ($IFS), which determines how Bash recognizes fields, or word boundaries.

$IFS defaults to whitespace (space, tab, and newline).

So when trying to process lines in a file (for example), it will iterate over each word. This may be what you want. But if not, modifying the $IFS allows the behaviour to match what is required.

This is fine for simple text processing. For more complex manipulations, its probably best not to attempt to bend Bash to your will, but instead reach for a more suitable tool (sed, awk, perl, python, ruby etc..).

Line-by-Line Processing Example

See nevermores.sh.

• it first demonstrates default for..in behaviour
• then modifies the IFS to scan The Raven line by line and do some processing

$ ./nevermores.sh

Using for..in with default IFS, we get words not lines:
> The
> Raven
> -
> Edgar
> Allan
> Poe

Reset the IFS for newline, then for..in gives us lines:
> Quoth the Raven, "Nevermore."
> With such name as "Nevermore."
> Then the bird said, "Nevermore."
> Of 'Never- nevermore'."
> Meant in croaking "Nevermore."
> She shall press, ah, nevermore!
> Quoth the Raven, "Nevermore."
> Quoth the Raven, "Nevermore."
> Quoth the Raven, "Nevermore."
> Quoth the Raven, "Nevermore."
> Shall be lifted- nevermore!

Credits and References

• 9.1. Internal Variables - Advanced Bash-Scripting Guide
• 11. Loops and Branches - Advanced Bash-Scripting Guide
• 7. Loops for, while and until
• Octal Escape Sequences
• The extended ASCII table
• The Raven