Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a space-delimited file that's about 3200 lines long. Each line contains 7+ fields.

What I would like to do is edit the file with sed so that every line containing a certain variable in field 5 would have its field 1 changed to X.

What I'm thinking is to do something like:

for variable in `cat word.list.file`  
sed 's/line_with_$variable_in_field5/replace_field1_with_X/g' old.file > new.file  
cp new.file old.file  

Is this correct? Is there a better way?

What I need help with is filling in the sed command or finding an alternative way to accomplish the same thing.

I can easily convert the space-delimited file to a comma-delimited file if it would make things easier.

Let me know if any clarification is needed.

share|improve this question
Could you provide some sample data? – John T Feb 8 '11 at 20:08
up vote 6 down vote accepted

This prevents having to read each file many times. It reads each file just once.

awk 'NR == FNR {a[$1]=1;next} $5 in a {$1="XYZ"} {print}' word.list.file old.file > new.file && mv new.file old.file


# if the current record number is the same as the record number in the file
# which means "if we're reading the first file"
NR == FNR {
    a[$1]=1  # put a flag in an array indexed by the contents of the first field
    next     # read the next line in the file and continue at the top of the script

# Now we're processing the second file
# if field 5 exists as an index in the array named "a" (it's a word from the first file)
$5 in a {
    $1="XYZ"  # replace the first field with new contents

# for all lines in the second file, changed or not
    print    # print them
}' \
    word.list.file old.file \
    > new.file && \
    mv new.file old.file

Use the files "word.list.file" and "old.file" as input. Write output to "new.file". If the whole operation doesn't produce an error (&&), then rename "new.file" back to "old.file". The part described in this paragraph is the only part of the whole thing that is Bash (or shell). The part in the original command at the top and described by the comment lines is an AWK script. AWK is a programming language on its own and is independent of the shell.

share|improve this answer
Nice! Didn't know awk handled multiple input files like that. – Mikel Feb 8 '11 at 21:00
@Mikel: Ordinarily, it handles them as if they'd been cat-ted together. Checking to see if the record number is equal to the file record number is what separates the first file so it can be stored in the array. It could also be done in a BEGIN block using a different technique involving ARGV and getline. – Dennis Williamson Feb 8 '11 at 21:07
Yeah, I figured that bit out, I've just never thought to use it like that before. Thanks for teaching me something. Again. ;-) – Mikel Feb 8 '11 at 21:08
thanks everyone for there quick response all of it was helpful – rp.sullivan Feb 8 '11 at 22:04
this worked perfectly for what i needed. im new to bash scripting so for educational purposes can you break this script into parts and explain to me what is going on? – rp.sullivan Feb 8 '11 at 22:13

There are many ways to do this.

Here's a way using only bash:


# read word.list.file into words
# read line-by-line, each space-separated field goes into an array called fields
while IFS=$' \n' read -r -a fields; do
    # could possibly be an associative array to make it faster
    for word in $words; do
        # zero-indexed, so 4 means the fifth field
        if test "${fields[4]}" = "$word"; then
            # change the first field to "X"
    echo "${fields[*]}"
done <old.file >new.file
mv new.file old.file

And here's a solution using sed:


# bash-only syntax: read word.list.file into an array...
words=( $(<word.list.file) )
# ...and make a variable called "wordpattern"
# that contains a sed extended regular expression that matches
# any of those words, i.e. "word1|word2|word3..."

# sed -r makes sed use extended re, which makes the pattern easier to read,
# but might only work on GNU/Linux and FreeBSD systems
# /...$wordpattern/ matches four words followed by a fifth word from word.list.file
# then the s/.../.../ makes a replacement on only those lines
# note that we have to use double quotes rather than single quotes
# so the shell can expand $wordpattern
sed -r -e "/^([^ ]* ){4}$wordpattern\>/s/^([^ ]*)(.*)/X\2/" old.file >new.file
mv new.file old.file

And a version in (rusty) Perl for good measure:

#!/usr/bin/env perl

my $wordfile = "word.list.file";
open WORDS, "<$wordfile"
    or die "Cannot open $wordfile: $!\n";

my @words;
while (my $word = <WORDS>) {
    chomp $word;
    push @words, $word;
my $wordpattern = join '|', @words;
close WORDS;

my $oldfile = "old.file";
open IN, "<$oldfile"
    or die "Cannot open $oldfile: $!\n";

my $newfile = "new.file";
open OUT, ">$newfile"
    or die "Cannot open $newfile for writing: $!\n";
# output now goes to the OUT file handle (meaning $newfile) by default
select OUT;

while (my $line = <IN>) {
    chomp $line;
    my @fields = split / /, $line;
    if ($fields[4] =~ /$wordpattern/) {
        $fields[0] = "X";
    $line = join ' ', @fields;
    print $line . "\n";

close OUT;
close IN;

rename $newfile, $oldfile
    or die "Cannot rename $newfile to $oldfile: $!\n";
share|improve this answer

This would be a good application for awk. As a simple example:

for variable in $(word.list.file)
    awk -v pat=$variable '$5 ~ pat {$1 = "X"}1' file1 > tmp
    mv tmp > file1
share|improve this answer
Can you use $1 as a left hand side in Awk? I don't think you can. – Mikel Feb 8 '11 at 20:51
Nevermind, it does seem to work. Sorry. – Mikel Feb 8 '11 at 20:57
This way will be slower because it makes multiple passes over old.file. – Mikel Feb 8 '11 at 21:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .