5

I have a space-delimited file that's about 3200 lines long. Each line contains 7+ fields.

What I would like to do is edit the file with sed so that every line containing a certain variable in field 5 would have its field 1 changed to X.

What I'm thinking is to do something like:

for variable in `cat word.list.file`  
do  
sed 's/line_with_$variable_in_field5/replace_field1_with_X/g' old.file > new.file  
cp new.file old.file  
done

Is this correct? Is there a better way?

What I need help with is filling in the sed command or finding an alternative way to accomplish the same thing.

I can easily convert the space-delimited file to a comma-delimited file if it would make things easier.

Let me know if any clarification is needed.

1
  • 2
    Could you provide some sample data?
    – user1931
    Feb 8, 2011 at 20:08

3 Answers 3

6

This prevents having to read each file many times. It reads each file just once.

awk 'NR == FNR {a[$1]=1;next} $5 in a {$1="XYZ"} {print}' word.list.file old.file > new.file && mv new.file old.file

Explanation:

# if the current record number is the same as the record number in the file
# which means "if we're reading the first file"
NR == FNR {
    a[$1]=1  # put a flag in an array indexed by the contents of the first field
    next     # read the next line in the file and continue at the top of the script
}

# Now we're processing the second file
# if field 5 exists as an index in the array named "a" (it's a word from the first file)
$5 in a {
    $1="XYZ"  # replace the first field with new contents
}

# for all lines in the second file, changed or not
{
    print    # print them
}' \
    word.list.file old.file \
    > new.file && \
    mv new.file old.file

Use the files "word.list.file" and "old.file" as input. Write output to "new.file". If the whole operation doesn't produce an error (&&), then rename "new.file" back to "old.file". The part described in this paragraph is the only part of the whole thing that is Bash (or shell). The part in the original command at the top and described by the comment lines is an AWK script. AWK is a programming language on its own and is independent of the shell.

6
  • Nice! Didn't know awk handled multiple input files like that.
    – Mikel
    Feb 8, 2011 at 21:00
  • @Mikel: Ordinarily, it handles them as if they'd been cat-ted together. Checking to see if the record number is equal to the file record number is what separates the first file so it can be stored in the array. It could also be done in a BEGIN block using a different technique involving ARGV and getline. Feb 8, 2011 at 21:07
  • Yeah, I figured that bit out, I've just never thought to use it like that before. Thanks for teaching me something. Again. ;-)
    – Mikel
    Feb 8, 2011 at 21:08
  • thanks everyone for there quick response all of it was helpful Feb 8, 2011 at 22:04
  • this worked perfectly for what i needed. im new to bash scripting so for educational purposes can you break this script into parts and explain to me what is going on? Feb 8, 2011 at 22:13
3

There are many ways to do this.

Here's a way using only bash:

#!/bin/bash

# read word.list.file into words
words=$(<word.list.file)
# read line-by-line, each space-separated field goes into an array called fields
while IFS=$' \n' read -r -a fields; do
    # could possibly be an associative array to make it faster
    for word in $words; do
        # zero-indexed, so 4 means the fifth field
        if test "${fields[4]}" = "$word"; then
            # change the first field to "X"
            fields[0]="X"
        fi
    done
    echo "${fields[*]}"
done <old.file >new.file
mv new.file old.file

And here's a solution using sed:

#!/bin/bash

# bash-only syntax: read word.list.file into an array...
words=( $(<word.list.file) )
OIFS="$IFS"
IFS=$'|'
# ...and make a variable called "wordpattern"
# that contains a sed extended regular expression that matches
# any of those words, i.e. "word1|word2|word3..."
wordpattern="${words[*]}"
IFS="$OIFS"

# sed -r makes sed use extended re, which makes the pattern easier to read,
# but might only work on GNU/Linux and FreeBSD systems
# /...$wordpattern/ matches four words followed by a fifth word from word.list.file
# then the s/.../.../ makes a replacement on only those lines
# note that we have to use double quotes rather than single quotes
# so the shell can expand $wordpattern
sed -r -e "/^([^ ]* ){4}$wordpattern\>/s/^([^ ]*)(.*)/X\2/" old.file >new.file
mv new.file old.file

And a version in (rusty) Perl for good measure:

#!/usr/bin/env perl

my $wordfile = "word.list.file";
open WORDS, "<$wordfile"
    or die "Cannot open $wordfile: $!\n";

my @words;
while (my $word = <WORDS>) {
    chomp $word;
    push @words, $word;
}
my $wordpattern = join '|', @words;
close WORDS;

my $oldfile = "old.file";
open IN, "<$oldfile"
    or die "Cannot open $oldfile: $!\n";

my $newfile = "new.file";
open OUT, ">$newfile"
    or die "Cannot open $newfile for writing: $!\n";
# output now goes to the OUT file handle (meaning $newfile) by default
select OUT;

while (my $line = <IN>) {
    chomp $line;
    my @fields = split / /, $line;
    if ($fields[4] =~ /$wordpattern/) {
        $fields[0] = "X";
    }
    $line = join ' ', @fields;
    print $line . "\n";
}

close OUT;
close IN;

rename $newfile, $oldfile
    or die "Cannot rename $newfile to $oldfile: $!\n";
1

This would be a good application for awk. As a simple example:

for variable in $(word.list.file)
do   
    awk -v pat=$variable '$5 ~ pat {$1 = "X"}1' file1 > tmp
    mv tmp > file1
done
3
  • Can you use $1 as a left hand side in Awk? I don't think you can.
    – Mikel
    Feb 8, 2011 at 20:51
  • Nevermind, it does seem to work. Sorry.
    – Mikel
    Feb 8, 2011 at 20:57
  • This way will be slower because it makes multiple passes over old.file.
    – Mikel
    Feb 8, 2011 at 21:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .