Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a directory of tab-delimited text files and some have comments in the first few lines that I would like to delete. I know that the first good line starts with "Mark" so I can use /^Mark/,$!d to delete these comments. After this deletion I have several other replacements that I make in the (new) first line that has variable names.

My question is, why do I have to save sed's output to get my script to work? I understand that if I line is deleted, then the output doesn't proceed downstream because there is no output. But if I don't delete (i.e., !d) then why do I have to save to file? Thanks!

Here is my shell script. (I'm a sed newbie, so any other feedback is also appreciated.)

#!/bin/bash
for file in *.txt; do
    mv $file $file.old1
    sed -e '/^Mark/,$!d' $file.old1 > $file.old2
    sed -e '1s/\([Ss]\)hareholder/\1hrhldr/g'\
        -e '1s/\([Ii]\)mmediate/\1mmdt/g'\
        -e '1s/\([Nn]\)umber/\1o/g'\
        -e '1s/\([Cc]\)ompany/\1o/g'\
        -e '1s/\([Ii]\)nformation/\1nfo/g'\
        -e '1s/\([Pp]\)ercentage/\1ct/g'\
        -e '1s/\([Dd]\)omestic/\1om/g'\
        -e '1s/\([Gg]\)lobal/\1lbl/g'\
        -e '1s/\([Cc]\)ountry/\1ntry/g'\
        -e '1s/\([Ss]\)ource/\1rc/g'\
        -e '1s/\([Oo]\)wnership/\1wnrshp/g'\
        -e '1s/\([Uu]\)ltimate/\1ltmt/g'\
        -e '1s/\([Ii]\)ncorporation/\1ncorp/g'\
        -e '1s/\([Tt]\)otal/\1ot/g'\
        -e '1s/\([Dd]\)irect/\1ir/g'\
        $file.old2 > $file
        rm -f $file.old*
done
share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

Using sponge is possible the safest way to overwrite your input file. sponge handles associated links to the file properly, sed -i doesn't. see this link: Is there a way to modify a file in-place?

Test file

echo "comment
comment
Mark Global Ownership
Shareholder
Domestic
Last line" >test.txt
file=test.txt
cat "$file"
echo =====

The script without the loop.. (Just add the loop)

sed -n '/^Mark/,${  # check for "first time"
          G         # append the hold space, looking for "\nX" 
          /\n$/{    # nothing in the hold space, so "first time"
            s/\n$// # remove \n introduced by G
            s/\([Ss]\)hareholder/\1hrhldr/g
            s/\([Ii]\)mmediate/\1mmdt/g
            s/\([Nn]\)umber/\1o/g
            s/\([Cc]\)ompany/\1o/g
            s/\([Ii]\)nformation/\1nfo/g
            s/\([Pp]\)ercentage/\1ct/g
            s/\([Dd]\)omestic/\1om/g
            s/\([Gg]\)lobal/\1lbl/g
            s/\([Cc]\)ountry/\1ntry/g
            s/\([Ss]\)ource/\1rc/g
            s/\([Oo]\)wnership/\1wnrshp/g
            s/\([Uu]\)ltimate/\1ltmt/g
            s/\([Ii]\)ncorporation/\1ncorp/g
            s/\([Tt]\)otal/\1ot/g
            s/\([Dd]\)irect/\1ir/g
            p           # print
            s/.*/X/; h  # set a "flag" in the hold space 
            b           # branch unconditionally (ie, next)
          };
          P;b   # print line (before "\nX") and branch            
        };d' "$file" | sponge "$file"

cat "$file"

The output

comment
comment
Mark Global Ownership
Shareholder
Domestic
Last line
=====
Mark Glbl Ownrshp
Shareholder
Domestic
Last line
share|improve this answer
    
Wow. Maybe it's easier to do the file write :). I guess I'm just puzzled why I have to write to file when I'm not deleting. There are lots of good tricks here for me. Thanks! –  Richard Herron Jun 18 '12 at 17:19
1  
Technically you may not need to, but it is fiddly and fraught with the "unexpected" (like when it needs to be modified for longer text), and (technically) sed just doesn't work that way; it is a text Stream EDitor, designed as per the nix paradigm of "do one thing and do it well" (and then pipe it to something else).. But, really, the overhead is not that great, you still need to write the data (all of it after the first changed byte) and then 1 file delete and 1 file rename.. On the other hand, running sed twice sequentially per file is definite overhead ...(sed | sed is reasonable). –  Peter.O Jun 18 '12 at 18:05
add comment

It is not necessary to save the standard ouput to a file. You can pipe(|) the output to the next sed process. Other option is to use sed's edit in place option(-i)

From man sed,

-i[SUFFIX], --in-place[=SUFFIX]
    edit files in place (makes backup if extension supplied) 
share|improve this answer
    
Thanks! If I run every command as -ie will it run really slow? It seems that would write to file after every operation? –  Richard Herron Jun 18 '12 at 17:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.