Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a directory of tab-delimited text files and some have comments in the first few lines that I would like to delete. I know that the first good line starts with "Mark" so I can use /^Mark/,$!d to delete these comments. After this deletion I have several other replacements that I make in the (new) first line that has variable names.

My question is, why do I have to save sed's output to get my script to work? I understand that if I line is deleted, then the output doesn't proceed downstream because there is no output. But if I don't delete (i.e., !d) then why do I have to save to file? Thanks!

Here is my shell script. (I'm a sed newbie, so any other feedback is also appreciated.)

for file in *.txt; do
    mv $file $file.old1
    sed -e '/^Mark/,$!d' $file.old1 > $file.old2
    sed -e '1s/\([Ss]\)hareholder/\1hrhldr/g'\
        -e '1s/\([Ii]\)mmediate/\1mmdt/g'\
        -e '1s/\([Nn]\)umber/\1o/g'\
        -e '1s/\([Cc]\)ompany/\1o/g'\
        -e '1s/\([Ii]\)nformation/\1nfo/g'\
        -e '1s/\([Pp]\)ercentage/\1ct/g'\
        -e '1s/\([Dd]\)omestic/\1om/g'\
        -e '1s/\([Gg]\)lobal/\1lbl/g'\
        -e '1s/\([Cc]\)ountry/\1ntry/g'\
        -e '1s/\([Ss]\)ource/\1rc/g'\
        -e '1s/\([Oo]\)wnership/\1wnrshp/g'\
        -e '1s/\([Uu]\)ltimate/\1ltmt/g'\
        -e '1s/\([Ii]\)ncorporation/\1ncorp/g'\
        -e '1s/\([Tt]\)otal/\1ot/g'\
        -e '1s/\([Dd]\)irect/\1ir/g'\
        $file.old2 > $file
        rm -f $file.old*
share|improve this question
up vote 1 down vote accepted

Using sponge is possible the safest way to overwrite your input file. sponge handles associated links to the file properly, sed -i doesn't. see this link: Is there a way to modify a file in-place?

Test file

echo "comment
Mark Global Ownership
Last line" >test.txt
cat "$file"
echo =====

The script without the loop.. (Just add the loop)

sed -n '/^Mark/,${  # check for "first time"
          G         # append the hold space, looking for "\nX" 
          /\n$/{    # nothing in the hold space, so "first time"
            s/\n$// # remove \n introduced by G
            p           # print
            s/.*/X/; h  # set a "flag" in the hold space 
            b           # branch unconditionally (ie, next)
          P;b   # print line (before "\nX") and branch            
        };d' "$file" | sponge "$file"

cat "$file"

The output

Mark Global Ownership
Last line
Mark Glbl Ownrshp
Last line
share|improve this answer
Wow. Maybe it's easier to do the file write :). I guess I'm just puzzled why I have to write to file when I'm not deleting. There are lots of good tricks here for me. Thanks! – Richard Herron Jun 18 '12 at 17:19
Technically you may not need to, but it is fiddly and fraught with the "unexpected" (like when it needs to be modified for longer text), and (technically) sed just doesn't work that way; it is a text Stream EDitor, designed as per the nix paradigm of "do one thing and do it well" (and then pipe it to something else).. But, really, the overhead is not that great, you still need to write the data (all of it after the first changed byte) and then 1 file delete and 1 file rename.. On the other hand, running sed twice sequentially per file is definite overhead ...(sed | sed is reasonable). – Peter.O Jun 18 '12 at 18:05

It is not necessary to save the standard ouput to a file. You can pipe(|) the output to the next sed process. Other option is to use sed's edit in place option(-i)

From man sed,

-i[SUFFIX], --in-place[=SUFFIX]
    edit files in place (makes backup if extension supplied) 
share|improve this answer
Thanks! If I run every command as -ie will it run really slow? It seems that would write to file after every operation? – Richard Herron Jun 18 '12 at 17:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .