Very Simple Script to Remove Last 5 Lines of a Series of Text Files

Question

For some reason this script outputs three files for each original, rather than one.

Must have made some trivial error - I'm new to this!

Much obliged if anyone could explain why this happens.

Script:-

for f in *.txt
do
   noOfRows=$(cat $f | wc -l)
   relevantRows=$(expr $noOfRows - 5)
   head -n $relevantRows $f | tee ${f%.txt}-Amended.txt
done

Result of ls command:-

E12-5_F2_NEG-Amended-Amended-Amended.txt  E12-5_M3_POS-Amended-Amended-Amended.txt
E12-5_F2_NEG-Amended-Amended.txt          E12-5_M3_POS-Amended-Amended.txt
E12-5_F2_NEG-Amended.txt                  E12-5_M3_POS-Amended.txt
E12-5_F2_NEG.txt                          E12-5_M3_POS.txt
E12-5_F2_POS-Amended-Amended-Amended.txt  E12-5_M4_NEG-Amended-Amended-Amended.txt
E12-5_F2_POS-Amended-Amended.txt          E12-5_M4_NEG-Amended-Amended.txt
E12-5_F2_POS-Amended.txt                  E12-5_M4_NEG-Amended.txt
E12-5_F2_POS.txt                          E12-5_M4_NEG.txt
E12-5_F5_NEG-Amended-Amended-Amended.txt  E12-5_M4_POS-Amended-Amended-Amended.txt
E12-5_F5_NEG-Amended-Amended.txt          E12-5_M4_POS-Amended-Amended.txt
E12-5_F5_NEG-Amended.txt                  E12-5_M4_POS-Amended.txt
E12-5_F5_NEG.txt                          E12-5_M4_POS.txt
E12-5_F5_POS-Amended-Amended-Amended.txt  E12-5_M7_NEG-Amended-Amended-Amended.txt
E12-5_F5_POS-Amended-Amended.txt          E12-5_M7_NEG-Amended-Amended.txt
E12-5_F5_POS-Amended.txt                  E12-5_M7_NEG-Amended.txt
E12-5_F5_POS.txt                          E12-5_M7_NEG.txt
E12-5_M3_NEG-Amended-Amended-Amended.txt  E12-5_M7_POS-Amended-Amended-Amended.txt
E12-5_M3_NEG-Amended-Amended.txt          E12-5_M7_POS-Amended-Amended.txt
E12-5_M3_NEG-Amended.txt                  E12-5_M7_POS-Amended.txt
E12-5_M3_NEG.txt                          E12-5_M7_POS.txt

Many thanks, Adam

Answer 1

this script outputs three files for each original ... Much obliged if anyone could explain why this happens.

Since E12-5_F2_NEG-Amended.txt ends in .txt it will be picked up by your script next time you run it.

The triple results indicate you ran your script three times whilst debugging it.

If the script output to $f.new rather than to ${f%.txt}-Amended.txt, you wouldn't have this problem.

Alternatively put rm *Amended.txt at the start of the program. If you have a very large number of files an a directory this can be slow on older Unix variants.

Another option is to output the files into a subdirectory (so something like "new/${f%.txt}.Amended.txt")

Answer 2

You can do what your script is trying to achieve in a single line:

head --lines=-5 input.txt > output.txt

In a for loop:

for f in *.txt; do head --lines=-5 "$f" > "${f%.txt}-Amended.txt"; done

You can use -n -5 instead of --lines=-5 to save on typing if you want.

As RedGrittyBrick points out, the reason you have three files per input is probably because you ran the script multiple times, and since the outputs end with .txt, they were picked up by the *.txt glob of the successive scripts.

Now I will critique your specific script.

noOfRows=$(cat $f | wc -l)

This is a truly useless use of cat; rather than cat $f | wc -l, use wc -l "$f". It's probably not so important in this particular script, but it's good not to develop bad habits. Speaking about bad habits: Always quote variables, e.g. "$f". This will make sure the file name is treated as a single argument even if it contains whitespace.

relevantRows=$(expr $noOfRows - 5)

There's nothing actually wrong here, though I would generally prefer to use something like

relevantRows=$((noOfRows-5))

AFAIK there's no performance difference between the two, but I find the bash way more visually pleasing; and more importantly, the way I described is defined in POSIX, and is thus more portable. Within bash only (so don't use this if you'll need to port the script to a different shell), the best way to do this in a script would be to use let:

let noOfRows-=5

...which would subtract 5 from the number contained in the variable $noOfRows, meaning that there's no need to create the variable $relevantRows.

head -n $relevantRows $f | tee ${f%.txt}-Amended.txt

This is the correct thing to do if you want to have the output displayed on the command-line as well as putting it into your output file. Otherwise, just use > to redirect stdout to a file.