1

I am using following code to extract information from a bunch of text files (foo*.txt).

for file in foo*.txt; do 
grep "some_text" $file | tail -n5 | awk '{print $2}' >> bar.csv
done

This comment prints the numbers I want from a bunch of files (foo*.txt). When I try to print both file name (in one column of the csv file) and the number (in the next coloum of the csv file), I tried following at the Terminal.

for file in foo*.txt; do 
echo $file
grep "some_text" $file | tail -n5 | awk '{print $2}' >> bar2.csv 
done

This prints the names of the files at the terminal. The csv file contains the numbers I want. How this code can be modified so that the name of the file is printed on one coloumn and the extracted numbers in the next coloumn of the csv file?

Another problem in this code is sorting problem. For example, consider file names foo_01_s.txt, foo_02_s.txt, foo_03_s.txt.....foo_100_s.txt. If I want to extract information (using above comments) the last file (foo_100_s.txt) does not come after foo_99_s.txt.

Solution using Python/Perl also would be helpful.

  • 1
    By the way, the only tag you need here is bash I think. The rest are irrelevant details. – kabanus Sep 29 '17 at 8:25
  • 1
    Could be done with paste too for file in foo*.txt; do paste <(echo $file) <(grep "some_text" $file | tail -n5 | awk '{print $2}') >> bar2.csv; done. – Paulo Sep 29 '17 at 13:44
3

You have to understand your the >> will only redirect the part of the current command - basically just the number which is the result of the command starting with grep, and piped a few times. echo $file is a separate command (you use ;) and thus will normally direct to stdout. All you need to do is redirect after the entire loop:

for file in foo*.txt; do 
    echo $file
    grep "some_text" $file | tail -n5 | awk '{print $2}'
done > bar2.csv

If you want to "version" sort your files (this is appropriate name) you can list them after sorting:

for file in $(ls foo*.txt | sort -V); do

for running something small quickly (few minutes for ~1000 files) this should be fine.

EDIT

Following your comment, there are a few solutions. I'm guessing you want:

file1 1
      2
      3

etc. Just drop the echo and amend the echo line:

for file in foo*.txt; do 
    grep "some_text" $file | tail -n5 | awk -v f=$file '{if(NR==1) {printf("%-20s %-5s\n",f,\$2)} else {printf("%-20s %-5s\n","",$2)}}'
done > bar2.csv

I let awk do the printing for me. Using -v allows me to pass in a variable in f. For the print familiarize yourself with printf syntax (you can use man printf in the shell. Basically I'm assuming two fields, one with with 20, the other 5, and a space between. Negative sign left justifies. You can play with it. This would have fixed your initial problem, since now you can pipe that single line.

If you want the file just to be:

file1,1
file1,2
...
file2,1

you can either drop the if in my awk statement, or leave the initial solution with echo, but use,

echo -n "$file,"

where -n ensures no newline is printed.

  • I would like to print the file names in one column and the number extracted from those file in the next column (for a quick verification of the data). Now, those (file name and the data extracted) are printed in the subsequent rows. – phenomenon Sep 29 '17 at 9:20
  • @phenomenon I see, I'll amend the answer. – kabanus Sep 29 '17 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.