Looping through files and collecting data

Question

Ok, I have a problem that I think you guys can help with.

I have 24 folders. Labeled Angle1, Angle2, ... , Angle24

Each folder has a file name output.txt

In each file named output.txt, I want to take certain information.

There are 10 columns of data with numbers, but the numbers are variable length. I want the 9th column of numbers, but this does not mean that this is just the 9th column by characters. It could be 46 columns in or something. Anyways, each column is separated by a blank space so perhaps that would be of use.

I want to loop through Angle1, Angle2 etc. and get the 9th column of data from each and put the total of 24 columns of data into a new file in parent directory called total.txt.

This may not be possible, but just curious - is it possible?

Note that I am using Windows PowerShell but could use Ubuntu if that is easier. If any answer might not work in both environments, please indicate which environment it is for.

Answer 1

You do not specify the output format and/or if you want some aggregation functions, but one example script which will create as exit one column file is:

for ((i=1;i<25;i++))
do
awk '{print $9}' Angle${i}/output.txt >>total.txt
done

If you want to have 9th column from each file as column you can use script like this:

awk '{print $9}' Angle1/output.txt >>tmp
for ((i=2;i<25;i++))
do
awk '{print $9}' Angle${i}/output.txt |paste -d " " tmp - >>total.txt
mv total.txt tmp
done
mv tmp total.txt

But this will work well only it you have same number of rows in input files.

These scripts will run with any Bash/Korn shell, and will work on Linux and Cygwin.

Here are my tests. Source files:

[romeo@localhost tmp]$ cat PKA1/output.txt 
1 2 3 4 5 6 7 8 9 0
2 3 4 5 6 7 8 9 0 1
3 4 5 6 7 8 9 0 1 2
[romeo@localhost tmp]$ cat PKA2/output.txt 
3 4 5 6 7 8 9 0 1 2 
4 5 6 7 8 9 0 1 2 3
5 6 7 8 9 0 1 2 3 4
[romeo@localhost tmp]$ cat PKA3/output.txt 
6 7 8 9 0 1 2 3 4 5
7 8 9 0 1 2 3 4 5 6
8 9 0 1 2 3 4 5 6 7

My scripts (edited to represent my environment):

[romeo@localhost tmp]$ cat z1
for ((i=1;i<4;i++))
do
awk '{print $9}' PKA${i}/output.txt >>total.txt
done
[romeo@localhost tmp]$ cat z2
awk '{print $9}' PKA1/output.txt >>tmp
for ((i=2;i<4;i++))
do
awk '{print $9}' PKA${i}/output.txt |paste -d " " tmp - >>total.txt
mv total.txt tmp
done
mv tmp total.txt

And my runs:

[romeo@localhost tmp]$ rm total.txt 
[romeo@localhost tmp]$ bash z1
[romeo@localhost tmp]$ cat total.txt 
9
0
1
1
2
3
4
5
6
[romeo@localhost tmp]$ rm total.txt 
[romeo@localhost tmp]$ bash z2
[romeo@localhost tmp]$ cat total.txt 
9 1 4
0 2 5
1 3 6