Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a directory structure with 14 directories containing a bunch of files containing data in a three-column format (separated with tabs). I intended to use find and awk to extract the second column from each of those files and output it with the same filename but under a different root folder. Here the sketch of my directory.

data/all -> AA, AB, AC, AD ... (A* being folders containing the files with data stored in a 3-column format, e.g. AA100.txt, AA101.txt ...)

i want to have the modified (one-column) files with the same name, but all under a new root directory data/pos (as opposed to data/all/) -> AA, AB, AC, AD ... (again, each containing A*100.txt, A*101...)

My try was to use find -exec and to give it the awk command, but I'm having issues with outputting the file to the right place.

when being in data/all/

find * -type f -exec awk '{print$2}' '{}' > ../pos/'{}' \;

However {} as wildcard for the input file doesn't seem to work when outputting the file?

What am I doing wrong? (I'm on a ubuntu server btw)

share|improve this question

2 Answers 2

up vote 0 down vote accepted

You could try without find, if all you want is all the files. While in data/all/, run this:

for file in ./*; do awk '{print$2}' "$file" > "../pos/$(basename $file)"; done

If you want to cover the files in the whole hierarchy under /data/all, you can enable the globstar option if you are using bash (I believe this would "just work" on zsh) and then use ** to match all files:

shopt -s globstar
for file in ./**; do awk '{print$2}' "$file" > "../pos/$(basename $file)"; done
share|improve this answer
    
Using for */* did the trick for me though, the globstar option wasn't available for me. –  Jonathan Dec 19 '13 at 19:44

What am I doing wrong?

You are using the redirection > ../pos/'{}' as if it was handled by find or awk but redirections are handled by the shell. In your case it means that you can redirect only output of the whole find (not output of awk).

Note that you do not usually need to use wildcard like * for the starting path of find. Is the common way find . what you wanted to do or is there any reason for find *?

Solutions

Here we will keep the flexibility of find in contrast to the solution by Jacobo de Vera. Run awk in a shell loop:

find . -type f -print0 |
  while read -r -d $'\0' x; do
    awk '{print $2}' "$x" > "../pos/$(basename "$x")"
  done

The original way with -exec will be less efficient because for every file a shell will be launched in addition to awk and the multi-level escaping is pretty complicated here:

find . -type f -exec sh -c 'awk "{print \$2}" "{}" > "../pos/{}"' \;

There could also be an alternative solution of doing the redirection inside awk.

share|improve this answer
    
I used the wildcard * in order to avoid the avoid the ./ in the path (as it gave me an error message that it could not find the specified path, which was something like pos/./AA/AA101.txt -> I had assumed the error to be there, that's why I used the * instead) –  Jonathan Dec 19 '13 at 19:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.