using awk with find -exec

Question

I have a directory structure with 14 directories containing a bunch of files containing data in a three-column format (separated with tabs). I intended to use find and awk to extract the second column from each of those files and output it with the same filename but under a different root folder. Here the sketch of my directory.

data/all -> AA, AB, AC, AD ... (A* being folders containing the files with data stored in a 3-column format, e.g. AA100.txt, AA101.txt ...)

i want to have the modified (one-column) files with the same name, but all under a new root directory data/pos (as opposed to data/all/) -> AA, AB, AC, AD ... (again, each containing A*100.txt, A*101...)

My try was to use find -exec and to give it the awk command, but I'm having issues with outputting the file to the right place.

when being in data/all/

find * -type f -exec awk '{print$2}' '{}' > ../pos/'{}' \;

However {} as wildcard for the input file doesn't seem to work when outputting the file?

What am I doing wrong? (I'm on a ubuntu server btw)

Answer 1

You could try without find, if all you want is all the files. While in data/all/, run this:

for file in ./*; do awk '{print$2}' "$file" > "../pos/$(basename $file)"; done

If you want to cover the files in the whole hierarchy under /data/all, you can enable the globstar option if you are using bash (I believe this would "just work" on zsh) and then use ** to match all files:

shopt -s globstar
for file in ./**; do awk '{print$2}' "$file" > "../pos/$(basename $file)"; done

Answer 2

What am I doing wrong?

You are using the redirection > ../pos/'{}' as if it was handled by find or awk but redirections are handled by the shell. In your case it means that you can redirect only output of the whole find (not output of awk).

Note that you do not usually need to use wildcard like * for the starting path of find. Is the common way find . what you wanted to do or is there any reason for find *?

Solutions

Here we will keep the flexibility of find in contrast to the solution by Jacobo de Vera. Run awk in a shell loop:

find . -type f -print0 |
  while read -r -d $'\0' x; do
    awk '{print $2}' "$x" > "../pos/$(basename "$x")"
  done

The original way with -exec will be less efficient because for every file a shell will be launched in addition to awk and the multi-level escaping is pretty complicated here:

find . -type f -exec sh -c 'awk "{print \$2}" "{}" > "../pos/{}"' \;

There could also be an alternative solution of doing the redirection inside awk.

Answer 3

find /path -exec ls -l {} \; | awk '{print$1}'