Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Whilst I have been using BASH for several years, my experience with BASH scripting is relatively limited.

My code is as below. It should grab the entire directory structure from within the current directory and replicate it into $OUTDIR.

for DIR in `find . -type d -printf "\"%P\"\040"`
do
  echo mkdir -p \"${OUTPATH}${DIR}\"        # Using echo for debug; working script will simply execute mkdir
  echo Created $DIR
done

The problem is, here is a sample of my file structure:

$ ls
Expect The Impossible-Stellar Kart
Five Iron Frenzy - Cheeses...
Five Score and Seven Years Ago-Relient K
Hello-After Edmund
I Will Go-Starfield
Learning to Breathe-Switchfoot
MMHMM-Relient K

Note the spaces :-S And for takes parameters word by word, so my script's output looks something like this:

Creating directory structure...
mkdir -p "/myfiles/multimedia/samjmusicmp3test/Learning"
Created Learning
mkdir -p "/myfiles/multimedia/samjmusicmp3test/to"
Created to
mkdir -p "/myfiles/multimedia/samjmusicmp3test/Breathe-Switchfoot"
Created Breathe-Switchfoot

But I need it to grab whole filenames (one line at a time) from the output of find. I have also tried making find put double-quotes around each filename. But this doesn't help.

for DIR in `find . -type d -printf "\"%P\"\040"`

And output with this changed line:

Creating directory structure...
mkdir -p "/myfiles/multimedia/samjmusicmp3test/"""
Created ""
mkdir -p "/myfiles/multimedia/samjmusicmp3test/"Learning"
Created "Learning
mkdir -p "/myfiles/multimedia/samjmusicmp3test/to"
Created to
mkdir -p "/myfiles/multimedia/samjmusicmp3test/Breathe-Switchfoot""
Created Breathe-Switchfoot"

Now, I need some way that I can iterate through like this, because I also wish to run a more complicated command involving gstreamer on each file in a following similar structure. How should I be doing this?

Edit: I need a code structure which will allow me to run multiple lines of code for each directory/file/loop. Sorry if I was unclear.

Solution: I initially tried:

find . -type d | while read DIR
do
  mkdir -p "${OUTPATH}${DIR}"
  echo Created $DIR
done

This worked fine for the most part. However, I later found that since the pipe results in the while loop running in a subshell, any variables set in the loop were later unavailable which made implementing an error counter quite difficult. My final solution (from this answer on SO):

while read DIR
do
  mkdir -p "${OUTPATH}${DIR}"
  echo Created $DIR
done < <(find . -type d)

This later allowed me to conditionally increment variables within the loop which would remain available later in the script.

share
    
Why_would_you_ever_need_a_space_in_a_file_name? –  Kevin Panko Feb 25 at 17:48
    
True, not my preference. Though, to remove spaces, you need to handle files with spaces first ;) –  samjetski Mar 17 at 3:16
    
Actually, file names should allow spaces. I would allow anything but / and unprintable chars. But anything is allowed except / and \0 so you must allow them. –  Kevin Panko Mar 17 at 3:21
add comment

8 Answers

up vote 11 down vote accepted

You need to pipe the find into a while loop.

find ... | while read -r dir
do
    something with "$dir"
done

Also, you won't need to use -printf in this case.

You can make this proof against files with newlines in their names, if you wish, by using a nullbyte delimiter (that being the only character which cannot appear in a *nix filepath):

find ... -print0 | while read -d '' -r dir
do
    something with "$dir"
done

You will also find using $() instead of backticks to be more versatile and easier. They can be nested much more easily and quoting can be done much more easily. This contrived example will illustrate these points:

echo "$(echo "$(echo "hello")")"

Try to do that with backticks.

share
2  
Also, rather than "$dir", it's preferable to use "${dir}" - it's easy to tell the difference between ${dir}name and ${dirname}, but $dirname could be interpreted either way. –  James Polley Dec 31 '09 at 0:00
    
The important thing here is that read reads an entire line into ${dir}, so the IFS doesn't matter. –  James Polley Dec 31 '09 at 0:04
1  
Thanks for finding the $/" typo. The braces aren't necessary if there's nothing following the variable name. –  Dennis Williamson Dec 31 '09 at 1:11
4  
This will handle pathnames with spaces (U+0020), but it will still fail to properly handle pathnames with line feeds (U+000A). I prefer find … -print0 | xargs -0 … because the delimiter it uses corresponds exactly to the only character that is not allowed in POSIX pathanames: NUL (U+0000). –  Chris Johnsen Dec 31 '09 at 5:19
2  
Perfect! Just what I was looking for. It had never occurred to me that you might be able to pipe to while. @Chris Johnsen: True, but even music ripping programs don't tend to put linefeeds in their filenames. And if they do, I want to know (ie: something goes wrong) and get rid of them immediately... –  samjetski Dec 31 '09 at 8:10
show 1 more comment

See this answer I wrote a few days ago for an example of a script that handles filenames with spaces.

There's a slightly more convoluted (but more concise) way to achieve what you're trying to do though:

find . -type d -print0 | xargs -0 -I {} mkdir -p ../theredir/{}

-print0 tells find to separate the arguments with a null; the -0 to xargs tells it to expect arguments seperated by nulls. This means that it handles spaces fine.

-I {} tells xargs to replace the string {} with the filename. This also implies that only one filename should be used per commandline (xargs will normally stuff as many as will fit on the line)

The rest should be obvious.

share
    
Dennis Williamson's suggestion is, however (aside from the typos) much more readable, and thus preferable in almost every way. –  James Polley Dec 31 '09 at 0:14
    
Works, for mkdir, but sorry I should have been more clear - I wish to run a series of commands for each file. You see, for my similar routine later I wish to generate an output filename based on the input filename (which involves stripping the .ogg extension and adding .mp3) and then use these multiple variables in my pipline when invoking gst-launch. –  samjetski Dec 31 '09 at 8:14
add comment

The issue you're encountering is the for statement is responding to the find as separate arguments. The space delimiter. You need to use bash's IFS variable to not split on space.

Here is a link that explains how to do this.

The IFS internal variable

One way around this problem is to change Bash's internal IFS (Internal Field Separator) variable so that it splits fields by something other than the default whitespace (space, tab, newline), in this case, a comma.

#!/bin/bash
IFS=$';'

for I in `find -type d -printf \"%P\"\;`
do
   echo "== $I =="
done

Set your find to output your field delimiter after the %P and set your IFS appropriately. I picked semi-colon since it's highly unlikely to found in your filenames.

The other alternative is to call mkdir from the find directly via -exec do you can skip the for loop altogether. That's if you don't need to do any additional parsing.

share
    
What if the filename contains the IFS? Then you have to choose a different one. But then, what if... –  Dennis Williamson Dec 30 '09 at 23:34
3  
You can pick / on POSIX, and : on DOS filesystems. There are illegal characters for different filesystems that you can pick for the IFS. Anything more complicated and you're better off using perl. –  Darren Hall Dec 30 '09 at 23:39
2  
The problem with using / is that it's the directory delimiter and find returns filenames with paths including a slash. Try changing the semicolon in your script to a slash and the echo will print the directory and the filename on separate lines. –  Dennis Williamson Dec 31 '09 at 1:10
    
That also looks quite useful. I've gone with the pipe to while option, but this also looks quite workable. Yes, in my similar structure later I needed to do further parsing. (The input filename would be .ogg, which would be passed as filesrc in the gst pipeline, but an equivalent ending in .mp3 based in the output directory would be generated and also passed to the pipeline as filesink, and this of course needs to be done for each file, along with some echo to the user.) –  samjetski Jan 1 '10 at 6:20
add comment

If the body of your loop is more than a single command, it is possible to use xargs to drive a shell script:

export OUTPATH=/some/where/else/
find . -type d -print0 | xargs -0 bash -c 'for DIR in "$@"; do
  printf "mkdir -p %q\\n" "${OUTPATH}${DIR}"        # Using echo for debug; working script will simply execute mkdir
  echo Created $DIR
done' -

Be sure to include the trailing dash (or some other ‘word’) if the shell is of the Bourne/POSIX variety (it is used to set $0 in the shell script). Also, care must be taken with quoting, since the shell script is being written inside a quoted string instead of directly at the prompt.

share
    
Another interesting concept. Thanks - I'm sure I'll find a use for this later :) –  samjetski Jan 1 '10 at 6:36
add comment

in your updated question you have

mkdir -p \"${OUTPATH}${DIR}\"

this should be

mkdir -p "${OUTPATH}${DIR}"
share
    
Thanks. Fixed. It was also reading to FILENAME instead of DIR - copy-paste :P –  samjetski Jan 2 '10 at 4:18
add comment

or to make the whole thing much less complicated:

% rsync -av --include='*/' --exclude='*' SRC DST

this replicates the directory structure of SRC into DST.

share
    
No, I need an iterative structure like so, which allows me to run multiple lines of code for each file. "Now, I need some way that I can iterate through like this, because I also wish to run a more complicated command involving gstreamer on each file in a following similar structure." Sorry if I was unclear. –  samjetski Jan 1 '10 at 6:27
    
the command i gave solves the problem you asked for, it does not matter if this is just a part of a bigger 'pipeline' on your side. for someone else having the problem as described in the question the rsync-approach will work. so, no need to be sorry about potential unclearity :) –  akira Jan 1 '10 at 13:10
    
Yea. No, I mean I would be using a similar while ... do ... done structure later to do similar processing from find, which would require several lines of code to be run on each file (modify string, echo, gst-launch, etc.) and rsync would not achieve this. That's why I specified that I needed to be able to run a more complicated set of commands within a similar strucutre. My script uses this loop structure twice, so for the question I posted the one with less crud in the middle. –  samjetski Jan 2 '10 at 4:26
add comment

If you have GNU Parallel http:// www.gnu.org/software/parallel/ installed you can do this:

find . -type d | parallel echo making {} ";" mkdir -p /tmp/outdir/{} ";" echo made {}

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

share
add comment
find . -type d -exec mkdir -p "{}\040" ';' -exec echo "Created {}\040" ';'
share
add comment

This site is currently not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .