Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a folder full of HTML files:

001.htm
002.htm
003.htm
…

I want to run Pandoc on them to convert them to similarly named Markdown files:

001.md
002.md
003.md

This command works on one of them:

pandoc -f html -t markdown 001.htm -o 001.md

And I want to use find and xargs to automatically run a similar command on every file in the folder.

I got as far as this:

find *.htm | xargs -I {} -n 1 pandoc -f html -t markdown -o {}

…which truncates every file in the directory, so now I'm asking before I really break something.

What is wrong with my command above, and/or what's a completely different / more efficient way to do this?

share|improve this question
up vote 3 down vote accepted

I managed to do that with this 1 liner. If you are flexible about the xargs and find part.

for f in ./*.htm; do pandoc -f html -t markdown "$f" -o "${f%.htm}.md"; done

If you want to act recursively (so: all .htm files in the current directory, and all subdirectories), then (assuming bash 4+) you can use the globstar shell option:

shopt -s globstar
for f in ./**/*.htm; do pandoc -f html -t markdown "$f" -o "${f%.htm}.md"; done
share|improve this answer
2  
+1. xargs doesn't allow you the same flexibility of filename modification. Do not replace *.htm with $(find...) -- filenames with spaces will be properly handled in the first case but not the second. – glenn jackman Jan 25 '13 at 19:35
1  
@glennjackman Unless you set the bash $IFS to $'\n' for that code section, in which case spaces aren't a problem -- newlines still are though. – Daniel Beck Jan 25 '13 at 20:05
    
Wow, there are two or three new things about the command line for me to learn from that snippet. Thanks! – 75th Trombone Jan 25 '13 at 23:44

Using {} isn't flexible enough for some situations. This appears to be one of those.

A possible workaround would be to -execa script from find, like so:

find . -name '*.htm' -exec ./convert-to-md.sh {} \;

The script file should look similar to this, depending on the exact pandoc command line:

#!/bin/bash
pandoc -f html -t markdown -o "${1/.htm/.md}" "${1}"

If you don't want to create and save a script file for this, you can always inline the bashscript code:

find . -name '*.htm' | xargs -n 1 bash -c 'pandoc -f html -t markdown -o "${1/.htm/.md}" "${1}"' -

The additional -at the end serves to fill $0in bash, which usually includes the name of the shell script, positional arguments starting at $0.

This allows you to keep using find(even with -print0and xargs -0 if you're handling weird file names), but doesn't require creation of a separate file.

share|improve this answer
    
Doesn't find handle weird file names on its own anyway? IIRC there's never a good reason to use find … -print0 | xargs -0 … – slhck Jan 25 '13 at 19:58
    
@slhck Newlines are valid file name components. The following demonstrates how this causes scripts to fail: touch "$( echo -e 'foo\nbar' )" ; find . -name 'foo*bar' | xargs -n 1 echo File: – Daniel Beck Jan 25 '13 at 20:00

You appear to be missing a {} in the pandoc command

find . -name \*.htm | xargs -I {} -n 1 pandoc -f html -t markdown {} -o {}.md

But then you'll have files named 001.htm.md -- you'll have to decide if this is a problem.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .