Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In a directory there are many folders .html inside, I would like all HTML parsed as new .txt with the name of the parent directory.

Example1/ Index.html>Example1.txt

Example2/ Index.html>Example2.txt

share|improve this question
How would you like the tags removed? – suspectus May 9 '13 at 7:01
Something like this works with individual files using sed, cat file | sed 's/<b>.*</b>//g' – z4nb0t May 9 '13 at 7:19
@z4nb0t it's generally accepted that using regex to parse HTML will lead to elder gods from before the start of time awakening from their eternal slumber to consume your computer and all of humanity. – evilsoup May 9 '13 at 10:50
@evilsoup: Haha, that's a great one. – mpy May 9 '13 at 11:09
up vote 2 down vote accepted

Obviously, you want to convert some HTML pages into plain text. Therefore I wouldn't strip the tags with a custom built solution (e.g. with some sed magic), but use a tool designed for that purpose like html2text; from its webpage:

html2text is a Python script that converts a page of HTML into clean, easy-to-read plain ASCII text. Better yet, that ASCII also happens to be valid Markdown (a text-to-HTML format).

To address your question of batch renaming:

find . -maxdepth 1 -type d -print0 | while IFS= read -r -d '' dirname
 do python path/to/html2text/ "${dirname}/index.html" > "${dirname}/${dirname}.txt"

Here the find command lists all directories only located in the current directory (i.e. not recursive) and the read command (in the while condition) assigns the values to the variable $dirname. Finally, the command(s) between do and done get(s) executed, here it converts the files according to your request. As pointed out by @slhck you need to use such a complex command, so that dirnames with whitespaces won't break anything.

[Edit]: Another variant to convert all HTML files under the current directory:

find . -iname "*.html" -print0 | while IFS= read -r -d '' filename
  do python path/to/html2text/ "${filename}" > "${filename%.*}.txt"

iname searches case insensitive for *.html.

${filename%.*}.txt strips the extention and appends .txt, i.e. if filename is some/path/index.html, ${filename%.*} is some/path/index and finally ${filename%.*}.txt is some/path/index.txt.

When you use the Z shell, you can use a much cleaner for loop, without braking at white spaces:

for i (*(/)) python path/to/html2text/ "${i}/index.html" > "${i}/${i}.txt"

The trick here is that *(/) does filename generation, but only returns directories (/).

[Edit]: Also in zsh syntax the variant to convert all HTML files under the current directory (you need the option EXTENDEDGLOB to be set):

for i ((#i)**/*.html) {
   python path/to/html2text/ "$i" > "${i:r}.txt"

(#i) uses case insensitive globbing, ** searches recursively, hence returning all HTML files under the current working directory. (If symbolic links should be followed, use three stars *** instead of two).

If you have more than one command inside the for loop, use curly { ... }brackets (unnecessary here, but they won't hurt).

${i:r} strips the extention (r for remove) from the variable $i.

share|improve this answer
@z4nb0t: Sorry, I forgot the redirection > in my first version. Now it should work as stated. (The error was, that html2text expects as an optional second parameter the encoding of the HTML page, but got the name of the txt file). – mpy May 9 '13 at 7:56
i just noticed some folders have multiple html files, is this going to be a problem? – z4nb0t May 9 '13 at 7:57
@z4nb0t: It's no problem, the command given does exactly what you asked for and takes only index.html in every folder. It's easy to use a second for loop to loop also over all html files in the dirs. But you need to specify how the renaming should be done then. – mpy May 9 '13 at 7:59
This breaks when file names or paths contain whitespace. You should use the exec option of find or pipe its output into a while loop (but only with the -print0 option. See: – slhck May 9 '13 at 8:25
@z4nb0t: zsh is very powerful, but of course you need some training time. If you are interested, I really recommend by Peter Stephenson, the current maintainer of zsh. Don't get frightened by its size, IMHO it is easy to read (at least the first chapters ;) – mpy May 9 '13 at 10:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .