Save parts of a web site as pure text - Super User

Save parts of a web site as pure text

2009-11-23T14:41:56Z

I hope I may ask this here.

I need to extract the contents of an existing web site (in charge of the web site owner) to word (or text) documents. For this, I only need the content from one DIV with a given ID.

Is there any tool for windows that can do this for me (ideally recursively)? I know wget and Web Site Downloader but both can "only" save the complete HTML.

Answer by joshhunt for Save parts of a web site as pure text

2009-11-23T14:46:30Z

I do not think something like this already exists. I think your best option would be to code something up yourself.

BeautifulSoup is a... beautiful Python library that will let you do this in very minimal code. For more help, I suggest you head over to Stack Overflow

Answer by eleven81 for Save parts of a web site as pure text

2009-11-23T14:53:03Z

Your best bet would be to build your own toolchain for this:

Use a tool such as wget to recursively download the HTML files from which content is needed. Pay special attention to options -r to specify recursive downloading, and -l to specify depth of the recursion. wget outputs plain text.
Use a tool such as grep to filter out everything except the line(s) containing the <DIV> you need. Pay special attention to options -r to specify recursive searching, and -e to specify a regular expression. Pipe grep's output to a file of your choice. grep outputs plain text if it is fed plain text.

Hint: It may be simpler to use grep multiple times to filter out things in smaller chunks. This depends entirely on how similar all of the various pages are, and how clean the code is.

Edit: Then again, perhaps using a regex is not a good way to parse HTML.

Answer by CarlF for Save parts of a web site as pure text

2009-11-23T14:58:51Z

I'm lazy. In the time it would take you to research and set up a special-purpose tool, surely you can just highlight the required text with a mouse, copy it, and paste it into a text editor?