How to parse a webpage into a text or excel file?

Question

I'm trying to figure out a way to grab 4 - 5 lines of text from multiple pages of the same site, the information is always in the same place in the webpage. How would I go about getting that information into a text or preferably, excel file?

Answer 1

traditionally, you download the pages HTML, and look for some constant that immediately precedes your data. unfortunately your question does not have a universal answer since every site is different. I've written scripts in python, Bash, and even C# that pull down a page, search it for the tag I want, and extract it.

Take a look at the pages source, and find your peice of data (firebug is very helpful for this), and determine the start and end delimiters for the content you want to extract.

here are some scripts people have written to scrape XKCD.com http://forums.xkcd.com/viewtopic.php?f=11&t=63037

Answer 2

You can try to scrape a page using Javascript (in form of a Greasemonkey script / Userscript / an extension / a bookmarklet / ...). The bookmarklet means that you have to jump manually to that page and execute it. The other methods allow for passing the data straight to your database (local storage when using a privileged extension or a web server).

Writing Javascript code requires knowledge on that field, you would look for DOM functions like document.getElementById, getElementsByClassName and querySelector or XPath. Since Etsy uses jQuery already, you can also use jQuery for getting the data.

As an alternative to scraping, consider using an API if available. A quick search revealed the following pages which may (or may not) help you:

• http://www.etsy.com/developers/documentation
• http://www.etsy.com/apps/

Answer 3

This depends a lot on what area your expertise lies in but in the past for this kind of operation I usually use PHP Simple HTML DOM Parser. It's really simple to use and the documentation is quite informative.

The syntax for querying the DOM is fairly similar to jQuery if you've used that before.

You can use the more verbose find('div[id=foo]'); and find('div[class=bar]'); but can also use the more compact jQuery-like method of defining selectors with find('#foo'); and find('.bar');.

I personally use Chrome Inspector to identify the elements that I want to query to find their id, class, tag etc.

Be aware that there is a small chance that you may encounter memory errors depending on the size of the file you intend to load into memory but if the page is similar to the one in your screenshot above you shouldn't have a problem.