Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to figure out a way to grab 4 - 5 lines of text from multiple pages of the same site, the information is always in the same place in the webpage. How would I go about getting that information into a text or preferably, excel file?

See image for example

share|improve this question
    
Try Javascript? –  Lekensteyn Mar 7 '13 at 20:41
    
@Lekensteyn I was under the impression that Javascript couldn't pull webpages that weren't from the same domain. I don't own the server this page is on. –  Cistoran Mar 7 '13 at 20:56
    
You can use Greasemonkey / a userscript / an extension / a bookmarklet to access the page. –  Lekensteyn Mar 7 '13 at 23:01
    
check out this answer on SO: stackoverflow.com/questions/8798260/… –  Peter Albert Mar 11 '13 at 7:42

3 Answers 3

traditionally, you download the pages HTML, and look for some constant that immediately precedes your data. unfortunately your question does not have a universal answer since every site is different. I've written scripts in python, Bash, and even C# that pull down a page, search it for the tag I want, and extract it.

Take a look at the pages source, and find your peice of data (firebug is very helpful for this), and determine the start and end delimiters for the content you want to extract.

here are some scripts people have written to scrape XKCD.com http://forums.xkcd.com/viewtopic.php?f=11&t=63037

share|improve this answer

You can try to scrape a page using Javascript (in form of a Greasemonkey script / Userscript / an extension / a bookmarklet / ...). The bookmarklet means that you have to jump manually to that page and execute it. The other methods allow for passing the data straight to your database (local storage when using a privileged extension or a web server).

Writing Javascript code requires knowledge on that field, you would look for DOM functions like document.getElementById, getElementsByClassName and querySelector or XPath. Since Etsy uses jQuery already, you can also use jQuery for getting the data.

As an alternative to scraping, consider using an API if available. A quick search revealed the following pages which may (or may not) help you:

share|improve this answer

This depends a lot on what area your expertise lies in but in the past for this kind of operation I usually use PHP Simple HTML DOM Parser. It's really simple to use and the documentation is quite informative.

The syntax for querying the DOM is fairly similar to jQuery if you've used that before.

You can use the more verbose find('div[id=foo]'); and find('div[class=bar]'); but can also use the more compact jQuery-like method of defining selectors with find('#foo'); and find('.bar');.

I personally use Chrome Inspector to identify the elements that I want to query to find their id, class, tag etc.

Be aware that there is a small chance that you may encounter memory errors depending on the size of the file you intend to load into memory but if the page is similar to the one in your screenshot above you shouldn't have a problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.