Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm looking for an application/tool that will allow me to spider a website where you have to scroll down vertically repeatedly for the whole page to load. I need to be able to define the starting URL, and define rules to require or disallow character patterns as to what URLs are saved to a file.

This task could be done with a VBScript (which runs natively under Windows) as is seen here: http://www.example-code.com/vbscript/spider.asp - but only for pages that completely load without the user scrolling down.

After brief searching apparently the 'load more content after scrolling down effect' can be implemented by a website with javascript, or jquery, and probably other methods. The application needs to to handle whatever is commonly used so the whole page loads (within a user-defined limit if an extensive amount of material would load.)

The application/tool needs to be freely available. (For example, VBScript runs native to Windows, you just save a text file as .vbs and execute it.)

The application/tool also needs to be able to log in to the website (https://) with a user name and password.

share|improve this question
    
The load content on scroll is most likely done via AJAX. It's not hard sto implement but i don't know VBS. In python there are modules for this. –  Gunnish Feb 20 '13 at 14:44
    
Since there are many different methods of implementing loading content on scroll, you would have to implement many different methods for scraping it, too. Some sites will fall-back to pure HTML gracefully and just load all the content in one page load when it sees your 'client' hasn't got JavaScript. Others will probably show you NONE of the dynamically loaded content. So there's no simple answer here. –  deed02392 Feb 20 '13 at 15:07
    
@deed02392 - Okay. What application/tool could I use that would just work the way Facebook implements it, for say listing a friends friend list. You have to keep vertically scrolling down to get all of it. –  user199340 Feb 20 '13 at 15:51
    
I am not sure, you will need to investigate how that is done. But I would be surprised if that kind of information couldn't be provided over the Facebook API. –  deed02392 Feb 20 '13 at 17:05
    
@deed02392 - You can't get that from Facebook FQL/API. See: developers.facebook.com/docs/reference/fql/friend - You get this message if you try: "Can only lookup for the logged in user or the logged in user's friends that are users of your app." –  user199340 Feb 20 '13 at 23:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.