Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able to get specific data like links or, in my case, products.
all of the products on my site have the following naming convention, website.com/uniqueAlphaNumericID.html
as far as i know, no dynamic content generation is being used and only one page per one item in the above format.
should i just be thinking about:
wget website.com | grep *.html
or should i be looking into spiders/crawlers?

share|improve this question
    
It's not clear what your question is. What exactly are you trying to do? –  Michelle Six Mar 7 '11 at 8:56
    
can i use a web-spider program myself on a web-page, or is this terminology used for only the search-engine itself getting data from pages? can i use a web-crawler program on mac osx, or any other OS for that matter and what is one you would recommend? what i am trying to do is find different ways to pull data from a website –  bboyreason Mar 7 '11 at 10:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.