Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able to get specific data like links or, in my case, products.
all of the products on my site have the following naming convention,
as far as i know, no dynamic content generation is being used and only one page per one item in the above format.
should i just be thinking about:
wget | grep *.html
or should i be looking into spiders/crawlers?

share|improve this question
It's not clear what your question is. What exactly are you trying to do? – Michelle Six Mar 7 '11 at 8:56
can i use a web-spider program myself on a web-page, or is this terminology used for only the search-engine itself getting data from pages? can i use a web-crawler program on mac osx, or any other OS for that matter and what is one you would recommend? what i am trying to do is find different ways to pull data from a website – fightermagethief Mar 7 '11 at 10:12

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.