i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able to get specific data like links or, in my case, products.
all of the products on my site have the following naming convention, website.com/uniqueAlphaNumericID.html
as far as i know, no dynamic content generation is being used and only one page per one item in the above format.
should i just be thinking about:
wget website.com | grep *.html
or should i be looking into spiders/crawlers?

  • It's not clear what your question is. What exactly are you trying to do? – Michelle Six Mar 7 '11 at 8:56
  • can i use a web-spider program myself on a web-page, or is this terminology used for only the search-engine itself getting data from pages? can i use a web-crawler program on mac osx, or any other OS for that matter and what is one you would recommend? what i am trying to do is find different ways to pull data from a website – fightermagethief Mar 7 '11 at 10:12

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.