Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to 'grab' data from a website.

Is there a way to just point a program at a URL and it grabs everything at that domain. How should i go about this on OSX 10.5? If really needed, I can use an ubuntu or windows box as a last resort.

share|improve this question

1 Answer 1

up vote 3 down vote accepted

You can use the wget command which is available for download for Mac OS X included with most Linux distributions to download the entire contents of a website, images, CSS, JavaScript, videos, Flash files, and all.

Once you have it, open a terminal. You probably want to call it like this:

wget -r -l0 -k http://www.example.com/

That will download everything from http://www.example.com/ accessible by links. The -r option turns on recursive downloading, so it downloads more than just the home page. The -l option sets how many pages deep it will look for links from and download, setting to 0 as I did will set it to go in as far as it can. Note that if the website uses dynamic page generation that could download a lot, as there could be many URLs that point to the same or very similar content. The -k command is optional, and will make wget convert all the links, image tags, etc. to the correct location on your local machine so you can view it with a web browser on your computer and it will work correctly.

Note that it will only download files from www.example.com, not any other domains. If you need it to go to other domains, use the -H switch to turn that on and then use the option -D switch to define the other domains to download from (e.g. -D comments.example.com,beta.example.com). Be careful, if you leave off the -D switch and set -l to 0/infinite you could very well try and download the entire World Wide Web!

Another switch that might be helpful is the -N switch, which will set the timestamp on the local file to the time provided by the HTTP Last-Modified header on the server, and not download files that haven't been changed on subsequent downloads.

For more information, consult the wget documentation.

share|improve this answer
    
OS X does in fact include curl out of the box, which does a lot of the same stuff as wget. –  NReilingh Mar 7 '11 at 4:15
    
@NReilingh: curl doesn't support recursive downloading. –  Patches Mar 7 '11 at 4:20
    
CocoaWget might be a bit easier to use, if not as flexible. –  Daniel Beck Mar 7 '11 at 6:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.