Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have several text files I'm wanting to grab from a web server at school and save them to my my local using a command line interface. This is not a homework problem, just me trying to do something tedious in an efficient way. So I have two things I'm trying to do:

  1. Recursively traverse a directory on a web server looking for some text file. I'm not sure if I'd be able to do this straight from the command line without writing some small custom app, but any ideas would be appreciated.

  2. Copy the text file and save on my local disk. I found this link, but I haven't been able to figure out how to cat it to a file or save it locally.

share|improve this question
do you not know of the URL of the file beforehand? How will you know when you have the text file you want? – qedi Nov 14 '09 at 17:37
I know the file URL ahead of time. A have a professor that posts a bunch of notes in text files on his website and periodically I'd like an efficient means of going out and grabbing a large set of them. – LJM Nov 14 '09 at 17:54
up vote 5 down vote accepted

wget is a good tool, and can recursively grab files from a webserver.

  wget -r 'http://url/'
share|improve this answer

To fetch files from the command line in linux there are two basic tools.

  1. WGet
  2. cURL

It is important what makes a file interesting for you to get it.

If its something to do with its path or name, things are easier and you can get around with just 'crawling' the site.

But, if you need to get a file because it contains something of interest for which you will search its contents (and you have not indexing for this -- like, say the Google search), then you will need to get each file and check it for matching your interests to retain or else delete (the local copy).

share|improve this answer

A simple way to download webpage recursively.

wget -r -p -k -c -np URL

say if you want to omit zip, tar.gz and rar files

wget -r -p -k -c -np -R zip,gz,rar URL

or you could use httrack, which has GUI (native and web) for configuration (like filter rules).

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .