Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have several text files I'm wanting to grab from a web server at school and save them to my my local using a command line interface. This is not a homework problem, just me trying to do something tedious in an efficient way. So I have two things I'm trying to do:

  1. Recursively traverse a directory on a web server looking for some text file. I'm not sure if I'd be able to do this straight from the command line without writing some small custom app, but any ideas would be appreciated.

  2. Copy the text file and save on my local disk. I found this link, but I haven't been able to figure out how to cat it to a file or save it locally.

share|improve this question
    
do you not know of the URL of the file beforehand? How will you know when you have the text file you want? –  qedi Nov 14 '09 at 17:37
    
I know the file URL ahead of time. A have a professor that posts a bunch of notes in text files on his website and periodically I'd like an efficient means of going out and grabbing a large set of them. –  LJM Nov 14 '09 at 17:54

2 Answers 2

up vote 4 down vote accepted

wget is a good tool, and can recursively grab files from a webserver.

  wget -r 'http://url/'
share|improve this answer

To fetch files from the command line in linux there are two basic tools.

  1. WGet
  2. cURL

It is important what makes a file interesting for you to get it.

If its something to do with its path or name, things are easier and you can get around with just 'crawling' the site.

But, if you need to get a file because it contains something of interest for which you will search its contents (and you have not indexing for this -- like, say the Google search), then you will need to get each file and check it for matching your interests to retain or else delete (the local copy).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.