Question

I have several text files I'm wanting to grab from a web server at school and save them to my my local using a command line interface. This is not a homework problem, just me trying to do something tedious in an efficient way. So I have two things I'm trying to do:

1. Recursively traverse a directory on a web server looking for some text file. I'm not sure if I'd be able to do this straight from the command line without writing some small custom app, but any ideas would be appreciated.

2. Copy the text file and save on my local disk. I found this link, but I haven't been able to figure out how to cat it to a file or save it locally.

Answer 1

wget is a good tool, and can recursively grab files from a webserver.

  wget -r 'http://url/'

Answer 2

To fetch files from the command line in linux there are two basic tools.

1. WGet
2. cURL

It is important what makes a file interesting for you to get it.

If its something to do with its path or name, things are easier and you can get around with just 'crawling' the site.

But, if you need to get a file because it contains something of interest for which you will search its contents (and you have not indexing for this -- like, say the Google search), then you will need to get each file and check it for matching your interests to retain or else delete (the local copy).