most recent 30 from superuser.com 2010-03-16T13:42:54Z http://superuser.com/feeds/question/70559 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://superuser.com/questions/70559/saving-a-text-file-from-web-to-local-from-linux-command-line 0 L. Moser http://superuser.com/users/16265 2009-11-14T17:25:18Z 2009-11-14T17:36:52Z

<p>I have several text files I'm wanting to grab from a web server at school and save them to my my local using a command line interface. This is not a homework problem, just me trying to do something tedious in an efficient way. So I have two things I'm trying to do:</p> <ol> <li><p>Recursively traverse a directory on a web server looking for some text file. I'm not sure if I'd be able to do this straight from the command line without writing some small custom app, but any ideas would be appreciated.</p></li> <li><p>Copy the text file and save on my local disk. I found <a href="http://superuser.com/questions/56109/launch-files-directories-using-gnome-from-command-line" title="this link">this link</a>, but I haven't been able to figure out how to cat it to a file or save it locally.</p></li> </ol>

http://superuser.com/questions/70559/saving-a-text-file-from-web-to-local-from-linux-command-line/70562#70562 2 nik http://superuser.com/users/263 2009-11-14T17:34:56Z 2009-11-14T17:34:56Z

<p>To fetch files from the command line in linux there are two basic tools.</p> <ol> <li><a href="http://www.gnu.org/software/wget/" rel="nofollow">WGet</a></li> <li><a href="http://curl.haxx.se/" rel="nofollow">cURL</a></li> </ol> <p>It is important <em>what</em> makes a file interesting for you to get it. </p> <p>If its something to do with its <em>path</em> or <em>name</em>, things are easier and you can get around with just 'crawling' the site. </p> <p>But, if you need to get a file because it <em>contains something of interest</em> for which you will search its contents (and you have not indexing for this -- like, say the Google search), then you will need to get each file and check it for matching your interests to retain or else delete (the local copy).</p>

http://superuser.com/questions/70559/saving-a-text-file-from-web-to-local-from-linux-command-line/70563#70563 3 qedi http://superuser.com/users/17134 2009-11-14T17:36:52Z 2009-11-14T17:36:52Z

<p>wget is a good tool, and can recursively grab files from a webserver.</p> <pre><code> wget -r 'http://url/' </code></pre>