Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to download a bunch of (lets say .txt files) from a website, now they're not very big (about 4-10kbs each) but there are about 70000 of them.

All I am after is the .txt, I don't want to clone the site like HTTrack etc...

Are there any programs out there which would help me out with this problem? ( and save me a ton of time!)

Cheers

share|improve this question
4  
Try wget for example. You should include further info into your question, how one can determine the names of the txt-files. And which OS you are using. –  mpy Mar 21 '13 at 12:58
    
Sorry I am using Windows 8 atm (forgot to say in the above) The .txt files are also on random pages within the site (also they have random names) I should of said this also in the above... sorry and thanks for the reply –  xencored Mar 21 '13 at 13:07
    
There's wget for Windows. –  Daniel Beck Mar 21 '13 at 13:16
1  
visit wget man page, gnu.org/software/wget/manual/wget.html, and look for acclist. –  Matt Mar 21 '13 at 13:18
    
Thanks I'll have a look, I was after something like this for files other than images exisoftware.com/news/download-all-images-from-a-website.html –  xencored Mar 21 '13 at 13:20

2 Answers 2

If you're OK with using a browser, you can use Firefox with the DownThemAll extension. You'll need to browse to the root of the directory and launch DownThemAll - you can filter just .txt files on the selection screen.

share|improve this answer
    
I'm not sure downthemall would be a good solution if they are spread out through a site because the OP has stated there are 700 txt files. Unless there is some option I am unfamiliar with please edit your post with an example. –  Matt Mar 21 '13 at 13:31
    
Wouldn't this take me just as long? I would have to vist each page (all 50000 of them)? –  xencored Mar 21 '13 at 13:32
1  
My apologies, I didn't realise they were spread across multiple pages. DTA would have worked if the files in question were centrally indexed/listed. –  Craig Watson Mar 21 '13 at 13:34
    
No worries thanks for the reply tho! –  xencored Mar 21 '13 at 13:36

Use the free HTTrack website copier. It lets you configure e.g. link depth, max file size, bandwith, include and exclude links etc. "It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure."

share|improve this answer
    
So how would the OP use HTTrack to download only the TXT files spread across the site? He has clearly stated he doesn't want to mirror the entire site. –  Karan Mar 21 '13 at 18:15
    
You're right, he cannot 'just' download the .txt because he needs the 'intermediate' .html too to gather the links. But that's as easy as afterwards "del *.html /s" (and a few more) or an xcopy of all *.txt files to another location. I've done this plenty times for other file types. –  Jan Doggen Mar 22 '13 at 7:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.