1

I am trying to download a bunch of (lets say .txt files) from a website, now they're not very big (about 4-10kbs each) but there are about 70000 of them.

All I am after is the .txt, I don't want to clone the site like HTTrack etc...

Are there any programs out there which would help me out with this problem? ( and save me a ton of time!)

Cheers

9

2 Answers 2

0

If you're OK with using a browser, you can use Firefox with the DownThemAll extension. You'll need to browse to the root of the directory and launch DownThemAll - you can filter just .txt files on the selection screen.

3
  • I'm not sure downthemall would be a good solution if they are spread out through a site because the OP has stated there are 700 txt files. Unless there is some option I am unfamiliar with please edit your post with an example. Mar 21, 2013 at 13:31
  • Wouldn't this take me just as long? I would have to vist each page (all 50000 of them)?
    – xencored
    Mar 21, 2013 at 13:32
  • 1
    My apologies, I didn't realise they were spread across multiple pages. DTA would have worked if the files in question were centrally indexed/listed. Mar 21, 2013 at 13:34
0

Use the free HTTrack website copier. It lets you configure e.g. link depth, max file size, bandwith, include and exclude links etc. "It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure."

2
  • So how would the OP use HTTrack to download only the TXT files spread across the site? He has clearly stated he doesn't want to mirror the entire site.
    – Karan
    Mar 21, 2013 at 18:15
  • You're right, he cannot 'just' download the .txt because he needs the 'intermediate' .html too to gather the links. But that's as easy as afterwards "del *.html /s" (and a few more) or an xcopy of all *.txt files to another location. I've done this plenty times for other file types.
    – Jan Doggen
    Mar 22, 2013 at 7:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .