Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Okay, I give up. How do I size limit which files are downloaded, like say I don't want any files bigger than 2 MB?

share|improve this question

3 Answers 3

up vote 5 down vote accepted

The only limitation option I know which wget supports is the -Q switch for quota. This is not what you want though, as it will stop after a combined limit of all files you've downloaded, not individually. Piping each link to it seperately with the -Q switch won't work either, as explained in the man page.

I don't know what environment you're using, but crawler supports file size limitations with max-length-bytes and runs on the Java platform.

from their user manual:

  • max-length-bytes

Maximum number of bytes to download per document. Will truncate file once this limit is reached.

By default this value is set to an extremely large value (in the exabyte range) that will never be reached in practice.

share|improve this answer
    
Hmmm. Okay. That reiterated a lot of what I found out but good answer anyway. I didn't know that Heritrix truncated files instead of skipping them, though. –  Nathaniel Mar 18 '10 at 22:28

If its about "downloading 2MB max" rather than "download files with max 2MB" you could just limit the output saved to disk.

wget -O - $url |head -c 1024 (with an optional > $SaveAsFile) -> saves the first KB and the rest gets truncated.

(enough to see a "OK:$Message", not killing my /tmp with tons of error messages from the remote ;-))

share|improve this answer

This possible with help of 3rd-party patches: http://yurichev.com/wget.html

share|improve this answer
    
Review this post, you must. –  KronoS Apr 3 '13 at 22:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.