Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to mirror a very large site but wget never seems to finish properly. I am using the command:

wget -r -l inf -nc -w 0.5 {the-site}

I have downloaded a good portion of the site, but not the whole thing. The content does not change fast enough to bother using time-stamping.

After running overnight, this message appears:

File `{filename}.html' already there; not retrieving.
File `{filename}.html' already there; not retrieving.
File `{filename}.html' already there; not retrieving.
File `{filename}.html' already there; not retrieving.
Killed

Does anyone know what is happening and how I can fix it?

share|improve this question

2 Answers 2

Have you tried using the '-m' option?
it is a short cut for,

-N -r -l inf --no-remove-listing

You can also experiment specifically with the site using a deeper URL for a limited set of files and avoid fetching parent paths with,

-np

share|improve this answer
    
I have not tried -m because it is not compatible with -nc and i dont want to overload the servers by downloading every single page when I already have most of them. I have also tried -np and it made little difference for this site. I will try to use a series of deeper urls. thanks. –  Evan Gill Jun 15 '10 at 17:16
    
@Evan: -m works in a similar way to -nc -- it will not re-download files unless the server has a newer version. Most web servers support those checks. –  grawity Jun 15 '10 at 20:19
    
Thanks. I have written up a script to use the -m flag and wget each level 2 directory separately, sleeping between each directory. I'll run it tonight and update the question/answers. –  Evan Gill Jun 15 '10 at 20:36
    
You might want to look at the -w option too. –  nik Jun 16 '10 at 2:27
    
Downloading each subdirectory separately solved the problem. I tried to get the entire site with one large recursive wget but it always used my entire 4GB of memory. In the end, I got everything 2 levels deep with -l 2, and ran a for loop to perform a recursive wget on all the directories in the site with this command: wget -m -w 1 --random-wait -np www.fakesite.com/directory –  Evan Gill Jun 22 '10 at 16:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.