Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to mirror a very large site but wget never seems to finish properly. I am using the command:

wget -r -l inf -nc -w 0.5 {the-site}

I have downloaded a good portion of the site, but not the whole thing. The content does not change fast enough to bother using time-stamping.

After running overnight, this message appears:

File `{filename}.html' already there; not retrieving.
File `{filename}.html' already there; not retrieving.
File `{filename}.html' already there; not retrieving.
File `{filename}.html' already there; not retrieving.
Killed

Does anyone know what is happening and how I can fix it?

share|improve this question

Have you tried using the '-m' option?
it is a short cut for,

-N -r -l inf --no-remove-listing

You can also experiment specifically with the site using a deeper URL for a limited set of files and avoid fetching parent paths with,

-np

share|improve this answer
    
I have not tried -m because it is not compatible with -nc and i dont want to overload the servers by downloading every single page when I already have most of them. I have also tried -np and it made little difference for this site. I will try to use a series of deeper urls. thanks. – Evan Gill Jun 15 '10 at 17:16
    
@Evan: -m works in a similar way to -nc -- it will not re-download files unless the server has a newer version. Most web servers support those checks. – grawity Jun 15 '10 at 20:19
    
Thanks. I have written up a script to use the -m flag and wget each level 2 directory separately, sleeping between each directory. I'll run it tonight and update the question/answers. – Evan Gill Jun 15 '10 at 20:36
    
You might want to look at the -w option too. – nik Jun 16 '10 at 2:27
    
Downloading each subdirectory separately solved the problem. I tried to get the entire site with one large recursive wget but it always used my entire 4GB of memory. In the end, I got everything 2 levels deep with -l 2, and ran a for loop to perform a recursive wget on all the directories in the site with this command: wget -m -w 1 --random-wait -np www.fakesite.com/directory – Evan Gill Jun 22 '10 at 16:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .