wget connection reset by peer

Question

I am using the following code to download a list of pdf files:

wget -i list.txt -A .pdf

Some pdf files are downloaded properly. However, some pdf files are not downloaded properly. When I check the log, I see the following report:

--2013-04-09 11:25:42--  http://amazon.com/111.pdf
Reusing existing connection to amazon.com:80.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Saving to: `111.pdf'


    [                                         <=>       ] 1,045       --.-K/s   in 2m 9s


2013-04-09 11:27:51 (8.11 B/s) - Read error at byte 1045 (Connection reset by peer).Retrying.


--2013-04-09 11:27:52--  (try: 2)  http://amazon.com/111.pdf
Connecting to amazon.com (amazon.com)|00.00.55.888|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2680728 (2.6M) [application/pdf]
Saving to: `111.pdf'


61% [==============================>                    ] 1,649,221   10.0K/s   in 2m 41s


2013-04-09 11:30:41 (10.0 KB/s) - Read error at byte 1649221/2680728 (Connection reset by peer). Retrying.


--2013-04-09 11:30:43--  (try: 3)  http://amazon.com/111.pdf
Connecting to amazon.com (amazon.com)|00.00.55.888|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2680728 (2.6M) [application/pdf]
Saving to: `111.pdf'


100%[==================================================>] 2,680,728   10.1K/s   in 4m 22s


2013-04-09 11:35:11 (10.0 KB/s) - `111.pdf' saved [2680728/2680728]

I wonder why I can not open the pdf file 111.pdf? The above report says that it is 100% downloaded. Is it because of the connection reset by the peer?

I wonder if it is possible to put the wget in a loop for every file, in such a way that it does not exit the loop, until the download is done with no error? I found the following loop, however, it gives an error. The code and the error is shown below:

Code:

while read -r link
do
        wget -A .pdf
        until [ $? = 0 ]
        do
            wget -A .pdf
        done
done < ./list.txt

Error:

Try `wget --help' for more options.
wget: missing URL
Usage: wget [OPTION]... [URL]...

I am using Cygwin on Windows.

Please let me know if you have other suggestions as well.

Thank you for your help.

Answer 1

Closed connections

Connection closed by peer usually means that the connection was deliberately closed by the target machine (or any other on the trace route).

Combined with the fact that the download works well in your browser, I suspect that the server is configured to forbid or slow down batch downloads and/or downloaders.

Corrupt file

In the first attempt, only 1,045 got downloaded and the headers state that the length was unspecified. I suspect that the first part of the file is actually some sort of error message (e.g., you have to wait X seconds between downloads).

Possible solutions

• Spoof the user agent of a major browser:

  --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22"
  

• Set reasonable or random waiting times between two downloads:

  --wait=15
  --random-wait