Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using the following code to download a list of pdf files:

wget -i list.txt -A .pdf

Some pdf files are downloaded properly. However, some pdf files are not downloaded properly. When I check the log, I see the following report:

--2013-04-09 11:25:42--
Reusing existing connection to
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Saving to: `111.pdf'

    [                                         <=>       ] 1,045       --.-K/s   in 2m 9s

2013-04-09 11:27:51 (8.11 B/s) - Read error at byte 1045 (Connection reset by peer).Retrying.

--2013-04-09 11:27:52--  (try: 2)
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2680728 (2.6M) [application/pdf]
Saving to: `111.pdf'

61% [==============================>                    ] 1,649,221   10.0K/s   in 2m 41s

2013-04-09 11:30:41 (10.0 KB/s) - Read error at byte 1649221/2680728 (Connection reset by peer). Retrying.

--2013-04-09 11:30:43--  (try: 3)
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2680728 (2.6M) [application/pdf]
Saving to: `111.pdf'

100%[==================================================>] 2,680,728   10.1K/s   in 4m 22s

2013-04-09 11:35:11 (10.0 KB/s) - `111.pdf' saved [2680728/2680728]

I wonder why I can not open the pdf file 111.pdf? The above report says that it is 100% downloaded. Is it because of the connection reset by the peer?

I wonder if it is possible to put the wget in a loop for every file, in such a way that it does not exit the loop, until the download is done with no error? I found the following loop, however, it gives an error. The code and the error is shown below:


while read -r link
        wget -A .pdf
        until [ $? = 0 ]
            wget -A .pdf
done < ./list.txt


Try `wget --help' for more options.
wget: missing URL
Usage: wget [OPTION]... [URL]...

I am using Cygwin on Windows.

Please let me know if you have other suggestions as well.

Thank you for your help.

share|improve this question
Are the file you can and can't download on the same server? What happens if you try to download them in your web browser? – Dennis Apr 11 '13 at 21:20
@Dennis When I download them in my browser, it seems flawless and I do not see anything strange. However, when I enter the URL (.com/111.pdf) in my browser (chrome), it does not open the pdf file in the browser, but it downloads the pdf file. The links seem to be on one server, however, I am not sure whether the server address is the same as the URL address. – Si14 Apr 12 '13 at 0:45
Any reason you're using wget from cygwin and not wget for Windows? – Karan Apr 12 '13 at 21:05
@Karan No particular reason. I am using Win 7 (64 bit) and I can use wget for Windows as well. – Si14 Apr 13 '13 at 1:49
up vote 4 down vote accepted

Closed connections

Connection closed by peer usually means that the connection was deliberately closed by the target machine (or any other on the trace route).

Combined with the fact that the download works well in your browser, I suspect that the server is configured to forbid or slow down batch downloads and/or downloaders.

Corrupt file

In the first attempt, only 1,045 got downloaded and the headers state that the length was unspecified. I suspect that the first part of the file is actually some sort of error message (e.g., you have to wait X seconds between downloads).

Possible solutions

  • Spoof the user agent of a major browser:

    --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22"
  • Set reasonable or random waiting times between two downloads:

share|improve this answer
Thank You. This solves the problem. – Si14 Apr 15 '13 at 19:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .