1

I am trying to download a list of URLs with cURL with multiple URLs per process. The following works but it returns a weird result if the xargs -L parameter is more than 1. I want to launch 8 processes with each getting 4 urls so I don't spawn too many.

curl url1...url4

cat urls.txt | xargs -n 1 -L 4 -P 8 curl -I -s -o /dev/null -w "%{http_code} %{url_effective}\n" 

The result is quite chaotic.

503 http://somewebsite.txt
404 http://somewebsite.txt
503 http://somewebsite.txt
404 http://somewebsite.txt
HTTP/1.1 404 Not Found
Server: nginx
Date: Thu, 24 Nov 2016 10:11:36 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Access-Control-Allow-Origin: *

404 http://somewebsite.txt
HTTP/1.1 404 Not Found
Server: nginx
Date: Thu, 24 Nov 2016 10:11:36 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Access-Control-Allow-Origin: *

404 http://somewebsite.txt
0

It does not look like a client side issue. It looks like a server issue, like a jsp which does not compile.

You can investigate with curl -vvv to have more information.

Plus, when issue solved, you likely want to use xargs -n 4 instead of "-n 1 -L 4".

|improve this answer|||||
  • Does -n4 concatenate lines? To "url 1 url 2 url 3 url 4" – Testr Nov 24 '16 at 10:22
  • This happens with any server btw. – Testr Nov 24 '16 at 10:27
  • Yes, "-n 4" concatenate lines. This is what you want if you want to decrease spawn. – Setop Nov 24 '16 at 12:47
  • This works for me against "en.wikipedia.org/wiki" server. Articles are downloaded and connections are reused. – Setop Nov 24 '16 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.