How to find bottleneck for parallel curl command?

Asked 11 months ago

Viewed 21 times

I got curl runing like this:

parallel -j0 curl -x 10.9.8.4:8080 --retry 1 --connect-timeout 3 -I https://{} --output {}

which results in 1000 parallel jobs. I've already incereased ulimit -n to 4096 so -j0 in parallel command runs 1000 parallel jobs.

Now I only got this:

while true; do ls | wc -l; sleep 1; done                                                               
62040
62242
62424
62603
62764
62920

Which equals around 120 - 200 request per second.

I got NVMe drives - my suspect is DNS resolver - I got local Bind DNS resolver with DNS top but with no useful results.

I've dumped cache of my local BIND and 80% of domains which will be requested are there - so maybe I am looking into wrong area.

Where should I look next?

I got enough computing power - 4.6GHz 64G RAM NVMe drives.

asked Dec 18 '18 at 12:51

David Strejc

1017 bronze badges

fork() is far from free... (though on Linux it's significantly lighter than on Windows)... If you wrote a quick Python script (or similar) that dealt with this all in one process (or with recycled workers) you'd likely have much better results. – Attie Dec 18 '18 at 13:58
It's easy enough to determine where the source of the problem might be... try running dig +short ${HOST} and curl ${URL} in a loop, and use time to get a feel for which takes longer. – Attie Dec 18 '18 at 14:00
In addition to that, your machine cannot execute 1000 threads simultaneously, so reducing that figure dramatically may actually improve your stats.. – Attie Dec 18 '18 at 14:01
You could also use curl's -w flag to get information on where the time is spent within the process (excludes the aforementioned fork(), etc...) – Attie Dec 18 '18 at 14:02

add a comment |

0 Your Answer

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy