I got curl runing like this:

parallel -j0 curl -x --retry 1 --connect-timeout 3 -I https://{} --output {}

which results in 1000 parallel jobs. I've already incereased ulimit -n to 4096 so -j0 in parallel command runs 1000 parallel jobs.

Now I only got this:

while true; do ls | wc -l; sleep 1; done                                                               

Which equals around 120 - 200 request per second.

I got NVMe drives - my suspect is DNS resolver - I got local Bind DNS resolver with DNS top but with no useful results.

I've dumped cache of my local BIND and 80% of domains which will be requested are there - so maybe I am looking into wrong area.

Where should I look next?

I got enough computing power - 4.6GHz 64G RAM NVMe drives.

  • fork() is far from free... (though on Linux it's significantly lighter than on Windows)... If you wrote a quick Python script (or similar) that dealt with this all in one process (or with recycled workers) you'd likely have much better results. – Attie Dec 18 '18 at 13:58
  • It's easy enough to determine where the source of the problem might be... try running dig +short ${HOST} and curl ${URL} in a loop, and use time to get a feel for which takes longer. – Attie Dec 18 '18 at 14:00
  • In addition to that, your machine cannot execute 1000 threads simultaneously, so reducing that figure dramatically may actually improve your stats.. – Attie Dec 18 '18 at 14:01
  • You could also use curl's -w flag to get information on where the time is spent within the process (excludes the aforementioned fork(), etc...) – Attie Dec 18 '18 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.