I am using Gatling to load test a spring application build using Spring webflux and mongo databse. My API just takes a string parameter and does a get call on the mongo reactive repository. My application is deployed in Google cloud GKE and GCE (mongo DB).
I am running gatling as a docker container using image https://github.com/denvazh/gatling . Gatling test access the API end point over internet using public IP.
Test works fine till the load 28000 concurrent users, when I inject 30K users at once , I get 'java.net.BindException: Address not available'. Can someone please give some idea on the issue.
I checked FD limit on both host machine and docker container , looks fine for me. I am pasting ulimit output at the end. I get below excpetion on gatling script stdout.
Request: getEligibilityMVCSimulation: KO i.n.c.AbstractChannel$AnnotatedSocketException: Address not available: /some_ip:80 Session: Session(getEligibilityMVCSimulation,30975,1556538559546,Map(gatling.http.ssl.sslContexts -> SslContexts(io.netty.handler.ssl.OpenSslClientContext@3ff8fa07,None), gatling.http.cache.dns -> io.gatling.http.cache.DnsCacheSupport$$anon$1@5273a181, gatling.http.cache.baseUrl -> http://localhost:8182/eligibility-service/v1/),0,KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$435/194107588@4249260d)
HTTP request: GET http://some_ip/eligibility-service/v1/8672 headers= Accept: application/json, text/javascript, /; q=0.01 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 origin: http://some_ip host: some_ip
HTTP response:
11:49:30.614 [WARN ] i.g.h.e.GatlingHttpListener - Request 'getEligibilityMVCSimulation' failed for user 30983
java.net.BindException: Address not available
... 34 common frames omitted
Wrapped by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Address not available: /some_ip
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254)
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:497)
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:298)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512)
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024)
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:259)
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
11:49:30.614 [WARN ] i.g.h.e.r.DefaultStatsProcessor - Request 'getEligibilityMVCSimulation' failed for user 30983: i.n.c.AbstractChannel$AnnotatedSocketException: Address not available: /some_ip:80
I am working on GCE VM with Debian 9 stretch. Initially ulimit -n was returning 1024 and after I followed all the steps in https://gatling.io/docs/current/general/operations/ ulimit -a is returning as 65535. But I am getting same exception.
ulimit -a gives below result,
joy123456@gatling-tests-vm-2:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 60087
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 60087
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I checked ulimit on docker container, and it gives below result,
joy123456@gatling-tests-vm-2:~$
sudo docker run -it --entrypoint sh denvazh/gatling
/opt/gatling # ulimit -a
-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) unlimited
-m: resident set size (kb) unlimited
-l: locked memory (kb) 64
-p: processes unlimited
-n: file descriptors 1048576
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0
Can someone please give some idea on the issue.
Thanks
