1

We have created a self managed kubernetes cluster using the kubeadm approach. It is a 3 node cluster hosted on AWS EC2 instances (1-Master and 2-Slaves).

We have created a separate NGINX server to route the requests for different applications deployed on K8S cluster.

This NGINX server is linked to AWS ALB along with an openssl self-signed certificate so that the Cluster is accessible only via HTTPS Listener PORT i.e. 443.

Also, we have linked one separate ALB with K8S Master to access K8S Dashboard / K8S Api Server.

We are using Jenkins to create jobs for automated deployment in K8S cluster. However, when we run the job intermittently we get TLS handshake timeout.

For instance:- if we run below command:

kubectl get pods --insecure-skip-tls-verify=true

We get an error:

Unable to connect to the server: net/http: TLS handshake timeout

On multiple re-runs the command runs successfully.

We are using the latest version of Kubernetes i.e. 1.15 version

We have been struggling to find a permanent solution to this.

How can I fix it?

4
  • Seems like a problem with your kube-apiserver. Make sure that the pod is not restarting (kubectl get po --namespace=kube-system), and take a look at the logs to find out a possible cause. Update your question with the info to get further help. This error is also susceptible if your instance is under high CPU pressure, or if there is a proxy capturing packets of your kubectl requests. Sep 18, 2019 at 13:47
  • @EduardoBaitello kube-apiserver pod is running. I checked the kube-apiserver pod log and found multiple entries of these lines - {"log":"I0919 10:38:50.217420 1 log.go:172] http: TLS handshake error from 172.22.84.126:58618: remote error: tls: bad certificate\n","stream":"stderr","time":"2019-09-19T10:38:50.217736958Z"} {"log":"E0919 10:44:50.020653 1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted\n","stream":"stderr","time":"2019-09-19T10:44:50.020957098Z"} Sep 19, 2019 at 11:05
  • Hi @user1425264 did you ever find a solution? I am unfortunately faced with the same problem with a K8S cluster in AWS. Any feedback you might have would be appreciated.
    – sbeskur
    Oct 3, 2019 at 2:45
  • @sbeskur : yes, we have resolved the time out issue Oct 4, 2019 at 9:15

1 Answer 1

0

Our Architecture is roughly as follows:-

EndUser <- ALB1 <- NGINX Server (where routing rules to access applications hosted on K8S Cluster are in place)

EndUser <- ALB2 <- K8S Master (used to access K8S Dashboard or send request to Kube API Server)

It looked like whenever a request was made to ALB2 from Jenkins Server to connect K8S Cluster at times packets were getting lost and hence we were getting intermittent time out issue.

We have replaced AWS ALB with Classic ELB and this has resolved out intermittent time-out issue.

Hope this helps.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .