If you are into DevOps, you must be aware of Kubernetes. Kubernetes is an open-source container orchestration system for automating application deployment, scaling, and management. In this short tutorial, we will know how to troubleshoot and fix the connection issue once Kubernetes master node IP has changed.
If your IP of the master node has been changed and while running `kubect get nodes`, you are getting the error “The connection to the server IP:6443 was refused – did you specify the right host or port?“, then this is the tutorial for you.
There are different methods to resolve this issue, but in this tutorial, we are telling the easiest and safest way to resolve the connection issue without resetting your Kubernetes cluster.
This issue can appear if you are not using DHCP or DNS server in your lab environment and every time you reboot the system, you might face this issue, so always use permanent IPs in your environment.
Just for your reference, we were using the below Kubernetes version and getting the below error.
[root@k8-master ~]# rpm -qa |grep -i kube kubernetes-cni-0.7.5-0.x86_64 kubeadm-1.17.4-0.x86_64 kubectl-1.17.4-0.x86_64 kubelet-1.17.4-0.x86_64
# kubectl get nodes The connection to the server 192.168.80.181:6443 was refused - did you specify the right host or port?
Steps To Resolve Connection Issue After Kubernetes Mater Server IP is Changed
1. First of all, change the IP address in all the files under /etc/kubernetes/ with your new IP of the master server and worker nodes. Listed down are the files where the IP will be present.
/etc/kubernetes/admin.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/scheduler.conf /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
Note: On worker node, you will only see “/etc/kubernetes/kubelet.conf” file.
2. Now, change the IP address on the file $HOME/.kube/config on your master server using your favorite editor. This is a very important step and if you forget this one, the master server will keep on giving you the error when you run kubectl command.
# vi $HOME/.kube/config
3. Once you have changed the IP in the above step, you will get the below certificate error while running kubectl command. If you are not getting certificate error you are good to skip this step.
[root@k8-master ~]# kubectl get nodes Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 192.168.80.159, not 192.168.80.181 [root@k8-master ~]#
Now you have to generate new certificates for apiserver and apiserver-kubelet-client located at /etc/kubernetes/pki.
Note: Please take a backup of the file before deleting them. Backup command not given below. You can use the “cp” command to backup.
cd /etc/kubernetes/pki rm -rf apiserver.* apiserver-kubelet-client.*
Now recreate those files using below commands.
# kubeadm init phase certs apiserver-kubelet-client
# kubeadm init phase certs apiserver
By default certificate will generate at /etc/kubernetes/pki/
[root@k8-master pki]# kubeadm init phase certs apiserver-kubelet-client I0408 21:41:51.332462 53981 version.go:251] remote version is much newer: v1.18.1; falling back to: stable-1.17 W0408 21:41:51.697632 53981 validation.go:28] Cannot validate kube-proxy config - no validator is available W0408 21:41:51.697705 53981 validation.go:28] Cannot validate kubelet config - no validator is available [certs] Generating "apiserver-kubelet-client" certificate and key [root@k8-master pki]# [root@k8-master pki]# kubeadm init phase certs apiserver I0408 21:42:26.566731 54171 version.go:251] remote version is much newer: v1.18.1; falling back to: stable-1.17 W0408 21:42:26.904598 54171 validation.go:28] Cannot validate kube-proxy config - no validator is available W0408 21:42:26.904623 54171 validation.go:28] Cannot validate kubelet config - no validator is available [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.80.181] [root@k8-master pki]# [root@k8-master pki]# ls /etc/kubernetes/pki/ apiserver.crt apiserver.key ca.crt front-proxy-ca.crt front-proxy-client.key apiserver-etcd-client.crt apiserver-kubelet-client.crt ca.key front-proxy-ca.key sa.key apiserver-etcd-client.key apiserver-kubelet-client.key etcd front-proxy-client.crt sa.pub
4. Now once you have generated the certificate, it is time to run kubectl command to check the nodes. We are sure, your issue must have resolved by now.
[root@k8-master pki]# kubectl get nodes NAME STATUS ROLES AGE VERSION cent Ready4h41m v1.17.4 k8-master Ready master 4h45m v1.17.4
In case you are still getting a certificate error, you can follow below series of commands:
# kubeadm init phase certs apiserver # kubeadm init phase certs apiserver-kubelet-client # systemctl stop kubelet # Delete the docker container with kubelet # docker container rm container_id # systemctl restart kubelet
Useful Tip
Sometime, you will run into the error “The connection to the server IP:6443 was refused – did you specify the right host or port?“, even without changing the IP, so that time just check the value of the “KUBECONFIG” variable in your master server. If it is not there, please use the below command to export it and you can also add it to your bash_profile if the issue is re-occurring.
export KUBECONFIG=/etc/kubernetes/admin.conf
We hope you like this solution and able to resolve the connection issue. In case of any problem or query, do let us know through your comments.
We hope you liked this article and if you do, please share it with your social network and do subscribe to our blog for more such troubleshooting articles on Kubernetes.
