Kubernetes Troubleshooting
This part introduces how to troubleshoot various problems on Kubernetes, includes
- Troubleshooting Clusters
- Troubleshooting Pods
- Troubleshooting Networking
- Troubleshooting Persistent Volumes
- Troubleshooting Windows Containers
- Troubleshooting Cloud Provider
- Tools for Troubleshooting
Remember kubectl
is always the most important tool when starting to troubleshoot any problems.
Listing Nodes
kubectl get nodes
kubectl describe node <node-name>
Listing Pods
kubectl get pods -o wide
kubectl -n kube-system get pods -o wide
Looking at Pod events
kubectl describe pod <pod-name>
Looking at kube-apiserver logs
PODNAME=$(kubectl -n kube-system get pod -l component=kube-apiserver -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system logs $PODNAME --tail 100
If kube-apiserver is not running as kubelet static Pods, then use journalctl -u kube-apiserver
command instead.
Looking at kube-controller-manager logs
PODNAME=$(kubectl -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system logs $PODNAME --tail 100
If kube-controller-manager is not running as kubelet static Pods, then use journalctl -u kube-controller-manager
command instead.
Looking at kube-scheduler logs
PODNAME=$(kubectl -n kube-system get pod -l component=kube-scheduler -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system logs $PODNAME --tail 100
If kube-scheduler is not running as kubelet static Pods, then use journalctl -u kube-scheduler
command instead.
Looking at kube-dns logs
Kube-dns is usually running as addons. Each kube-dns Pod contains three containers and kubedns
container's log could be got by
PODNAME=$(kubectl -n kube-system get pod -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system logs $PODNAME -c kubedns
Looking at kubelet logs
SSH to the Node and then run
journalctl -l -u kubelet
Looking at kube-proxy logs
Kube-proxy is usually running as daemonsets, whose logs could be got by
$ kubectl -n kube-system get pod -l component=kube-proxy
NAME READY STATUS RESTARTS AGE
kube-proxy-42zpn 1/1 Running 0 1d
kube-proxy-7gd4p 1/1 Running 0 3d
kube-proxy-87dbs 1/1 Running 0 4d
$ kubectl -n kube-system logs kube-proxy-42zpn